5 ms·
Scientific rigor proponents retract paper on benefits of scientific rigor
- delichon 2y agoMaybe there is a vast dark matter of uncompromising super rigorous science happening that is invisible to us because it is too rigorous to meet its own standard so never publishes anything. But they know things.
- zaik 2y agoOur math department does publish
- georgeburdell 2y agoThere was a prof in my department who routinely passed grad students without a single publication because his standards were extremely high. All of his publications going back to like 2005 were Science or Nature
- StableAlkyne 2y ago> who routinely passed grad students without a single publication I really hope these were just Masters level students, because at least they could get the benefit of the doubt that it was coursework or something. If it's PhDs, I feel so sorry for them: a major part of the degree is learning to communicate research. Without publications, there is no proof they can do research or communicate it.
- analog31 2y agoI know a professor like this, and I feel sorry for his students too.
- georgeburdell 2y agoPhDs, and they did present at conferences (which in my field were not very selective), but yes it does seem like they did not get the whole PhD experience
- robotresearcher 2y agoWhy discount the thesis, the traditional evidence of these things?
- kergonath 2y agoIt depends on the context. I don’t bother reading American theses anymore because there is usually not much of value in them. Even in countries where theses usually mean something, they are often seen as a box-ticking exercise that needs to be done in the quickest and easiest way possible. If we fail students who don’t have a good thesis, not many would pass.
- dotnet00 2y agoIn my experience, the thesis doesn't properly test the skills involved in being a researcher. All it shows is that you've done some research, and that you can write and talk about it given a large amount of prep time and guidance. Without any publications, it's hard to judge if the PhD knows how to do peer review for instance. On the other hand, after having gotten through a couple of publications, a couple of seminars and at least one peer review process, the thesis becomes almost just a formality. Since by then you've already had plenty of experience writing, dealing with criticism, and presenting your work to experts.
- samsartor 2y agoI think this may actually be true in a sense. A lot of good science gets published but never wildly noticed, just because good science is rarely sexy.
- fuzzfactor 2y agoWhen it comes to testing materials, ASTM publishes and maintains dozens of kilos of fine-print documentation covering laboratory methods and procedures, almost all of which conclude with statistical ratings of their repeatability and reproducibility. Very few would ever be published without a significant number of labs able to routinely achieve comparable performance.
- sitkack 2y agoThat is like your prior, man.
- kergonath 2y agoI have tons of ideas that are interesting and with good preliminary results, but that will never get published before a lack of time to support them well enough that they are accepted by my peers (i.e., interesting works without enough rigour to clear a partially self-imposed threshold). Everyone interested in their field of research is in the same situation. Lowering the threshold would increase the amount of ideas that get communicated, but it would also increase the likelihood of one of these ideas being bonkers and decrease the signal-to-noise ratio in the literature.
- achrono 2y agoAnd a prominent study on dishonesty was itself found to have fabricated data [1]. Like too much refined sugar, this is too much refined irony. What is going on? 1: https://www.theatlantic.com/science/archive/2023/08/gino-ariely-data-fraud-allegations/674891/ https://www.theatlantic.com/science/archive/2023/08/gino-ari...
- BarryMilo 2y agoWell, they are experts on dishonesty....
- giardini 2y agoBut they were caught! Wouldn't the best experts on dishonesty get away with it leaving others none the wiser? Perhaps delichon's hypothesis of "dark matter" dishonesty (post above) should be considered seriously.
- thedailymail 2y agoAs icing on the cake, one of the authors of the retracted study written about in Science is named Dr. Perfecto. https://www.nature.com/articles/s41562-023-01749-9#Sec8 https://www.nature.com/articles/s41562-023-01749-9#Sec8
- levocardia 2y agoDon't lump in actual fraud with incorrect data analysis. The former is far more serious than the latter. In a long career, most scientists (even very good ones) will probably make mistakes in data analysis. Very few will commit outright fraud.
- achrono 2y agoI'm drawing attention to the irony inherent in both these cases. To wit, a study on dishonesty suffering from dishonesty, and a study on rigor suffering from lack of rigor. And sure, mistakes in data analysis are entirely possible. But there are lines to be drawn, always. Ariely and Gino, and now Protzko, Krosnick and company are not in the category of reasonable and honest mistakes. On the other hand, even something with widespread effects such as the mess Excel created for genetics papers is not something that most people would find as deliberate sabotage.
- renewiltord 2y agoBehavioral science is too hard to get right. The effect sizes mostly don't seem to be big and it's too hard to run a trial, so to keep a career going scientists are forced to aim for some sexy outcome which isn't there from an underpowered experiment. But almost nothing really matters in that field. It's as good as plastic recycling. Still, we are forced to study it because there are interventions that have worked in the past: cigarette smoking, teen pregnancies, etc. And it's useful to know what made those so successful. But maybe the answer is luck.
- subroutine 2y agoThere are many basic things we know about the brain that can be attributed to behavioral science. For example there is a ton of solid research from lesion studies that provide insight on, for example, the function of almost every cortical and subcortical region in supporting memory, language, emotion, motor control, all aspects of perception, attention, etc.
- epistasis 2y agoThis is evidence of science working, and self correcting. Should we expect everything that gets published to be definitive, the end word, worthy of enshrining on marble and being unerring in perpetuity? Of course not. There is no more unscientific view of knowledge and of science. Journal articles are the communications between practitioners. Blog posts before there were blog posts. Sometimes they are revolutionary. Sometimes they are work-a-day, filling in essential knowledge but not revolutionary. Many are arduous journeys that are hated by all by that touch the manuscripts by the time they get through the process of writing and peer review. Some are wrong. The scientific literature is not a series of proofs and lemmas, it's an accumulation of evidence paired with contemporaneous interpretation of what that data might mean, and those interpretations should be viewed as fluid. The data should be forever (unless there is error or fraud). The interpretation is moving. I don't agree that need we tons more "rigor" in science, instead it would be better to have clearer communication along with less busy work.
- raverbashing 2y ago> This is evidence of science working Is it? Looks more like academic bureaucracy to me And herein lies a big problem with modern science, the two are conflated. Publishing by itself does little to advance science. I feel there would be more science done if scientists discussed in an online forum than through papers
- JellyBeanThief 2y agoIt couldn't be just any online forum, though. It would have to support scientists' specific needs. It would need to host datasets; make it easy to create and reference charts, diagrams, tables, figures; track post edits and the reasons for them; disincentivize both lengthy and low-content posts; provide advanced search and filter functionality; assist with jargon control; and support consensus building and measurement. And then there's need for accessibility, for users with physical or mental handicaps, and for users with limited computing and bandwidth resources.
- raverbashing 2y ago
- hdivider 2y agoReminds me of a conversation with a late friend. We wondered: what would the most academic book possible look like? It could have: one brief, vague, and hedged statement, supported by a crapton of superscript references -- which constitute the rest of the book. So like: "Chapter 1: Some things about the universe are at least partly knowable.<references 1-100,000> The End."
- lmpdev 2y agoReminds me of DFW’s Infinite Jest (although I wouldn’t describe it as brief or vague)
- endominus 2y agoThe philosopher in me immediately objects, "How can you tell if something is 'knowable'? What constitutes knowledge in this model?"
- AnimalMuppet 2y agoThat's covered in the footnotes...
- kergonath 2y agoIt would not be academic without self-congratulation and lofty promises.
- adammarples 2y ago1. The world* is* everything† that is the case‡
- ggm 2y agoI can imagine a world where they hedged the outcome sufficiently to not have to retract. "methodologies based on those used here may prove beneficial based on preliminary findings, subject to further refinement" type comment. I mean, without over-egging it you would not deploy peer review, based on the modern evidenced outcomes: the papers are demonstrably not better in volume, and have to be retracted, and peer-review has been massively destructive of career progression for academics because of Rei-ification. To me, it's analogous: the model is flawed? refine the model don't junk the principle. I think at one remove I agree with comments which go to "this is evidence that rigor is good, and self-policing is good"
- deleted 2y ago[deleted]
- fuzzfactor 2y agoVery misleading choice of artwork showing natural scientists in doubt when it's actually things like behavioral theorists who have orders of magnitude more irreproducible results.
- bordercases 2y agoThis reminds me of how the Dunning Kreguer results might have been misapplied statistics.
- deleted 2y ago[deleted]