5 ms·
It occurs to me that the replication crisis largely came into being because scientists are rewarded for publishing papers that demonstrate some effect and not f
by geofft 9y ago
It occurs to me that the replication crisis largely came into being because scientists are rewarded for publishing papers that demonstrate some effect and not for publishing papers that fail to demonstrate anything, and so the right fix would have been to fix the broken incentives. But apparently (if this article is to be believed) no such change in incentives has happened - the only thing is that there are now also incentives for demonstrating that some other study wasn't replicable. Steadily doing good science, where good science is expected not to produce results much of the time, continues to have no support. Is it any wonder that the state of science seems a little better (at least we can admit the replication crisis now) but people like Cuddy feel like they have no future in the field, that having the misfortune of the one-in-twenty chance of getting a really interesting false positive at p = 0.05 means you either have to double down on the study being right after all or you have to quit the field? Are schools actually looking at past p-hacking while continuing to demand a huge publication record for hiring? This doesn't seem like we've solved any fundamental problems.
(Also, as a side note, I'd fully expect that CS is at least as vulnerable to the same thing. McSherry et al.'s COST work is a good example of something that seems far worse than p-hacking: instead of having a clearly defined hypothesis and publishing the 1-in-20 papers that falsely state it's true, as in other fields, there's nothing clearly defined in scalable systems and 20 out of 20 people who build something excessively complicated can get a paper out of it.)
- fhood 9y agoIs this http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf the paper you referenced?
- geofft 9y agoYup. See also this recent followup from him: https://github.com/frankmcsherry/blog/blob/master/posts/2017-09-23.md https://github.com/frankmcsherry/blog/blob/master/posts/2017... Literally the papers are "I built a thing, and it gets faster when you do this thing to it." There's no condition on whether the base case is realistic. To be clear, I don't think the authors of the papers he's taking down are bad people, or maliciously trying to get publications for uninteresting work, or too stupid to implement things efficiently on a single core, or anything. I think they're genuinely interested in the work they're doing, but responding to the pressures of the environment that allows them to keep doing that work.
- srtjstjsj 9y ago> having the misfortune of the one-in-twenty chance of getting a really interesting false positive at p = 0.05 means you either have to double down on the study Or you could get a large sample size to bring the p down to 0.01 or whatever. You know, do science.
- sliverstorm 9y agoYou say that like it's free. It's entirely reasonable, IMO, to do more "exploratory" studies at p = 0.05, and then do a follow-up on promising studies at p = 0.01
- geofft 9y agoWell, the smaller your p, the higher your chance of false negatives, too; maybe there was an actual effect there that was worth more study, and you dismissed it. The problem is that a study that's significant at p = 0.05 but not p = 0.01 is worth something to science. It indicates something worth doing several more studies on, preferably by different teams from different institutes with non-overlapping confounding factors. But the practice of science gives you only two options once you find such a study, as a young scientist: promote it immediately and become visible for having discovered something, or aggressively ignore it and go find something else that you think is likely to be immediately promising. "Doing science," as you put it, is the worst option for you because it ensures you spend a lot more time on something for no reward -- and the rational self-interest here is perfectly rational, because if you don't get funding or tenure, you're not going to do any science any more at all. (As seems to have happened with Dr. Cuddy.) To be clear, I have no well-formed opinion on whether Dr. Cuddy is a good scientist, having only read this article. I assume she is, or at least that she's honest in her desires. But I think talented, honest people are nonetheless pressured by incentives, and fixing incentives is better for society (assuming we as a larger society wish our scientists to continue to do science) than assuming that people will be self-sacrificial and altruistic in pursuit of a tiny chance at accomplishing the greater good.
- killjoywashere 9y agoHere's the structural problem I see: there are acres, literally, acres of computer science types in SF and certain other enclaves, slaving away on ads when they would rather be curing cancer. But they don't know the first thing about cancer. Meanwhile, there are enclaves where biologists are slaving away capturing data and not even realizing it's data. "Bookkeeping data" they might call it. There should be mixers between the biologists and the computer scientists. Especially non-academic physicians and computer scientists. Look at Kaiser-Permanente, Sutter Health, or Defense Health Agency. Huge orgs, very little research. Data falling off the shelves and they don't know what to do with it.
- kansface 9y ago> that having the misfortune of the one-in-twenty chance of getting a really interesting false positive at p = 0.05 means you either have to double down on the study being right after all or you have to quit the field? There were (are?) enough degrees of freedom in the field in every study to always meet this threshold.
- sliverstorm 9y agoSteadily doing good science, where good science is expected not to produce results much of the time, continues to have no support. We say this, but I haven't heard any particularly good ideas on how to build such a system that isn't also easy to game. How do you ensure the failed studies were actually worth doing? Is peer review of study design feasible?
- geofft 9y agoHere's a system that's gameable, but no more gameable than other things our society seems to be fine with: have peers evaluate you for the quality of your work as opposed to your outputs. Retain people whose peers say "They do good work," even if they don't have a flashy success or novel discovery to show. This is roughly how private industry works. I'm of course asked in job interviews about my successes, but the primary thing that determines whether I get a job, how much I get paid, whether I get other people to work with, etc. isn't what my past successes are, it's my ability to convince my peers that I am a steady and good worker. If I maintained some tech stack for two years, my employer is happy to let me continue doing that, and other employers are happy to hire me to maintain similar tech stacks, without me having been like the one person in the world to figure something out. (On rare occasion, I am, in the sense that I'll fix something upstream that's been challenging, and employers do value that! But there's no, e.g., "fix upstream bugs or perish" culture.) As in private industry, it's gameable because it allows people doing poor-quality work to vouch for other people doing poor-quality work. But it's hard to game this long-term, because people interested in doing good-quality work will tend to find others who do, and in aggregate -- from a large number of such people working together over a period of time, not from any one individual -- these people will produce more measurable results.
- sliverstorm 9y agoHow do you allocate funding?
- geofft 9y agoGive funding and recognition/status to large organizations (groups of at least 100 researchers) that produce good work, without naming names except perhaps in truly exceptional cases. Refuse meddling from outside incentives that break the organizational abstraction barrier (prizes, fellowships, grant proposals that with individuals named on them, fellowships or grant proposals that stay with individuals across organizations, etc.). Create alternative structures that recognize organizations that are producing results with fame and fortune. Note that by "organizations" I don't mean "universities" or "institutions". Departments should probably be treated separately, as different departments within an institution can easily drift in terms of quality. Meanwhile, it should be possible recognize collaborations of researchers across researchers as an entity: if the collaboration wants to support a researcher moving between institutions and wants to keep supporting them, that doesn't lead to the same flaws, I think.
- Balgair 9y agoOh, so, 'Capital S' Science[0] has moved on from publish-or-perish to funding-or-famine. Most researchers are in 'famine' and their spouse is financing their lifestyle of crazy hours and stress (love is strange, for sure). Some are in 'funding' and they generally get all the cash from grant agencies. Where I am, each square foot of your research is ~$420/month for rent, including your office. Then you start buying lab equipment and do research. The incentives are heavily skewed towards making cash, no matter the method. Also, if you haven't been following the 'X-ray Carrots' fiasco, that is a BIG deal in science and Science. VERY short summery: Andrew Wansink at Cornell made up a bunch of data and results, he published them, he got caught. One of these papers was about calling carrots as 'x-ray carrots' makes kids eat more of them, hence the name of the fiasco. Here's a good way into the mess [1]. Anyway, some papers were retracted, there was an investigation, yadda yadda. Now, in 'normal' science, he would have been banished from the field for this nonsense. But in 'finding-or-famine', he got off essentially scot-free. Nothing happened, really. He's still the head of the department (AFAIK), he still brings in cash, the FDA still is using his findings. This is a huge blow to science and a huge boon to Science. The existential crisis is now out in the open: you can cheat with impunity, publish whatever, if you get caught nothing will happen, just bring in the dollars. Now is not a good time to go to grad school. I think they're just going to have to throw out ALL US based bio-science (at least) from about 2003 to whenever heads start rolling. It's all crap, maybe, who knows. [0] Science and science are different, though look similar. Big Science is the grant machine and the big money side of science. Splashy, barely replicated, large clean buildings, pensions, perfectly fit for corporate needs. Think Big Tobacco, BPA plastics stuff, power poses like Cuddy's, that kinda stuff. It's like the difference between the Military Industrial Complex and actually kicking down doors in Mosul. 'Little s' science is what you are probably thinking of when you think of science generally. [1]http://andrewgelman.com/2017/04/06/dear-cornell-university-public-relations-office/ http://andrewgelman.com/2017/04/06/dear-cornell-university-p...
- shoyer 9y ago> each square foot of your research is ~$420/month for rent I think you missed a decimal point or two here -- $42,000/month for a 100 square foot office can't be right :).
- AlexCoventry 9y agoYou don't have to quit the field. Her coauthor did not.