18 ms·

People are starting to learn that the vast majority of "science" are poorly-controlled white papers that get accepted and are never looked at again unless it is

• jdoliner 9y ago
  As someone who works in science what do you think is the underlying cause of this shift?
  • deleted 9y ago
    [deleted]
  • icelancer 9y ago
    I think it's multivariate. The rise of for-profit publishers certainly plays a large role, as does tenure track limitations, the booming business of education, and just the general pushback against openness. People fear the sunlight, no matter what "science" is supposed to be about. I don't think it's any one thing or that there's an easy fix. For my part, I refuse to publish in non-OA journals and strongly prefer those who require/strongly encourage both open data and open peer review. PeerJ is one such journal which also features very fast time to first decision AND low fees. I cannot recommend them enough.
• Chaebixi 9y ago
  > People are starting to learn that the vast majority of "science" are poorly-controlled white papers that get accepted and are never looked at again unless it is by a group of replication-crazed people (or what I like to call "actual scientists") reviewing conclusions drawn from decades-old papers. Correct me if I'm wrong, but I thought the main issue was that papers were being replicated, but the results weren't being published (especially if they were negative). These problems were always being discovered, but the record was never being corrected.
  • icelancer 9y ago
    >>Correct me if I'm wrong, but I thought the main issue was that papers were being replicated, but the results weren't being published (especially if they were negative). These problems were always being discovered, but the record was never being corrected. You may be right, but that's basically the same thing (and as you have sort of pointed out, it's not possible to tell either way).
  • justabystander 9y ago
    There's kind of a lot of issues. Especially with psychology and sociology and the areas they intersect. 1. Replicability of social psychology results are estimated at 25%. (Cited in article, from http://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248 http://www.nature.com/news/over-half-of-psychology-studies-f...) 2. People are not reporting non-significant results. Which means we get an incomplete picture of the validity of certain hypotheses. (from article) 3. The numbers don't line up when comparing observed power and significance of results. (from article) 4. Some results data is erroneous or falsified. The GRIM test that surfaced a while back shows numeric "results" that cannot possibly be derived from the sample data. (https://medium.com/@jamesheathers/the-grim-test-a-method-for-evaluating-published-research-9a4e5f05e870 https://medium.com/@jamesheathers/the-grim-test-a-method-for...) (https://www.vox.com/science-and-health/2016/9/30/13077658/statcheck-psychology-replication https://www.vox.com/science-and-health/2016/9/30/13077658/st...) 5. Many authors on the papers don't want to share the data from their papers - even if it's a condition of publication. (from a previous article about the GRIM test that I can't find) This doesn't directly equate to maliciousness or deceit, but it illustrates incorrect attitudes in the field. Some authors consider it undignified to have someone "check their work". Others are concerned of fallout if their data contains errors. (Generic mainstream blog/press article on reproducibility https://www.vox.com/2016/3/14/11219446/psychology-replication-crisis https://www.vox.com/2016/3/14/11219446/psychology-replicatio...) 6. Ridiculously small sample sizes are often used in experiments. Often this doesn't seem to affect how the results are received, though it should. (briefly discussed in article and comments) 7. Papers and studies are cited in articles and future papers before they've been peer reviewed, replicated, or proven sufficiently to justify their inclusion into the field. Thus the results are "baked in" to social psychology as fact at an uncomfortably early stage. (http://blogs.plos.org/mindthebrain/2015/12/30/should-have-seen-it-coming-once-high-flying-psychological-science-article-lies-in-pieces-on-the-ground/ http://blogs.plos.org/mindthebrain/2015/12/30/should-have-se...) This kind of feeds into the hype machine where people focus on significant results - even if the data needs massaging to get there. Even after being refuted (if refuted) the field and its fans never seem to fully recover and back away from bad science. By then it's already made its rounds on Facebook and it's even referenced in the worst of the college intro to psych/sociology courses. 8. Journals in general need to be more open with their data. Especially with publicly-funded research. Paywalls don't help progress. 9. Corporate/politically funded white papers publish studies without making the connections clear. The field needs a massive overhaul in the sense that its participants need to stop chasing controversial results in the press and instead focus on reproducible results and methods. All fields need some of this, but psychology needs it more than others.
• aaron-lebo 9y ago
  Completely agree. It scares me to dig into research papers and to find very obvious mistakes (anyone at a graduate level is capable of this) and even worse to see how heavily seasoned some of the horrific datasets are. This kind of stuff despite those flaws gets propagated without much skepticism. It makes me wonder how much stuff we "know" to be true is the result of an intuitive argument but has no real support.
  • icelancer 9y ago
    >>It scares me to dig into research papers and to find very obvious mistakes (anyone at a graduate level is capable of this) and even worse to see how heavily seasoned some of the horrific datasets are. It is scary, and you are right, but you can do your part by blogging about it. Doesn't have to be official, just post the text and your notes. The more people do this, the more we have informal peer-review and replication checking after the fact, and at least a loosely-connected database of blogs and articles will exist when people search the article title.
    • claytonjy 9y ago
      Say it louder! As a very interested non-scientist layperson, I have learned so much through blog posts critical of poor papers, which I often find through twitter. The more people doing that, the better the state of the field becomes, faster. It's fascinating to see the "new guard" emerging in psychology, actively acknowledging the poor practices that have preceded them and fighting back were possible.
• WalterSear 9y ago
  My graduate advisor told me that I should be designing experiments that had counterintuitive results that were solidly significant. I don't see how you can look for a specific kind of outcome from an experiment without fooling yourself, or cheating.
  • cryoshon 9y ago
    >My graduate advisor told me that I should be designing experiments that had counterintuitive results that were solidly significant. This is the advice of a true scientist. I'm refreshed to see that they're still out there.
    • Filligree 9y ago
      > This is the advice of a true scientist. I'm refreshed to see that they're still out there. It encourages students to cheat, because most of the time you won't get either. Not a significant result, and usually not a counterintuitive one either. You're pretty likely to get a significant result that agrees with existing theory, though.
      • richardwhiuk 9y ago
        By significant, you mean statistically, not to people in general.
        • vorotato 9y ago
          Kind of bizzare that we don't want to report where we weren't able to find a statistically significant relationship.
        • Filligree 9y ago
          Of course.
  • PeterisP 9y ago
    Before doing an experiment, you can often see that with your settings, if you'd get the counterintuitive result, then it wouldn't be statistically significant, so you'd need to plan a larger experiment or not do it at all. You can look at any experiment as having essentially two possible outcomes - you need to ensure that at least one of those outcomes is publishable, because there are many experiments where any possible outcome will be weak.
• meri_dian 9y ago
  The reality is that genuinely interesting, productive and novel results are hard to produce. Even for teams of highly intelligent researchers. The scientific industry is built on the assumption that such results are in greater supply - or more accessible - than they actually are.
  • icelancer 9y ago
    I don't think I could have put it better than that. Very well done. I am reminded of a parallel quote by Warren Buffett regarding the lack of truly good investments: "I could improve your ultimate financial welfare by giving you a ticket with only twenty slots in it so that you had twenty punches - representing all the investments that you got to make in a lifetime. And once you'd punched through the card, you couldn't make any more investments at all. Under those rules, you'd really think carefully about what you did, and you'd be forced to load up on what you'd really thought about. So you'd do so much better."
    • jon_richards 9y ago
      I'm really surprised he said that. It flies in the face of diversified risk and everything it affects, like Sharpe ratios.
      • davidgrenier 9y ago
        I think it's meant to be taken as the number of times you get to change your portfolio.
        • njarboe 9y ago
          I don't think so. Buffett is more of a buy to hold forever kind of investor.
      • Lon7 9y ago
        The common numbers thrown around are that with 20 stocks your portfolio risk is reduced by 70%. The corresponding ratios follow. So in that sense it's not that bad. Of course that's just one form of risk. The less attractive aspect of a 20 stock strategy comes from the fact that the majority of the market's returns come from very few stocks - The 80/20 rule applies pretty well here. With only 20 stocks you'll probably miss out on the few winners that contribute all the market's gains. It's very easy to end up with a 20 stock portfolio with low risk/variance and low returns. Of course if you are Buffet, then your goal is to pick 20 stocks that all outperform.
• ianai 9y ago
  What's the disruptive site for academic journals? I.e. There's got to be a way to ensure proper communication of positive/negative results and preferably less cruft.
  • icelancer 9y ago
    Well, Sci-Hub is the start of it all. Elbakyan deserves Nobel consideration if you ask me, and I am not being hyperbolic. She has done more for science around the world than the bottom quartile of laureates. Secondly, supporting OA-friendly journals is the next step. PeerJ is a very good initiative on that front for a lot of reasons - they open and document the peer-review process, which is unheard of. It's a tremendous asset to look through peer review and see how the "science" is really done. I highly recommend everyone find some papers that they are interested in on PeerJ and check the peer review logs.
    • Fomite 9y ago
      I confess that I'm seriously considering no longer reviewing for open peer review journals. I vastly prefer double-blinding.
      • icelancer 9y ago
        PeerJ is single-blinded in favor of the reviewers, if that makes you feel better. You can remove blinding at your discretion.
• sndean 9y ago
  > "Novelty" is literally written in the "guidelines for authors" sections of many journals. A while ago I had a manuscript rejected because I used the word "novel"... so hopefully the trend of emphasizing "novel" things [1] is changing. I'm trying to find the specific manuscript guidelines for that journal, but pretty sure they banned use of the word. [1] http://www.nature.com/news/novel-amazing-innovative-positive-words-on-the-rise-in-science-papers-1.19024 http://www.nature.com/news/novel-amazing-innovative-positive...
  • icelancer 9y ago
    Nature itself is to blame for this. Read their guide to authors. The criteria for publication of scientific papers (Articles and Letters) in Nature are that they: -report original scientific research (the main results and conclusions must not have been published or submitted elsewhere) -are of outstanding scientific importance -reach a conclusion of interest to an interdisciplinary readership. "Outstanding scientific importance" screams no boring replication studies that confirm pre-existing work.
    • stephengillie 9y ago
      > "Outstanding scientific importance" screams no boring replication studies that confirm pre-existing work. The note in the first bullet also forbids replication studies that confirm pre-existing work. > (the main results and conclusions must not have been published or submitted elsewhere)
      • icelancer 9y ago
        It can be interpreted that way, but usually that just means the work cannot be dual submitted for consideration or have appeared as a preprint without explicit permission.
    • naasking 9y ago
      > "Outstanding scientific importance" screams no boring replication studies that confirm pre-existing work. I don't know, "couldn't replicate this other major study" seems like it would qualify as outstanding scientific importance. It all depends how you frame "scientific importance". Replication seems like it should be high on the list of scientific goals.
• perlgeek 9y ago
  > People are starting to learn that the vast majority of "science" are poorly-controlled white papers that get accepted and are never looked at again unless it is by a group of replication-crazed people (or what I like to call "actual scientists") reviewing conclusions drawn from decades-old papers. I wouldn't generalize this to all branches of science. My own experience in physics contradict that, and I also get the impression that some of the large successes we've seen recently (like gravitational wave detections) couldn't have happened if the scenario you described were the norm.
  • Zaak 9y ago
    I was under the impression that gravitational wave detection had more to do with engineering advances than science per se.
    • spaceseaman 9y ago
      The most HackerNews of all comments. "Well it was the engineers who did most of the work..." Who do you think came up with some of those engineering advances? No need to play semantic scientist vs. engineer arguments. Some of the advances came from scientists and some came from engineers.
      • _dps 9y ago
        I was at Caltech when LIGO was being built. Kip Thorne gave a great lecture in the engineering school about how they went about knowing what to try to engineer in the first place. I'm not an expert on LIGO, but based on the seminars I attended it sounded like there was a lot of science that went into figuring out what the engineering team needed to do.
        • spaceseaman 9y ago
          I may have gone to a similar talk when Kip was at my school, what a coincidence!
      • KGIII 9y ago
• weberc2 9y ago
  I'm particularly interested in metastudies about fields like sociology which are dominated by people of one particular political ideology--I'm curious about the degree to which this homogeneity affects the quality of the work that the field outputs. I don't have a theory, but I am curious about what size grain of salt I should take when interpreting a particular finding.
  • icelancer 9y ago
    You have the arrow in the correct direction. The magnitude of the effect is unknowable. But sociology/psychology/etc are highly, highly susceptible to bias and political leanings.
    • weberc2 9y ago
      Is it unknowable? Couldn't you take a population of recent studies (because IIRC, the field has never been as homogeneous as it is today) and compare the reproducibility of non-political studies with political studies? Obviously some care needs to be taken with how we code "political" and "non-political", but I would think that could at least give a pretty good indication about the effect. It would also be interesting to repeat the experiment with studies from different points in time (when there was more ideological diversity) to see how the effect varies with homogeneity. My guess is that if there is an effect, it varies nonlinearly with diversity--in other words, a field with 90% political homogeneity might have an effect that is ten times that of the same field when it was only 70% homogeneous. I suspect that once you hit 90%, dissent is much more effectively suppressed than at 70%. But maybe not?
    • creaghpatr 9y ago
      I expect big data and AI to uncover many 'uncomfortable' sociological trends that were censored by the widely postmodernist-minded academics in the near future. The academics will deny these hidden truths but the greater population will (and already do) exploit them.
      • awkim 9y ago
• _e 9y ago
  This is also true for finance which is heavily dependent on Excel. Very few people want to audit and verify a model given the monstrosity that some spreadsheets turn into over time.
  • icelancer 9y ago
    Ha, very. My previous life as a data scientist and economist knows your comment all too well.
• tepe1 9y ago
  For somebody who claims to live science you really don't seem to understand how our why it works. Replication is not at all important to science. It never had been and never will be. What is important is falsifiability and predictive power. The article is a perfect example of science at work.
• snowAbstraction 9y ago
  Does this novelty driven research decline once tenure has been obtained? I would like tenured researchers to take on risks, long term projects and investigations that many companies will not, regardless they pursue novelty.
  • sevensor 9y ago
    That's how one might hope it works, but it doesn't. First of all, it's a lot to expect of anyone who's been struggling in a field for well over a decade (from the start of grad school to the grant of tenure) to suddenly turn around and stop doing the things (like chasing novelty at the expense of rigor) that have been key to academic survival up to that point. Second, the system doesn't incentivize that kind of behavior. If you put your grant pipeline at risk by making a big bet, you could well end up spending most of your time teaching undergraduates because academics who don't pull down lots of grant money are penalized with heavier teaching loads. (Not that this ought to be seen as a penalty, but that's another rant.)
  • Fomite 9y ago
    The pressures applied to getting tenure are the same pressures that get applied to keeping your lab funded, getting your graduates and postdocs jobs, etc.
• SubiculumCode 9y ago
  The lack of pure replication studies is frustrating, but I am compelled to mention that in many cases the experimental effect of the prior study often ends up becoming the baseline condition of the next study. IF the earlier experimental effect fails to replicate as baseline condition, then it isn't a good baseline in the new research, and thus fails to replicate, and thus the experiment is discontinued.
• speby 9y ago
  It is much easier to demand novelty rather than repeated validation since most academic-based research completely stops at trying to apply said research and found conclusions to a real thing. Because if they were to go and apply their findings to something in the real-world (a product, service, or some good that can be delivered and/or consumed) it would often quickly put their research to the flame, revealing either its flaws or its possible validity.
• megamindbrian 9y ago
  Doesn't science also apply to science? We should be sciencing our science which means being skeptical of all conclusions, even your own. I feel your stress. I use to even call myself a scientist, and regard myself as a scientist, but then I realized 90% of my work is entirely unmeasured and there is no sense of effectiveness of the remaining 10% that actually makes it to production.
  • jrochkind1 9y ago
    I think we need a lot more focus and funding on a field of study/discipline that studies how science it done. I'm not honestly sure who in the academy, if anyone, is doing this. "History of science" comes close (it's not always "in the past"), but generally scrupulously avoids making any determinations of the "validity" of the science, it tends to be like a _social_ study of how science it done, intentionally having no opinions on how to make it "better" or "worse". Are there any departments or disciplines or academic careers that can be made on scrupulous scientific study of how science goes right and wrong, including statistical as well as other issues? I think we need it.
  • isolli 9y ago
    Considering that scientists are reluctant to accept scientific evidence when it applies to them, I'd say no. https://www.wired.com/story/why-men-dont-believe-the-data-on-gender-bias-in-science/ https://www.wired.com/story/why-men-dont-believe-the-data-on...
• refurb 9y ago
  I worked as both an academic scientist and in industry. I can attest there is a lot of crap science published. So much so that when you find a really good publication you think "wow!!" The one thing about industry is that if the science doesn't work you don't make money. However there is little incentive to publish.
• AlexCoventry 9y ago
  > the vast majority of "science" are poorly-controlled white papers that get accepted and are never looked at again And worse, the vast majority of the evidence cited in major social science debates is deficient in this way.
• flipside 9y ago
  Scoring research by reliability in addition to novelty would help fix the incentives in research publications. Replicating research would boost your reliability score and the original's too which would boost credibility for future research.
• LordHumungous 9y ago
  I used to work in medical research and I've been saying this for years. It's amazing what tenuous results get published in fields like biology and psychology, and it's even more amazing how those tenuous results get spun into concrete "facts" by a credulous public.
  • tertius 9y ago
    This is an argument for denying man made climate change. Be careful.
    • feral 9y ago
      What are you trying to say here? That if a fact (such as that a particular field spins tenuous research into facts - let's assume that's a fact) happens to lead us to a conclusion we don't like, we should 'be careful' and pretend the fact is untrue? That's obviously horrifically unscientific. Or is this the point you are making, and you are expressing a climate-change-denying position, really obtusely? If so, it's on you to show climate science is 'spinning tenuous research into facts'.
      • tertius 9y ago
        > happens to lead us to a conclusion we don't like, we should 'be careful' and pretend the fact is untrue? Reduce that to: > happens to lead us to a conclusion we don't like, we should be careful especially given the immense political pressure both within science and in governments. Liking has little to do with it. Those who are purists also don't like that others question dogma.
      • lotsoflumens 9y ago
        I think it's just very subtle sarcasm that these are dangerous thoughts in a world where thought-crime is real.
    • icelancer 9y ago
      No, this is an argument about being skeptical about climate change effects, which is good. There are a lot of people weaponizing climate change to affect their careers in a positive way by jumping on the bandwagon. (And yes, it is my opinion that climate change is absolutely happening. Does not mean that we can't be skeptical of the effect sizes cited.)
• kbutler 9y ago
  Seems like we need to increase the visibility and prestige associated with disproving published results. Right now, submitting data that contradicts a published paper is likely to be ignored, treated with hostility, or swept under the rug. At very best, you'll get a small publication of a retraction. Disproving published results needs to be at least as prestigious as producing novel results. If we can't swing that - it would be great for graduate student research work...
  • colechristensen 9y ago
    Perhaps an open-access Disprove Journal?
    • lolptdr 9y ago
      That is an amazing and fascinating idea. It should be run on a public facing front with fact-checkers of these fact-checkers who are anonymous.
  • lliamander 9y ago
    A story I heard once was that, back in the 60's, IBM was having problems with engineering quality (quelle surprise). This lead to the formation of a testing team called the Black Team. The Black Team developed a culture that prided itself on breaking systems the engineers thought were rock solid. They delighted at making engineers cry. They even started coming to work dressed in all black and sporting sinister mustaches. Consequently, many of the quality issues abated. I don't know if it's true, but it sounds like it might be a good idea. How might we get funding though? Perhaps with a prediction market (people bet on whether scientific hypotheses will be disproved/confirmed)? It could be fun.
  • leksak 9y ago
    Whenever I start working somewhere new one of my side projects outside of whatever assigned work or feature work that I have is to write a test case that proves that something that was thought to be in a certain way isn't. After having done so I suddenly feel way more comfortable at my desk. I'd love to prove things wrong for my master's thesis but novelty is a grading criterion so that's a no-go.
• tostitos1979 9y ago
  Agreed. I'd also like to point out that the review process itself is pretty broken. People sometimes behave like sociopaths (e.g. torpedo a paper that is competing with your work), personal grudges etc.
• DoofusOfDeath 9y ago
  > It's incredibly frustrating being someone who loves science, works in the field of science, and is skeptical about the system, which used to be a prerequisite and is now looked at like luddite behavior. I'm concerned about it from a different angle: How much public policy, medical treatment, and follow-on science is seriously suboptimal due to this house of cards?
  • icelancer 9y ago
    >>How much public policy, medical treatment, and follow-on science is seriously suboptimal due to this house of cards? This you do not want to know. Here is a good question, the politics and bad science of which were uncovered by the documentary "Bigger, Faster, Stronger: The Side Effects of Being American" - why are exogenous testosterone and anabolic steroids strictly controlled and essentially banned? What is your first reaction to this question? That they are unhealthy, that they have gross and terrible side effects? Likely this is what you have heard. Nearly 100% of what is popularly "known" about anabolic steroids and use of testosterone is absolutely false. Exogenous testosterone was banned because a father of a boy who committed suicide (who was taking steroids [0]) rode a publicity wave all the way to Congress, despite the recommendation against the ban by the AMA and other medical experts. Now, think of what else this may be true of? [0]: The son was taking steroids, yes. He was also on benzos, drinking alcohol, and doing other recreational drugs.
• vanderZwan 9y ago
  Relevant example: > The most widely cited test was a 1987 study for Bicycling magazine by engineering professor Chester Kyle, one of the pioneers of cycling aerodynamics. He found that leg-shaving reduced drag by 0.6 per cent, enough to save about 5 seconds over the course of one hour at the brisk speed of 37 kilometres per hour. At slower speeds, the savings would be less. > [More recent tests in a modern windtulle show] that [shaving legs reduces drag] by about 7 per cent (...). In theory, that translates to a 79-second advantage over a 40-kilometre time trial that takes about one hour. > [The aerodynamicists in charge of the windtunnel contacted Kyle], to ask if he had any ideas about the discrepancy between the two results. It turned out that the 1987 test involved a fake lower leg in a miniature wind tunnel with or without hair glued onto it – hardly a definitive test, and yet it was enough to persuade most people not to bother with further tests for the next three decades. https://beta.theglobeandmail.com/life/health-and-fitness/health/the-curious-case-of-the-cyclists-unshaved-legs/article20370814/?ref=http://www.theglobeandmail.com& https://beta.theglobeandmail.com/life/health-and-fitness/hea...
  • infinite8s 9y ago
    It's interesting because that would have been obvious to anyone looking at the methodology section of the original paper. Unfortunately it only takes one review paper or two paraphrasing or quoting the the original before it becomes standard lore (without the methodological caveats).
  • Hermel 9y ago
    Interestingly, bicyclists have been shaving their legs anyway, ignoring the scientific consensus. Maybe common sense and personal experience is underrated.
    • vanderZwan 9y ago
      I don't think this example generalises quite to that conclusion: as the tests show, the differences are huge, having bigger advantages than getting top-of-the-line bike equipment. Assuming that we're already reaching the tail-end of the bell curve for human performance in cycling sport (which seems likely to me), suggesting minimal individual performance differences, then being at among that group despite a 7% performance disadvantage seems downright impossible. So especially at the top there would be an incredible selection against anyone with hairy legs. But I guess you could say that natural selection beats human belief.
• c3534l 9y ago
  But if there were high academic standards, how would you manage to get enough published to meet university demands?
• avip 9y ago
  What's the relation between the "scientific industry" and science? I tend to say it's orthogonal. To arrive at the simplest truth, as Newton knew and practiced, requires years of contemplation. Not activity. Not reasoning. Not calculating. Not busy behaviour of any kind. Not reading. Not talking. Not making an effort. Not thinking. Simply bearing in mind what it is one needs to know
  • paulddraper 9y ago
    Uh huh. So you're going to understand cancer treatments just by thinking really hard?
    • avip 9y ago
      Probably by contemplating the results of a double-blinded clinical trial.