10 ms·

Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

330 points by curtis 8y ago

taruz 8y ago
I want a chart like that but for journalism.
- clircle 8y ago
  I don't follow this comment. What would you plot on the vertical axis?
  - nostrademons 8y ago
    Clickthroughs, presumably. I assume that taruz is saying that journalists should be required to say what they're investigating before publishing an investigative report. Right now, they start investigating, and if there's something outrageous (even if it wasn't what they were initially looking for), they publish. Sometimes they even skip the "start investigating" part and just put up a tip-line for anyone who has a beef to get a story out there. This is great for manufacturing outrage and hence clicks, but it gives the public a hugely skewed perspective on how the world is. Imagine that 0.01% (1 in 10,000) of all peoples' actions are outrageous and will piss off a large portion of planet. Most people, by those numbers, would say that the majority of folks are decent, law-abiding citizens. Now imagine that a news outlet is allowed to freely go over someone's life, and they end up evaluating 1000 actions. There's a 10% chance that they'll find something outrageous. Now imagine that 10 such reporters do this to 10 people, and if any one of them finds something publishable (= outrageous), they go to press. Suddenly there's a 64% chance that one of them will find something, and you've likely got your news cycle for the day. With the millions of people looking for something bad that a tip-line can generate, outrage is virtually assured. And that's where journalism is today. The world isn't actually a worse place than it was in 1980; in fact, by most metrics it's significantly better. But we've increased the amount of unpleasantness that people can be exposed to by 3-4 orders of magnitude and then implemented a selection filter that ensures that only the worst stories go viral. Of course we get only bad news; that's all that's economically viable, and we have such a large sample size that we can surely find it.
    - teddyh 8y ago
      There’s also this to consider: “If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him.” – Commonly attributed to Cardinal Richelieu I.e. any fact or action can be willfully misinterpreted to fit almost any narrative.
    - jernfrost 8y ago
      That is a natural outcome of the hyper capitalist world we currently live in. I have a mother who has been a journalist for over 40 years. I've heard for years about the pressure sell. Article titles often get changed against the will of journalists to be more click-bait like to sell more. As people pay less for good journalism, and ad revenue is shrinking, the media is getting desperate to stay afloat. Journalists today have significantly less independence than they had 15 years ago. Everything is far more controlled and geared towards sales. I would like to see more alternative finance models for quality journalism. Given how social media is actually starting to destroy democracy, it may be worth considering government grants to independent media organizations as a part of national defense. A democracy cannot function if the media is utterly broken. That means citizens are no longer capable of making informed decisions.
- rossdavidh 8y ago
  Good idea, but you'd have to have an objective way of deciding how accurate the article was.
- njstraub608 8y ago
  I'd have to agree with this; every news outlet tries to treat a lack of proper controls as a breaking story / massive conspiracy theory to get a buzz going around the topic. I think it's important to inform people but unfortunately, most people would rather buy into the idea that there was malicious intent rather than also evaluating the possibility that, like most evolving business areas, it takes time to build in the proper procedures to attain measurably objective results. This happens in every industry and isn't unique to science, however, in this case, the physicians recommending these drugs to their patients should have been able to read between the lines or at the very least have a managed treatment plan to evaluate the efficacy on those they're treating.
caymanjim 8y ago
I'm cynical and have no trouble believing that results were twisted immensely prior to the rule change in 2000. The incentives are huge: profit, prestige, or simply job security. I do wonder if this chart misrepresents something, though: there are studies that produce incidental--but genuinely valuable--discoveries. It's unclear to me if that accounts for the pre-2000 results or not. With the new rules, would there have to be another study stating the new objective? I'm not questioning that bad science occurs, but I am questioning what this graph really tells us.
- rdiddly 8y ago
  You're right, the new rule doesn't filter out dishonest discoveries, only incidental ones (which may be dishonest). If you wanted to show benefit from an incidental discovery, you would have to run another study with that as its declared objective. The author is kind of doing the same thing here that they accuse others of doing...
- scotty79 8y ago
  > With the new rules, would there have to be another study stating the new objective? I think it's only fair to force you to replicate at least once the positive result you think you see in the data you collected for another purpose before you can claim you got something.
  - Balgair 8y ago
    Yes, the linked article is saying that researchers used to employ techniques like 'p-hacking' (among others) in order to report results that were favorable/novel. That the scientists and clinicians in charge teased the data too much.
smrtinsert 8y ago
Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.
- function_seven 8y ago
  Aren't they equivalent in their potential to allow post-hoc data mining? Whether it's a clinical trial, or a psychological study with n participants, or a look at how changes in x soil conditions affects the y tree. Anything where data samples are taken and statistically analyzed is prone to p-hacking and after-the-fact hypothesis changes. This example happens to be for a narrow set of studies: clinical trials for cardiovascular drugs/supplements. The take away from that graph applies broadly.
  - L_Rahman 8y ago
    And in fact the replication crisis in psychology suggests that the practice goes beyond just clinical trials.
    - gaius 8y ago
      The replication crisis in every field of science. Two-thirds of reported results cannot be reproduced.
      - dwd 8y ago
        If you haven't read it already, this article on the placebo effect a few days ago was interesting in how we on one hand use it as a baseline for measuring the effectiveness of drug treatments but actually can vary wildly between individuals. It's not surprising that results are so inconsistent. https://news.ycombinator.com/item?id=18398342 https://news.ycombinator.com/item?id=18398342
- geggam 8y ago
  Explain that to the layman in a way the entire Internet will understand. Until you can ... they are for all intents and purposes.
cup 8y ago
The downside of this though is that unexpected positive results sometimes get buried. For instance, clinical trials have to report all adverse effects but not positive effects. We've had drugs tested for symptom X fail to have any effect, while causing bald people to have their hair grow back. Yet the company we were testing it for wasn't interested and those results never made it into the public domain.
- rickycook 8y ago
  i would assume that the positive results still get reported somewhere in the paper? so other people could still pick up on it in the future. if they’re positive enough then the drug company will fund another study with that as the primary outcome. that seems prudent too: it should be the primary thing you’re examining so that you can design the study correctly, rather than a simple “oh and by the way” side note
- jonathankoren 8y ago
  That wouldn't happen. You would simply declare, that you want to check if the drug does cause hair growth, and then run a test explicitly looking for that. Unless you're looking for the effect from the outset, you can't be sure that what you saw wasn't actually random. There's a term for what you're describing, it's p-hacking, and it's explicitly the very thing declaring what you're looking for before you run the test is designed to prevent. Like many science related topics, there's a XKCD about p-hacking that describes a similar scenario to your example: https://xkcd.com/882/ https://xkcd.com/882/
epistasis 8y ago
I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials), science is somewhat additive. Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? With the advances in treatments for cardiovascular disease couldn't the problem just be harder? And look at that "harm" study. Turns out it's one of many from the Women's Health Initiative [1], and it's the test of whether hormone therapy for post-menopausal women causes heart problems: that is, it's not a test of a drug to prevent heart conditions, but a test of side effects. How many other of these studies are like that: studies about effects on the heart, not trials for new drugs to treat the heart? How did the ratio change before and after 2000? In any case, don't trust every popular news article you read about science, particularly if it's written by Kevin Drum and posted on Mother Jones. [1] https://www.whi.org/SitePages/WHI%20Home.aspx https://www.whi.org/SitePages/WHI%20Home.aspx
- prakashk 8y ago
  > Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? > However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. Is it possible that your own preconceived notions about the author and the publication may have caused you to judge this way? In any case, don't trust every comment posted on Hacker News (including my own) :)
  - Natsu 8y ago
    > The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc.
certmd 8y ago
While many on this forum may understand why what this analysis shows is not good science, the article does a poor job of explaining why finding positive results one wasn't originally looking for may not be reliable. I can imagine a non-statistically minded person thinking "So what it's not what they were looking for originally? We're missing positive findings now. This is a terrible regulation." When in reality these "positive" findings were p-hacked to meet minimum criteria for statistical significance and likely arose by chance since luck would have it that in any study there will be some set of data that by chance is a statistical aberration.
nemo44x 8y ago
Good science has its methods published, is reproducible and is peer reviewed. Another, perhaps better, headline would be: "Why You Shouldn't Trust Every 'Study' You See".
- jacksnipe 8y ago
  No True Scotsman. In fact, I’m pretty sure this is just another form of post-hoc reasoning.
DrNuke 8y ago
From a science perspective, positive results are overrated; the most important aspect is to investigate a “good” question. Researchers understand “good” has not the same meaning everywhere or to every stakeholder. More at large, blurred journalism is attacking science to undermine the scientific method, aka meaningful questions + proper scrutiny + full transparence. This is a sign of the times, though.
superjan 8y ago
One could argue that this chart too is an example of post-hoc data mining. I still find the effect plausible, though.
- ineedasername 8y ago
  Post-Hoc or not would depend on what hypothesis the researcher had before looking at the data. We don't really know, or at least I couldn't tell from the article.
rossdavidh 8y ago
While some have pointed out that this very article could be considered post-hoc data mining, I think it is not. This exact kind of affect was what was intended by the requirement to pre-register. Looking afterwards to see if the intended affect (change in how many studies report benefits) appears to have happened in reality, makes perfect sense. They didn't institute the requirement at random, and then later consider that maybe it could have an impact of reducing (perhaps spurious) findings of significant benefit; that was more or less the only reason for doing so.
- LanceH 8y ago
  The article claims the drugs are worthless. The studies only show they aren't successful in a single pre-registered result.
leesec 8y ago
Reminds me of this old classic, one of my favorites: http://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ http://slatestarcodex.com/2014/12/12/beware-the-man-of-one-s...
corysama 8y ago
This would seem to indicate that the results of drug research are unpredictable. Therefore, if you are required to predict your results, you are set up to fail. My question is: Was the free-range research actually effective? Or, was it "technically-correct effective just so you can't call me out on a failure, moving on..."
qwerty456127 8y ago
> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductive limitation? Significant benefits found without knowing what are they going to be in advance are still significant benefits and if they are observed scientifically and proven reproducible I'm glad we've found them. > Once they had to explain beforehand what primary outcome they were looking for, practically every study came up null. The drugs turned out to be useless. Aren't newly discovered drugs meant to undergo strict and targeted clinical trials? How can they even be considered being drugs before this? And how can they turn out to be useless after passing this stage? Also in some cases when nobody wants to fund clinical trials despite very interesting life-enhancing effects supposed or when it's clear the time to general availability through the fully white research and approval chain is going to take longer than people want to wait some non-approved substances happen to be sold on eBay (or, for more questionable substances, on black market), hundreds or thousands of people buy it and report their experience on reddit and this data can be a source for further clues for research.
- bgaragan 8y ago
  I'm curious about this as well. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? It seems counterproductive indeed to waste time and money like that. I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find. Please correct me if I'm misguided on any of this.
  - nkurz 8y ago
    I presume you are being genuine, but you should know that your comment is almost indistinguishable from satire. More specifically (and hopefully more helpfully) it sounds like the comments made by Brian Wansink before his recent fall. Google for his story if you are unfamiliar. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Assuming that the effects are legitimate is exactly the problem. You are right, if we somehow know that the effects found are real and reproducible, then all is good. The problem is that we almost never know this. Presumably what you mean is "If the results are reproducible, what's the harm?". I'd agree with this, but the problem is how to know ahead of time that the results are going to be reproducible. Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? If they did the statistics correctly (accounting for the multiple inferences, all assumptions about iid data met, no biased dropouts, everything else aboveboard) and got a really solid result, then yes, it's theoretically likely that the results would be reproducible. The problem is that they almost certainly didn't do the statistics correctly, and intentionally or not they probably violated a lot assumptions. In too many cases, even the main line conclusions can't be replicated. It's rarely "safe" to assume that an effect is real until it's actually been replicated, and almost never safe to make this assumption for result obtained by sifting the data after the fact looking for correlations. I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find. There are ways to do it this way, but it usually takes larger sample sizes than are available. It also requires "bespoke" statistics that are easy to get wrong. In practice, it's usually better to use the incidental "results" as idea generators for future experiments, rather than assuming that the findings are real and don't require further testing.
amirmasoudabdol 8y ago
Although I agree with the argument but their chart and their data actually do not represent what they are claiming. I’ve checked their paper while back [1] and contacted them about it but haven’t heard anything back. Their argument is sound but only because! [1]: https://amirmasoudabdol.name/likelihood-of-finding-null-effects-over-time/ https://amirmasoudabdol.name/likelihood-of-finding-null-effe...
fizzigy 8y ago
The article seems to suggest that scientific studies aren't reliable, but fails to point out the key takeaway that now (post registration requirement), studies are much more likely to be valid/useful than before. So the headline really should read something like "Why you should trust scientific studies a lot more now than you did before"
whack 8y ago
Is it possible that prior to 2000, the studies that found a null result, were simply not published? Hence why they are underrepresented in that chart? My understanding is that it's very hard to get null results published in scientific journals, though I'm not sure if that applies to clinical studies as well.
- deleted 8y ago
  [deleted]
- Fomite 8y ago
  One interesting effect that pre-registration has is that there are now several journals that allow you to publish your protocol, and agree to publish the followup paper about the results regardless of the outcome.
thedirt0115 8y ago
This is very similar to A/B Experimentation -- Gotta force the user to select a primary metric BEFORE the experiment starts to keep 'em honest.
ineedasername 8y ago
I'd say a good rule of thumb is that the smaller the number of observations, the less you should trust any research that hasn't been reproduced. And when it comes to publication of false positive results, always remember your XKCD: https://xkcd.com/882/ https://xkcd.com/882/
sudofail 8y ago
I have a general rule to ignore any statistical interpretation in articles like these. Data interpretation is hard. There are a lot of variables that need to be accounted for, and under the best circumstances of academic peer review, we still get it wrong. This article also feels like it has an agenda. Maybe I'm not familiar with MotherJones, but the tone of the article strikes me as unprofessional. And the headline, that's obviously correct and means nothing. It's like saying, "Why You Shouldn't Trust Every Stranger You Meet".
monochromatic 8y ago
>Post-hoc data mining is very bad, except when you’re gunning up evidence of anthropogenic climate change, which is very good
- noname120 8y ago
  For the record, I believe that you didn't get downvoted for suggesting that there is post-hoc data mining happening around anthropogenic climate change. Instead, it's because you're providing no evidence for this and it's in addition offtopic.
- jernfrost 8y ago
  I down voted because this is (1) whataboutism, (2) I can't see the analogy. That increased CO2 emissions should cause a temperature increase has been proposed over 100 years ago. They did not look at climate change recently and suddenly decide "lets blame CO2". (3) the man made climate change theory, does not rest of a single study, it rests on countless ones. (4) There is actually a scientific model explaining why we see the measurements we get. They are not digging around at random. The case for man made climate change is pretty strong. Look at these graphs and tell me which graph correlates most strongly with global temperature increases: https://www.bloomberg.com/graphics/2015-whats-warming-the-world/ https://www.bloomberg.com/graphics/2015-whats-warming-the-wo...
brownbat 8y ago
In our world, it's considered malpractice if you have the option to include double blinding in your study design, but opt not to for convenience. In some alternate universe, it is considered malpractice for those who design the study to be the same group that runs the study. I don't think we can get there from here, but if we had a core track of theorists who designed studies, and a second equally prestigious track of practitioners, who independently tested and ran studies, experimental science would be much more rigorous. Your prestige should be tied to your ability to identify novel experiments to try, or in rigorous testing procedures, never tied to your ability to shape data to make your claims appear grand.
qyz721 8y ago
I often wonder about the kind of research done in computer science--how much is it influenced by the kinds of things that easily get you a Ph.D., versus the kinds of things that are useful but less apparently flashy. A lot of the PL students at my school are extremely wary of doing any follow-up work on ideas that have been published before, even if the implementations of those things are obviously shoddy and don't really demonstrate that the idea works. There's a lot of novelty chasing, which is part of what's pushing people to include deep learning in their work, since it often allows them to claim novelty, even if their results aren't very good.
- SomewhatLikely 8y ago
  Another big influence is what problems can attract funding.
icelancer 8y ago
"Before 2000, researchers cheated outrageously. They tortured their data relentlessly until they found something—anything—that could be spun as a positive result, even if it had nothing to do with what they were looking for in the first place. After that behavior was banned, they stopped finding positive results. Once they had to explain beforehand what primary outcome they were looking for, practically every study came up null. The drugs turned out to be useless." This is a ridiculous "plain English" description of what is happening here, and I say that as someone who is regularly very critical of academia, drug trials, and research science (I've lived it; check my bio). Clinical trials and mandatory registration are great things. It does not mean that researchers were massively cheating in the past, however - it means that they were finding secondary and tertiary findings and reporting them instead of the main investigative thrust of the research. Yes, some blatant cheating happened, as did p-hacking (though this problem still exists), but to act like the clinical registration database completely stopped a massive ring of fraud is ridiculous and the data does not support that, merely a conspiratorial narrative around it.

330 points by curtis 8y ago

taruz 8y ago

I want a chart like that but for journalism.

clircle 8y ago
I don't follow this comment. What would you plot on the vertical axis?
- nostrademons 8y ago
  Clickthroughs, presumably. I assume that taruz is saying that journalists should be required to say what they're investigating before publishing an investigative report. Right now, they start investigating, and if there's something outrageous (even if it wasn't what they were initially looking for), they publish. Sometimes they even skip the "start investigating" part and just put up a tip-line for anyone who has a beef to get a story out there. This is great for manufacturing outrage and hence clicks, but it gives the public a hugely skewed perspective on how the world is. Imagine that 0.01% (1 in 10,000) of all peoples' actions are outrageous and will piss off a large portion of planet. Most people, by those numbers, would say that the majority of folks are decent, law-abiding citizens. Now imagine that a news outlet is allowed to freely go over someone's life, and they end up evaluating 1000 actions. There's a 10% chance that they'll find something outrageous. Now imagine that 10 such reporters do this to 10 people, and if any one of them finds something publishable (= outrageous), they go to press. Suddenly there's a 64% chance that one of them will find something, and you've likely got your news cycle for the day. With the millions of people looking for something bad that a tip-line can generate, outrage is virtually assured. And that's where journalism is today. The world isn't actually a worse place than it was in 1980; in fact, by most metrics it's significantly better. But we've increased the amount of unpleasantness that people can be exposed to by 3-4 orders of magnitude and then implemented a selection filter that ensures that only the worst stories go viral. Of course we get only bad news; that's all that's economically viable, and we have such a large sample size that we can surely find it.
  - teddyh 8y ago
    There’s also this to consider: “If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him.” – Commonly attributed to Cardinal Richelieu I.e. any fact or action can be willfully misinterpreted to fit almost any narrative.
  - jernfrost 8y ago
    That is a natural outcome of the hyper capitalist world we currently live in. I have a mother who has been a journalist for over 40 years. I've heard for years about the pressure sell. Article titles often get changed against the will of journalists to be more click-bait like to sell more. As people pay less for good journalism, and ad revenue is shrinking, the media is getting desperate to stay afloat. Journalists today have significantly less independence than they had 15 years ago. Everything is far more controlled and geared towards sales. I would like to see more alternative finance models for quality journalism. Given how social media is actually starting to destroy democracy, it may be worth considering government grants to independent media organizations as a part of national defense. A democracy cannot function if the media is utterly broken. That means citizens are no longer capable of making informed decisions.
rossdavidh 8y ago
Good idea, but you'd have to have an objective way of deciding how accurate the article was.
njstraub608 8y ago
I'd have to agree with this; every news outlet tries to treat a lack of proper controls as a breaking story / massive conspiracy theory to get a buzz going around the topic. I think it's important to inform people but unfortunately, most people would rather buy into the idea that there was malicious intent rather than also evaluating the possibility that, like most evolving business areas, it takes time to build in the proper procedures to attain measurably objective results. This happens in every industry and isn't unique to science, however, in this case, the physicians recommending these drugs to their patients should have been able to read between the lines or at the very least have a managed treatment plan to evaluate the efficacy on those they're treating.

caymanjim 8y ago

I'm cynical and have no trouble believing that results were twisted immensely prior to the rule change in 2000. The incentives are huge: profit, prestige, or simply job security. I do wonder if this chart misrepresents something, though: there are studies that produce incidental--but genuinely valuable--discoveries. It's unclear to me if that accounts for the pre-2000 results or not. With the new rules, would there have to be another study stating the new objective? I'm not questioning that bad science occurs, but I am questioning what this graph really tells us.

rdiddly 8y ago
You're right, the new rule doesn't filter out dishonest discoveries, only incidental ones (which may be dishonest). If you wanted to show benefit from an incidental discovery, you would have to run another study with that as its declared objective. The author is kind of doing the same thing here that they accuse others of doing...
scotty79 8y ago
> With the new rules, would there have to be another study stating the new objective? I think it's only fair to force you to replicate at least once the positive result you think you see in the data you collected for another purpose before you can claim you got something.
- Balgair 8y ago
  Yes, the linked article is saying that researchers used to employ techniques like 'p-hacking' (among others) in order to report results that were favorable/novel. That the scientists and clinicians in charge teased the data too much.

smrtinsert 8y ago

Clinical trials != scientific studies in general. Motherjones is doing a great disservice implying equivalence.

function_seven 8y ago
Aren't they equivalent in their potential to allow post-hoc data mining? Whether it's a clinical trial, or a psychological study with n participants, or a look at how changes in x soil conditions affects the y tree. Anything where data samples are taken and statistically analyzed is prone to p-hacking and after-the-fact hypothesis changes. This example happens to be for a narrow set of studies: clinical trials for cardiovascular drugs/supplements. The take away from that graph applies broadly.
- L_Rahman 8y ago
  And in fact the replication crisis in psychology suggests that the practice goes beyond just clinical trials.
  - gaius 8y ago
    The replication crisis in every field of science. Two-thirds of reported results cannot be reproduced.
    - dwd 8y ago
      If you haven't read it already, this article on the placebo effect a few days ago was interesting in how we on one hand use it as a baseline for measuring the effectiveness of drug treatments but actually can vary wildly between individuals. It's not surprising that results are so inconsistent. https://news.ycombinator.com/item?id=18398342 https://news.ycombinator.com/item?id=18398342
geggam 8y ago
Explain that to the layman in a way the entire Internet will understand. Until you can ... they are for all intents and purposes.

cup 8y ago

The downside of this though is that unexpected positive results sometimes get buried. For instance, clinical trials have to report all adverse effects but not positive effects. We've had drugs tested for symptom X fail to have any effect, while causing bald people to have their hair grow back. Yet the company we were testing it for wasn't interested and those results never made it into the public domain.

rickycook 8y ago
i would assume that the positive results still get reported somewhere in the paper? so other people could still pick up on it in the future. if they’re positive enough then the drug company will fund another study with that as the primary outcome. that seems prudent too: it should be the primary thing you’re examining so that you can design the study correctly, rather than a simple “oh and by the way” side note
jonathankoren 8y ago
That wouldn't happen. You would simply declare, that you want to check if the drug does cause hair growth, and then run a test explicitly looking for that. Unless you're looking for the effect from the outset, you can't be sure that what you saw wasn't actually random. There's a term for what you're describing, it's p-hacking, and it's explicitly the very thing declaring what you're looking for before you run the test is designed to prevent. Like many science related topics, there's a XKCD about p-hacking that describes a similar scenario to your example: https://xkcd.com/882/ https://xkcd.com/882/

epistasis 8y ago

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials), science is somewhat additive. Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? With the advances in treatments for cardiovascular disease couldn't the problem just be harder? And look at that "harm" study. Turns out it's one of many from the Women's Health Initiative [1], and it's the test of whether hormone therapy for post-menopausal women causes heart problems: that is, it's not a test of a drug to prevent heart conditions, but a test of side effects. How many other of these studies are like that: studies about effects on the heart, not trials for new drugs to treat the heart? How did the ratio change before and after 2000? In any case, don't trust every popular news article you read about science, particularly if it's written by Kevin Drum and posted on Mother Jones. [1] https://www.whi.org/SitePages/WHI%20Home.aspx https://www.whi.org/SitePages/WHI%20Home.aspx

prakashk 8y ago
> Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? > However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. Is it possible that your own preconceived notions about the author and the publication may have caused you to judge this way? In any case, don't trust every comment posted on Hacker News (including my own) :)
- Natsu 8y ago
  > The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc.

certmd 8y ago

While many on this forum may understand why what this analysis shows is not good science, the article does a poor job of explaining why finding positive results one wasn't originally looking for may not be reliable. I can imagine a non-statistically minded person thinking "So what it's not what they were looking for originally? We're missing positive findings now. This is a terrible regulation." When in reality these "positive" findings were p-hacked to meet minimum criteria for statistical significance and likely arose by chance since luck would have it that in any study there will be some set of data that by chance is a statistical aberration.

nemo44x 8y ago

Good science has its methods published, is reproducible and is peer reviewed. Another, perhaps better, headline would be: "Why You Shouldn't Trust Every 'Study' You See".

jacksnipe 8y ago
No True Scotsman. In fact, I’m pretty sure this is just another form of post-hoc reasoning.

DrNuke 8y ago

From a science perspective, positive results are overrated; the most important aspect is to investigate a “good” question. Researchers understand “good” has not the same meaning everywhere or to every stakeholder. More at large, blurred journalism is attacking science to undermine the scientific method, aka meaningful questions + proper scrutiny + full transparence. This is a sign of the times, though.

superjan 8y ago

One could argue that this chart too is an example of post-hoc data mining. I still find the effect plausible, though.

ineedasername 8y ago
Post-Hoc or not would depend on what hypothesis the researcher had before looking at the data. We don't really know, or at least I couldn't tell from the article.

rossdavidh 8y ago

While some have pointed out that this very article could be considered post-hoc data mining, I think it is not. This exact kind of affect was what was intended by the requirement to pre-register. Looking afterwards to see if the intended affect (change in how many studies report benefits) appears to have happened in reality, makes perfect sense. They didn't institute the requirement at random, and then later consider that maybe it could have an impact of reducing (perhaps spurious) findings of significant benefit; that was more or less the only reason for doing so.

LanceH 8y ago
The article claims the drugs are worthless. The studies only show they aren't successful in a single pre-registered result.

leesec 8y ago

Reminds me of this old classic, one of my favorites: http://slatestarcodex.com/2014/12/12/beware-the-man-of-one-study/ http://slatestarcodex.com/2014/12/12/beware-the-man-of-one-s...

corysama 8y ago

This would seem to indicate that the results of drug research are unpredictable. Therefore, if you are required to predict your results, you are set up to fail. My question is: Was the free-range research actually effective? Or, was it "technically-correct effective just so you can't call me out on a failure, moving on..."

qwerty456127 8y ago

> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductive limitation? Significant benefits found without knowing what are they going to be in advance are still significant benefits and if they are observed scientifically and proven reproducible I'm glad we've found them. > Once they had to explain beforehand what primary outcome they were looking for, practically every study came up null. The drugs turned out to be useless. Aren't newly discovered drugs meant to undergo strict and targeted clinical trials? How can they even be considered being drugs before this? And how can they turn out to be useless after passing this stage? Also in some cases when nobody wants to fund clinical trials despite very interesting life-enhancing effects supposed or when it's clear the time to general availability through the fully white research and approval chain is going to take longer than people want to wait some non-approved substances happen to be sold on eBay (or, for more questionable substances, on black market), hundreds or thousands of people buy it and report their experience on reddit and this data can be a source for further clues for research.

bgaragan 8y ago
I'm curious about this as well. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? It seems counterproductive indeed to waste time and money like that. I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find. Please correct me if I'm misguided on any of this.
- nkurz 8y ago
  I presume you are being genuine, but you should know that your comment is almost indistinguishable from satire. More specifically (and hopefully more helpfully) it sounds like the comments made by Brian Wansink before his recent fall. Google for his story if you are unfamiliar. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Assuming that the effects are legitimate is exactly the problem. You are right, if we somehow know that the effects found are real and reproducible, then all is good. The problem is that we almost never know this. Presumably what you mean is "If the results are reproducible, what's the harm?". I'd agree with this, but the problem is how to know ahead of time that the results are going to be reproducible. Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? If they did the statistics correctly (accounting for the multiple inferences, all assumptions about iid data met, no biased dropouts, everything else aboveboard) and got a really solid result, then yes, it's theoretically likely that the results would be reproducible. The problem is that they almost certainly didn't do the statistics correctly, and intentionally or not they probably violated a lot assumptions. In too many cases, even the main line conclusions can't be replicated. It's rarely "safe" to assume that an effect is real until it's actually been replicated, and almost never safe to make this assumption for result obtained by sifting the data after the fact looking for correlations. I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find. There are ways to do it this way, but it usually takes larger sample sizes than are available. It also requires "bespoke" statistics that are easy to get wrong. In practice, it's usually better to use the incidental "results" as idea generators for future experiments, rather than assuming that the findings are real and don't require further testing.

amirmasoudabdol 8y ago

Although I agree with the argument but their chart and their data actually do not represent what they are claiming. I’ve checked their paper while back [1] and contacted them about it but haven’t heard anything back. Their argument is sound but only because! [1]: https://amirmasoudabdol.name/likelihood-of-finding-null-effects-over-time/ https://amirmasoudabdol.name/likelihood-of-finding-null-effe...

fizzigy 8y ago

The article seems to suggest that scientific studies aren't reliable, but fails to point out the key takeaway that now (post registration requirement), studies are much more likely to be valid/useful than before. So the headline really should read something like "Why you should trust scientific studies a lot more now than you did before"

whack 8y ago

Is it possible that prior to 2000, the studies that found a null result, were simply not published? Hence why they are underrepresented in that chart? My understanding is that it's very hard to get null results published in scientific journals, though I'm not sure if that applies to clinical studies as well.

deleted 8y ago
[deleted]
Fomite 8y ago
One interesting effect that pre-registration has is that there are now several journals that allow you to publish your protocol, and agree to publish the followup paper about the results regardless of the outcome.

thedirt0115 8y ago

This is very similar to A/B Experimentation -- Gotta force the user to select a primary metric BEFORE the experiment starts to keep 'em honest.

ineedasername 8y ago

I'd say a good rule of thumb is that the smaller the number of observations, the less you should trust any research that hasn't been reproduced. And when it comes to publication of false positive results, always remember your XKCD: https://xkcd.com/882/ https://xkcd.com/882/

sudofail 8y ago

I have a general rule to ignore any statistical interpretation in articles like these. Data interpretation is hard. There are a lot of variables that need to be accounted for, and under the best circumstances of academic peer review, we still get it wrong. This article also feels like it has an agenda. Maybe I'm not familiar with MotherJones, but the tone of the article strikes me as unprofessional. And the headline, that's obviously correct and means nothing. It's like saying, "Why You Shouldn't Trust Every Stranger You Meet".

monochromatic 8y ago

>Post-hoc data mining is very bad, except when you’re gunning up evidence of anthropogenic climate change, which is very good

noname120 8y ago
For the record, I believe that you didn't get downvoted for suggesting that there is post-hoc data mining happening around anthropogenic climate change. Instead, it's because you're providing no evidence for this and it's in addition offtopic.
jernfrost 8y ago
I down voted because this is (1) whataboutism, (2) I can't see the analogy. That increased CO2 emissions should cause a temperature increase has been proposed over 100 years ago. They did not look at climate change recently and suddenly decide "lets blame CO2". (3) the man made climate change theory, does not rest of a single study, it rests on countless ones. (4) There is actually a scientific model explaining why we see the measurements we get. They are not digging around at random. The case for man made climate change is pretty strong. Look at these graphs and tell me which graph correlates most strongly with global temperature increases: https://www.bloomberg.com/graphics/2015-whats-warming-the-world/ https://www.bloomberg.com/graphics/2015-whats-warming-the-wo...

brownbat 8y ago

In our world, it's considered malpractice if you have the option to include double blinding in your study design, but opt not to for convenience. In some alternate universe, it is considered malpractice for those who design the study to be the same group that runs the study. I don't think we can get there from here, but if we had a core track of theorists who designed studies, and a second equally prestigious track of practitioners, who independently tested and ran studies, experimental science would be much more rigorous. Your prestige should be tied to your ability to identify novel experiments to try, or in rigorous testing procedures, never tied to your ability to shape data to make your claims appear grand.

qyz721 8y ago

I often wonder about the kind of research done in computer science--how much is it influenced by the kinds of things that easily get you a Ph.D., versus the kinds of things that are useful but less apparently flashy. A lot of the PL students at my school are extremely wary of doing any follow-up work on ideas that have been published before, even if the implementations of those things are obviously shoddy and don't really demonstrate that the idea works. There's a lot of novelty chasing, which is part of what's pushing people to include deep learning in their work, since it often allows them to claim novelty, even if their results aren't very good.

SomewhatLikely 8y ago
Another big influence is what problems can attract funding.

icelancer 8y ago

"Before 2000, researchers cheated outrageously. They tortured their data relentlessly until they found something—anything—that could be spun as a positive result, even if it had nothing to do with what they were looking for in the first place. After that behavior was banned, they stopped finding positive results. Once they had to explain beforehand what primary outcome they were looking for, practically every study came up null. The drugs turned out to be useless." This is a ridiculous "plain English" description of what is happening here, and I say that as someone who is regularly very critical of academia, drug trials, and research science (I've lived it; check my bio). Clinical trials and mandatory registration are great things. It does not mean that researchers were massively cheating in the past, however - it means that they were finding secondary and tertiary findings and reporting them instead of the main investigative thrust of the research. Yes, some blatant cheating happened, as did p-hacking (though this problem still exists), but to act like the clinical registration database completely stopped a massive ring of fraud is ridiculous and the data does not support that, merely a conspiratorial narrative around it.