5 ms·

> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data after

> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductive limitation? Significant benefits found without knowing what are they going to be in advance are still significant benefits and if they are observed scientifically and proven reproducible I'm glad we've found them. > Once they had to explain beforehand what primary outcome they were looking for, practically every study came up null. The drugs turned out to be useless. Aren't newly discovered drugs meant to undergo strict and targeted clinical trials? How can they even be considered being drugs before this? And how can they turn out to be useless after passing this stage? Also in some cases when nobody wants to fund clinical trials despite very interesting life-enhancing effects supposed or when it's clear the time to general availability through the fully white research and approval chain is going to take longer than people want to wait some non-approved substances happen to be sold on eBay (or, for more questionable substances, on black market), hundreds or thousands of people buy it and report their experience on reddit and this data can be a source for further clues for research.

bgaragan 8y ago
I'm curious about this as well. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? It seems counterproductive indeed to waste time and money like that. I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find. Please correct me if I'm misguided on any of this.
- nkurz 8y ago
  I presume you are being genuine, but you should know that your comment is almost indistinguishable from satire. More specifically (and hopefully more helpfully) it sounds like the comments made by Brian Wansink before his recent fall. Google for his story if you are unfamiliar. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Assuming that the effects are legitimate is exactly the problem. You are right, if we somehow know that the effects found are real and reproducible, then all is good. The problem is that we almost never know this. Presumably what you mean is "If the results are reproducible, what's the harm?". I'd agree with this, but the problem is how to know ahead of time that the results are going to be reproducible. Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? If they did the statistics correctly (accounting for the multiple inferences, all assumptions about iid data met, no biased dropouts, everything else aboveboard) and got a really solid result, then yes, it's theoretically likely that the results would be reproducible. The problem is that they almost certainly didn't do the statistics correctly, and intentionally or not they probably violated a lot assumptions. In too many cases, even the main line conclusions can't be replicated. It's rarely "safe" to assume that an effect is real until it's actually been replicated, and almost never safe to make this assumption for result obtained by sifting the data after the fact looking for correlations. I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find. There are ways to do it this way, but it usually takes larger sample sizes than are available. It also requires "bespoke" statistics that are easy to get wrong. In practice, it's usually better to use the incidental "results" as idea generators for future experiments, rather than assuming that the findings are real and don't require further testing.
abrichr 8y ago
The problem is that given enough data, you can always find patterns. Without a causal relationship, however, those patterns are likely spurious. For a more in-depth analysis, see "The Deluge of Spurious Correlations in Big Data": https://www.di.ens.fr/users/longo/files/BigData-Calude-LongoAug21.pdf https://www.di.ens.fr/users/longo/files/BigData-Calude-Longo... Here's the abstract: Very large databases are a major opportunity for science and data analytics is a remarkable new field of investigation in computer science. The effectiveness of these tools is used to support a ‘‘philosophy’’ against the scientific method as developed throughout history. According to this view, computer-discovered correlations should replace understanding and guide prediction and action. Consequently, there will be no need to give scientific meaning to phenomena, by proposing, say, causal relations, since regularities in very large databases are enough: ‘‘with enough data, the numbers speak for themselves’’. The ‘‘end of science’’ is proclaimed. Using classical results from ergodic theory, Ramsey theory and algorithmic information theory, we show that this ‘‘philosophy’’ is wrong. For example, we prove that very large databases have to contain arbitrary correlations. These correlations appear only due to the size, not the nature, of data. They can be found in ‘‘randomly’’ generated, large enough databases, which—as we will prove—implies that most correlations are spurious. Too much information tends to behave like very little information. The scientific method can be enriched by computer mining in immense databases, but not replaced by it.
bluGill 8y ago
Because when you analyse a lot of data after the fact you will find something. It is really easy/cheap to measure things not part of the study. If you look at 100 things and find 5 that are interesting with only a 5% chance of being a false positive they are all false positives because you expect in a sample of 100 to find 5 false positives. Analysis after the fact isn't useless. However the bar of significance needs to be much higher. You need a much larger sample size, or better yet design a new study (controlling for things the original didn't) to draw conclusions. Note, technically statistics do not expect to find 5 false positives in 100. It is close enough for discussion and makes intuitive sense. If you want the real truth be prepared for a lot of math.
monktastic1 8y ago
Are you familiar with the multiple comparisons problem[1]? The problem is that in any sufficiently rich dataset, you can find something unusual-looking, and if you're not straightforward about how much digging you had to do to find it, it will look much more special than it actually is. So if a benefit is found by chance, that's fine, but it should only motivate further research which specifically look for that effect. [1] https://en.wikipedia.org/wiki/Multiple_comparisons_problem https://en.wikipedia.org/wiki/Multiple_comparisons_problem
- qwerty456127 8y ago
  > but it should only motivate further research which specifically look for that effect Indeed. But this doesn't disqualify looking for "something" as a valid and useful method of research to be a stage of the whole research chain. That ought to be allowed although research papers produced this way should make this clear.
  - lolc 8y ago
    It's called exploratory research and everybody is fine with it when it's labeled as such.
xyzzyz 8y ago
"Significant" here doesn't mean what you think it means, but rather it has a technical meaning. Basically, it means that if the intervention has absolutely no effect, then there's less than 5% probability that measured "benefit" is just a random fluke -- thus, since you actually measured benefit, you are now left wondering whether the intervention actually has an effect, or you just happen to be in 5% of possible universes where the randomness just happened to align this way. Of course, the first one is usually more likely, but 5% probability of getting result if no effect actually exists is still pretty high. The simplest, most straightforward explanation for the mechanism, and why it's bad, is this: https://xkcd.com/882/ https://xkcd.com/882/ In the old scheme of things, the researchers would have found and reported "significant" bad outcomes connected to green jelly beans. With required preregistration, they must preregister 20 studies, so if 19 similar studies show no effect, and 1 of them shows "significant" effect, then it's likely that you're just dealing with random fluke.
- qwerty456127 8y ago
  > since you actually measured benefit, you are now left wondering whether the intervention actually has an effect, or you just happen to be in 5% of possible universes where the randomness just happened to align this way. That still seems a good first step. It may also be curious to find out what were the things that have actually lead to the effect if not the one we were checking. E.g. it may happen that it the substance researched only affects subjects with particular features or that something else they were doing has produced the result on itself and knowing any of these still seems useful (some really important discoveries were made this way AFAIK).
  - xyzzyz 8y ago
    > It may also be curious to find out what were the things that have actually lead to the effect if not the one we were checking. This sounds like a good idea in theory, but in practice, it is usually even harder to answer this question than the original question that yielded the data. This type of question requires extremely careful study designs, large sample sizes, accurate power estimates, etc.
ineedasername 8y ago
It's not a counterproductive limitation. In fact it should lead to more research: In this type of clinical experiment, if your data happens to show a significant result that wasn't in the pre-planned outcomes, the proper process to follow is to formulate a second course of research designed around validating that new hypothesis.
- BurningFrog 8y ago
  That's a nice theory. I worry that in reality a lot of those second research projects will never happen.
  - ineedasername 8y ago
    You're right, it may not have had that result. But I think if the pharmaceutical companies had potentially significant results there would be a strong financial incentive to follow up with a another targeted study. It would be interesting to test, look and see whether the average number of trials per unit of time, probably year, increase after this change was made. I'm curious, but not enough to actually do the work :) Always more interesting question out there than time to follow up on them.

bgaragan 8y ago

I'm curious about this as well. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? It seems counterproductive indeed to waste time and money like that. I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find. Please correct me if I'm misguided on any of this.

nkurz 8y ago
I presume you are being genuine, but you should know that your comment is almost indistinguishable from satire. More specifically (and hopefully more helpfully) it sounds like the comments made by Brian Wansink before his recent fall. Google for his story if you are unfamiliar. Assuming the effects they found were actually legitimate (that is, the alleged "torture" of the data was statistically sound), how is this at all a bad thing? Assuming that the effects are legitimate is exactly the problem. You are right, if we somehow know that the effects found are real and reproducible, then all is good. The problem is that we almost never know this. Presumably what you mean is "If the results are reproducible, what's the harm?". I'd agree with this, but the problem is how to know ahead of time that the results are going to be reproducible. Shouldn't it be safe to assume that they could simply repeat the trial with a different stated objective and successfully yield the positive result? If they did the statistics correctly (accounting for the multiple inferences, all assumptions about iid data met, no biased dropouts, everything else aboveboard) and got a really solid result, then yes, it's theoretically likely that the results would be reproducible. The problem is that they almost certainly didn't do the statistics correctly, and intentionally or not they probably violated a lot assumptions. In too many cases, even the main line conclusions can't be replicated. It's rarely "safe" to assume that an effect is real until it's actually been replicated, and almost never safe to make this assumption for result obtained by sifting the data after the fact looking for correlations. I'm having a hard time seeing the problem with taking an exploratory approach and just testing placebo vs. some treatment and reporting whatever you find. There are ways to do it this way, but it usually takes larger sample sizes than are available. It also requires "bespoke" statistics that are easy to get wrong. In practice, it's usually better to use the incidental "results" as idea generators for future experiments, rather than assuming that the findings are real and don't require further testing.

abrichr 8y ago

The problem is that given enough data, you can always find patterns. Without a causal relationship, however, those patterns are likely spurious. For a more in-depth analysis, see "The Deluge of Spurious Correlations in Big Data": https://www.di.ens.fr/users/longo/files/BigData-Calude-LongoAug21.pdf https://www.di.ens.fr/users/longo/files/BigData-Calude-Longo... Here's the abstract: Very large databases are a major opportunity for science and data analytics is a remarkable new field of investigation in computer science. The effectiveness of these tools is used to support a ‘‘philosophy’’ against the scientific method as developed throughout history. According to this view, computer-discovered correlations should replace understanding and guide prediction and action. Consequently, there will be no need to give scientific meaning to phenomena, by proposing, say, causal relations, since regularities in very large databases are enough: ‘‘with enough data, the numbers speak for themselves’’. The ‘‘end of science’’ is proclaimed. Using classical results from ergodic theory, Ramsey theory and algorithmic information theory, we show that this ‘‘philosophy’’ is wrong. For example, we prove that very large databases have to contain arbitrary correlations. These correlations appear only due to the size, not the nature, of data. They can be found in ‘‘randomly’’ generated, large enough databases, which—as we will prove—implies that most correlations are spurious. Too much information tends to behave like very little information. The scientific method can be enriched by computer mining in immense databases, but not replaced by it.

bluGill 8y ago

Because when you analyse a lot of data after the fact you will find something. It is really easy/cheap to measure things not part of the study. If you look at 100 things and find 5 that are interesting with only a 5% chance of being a false positive they are all false positives because you expect in a sample of 100 to find 5 false positives. Analysis after the fact isn't useless. However the bar of significance needs to be much higher. You need a much larger sample size, or better yet design a new study (controlling for things the original didn't) to draw conclusions. Note, technically statistics do not expect to find 5 false positives in 100. It is close enough for discussion and makes intuitive sense. If you want the real truth be prepared for a lot of math.

monktastic1 8y ago

Are you familiar with the multiple comparisons problem[1]? The problem is that in any sufficiently rich dataset, you can find something unusual-looking, and if you're not straightforward about how much digging you had to do to find it, it will look much more special than it actually is. So if a benefit is found by chance, that's fine, but it should only motivate further research which specifically look for that effect. [1] https://en.wikipedia.org/wiki/Multiple_comparisons_problem https://en.wikipedia.org/wiki/Multiple_comparisons_problem

qwerty456127 8y ago
> but it should only motivate further research which specifically look for that effect Indeed. But this doesn't disqualify looking for "something" as a valid and useful method of research to be a stage of the whole research chain. That ought to be allowed although research papers produced this way should make this clear.
- lolc 8y ago
  It's called exploratory research and everybody is fine with it when it's labeled as such.

xyzzyz 8y ago

"Significant" here doesn't mean what you think it means, but rather it has a technical meaning. Basically, it means that if the intervention has absolutely no effect, then there's less than 5% probability that measured "benefit" is just a random fluke -- thus, since you actually measured benefit, you are now left wondering whether the intervention actually has an effect, or you just happen to be in 5% of possible universes where the randomness just happened to align this way. Of course, the first one is usually more likely, but 5% probability of getting result if no effect actually exists is still pretty high. The simplest, most straightforward explanation for the mechanism, and why it's bad, is this: https://xkcd.com/882/ https://xkcd.com/882/ In the old scheme of things, the researchers would have found and reported "significant" bad outcomes connected to green jelly beans. With required preregistration, they must preregister 20 studies, so if 19 similar studies show no effect, and 1 of them shows "significant" effect, then it's likely that you're just dealing with random fluke.

qwerty456127 8y ago
> since you actually measured benefit, you are now left wondering whether the intervention actually has an effect, or you just happen to be in 5% of possible universes where the randomness just happened to align this way. That still seems a good first step. It may also be curious to find out what were the things that have actually lead to the effect if not the one we were checking. E.g. it may happen that it the substance researched only affects subjects with particular features or that something else they were doing has produced the result on itself and knowing any of these still seems useful (some really important discoveries were made this way AFAIK).
- xyzzyz 8y ago
  > It may also be curious to find out what were the things that have actually lead to the effect if not the one we were checking. This sounds like a good idea in theory, but in practice, it is usually even harder to answer this question than the original question that yielded the data. This type of question requires extremely careful study designs, large sample sizes, accurate power estimates, etc.

ineedasername 8y ago

It's not a counterproductive limitation. In fact it should lead to more research: In this type of clinical experiment, if your data happens to show a significant result that wasn't in the pre-planned outcomes, the proper process to follow is to formulate a second course of research designed around validating that new hypothesis.

BurningFrog 8y ago
That's a nice theory. I worry that in reality a lot of those second research projects will never happen.
- ineedasername 8y ago
  You're right, it may not have had that result. But I think if the pharmaceutical companies had potentially significant results there would be a strong financial incentive to follow up with a another targeted study. It would be interesting to test, look and see whether the average number of trials per unit of time, probably year, increase after this change was made. I'm curious, but not enough to actually do the work :) Always more interesting question out there than time to follow up on them.