5 ms·

Only badly designed experiments overrely on p-values. A robustly designed experiment should produce repeatable, intuitive results. Anyone looking at the presen

Only badly designed experiments overrely on p-values. A robustly designed experiment should produce repeatable, intuitive results. Anyone looking at the presented data should be able to reason the result without needing to conduct a statistical test. Obviously, these tests are important, but people focus too much on the p-value rather than designing an experiment which will produce a meaningful result (regardless of whether the result is positive or negative). Proper experimental design is like a cake, and p-values are the frosting. Everybody wants cake, most often with frosting. Nobody wants frosting without the cake, or at least those that do have questionable taste.

tlb 9y ago
Not all phenomena are strong enough to allow such experiments. Some make only a small difference to an inherently noisy process. These can only be discovered with statistical tests on substantial samples.
- toufka 9y ago
  Certainly, but a well-designed experiment can identify the wedges that differentiate hypothesis A from hypothesis B without requiring a statistical analysis of noisy data. And true, there are certain kinds of experiments that can only be done statistically. However, depending on the field, it can just be easier to do, and really should just be considered lazy science - again the details matter. In my field (biochemistry/genetics) there really is a very real effort to design experiments that do not require statistics to verify the result. And that effort in design is rewarded amply when the result of the experiment becomes quite clearly binary in nature, from multiple angles, rather that a statistical "probably".
  - tlb 9y ago
    Are there many real examples of studies where statistical tests are needed to see the effect, but a different experiment could demonstrate the same effect without any statistics? I haven't been able to think of one in the fields I know well.
    - toufka 9y ago
      In my field it often comes down to 'mechanism'. If you see an effect of part X on part Y, via statistics, you could write a paper, and do good science. However, if you have an experiment that pulls X from the system, watches Y fail, then readds X in a new way to recreate the system, you can be much more certain that X influences Y. That becomes a much desired, "elegant" experiment. The later experiment is often harder, and might not even be proposable until you've seen some statistical effect. However the scientific results from that experiment really don't hinge on statistical signifigances. The results are either binary in result, or better yet, contradictory - elucidating hitherto unknown variables (something a purely statistical result won't help with). And if mechanism is shown in multiple systems, with multiple techniques, the results can very quickly become near-definitive.
      - sndean 9y ago
        > However, if you have an experiment that pulls X from the system, watches Y fail, then readds X in a new way to recreate the system, you can be much more certain that X influences Y. That becomes a much desired, "elegant" experiment. Yeah, in molecular biology this always seemed to be the most convincing evidence. Deleting a gene and then adding it back. Sometimes going a step further and overexpressing the gene. You can generally publish that data without any statistical analysis. > The later experiment is often harder, and might not even be proposable until you've seen some statistical effect. I think the better journals/reviewers can push back at this step, though: If you don't have a demonstrable mechanism, you can't publish. Don't let people stop at "...this gene is important (p < 0.005)." (I know not all things are as simple as gene deletion and complementation)
    - ianamartin 9y ago
      I think examples would be when identifying necessary and sufficient causes for a given effect. A trivial case would be figuring out how fire works. You have have heat, fuel, and oxygen. Take any of those away, and there is no fire. This can be demonstrated by direct experiment. But it’s unfortunately a pretty blunt tool. Because the real (and non-trivial) case is that this isn’t strictly true. What you need is not specifically oxygen, but rather anything that can function as an oxidizing agent. Still demonstrable by direct experiment, but more difficult. And how you want to understand fire is directly related to what you want to do with it: create it? Or make it go away? If you want to make fire, it might be enough to understand the basic three element. You stumble onto the fact that magnesium will burn, and that’s good enough that you don’t look any further. If you’re interested in putting fires out, it’s also “easy” to show that sumberging a burning object in water puts out the fire by direct experiment. Until you try to put out a magnesium fire like that. Similarly a phone battery that’s burning up will get worse if you dunk it in water. Then on top of that, there are things that act like fire but are fundamentally different. Nuclear reaction as opposed to chemical ones. Now, of course, we have a pretty good idea of how all of this works at a molecular or atomic level, but direct experimentation gets us a certain way down the path of investigating what’s really going on, even though it’s pretty imperfect. But pretend that you have no knowledge of chemistry or physics, and you goal is to understand what causes fire. What you do know is some math and some statistics. How do you design a statistical approach to figuring out what makes fire? Or what makes fire go away? You know that fire is related to heat because it’s hot. But you don’t know how. We also know that fire is related to wood because that our most frequent experience with fire. We think fire causes heat because everything that’s on fire is hot, and only some things not on fire are hot. So there’s our null hypothesis: fire causes heat. The obvious alternative hypothesis is that heat causes fire. And because we’re somewhat primitive in this example, we are really focusing on wood. So we sample as many different types of burning wood as we can find. We go through the effort of finding samples of 1800 different types of tree on fire. (Doesn’t matter how the fire started, it just happened). We find in every case that the burning wood was associated with heat of a certain temperature. So we accept the null hypothesis and reject the alternative. Obviously, there’s a shit-ton of things that are obviously wrong with that statistical approach. And it probably seems comically bad to everyone. But the fact is that once you wade through the jargon specific to a given field, that’s really what you’re dealing with. Null and alternative hypotheses that are causally confused, accepting rather than failing to reject . . . The whole thing. It happens mostly in fields where you cannot discover causal mechanisms through greater understanding of the platform. Social sciences, Econ, neuroscience, Climatology, etc. And in the bad case above, note that the p-value would make no difference. The design itself and the misinterpretation is to blame. Not the p-value itself. Yes, I think p-values and hacking them is a huge problem. But not because people have a habit of creatively interpreting data to get published. It’s because a focus on p-values encourages bad bad experimental designs, and reviewers don’t pay enough attention to that.
- Bromskloss 9y ago
  Wasn't there some famous scientist (Feynman?) who said that if it requires a statistical test, it's not real science? (Or am I making this up?) In any case, while I can't say I agree (and you gave the reason why), it sure is a more _exiting_ experiment if the result is immediately apparent!
  - jamessb 9y ago
    Ernest Rutherford: "If your experiment needs statistics, you ought to have done a better experiment." (Gaither's Dictionary of Scientific Quotations gives the attribution "In N.T. Bailey The Mathematical Approach to Biology and Medicine")
    - searine 9y ago
      Exactly. I need this tattoo'd on my forehead.
ak_yo 9y ago
> Anyone looking at the presented data should be able to reason the result without needing to conduct a statistical test. Part of the problem, though, is that scientific results are often counterintuitive, or (particularly in the social sciences) are easy to view as obvious after we know what the answer is. One famous example: the sociologist Paul Lazarsfeld once gave a talk where he asked whether urban or rural men had adjusted more easily to the day-to-day routines of military service in World War II. There are equally obvious reasons why rural men would adjust more easily (e.g. they have more experience with physical labor, have experience with guns) or urban men would (e.g. they're used to crowded conditions, have more experience with a hierarchical division of labor).
- searine 9y ago
  >Part of the problem, though, is that scientific results are often counterintuitive The only reason they are counterintuitive is because the current model of intuition is flawed. A good experiment creates new intuition by creating a new model by which once counterintuitive results become intuitive.
KGIII 9y ago
I'm not sure that the results should be intuitive. I have a moderate fascination with quantum mechanics and I don't find much of that to really be intuitive. Some of it is pretty hard for me to fully grasp.
- searine 9y ago
  >I'm not sure that the results should be intuitive. This is an appeal to complexity fallacy. A result may initially be counterintuitive, but it is the the duty of the experimenter to alter their model to fit the new data. To explain the results in terms of known phenomenon and develop a new intuition which includes the unexpected results in the model. If you are having trouble fitting a result into your current intuition, it is because your intuition is an using outdated or improper model of the phenomenon. Nobody is born with intuition. It is learned, via bayesian style updates of information to model reality. The more effort we put into quality updates of our intuitive model, the better our intuition of reality becomes.

tlb 9y ago

Not all phenomena are strong enough to allow such experiments. Some make only a small difference to an inherently noisy process. These can only be discovered with statistical tests on substantial samples.

toufka 9y ago
Certainly, but a well-designed experiment can identify the wedges that differentiate hypothesis A from hypothesis B without requiring a statistical analysis of noisy data. And true, there are certain kinds of experiments that can only be done statistically. However, depending on the field, it can just be easier to do, and really should just be considered lazy science - again the details matter. In my field (biochemistry/genetics) there really is a very real effort to design experiments that do not require statistics to verify the result. And that effort in design is rewarded amply when the result of the experiment becomes quite clearly binary in nature, from multiple angles, rather that a statistical "probably".
- tlb 9y ago
  Are there many real examples of studies where statistical tests are needed to see the effect, but a different experiment could demonstrate the same effect without any statistics? I haven't been able to think of one in the fields I know well.
  - toufka 9y ago
    In my field it often comes down to 'mechanism'. If you see an effect of part X on part Y, via statistics, you could write a paper, and do good science. However, if you have an experiment that pulls X from the system, watches Y fail, then readds X in a new way to recreate the system, you can be much more certain that X influences Y. That becomes a much desired, "elegant" experiment. The later experiment is often harder, and might not even be proposable until you've seen some statistical effect. However the scientific results from that experiment really don't hinge on statistical signifigances. The results are either binary in result, or better yet, contradictory - elucidating hitherto unknown variables (something a purely statistical result won't help with). And if mechanism is shown in multiple systems, with multiple techniques, the results can very quickly become near-definitive.
    - sndean 9y ago
      > However, if you have an experiment that pulls X from the system, watches Y fail, then readds X in a new way to recreate the system, you can be much more certain that X influences Y. That becomes a much desired, "elegant" experiment. Yeah, in molecular biology this always seemed to be the most convincing evidence. Deleting a gene and then adding it back. Sometimes going a step further and overexpressing the gene. You can generally publish that data without any statistical analysis. > The later experiment is often harder, and might not even be proposable until you've seen some statistical effect. I think the better journals/reviewers can push back at this step, though: If you don't have a demonstrable mechanism, you can't publish. Don't let people stop at "...this gene is important (p < 0.005)." (I know not all things are as simple as gene deletion and complementation)
  - ianamartin 9y ago
    I think examples would be when identifying necessary and sufficient causes for a given effect. A trivial case would be figuring out how fire works. You have have heat, fuel, and oxygen. Take any of those away, and there is no fire. This can be demonstrated by direct experiment. But it’s unfortunately a pretty blunt tool. Because the real (and non-trivial) case is that this isn’t strictly true. What you need is not specifically oxygen, but rather anything that can function as an oxidizing agent. Still demonstrable by direct experiment, but more difficult. And how you want to understand fire is directly related to what you want to do with it: create it? Or make it go away? If you want to make fire, it might be enough to understand the basic three element. You stumble onto the fact that magnesium will burn, and that’s good enough that you don’t look any further. If you’re interested in putting fires out, it’s also “easy” to show that sumberging a burning object in water puts out the fire by direct experiment. Until you try to put out a magnesium fire like that. Similarly a phone battery that’s burning up will get worse if you dunk it in water. Then on top of that, there are things that act like fire but are fundamentally different. Nuclear reaction as opposed to chemical ones. Now, of course, we have a pretty good idea of how all of this works at a molecular or atomic level, but direct experimentation gets us a certain way down the path of investigating what’s really going on, even though it’s pretty imperfect. But pretend that you have no knowledge of chemistry or physics, and you goal is to understand what causes fire. What you do know is some math and some statistics. How do you design a statistical approach to figuring out what makes fire? Or what makes fire go away? You know that fire is related to heat because it’s hot. But you don’t know how. We also know that fire is related to wood because that our most frequent experience with fire. We think fire causes heat because everything that’s on fire is hot, and only some things not on fire are hot. So there’s our null hypothesis: fire causes heat. The obvious alternative hypothesis is that heat causes fire. And because we’re somewhat primitive in this example, we are really focusing on wood. So we sample as many different types of burning wood as we can find. We go through the effort of finding samples of 1800 different types of tree on fire. (Doesn’t matter how the fire started, it just happened). We find in every case that the burning wood was associated with heat of a certain temperature. So we accept the null hypothesis and reject the alternative. Obviously, there’s a shit-ton of things that are obviously wrong with that statistical approach. And it probably seems comically bad to everyone. But the fact is that once you wade through the jargon specific to a given field, that’s really what you’re dealing with. Null and alternative hypotheses that are causally confused, accepting rather than failing to reject . . . The whole thing. It happens mostly in fields where you cannot discover causal mechanisms through greater understanding of the platform. Social sciences, Econ, neuroscience, Climatology, etc. And in the bad case above, note that the p-value would make no difference. The design itself and the misinterpretation is to blame. Not the p-value itself. Yes, I think p-values and hacking them is a huge problem. But not because people have a habit of creatively interpreting data to get published. It’s because a focus on p-values encourages bad bad experimental designs, and reviewers don’t pay enough attention to that.
Bromskloss 9y ago
Wasn't there some famous scientist (Feynman?) who said that if it requires a statistical test, it's not real science? (Or am I making this up?) In any case, while I can't say I agree (and you gave the reason why), it sure is a more _exiting_ experiment if the result is immediately apparent!
- jamessb 9y ago
  Ernest Rutherford: "If your experiment needs statistics, you ought to have done a better experiment." (Gaither's Dictionary of Scientific Quotations gives the attribution "In N.T. Bailey The Mathematical Approach to Biology and Medicine")
  - searine 9y ago
    Exactly. I need this tattoo'd on my forehead.

ak_yo 9y ago

> Anyone looking at the presented data should be able to reason the result without needing to conduct a statistical test. Part of the problem, though, is that scientific results are often counterintuitive, or (particularly in the social sciences) are easy to view as obvious after we know what the answer is. One famous example: the sociologist Paul Lazarsfeld once gave a talk where he asked whether urban or rural men had adjusted more easily to the day-to-day routines of military service in World War II. There are equally obvious reasons why rural men would adjust more easily (e.g. they have more experience with physical labor, have experience with guns) or urban men would (e.g. they're used to crowded conditions, have more experience with a hierarchical division of labor).

searine 9y ago
>Part of the problem, though, is that scientific results are often counterintuitive The only reason they are counterintuitive is because the current model of intuition is flawed. A good experiment creates new intuition by creating a new model by which once counterintuitive results become intuitive.

KGIII 9y ago

I'm not sure that the results should be intuitive. I have a moderate fascination with quantum mechanics and I don't find much of that to really be intuitive. Some of it is pretty hard for me to fully grasp.

searine 9y ago
>I'm not sure that the results should be intuitive. This is an appeal to complexity fallacy. A result may initially be counterintuitive, but it is the the duty of the experimenter to alter their model to fit the new data. To explain the results in terms of known phenomenon and develop a new intuition which includes the unexpected results in the model. If you are having trouble fitting a result into your current intuition, it is because your intuition is an using outdated or improper model of the phenomenon. Nobody is born with intuition. It is learned, via bayesian style updates of information to model reality. The more effort we put into quality updates of our intuitive model, the better our intuition of reality becomes.