36 ms·

It doesn't limit the size of the elephant. Any observation is consistent with an infinite number of models. Eg., your sight is defective, ie., in many cases:

It doesn't limit the size of the elephant. Any observation is consistent with an infinite number of models. Eg., your sight is defective, ie., in many cases: your sample is biased, not big enough, etc. And that A correlates with B, or fails to, to "some significance" is consistent with any causal relationship between A and B. > Surveys of hundreds of articles have found that statistically non-significant results are interpreted as indicating ‘no difference’ or ‘no effect’ in around half (see ‘Wrong interpretations’ and Supplementary Information).

whatshisface 8y ago
>Eg., your sight is defective, ie., in many cases: your sample is biased, not big enough, etc. You can't expect a statistical test to comb through your experimental procedure to check whether or not you did it right. (Least of all, to comb through your personal medical history. ;) ) >Any observation is consistent with an infinite number of models. I'm afraid that slips below the "get out of bed in the morning" pragmatism bar. Although it's a provocative idea, concluding that conclusions are impossible because you could be making a mistake or hallucinating would lead to, literally, not getting out of bed in the morning, if you took it seriously. At the end of the day you just have to hope that the human beings reading your paper will be able to tell whether or not you did the experiment right. Any statistical test just walls off one way (or a few ways) in which you could fool yourself. >Surveys of hundreds of articles have found that statistically non-significant results are interpreted as indicating ‘no difference’ or ‘no effect’ in around half (see ‘Wrong interpretations’ and Supplementary Information). What we really want to avoid here is requiring every paper to have a twelve paragraph Sophist disclaimer about how knowledge is impossible. ;) Just because the text doesn't say "oh and by the way there could be gnomes in the LHC that are fooling us to entertain themselves," doesn't mean that the author wouldn't begrudgingly acknowledge it as an epistemic risk.
- mjburgess 8y ago
  Well your "pragmatism bar" is higher that all future science. It isnt a trivial sceptical point. Does CO2 cause climate change, or climate change cause C02, or are they both caused by a hidden factor (one of many) or do they co-vary accidentally (ie., no joint cause)? No "statistical significance" of any kind can decide between them, yet this is how it is used. There is no way to do so with statistics; it requires repeated experimentation and refutation of all plausible causal models until the one remains stands as "the most plausible". "Statistics" is being used in place of the scientific method which it cannot replace. Covariation is not causation: the covarying of any number of variables is consistent with any number of causal models. No "test" decides.
  - whatshisface 8y ago
    The evidence for anthropogenic climate change involves a lot more than correlation, so maybe that isn't the best example. In any case, even if everyone on Earth is misinterpreting p-values, that's not a problem with p-values.
  - mch82 8y ago
    The point of statistics isn’t to attain absolute certainty or for the statistical test to make a decision. The point is for people to make a decision using probability as one available decision aid, and then deciding whether to place a bet.
- pmoriarty 8y ago
  "concluding that conclusions are impossible because you could be making a mistake or hallucinating would lead to, literally, not getting out of bed in the morning, if you took it seriously." Not if you don't have free will, which is apparently the conclusion that a lot of scientists seem to be coming to recently.
  - whatshisface 8y ago
    Haha, then allow me to rephrase. A brain whose molecules react that conclusions are impossible will no longer chemically get out of bed in the morning.
    - pmoriarty 8y ago
      Why can't you act in ways that are at odds with your conclusions? When I look around me I see people doing this every day.
  - xv10011 8y ago
    I don't believe that, but I concede the possibility that scientists who do unimaginative mainstream studies like gut bacteria or MRI free will "experiments" do in fact not have free will.
  - xbbx11 8y ago
    Again, a science critical comment was censored here. The parasite mafia needs to keep up appearances.
EpiMath 8y ago
This is why we report statistical power. It does in fact put probabalistic limits on the size of the elephant. In effect, we can make a statement like "at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table". It's not reasonable to interpret the results of a single study without consideration of power.
- blotter_paper 8y ago
  > It does in fact put probabalistic limits on the size of the elephant. Emphasis added (obviously). This word was missing from the analysis of whatshisface (GGP). While all of our models are probabilistic, and it would be silly to constantly describe everything this way in normal parlance, a discussion of how we want to go about building and interpreting models of the world is exactly the domain where this needs to be explicitly stated. You're not wrong here, but neither is GP. GGP is wrong, though not provably so (again, obviously). > In effect, we can make a statement like "at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table". The "in effect" disclaimer makes this statement arguable, but fundamentally we still can't say that. We can only say whether or not an elephant was detected, since detection (in this sense) is subjective. We can't say whether elephants actually exist or not. Perhaps all tables have invisible elephants on them, or perhaps all elephants are hallucinations shared by multiple humans through crazy coincidence. If 50% of elephants that actually exist on tables are invisible, and 10% of visible elephants are not detected due to error, then only 45% of studies will detect an elephant even though it will still seem like 90% to best of our knowledge. Since we don't know what percentage of elephants are currently detectable we need to be even more ridiculous and say something like "we guesstimate that there is a 99ish% chance that at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table". Edit: s/whathisface/whatshisface
  - EpiMath 8y ago
    Yes, okay. There are implicit assumptions in scientific studies ( e.g.: there are no invisible elephants, or the study we are doing is actually related to the question we are investigating! ). Power calculations refer to the model and not to whether it is the correct model. To some extent we routinely worry about certain types of "hidden from observation" problems: we have zero-inflated poisson models, or we worry about what happens if there is a limited susceptible subpopulation ( that could deplete over time differentially depending on treatment, etc ). But it is not correct to suggest that if a study of whatever power does not find an effect, then a huge effect and no effect are equally plausible. I have seen outrageous examples of "accepting the null hypothesis" many times, but many negative result studies have great value and even a single negative study can provide evidence against a very large effect.

whatshisface 8y ago

>Eg., your sight is defective, ie., in many cases: your sample is biased, not big enough, etc. You can't expect a statistical test to comb through your experimental procedure to check whether or not you did it right. (Least of all, to comb through your personal medical history. ;) ) >Any observation is consistent with an infinite number of models. I'm afraid that slips below the "get out of bed in the morning" pragmatism bar. Although it's a provocative idea, concluding that conclusions are impossible because you could be making a mistake or hallucinating would lead to, literally, not getting out of bed in the morning, if you took it seriously. At the end of the day you just have to hope that the human beings reading your paper will be able to tell whether or not you did the experiment right. Any statistical test just walls off one way (or a few ways) in which you could fool yourself. >Surveys of hundreds of articles have found that statistically non-significant results are interpreted as indicating ‘no difference’ or ‘no effect’ in around half (see ‘Wrong interpretations’ and Supplementary Information). What we really want to avoid here is requiring every paper to have a twelve paragraph Sophist disclaimer about how knowledge is impossible. ;) Just because the text doesn't say "oh and by the way there could be gnomes in the LHC that are fooling us to entertain themselves," doesn't mean that the author wouldn't begrudgingly acknowledge it as an epistemic risk.

mjburgess 8y ago
Well your "pragmatism bar" is higher that all future science. It isnt a trivial sceptical point. Does CO2 cause climate change, or climate change cause C02, or are they both caused by a hidden factor (one of many) or do they co-vary accidentally (ie., no joint cause)? No "statistical significance" of any kind can decide between them, yet this is how it is used. There is no way to do so with statistics; it requires repeated experimentation and refutation of all plausible causal models until the one remains stands as "the most plausible". "Statistics" is being used in place of the scientific method which it cannot replace. Covariation is not causation: the covarying of any number of variables is consistent with any number of causal models. No "test" decides.
- whatshisface 8y ago
  The evidence for anthropogenic climate change involves a lot more than correlation, so maybe that isn't the best example. In any case, even if everyone on Earth is misinterpreting p-values, that's not a problem with p-values.
- mch82 8y ago
  The point of statistics isn’t to attain absolute certainty or for the statistical test to make a decision. The point is for people to make a decision using probability as one available decision aid, and then deciding whether to place a bet.
pmoriarty 8y ago
"concluding that conclusions are impossible because you could be making a mistake or hallucinating would lead to, literally, not getting out of bed in the morning, if you took it seriously." Not if you don't have free will, which is apparently the conclusion that a lot of scientists seem to be coming to recently.
- whatshisface 8y ago
  Haha, then allow me to rephrase. A brain whose molecules react that conclusions are impossible will no longer chemically get out of bed in the morning.
  - pmoriarty 8y ago
    Why can't you act in ways that are at odds with your conclusions? When I look around me I see people doing this every day.
- xv10011 8y ago
  I don't believe that, but I concede the possibility that scientists who do unimaginative mainstream studies like gut bacteria or MRI free will "experiments" do in fact not have free will.
- xbbx11 8y ago
  Again, a science critical comment was censored here. The parasite mafia needs to keep up appearances.

EpiMath 8y ago

This is why we report statistical power. It does in fact put probabalistic limits on the size of the elephant. In effect, we can make a statement like "at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table". It's not reasonable to interpret the results of a single study without consideration of power.

blotter_paper 8y ago
> It does in fact put probabalistic limits on the size of the elephant. Emphasis added (obviously). This word was missing from the analysis of whatshisface (GGP). While all of our models are probabilistic, and it would be silly to constantly describe everything this way in normal parlance, a discussion of how we want to go about building and interpreting models of the world is exactly the domain where this needs to be explicitly stated. You're not wrong here, but neither is GP. GGP is wrong, though not provably so (again, obviously). > In effect, we can make a statement like "at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table". The "in effect" disclaimer makes this statement arguable, but fundamentally we still can't say that. We can only say whether or not an elephant was detected, since detection (in this sense) is subjective. We can't say whether elephants actually exist or not. Perhaps all tables have invisible elephants on them, or perhaps all elephants are hallucinations shared by multiple humans through crazy coincidence. If 50% of elephants that actually exist on tables are invisible, and 10% of visible elephants are not detected due to error, then only 45% of studies will detect an elephant even though it will still seem like 90% to best of our knowledge. Since we don't know what percentage of elephants are currently detectable we need to be even more ridiculous and say something like "we guesstimate that there is a 99ish% chance that at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table". Edit: s/whathisface/whatshisface
- EpiMath 8y ago
  Yes, okay. There are implicit assumptions in scientific studies ( e.g.: there are no invisible elephants, or the study we are doing is actually related to the question we are investigating! ). Power calculations refer to the model and not to whether it is the correct model. To some extent we routinely worry about certain types of "hidden from observation" problems: we have zero-inflated poisson models, or we worry about what happens if there is a limited susceptible subpopulation ( that could deplete over time differentially depending on treatment, etc ). But it is not correct to suggest that if a study of whatever power does not find an effect, then a huge effect and no effect are equally plausible. I have seen outrageous examples of "accepting the null hypothesis" many times, but many negative result studies have great value and even a single negative study can provide evidence against a very large effect.