4 ms·

At the very least, Romain Brette suggests to change the wording to "statistically detectable". I 'm not sure if abandoning tests altogether is good though. Wha

At the very least, Romain Brette suggests to change the wording to "statistically detectable". I 'm not sure if abandoning tests altogether is good though. What does it mean "it's detectable but not clear" for communication? How do you e.g. communicate global warming like that?

hannob 7y ago
I usually don't see climate science being communicated with p-values and poor significance cutoffs. The place where you usually see this used is for communicating single study results (which often enough are things that you probably shouldn't communicate at all). To get an idea how climate science is trying to communicate look at the summary for policymakers of the SR15 report: https://report.ipcc.ch/sr15/pdf/sr15_spm_final.pdf https://report.ipcc.ch/sr15/pdf/sr15_spm_final.pdf They have different confidence levels they indicate instead of a single cutoff.
- buboard 7y ago
  That's not different than communicating significance with p < 0.05 , p < 0.01, p < 0.001 etc as is usually done in paper. There is again a cutoff.
  - Angostura 7y ago
    It's different in that the cut-offs are communicated with careful wording, rather than a figure.
    - buboard 7y ago
      scientific papers communicate with numbers, errors, p-values next to numebrs, name of the statistical test used, and figures are usually annotated to indicate the level of significance. That's pretty careful.
      - Angostura 7y ago
        But the fundamental problem is here is that people misunderstand the significance (sic) of p-values.
  - jerf 7y ago
    Cutoffs aren't the problem. They are inevitable; the logic is virtually identical to the question "Why can you drink at 21 and vote at 18?" Of course the Responsibility Fairy doesn't visit you that night and make you suddenly able to handle it when you weren't the day before, it's just that at scale, you don't have much choice but to operate that way because everything else is just too expensive. The problem with p-values is that the shape of the resulting cutoff just isn't what we're looking for. We want something along the lines of "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false?" is only an approximation at the best of times, and at worst, downright misleading. That's true even before we consider some of the other issues that my English kind of elides over, but the math contains; one of my problems with significance testing as it is commonly done is that there are actually ranges of hypotheses, and it really overprivileges "the" "null" hypothesis; I could write a decently-sized HN post just criticizing those two quoted words. There is no rigid process that can produce the answer we really want, but I think we can provide a selection of better default tools. An example of one that has already been deployed to some extent is "power analysis", which is not a direct answer but lets people crafting studies analyze how big their studies will have to be before running them. We can build more tools like that.
    - buboard 7y ago
      > "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false? The first is impossible to calculate by definition. The second can be derived from p value. Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used.
      - jerf 7y ago
        You've mangled my sentence in your quote. The rest matters. It may not be a masterpiece of the English language, but I already said the first is not directly possible and I wrote the second as a description of p-values quite on purpose. "Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used." That's begging the question. The entire topic of conversation is whether or not the standards of justification are adequate.

hannob 7y ago

I usually don't see climate science being communicated with p-values and poor significance cutoffs. The place where you usually see this used is for communicating single study results (which often enough are things that you probably shouldn't communicate at all). To get an idea how climate science is trying to communicate look at the summary for policymakers of the SR15 report: https://report.ipcc.ch/sr15/pdf/sr15_spm_final.pdf https://report.ipcc.ch/sr15/pdf/sr15_spm_final.pdf They have different confidence levels they indicate instead of a single cutoff.

buboard 7y ago
That's not different than communicating significance with p < 0.05 , p < 0.01, p < 0.001 etc as is usually done in paper. There is again a cutoff.
- Angostura 7y ago
  It's different in that the cut-offs are communicated with careful wording, rather than a figure.
  - buboard 7y ago
    scientific papers communicate with numbers, errors, p-values next to numebrs, name of the statistical test used, and figures are usually annotated to indicate the level of significance. That's pretty careful.
    - Angostura 7y ago
      But the fundamental problem is here is that people misunderstand the significance (sic) of p-values.
- jerf 7y ago
  Cutoffs aren't the problem. They are inevitable; the logic is virtually identical to the question "Why can you drink at 21 and vote at 18?" Of course the Responsibility Fairy doesn't visit you that night and make you suddenly able to handle it when you weren't the day before, it's just that at scale, you don't have much choice but to operate that way because everything else is just too expensive. The problem with p-values is that the shape of the resulting cutoff just isn't what we're looking for. We want something along the lines of "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false?" is only an approximation at the best of times, and at worst, downright misleading. That's true even before we consider some of the other issues that my English kind of elides over, but the math contains; one of my problems with significance testing as it is commonly done is that there are actually ranges of hypotheses, and it really overprivileges "the" "null" hypothesis; I could write a decently-sized HN post just criticizing those two quoted words. There is no rigid process that can produce the answer we really want, but I think we can provide a selection of better default tools. An example of one that has already been deployed to some extent is "power analysis", which is not a direct answer but lets people crafting studies analyze how big their studies will have to be before running them. We can build more tools like that.
  - buboard 7y ago
    > "What is the probability that this hypothesis is true, and how true is it?", and "What is the probability that this result could have happened even if the hypothesis is false? The first is impossible to calculate by definition. The second can be derived from p value. Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used.
    - jerf 7y ago
      You've mangled my sentence in your quote. The rest matters. It may not be a masterpiece of the English language, but I already said the first is not directly possible and I wrote the second as a description of p-values quite on purpose. "Virtually all journals require rigorous reporting of p values along with averages, and the justification of the statistical test used." That's begging the question. The entire topic of conversation is whether or not the standards of justification are adequate.