4 ms·

I would be ashamed to publish the graph with the "trend", the fit is so incredibly bad.

by syats 8y ago

wallace_f 8y ago
Question: if I analyze 100 junk datasets and come up with 1 which has a statistically significant relationship (especially following some horrid-fit trendline like this) isn't there going to be some name for this? After 100 datasets you're virtually necessarily going to come across these misleading conclusions, no?
- duality 8y ago
  I've heard that called "data dredging."
- idm 8y ago
  It's called the "file drawer problem" in which research is conducted but not published because it wasn't significant. You're referring to "Type I error" in which a statistically significant result is obtained by chance, not because the effect is real. This article does not appear to be an example of the hypothetical scenario you proposed. The trend line is not a "horrid-fit," as attested to by the statistical analysis published in the article - which is open access and may be read for free. This is a scientific article - and unless you're a scientist, it may be difficult to understand the entire article.
- SmileyJames 8y ago
  https://xkcd.com/882/ https://xkcd.com/882/ I've heard it called p-value hacking, Wikipedia refers to it as data-dredging.
- derf_ 8y ago
  Yes, it is the "Family-wise error rate". You need to perform "correction for multiple tests". The simplest, most conservative form is Bonferroni correction [1], where you just divide the required significance threshold by the number of tests you perform. In your example, if you required significance at the 99% level (1% chance a false positive, or "Type I error"), then for 100 datasets, each individual one would require significance at the 99.99% level (0.01% false positives for each data set). (99.99%)^100 = 99.005%. This does not require any assumptions about the independence of the tests. Independent tests are the worst case, so this over-corrects if there is actual correlation between the measurements. There are other, more complicated correction methods that do not give up as much statistical power (i.e., produce fewer false negatives), but that may require making more assumptions. [1] https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction
- ims 8y ago
  Andrew Gelman calls the larger problem the "Garden of Forking Paths" - http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
  - sn41 8y ago
    IIRC "The Garden of Forking Paths" is an interesting story by J. L. Borges, from where the title of the paper is probably taken.
- FabHK 8y ago
  If you're using a 5% significance level, you'd expect 5 datasets of 100 to come up with a statistically significant result even though there's no there there (ie even though the null hypothesis is true for all the datasets). That's by definition. This is even before p-hacking. Having said that, the study reported on here has significance levels better than 1%: > The farther away a municipality (in dots) is from a historical mission, the lower its literacy level today. This unconditional relationship is negative and highly significant with a t-statistic of -4.36. So, notwithstanding the experts here eyeballing the graph and dismissing it as embarrassing, I'd say there is something going on here. Paper: Caicedo, F. V. (2018). The Mission: Human Capital Transmission, Economic Persistence, and Culture in South America*. The Quarterly Journal of Economics. doi:10.1093/qje/qjy024
- asgraham 8y ago
  It's called multiple testing. You're not guaranteed to get a misleading result, but it would be expected. The definition of the 5% significance level used in p-value testing is that given 100 junk datasets (distributed similarly), you would only expect 5 to show as strong an effect. However, I don't really see what this study has to do with multiple testing. As far as I can tell, they had one dataset, and they tested that, and they got a significant result. Nothing fishy there. Sure the data is noisy, but wouldn't you expect that given how coarse the data is, and how many confounding factors there must be? [1] https://en.wikipedia.org/wiki/Multiple_comparisons_problem https://en.wikipedia.org/wiki/Multiple_comparisons_problem
  - DavidSJ 8y ago
    Publication bias is a sort of multiple testing at the meta level: different groups of scientists try out different hypotheses — but only one each — and some of them just barely pass the statistical tests and thus get published. Not saying that happened here! I haven’t even read the paper. But it is a valid concern when mining data for patterns, especially when those patterns have a questionable theoretical basis.
    - asgraham 8y ago
      Oh absolutely! I totally understand you're not saying that's what's happening here, but just to clarify for other readers, I'd say in this case there are a few reasons we don't need quite that level of suspicion: 1. There is a seemingly-sound theoretical basis for the observation (moreover, there doesn't seem to have been any element of fishing. Sometimes researchers think fishing is OK if they just do it once to avoid multiple testing. But then you certainly have multiple testing at the meta level). 2. This study agrees with at least one other completely independent, reputable study [1] (even different methodologies, apparently) that supported the underlying theory. I do think the problem of multiple testing at the meta level is vastly underrated by the scientific community. Just, in this case, I think there are reasons to reject that hypothesis. [1] https://scholar.princeton.edu/sites/default/files/lwantche/files/beninwkn-qje-final.pdf https://scholar.princeton.edu/sites/default/files/lwantche/f...
- blauditore 8y ago
  I've definitely seen it happening, and I think it's been referred to as "p-fishing".
idm 8y ago
This is an open access article and the statistics are published. See p.41 for a per-country analysis of the effect on literacy. I would be ashamed to publish criticism about an open access scientific article without having any understanding of statistics.
- darkhorn 8y ago
  It looks like variance is increasing as distance increases. This is bad statistics.
  - Terretta 8y ago
    > It looks like variance is increasing as distance increases. Given the topic, how would it not?
    - darkhorn 8y ago
      And how would you validate Bartlett’s test?
  - ineedasername 8y ago
    Variance can increase with time without invalidating the trend. Signals can attenuate with time and nonetheless still be a signal. This is very frequently the case with longitudinal studies that track outcomes from initial conditions.
    - darkhorn 8y ago
      And how you validate Bartlett’s test in this case?
      - ineedasername 8y ago
        Personally I prefer levene's test (it's more robust in situations where the distribution may not be normal). Either way though, typical practice would be to run the tests both assuming equal variance and again not assuming equal variance. If both are statistically significant then it's a moot point, the results are statistically significant regardless of whether the variance in populations is equivalent.

I would be ashamed to publish the graph with the "trend", the fit is so incredibly bad.

by syats 8y ago

I would be ashamed to publish the graph with the "trend", the fit is so incredibly bad.

Question: if I analyze 100 junk datasets and come up with 1 which has a statistically significant relationship (especially following some horrid-fit trendline like this) isn't there going to be some name for this? After 100 datasets you're virtually necessarily going to come across these misleading conclusions, no?

duality 8y ago
I've heard that called "data dredging."
idm 8y ago
It's called the "file drawer problem" in which research is conducted but not published because it wasn't significant. You're referring to "Type I error" in which a statistically significant result is obtained by chance, not because the effect is real. This article does not appear to be an example of the hypothetical scenario you proposed. The trend line is not a "horrid-fit," as attested to by the statistical analysis published in the article - which is open access and may be read for free. This is a scientific article - and unless you're a scientist, it may be difficult to understand the entire article.
SmileyJames 8y ago
https://xkcd.com/882/ https://xkcd.com/882/ I've heard it called p-value hacking, Wikipedia refers to it as data-dredging.
derf_ 8y ago
Yes, it is the "Family-wise error rate". You need to perform "correction for multiple tests". The simplest, most conservative form is Bonferroni correction [1], where you just divide the required significance threshold by the number of tests you perform. In your example, if you required significance at the 99% level (1% chance a false positive, or "Type I error"), then for 100 datasets, each individual one would require significance at the 99.99% level (0.01% false positives for each data set). (99.99%)^100 = 99.005%. This does not require any assumptions about the independence of the tests. Independent tests are the worst case, so this over-corrects if there is actual correlation between the measurements. There are other, more complicated correction methods that do not give up as much statistical power (i.e., produce fewer false negatives), but that may require making more assumptions. [1] https://en.wikipedia.org/wiki/Bonferroni_correction https://en.wikipedia.org/wiki/Bonferroni_correction
ims 8y ago
Andrew Gelman calls the larger problem the "Garden of Forking Paths" - http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
- sn41 8y ago
  IIRC "The Garden of Forking Paths" is an interesting story by J. L. Borges, from where the title of the paper is probably taken.
FabHK 8y ago
If you're using a 5% significance level, you'd expect 5 datasets of 100 to come up with a statistically significant result even though there's no there there (ie even though the null hypothesis is true for all the datasets). That's by definition. This is even before p-hacking. Having said that, the study reported on here has significance levels better than 1%: > The farther away a municipality (in dots) is from a historical mission, the lower its literacy level today. This unconditional relationship is negative and highly significant with a t-statistic of -4.36. So, notwithstanding the experts here eyeballing the graph and dismissing it as embarrassing, I'd say there is something going on here. Paper: Caicedo, F. V. (2018). The Mission: Human Capital Transmission, Economic Persistence, and Culture in South America*. The Quarterly Journal of Economics. doi:10.1093/qje/qjy024
asgraham 8y ago
It's called multiple testing. You're not guaranteed to get a misleading result, but it would be expected. The definition of the 5% significance level used in p-value testing is that given 100 junk datasets (distributed similarly), you would only expect 5 to show as strong an effect. However, I don't really see what this study has to do with multiple testing. As far as I can tell, they had one dataset, and they tested that, and they got a significant result. Nothing fishy there. Sure the data is noisy, but wouldn't you expect that given how coarse the data is, and how many confounding factors there must be? [1] https://en.wikipedia.org/wiki/Multiple_comparisons_problem https://en.wikipedia.org/wiki/Multiple_comparisons_problem
- DavidSJ 8y ago
  Publication bias is a sort of multiple testing at the meta level: different groups of scientists try out different hypotheses — but only one each — and some of them just barely pass the statistical tests and thus get published. Not saying that happened here! I haven’t even read the paper. But it is a valid concern when mining data for patterns, especially when those patterns have a questionable theoretical basis.
  - asgraham 8y ago
    Oh absolutely! I totally understand you're not saying that's what's happening here, but just to clarify for other readers, I'd say in this case there are a few reasons we don't need quite that level of suspicion: 1. There is a seemingly-sound theoretical basis for the observation (moreover, there doesn't seem to have been any element of fishing. Sometimes researchers think fishing is OK if they just do it once to avoid multiple testing. But then you certainly have multiple testing at the meta level). 2. This study agrees with at least one other completely independent, reputable study [1] (even different methodologies, apparently) that supported the underlying theory. I do think the problem of multiple testing at the meta level is vastly underrated by the scientific community. Just, in this case, I think there are reasons to reject that hypothesis. [1] https://scholar.princeton.edu/sites/default/files/lwantche/files/beninwkn-qje-final.pdf https://scholar.princeton.edu/sites/default/files/lwantche/f...
blauditore 8y ago
I've definitely seen it happening, and I think it's been referred to as "p-fishing".

idm 8y ago

This is an open access article and the statistics are published. See p.41 for a per-country analysis of the effect on literacy. I would be ashamed to publish criticism about an open access scientific article without having any understanding of statistics.

darkhorn 8y ago
It looks like variance is increasing as distance increases. This is bad statistics.
- Terretta 8y ago
  > It looks like variance is increasing as distance increases. Given the topic, how would it not?
  - darkhorn 8y ago
    And how would you validate Bartlett’s test?
- ineedasername 8y ago
  Variance can increase with time without invalidating the trend. Signals can attenuate with time and nonetheless still be a signal. This is very frequently the case with longitudinal studies that track outcomes from initial conditions.
  - darkhorn 8y ago
    And how you validate Bartlett’s test in this case?
    - ineedasername 8y ago
      Personally I prefer levene's test (it's more robust in situations where the distribution may not be normal). Either way though, typical practice would be to run the tests both assuming equal variance and again not assuming equal variance. If both are statistically significant then it's a moot point, the results are statistically significant regardless of whether the variance in populations is equivalent.