6 ms·

"The effect size is too large" seems like a weak heuristic too me. Feynmann talks about the kind of problems such a heuristic can cause: > It's interesting to

"The effect size is too large" seems like a weak heuristic too me. Feynmann talks about the kind of problems such a heuristic can cause: > It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. > Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that... https://hsm.stackexchange.com/questions/264/timeline-of-measurements-of-the-electrons-charge https://hsm.stackexchange.com/questions/264/timeline-of-meas... A more theoretical argument is as follows. Suppose we systematically remove from analysis any data point which are more than 1.5 times the IQR below the first quartile or about the third quartile. (https://en.wikipedia.org/wiki/Outlier#Tukey's_fences https://en.wikipedia.org/wiki/Outlier#Tukey's_fences) Points censured by such a rule are often called "outliers," but because we do not yet know the true underlying probability distribution this terminology is misleading. Then our estimates for skew, kurtosis, and all higher moments will be biased downward, and furthermore the censured sample is more likely to "pass" normality tests such as Shapiro-Wilks which can lead to us applying inappropriate models. However, what is the interpretation of a model when applied to a new observation? If the data point is well within the Tukey fences, then we can use the model to make a valid prediction. However, if it is outside the fences, we can say nothing. What then is the population mean of the predicted value? It is the weighted mean of the values for those data points for which our model is valid and those which lie outside of it. But those that lie outside of it can be enormously influential. An example can make this concrete. Suppose an actuary estimates the amount of an insurance payouts for house insurance. 99% of the time, these effects are small, a few hundred or a few thousand dollars. But 1% of the time they are much larger, perhaps $500,000 or more. If the actuary applies Tukey's fences to his data and fits a model, he may infer that 1% of claims payout each year, and those payouts average $1,000 dollars. So he sets the price of the insurance at $20/year and makes a 100% profit margin. Until the first house burns down, and his company goes bankrupt. The point of this is merely to illustrate the dangers of ignoring any effect which simply seems "too large." The beneficial effects of penicillin are "impossibly large" - would we reject the evidence on those grounds? The change in resistance of a semiconductor in the presence of an electric field seem "impossibly large" until the solid state physics is carefully analyzed - would we reject this clear and highly reproducible evidence of a qualitative change in behavior because it is too evident? A much better approach is to scrutinize the methodology for, say, omitted variables. This is what the more detailed criticisms he cites in the third paragraph do. Or to attempt to reproduce the study, which is perhaps the strongest approach.

cultus 8y ago
It's not entirely just a heuristic, but does have a Bayesian interpretation in many circumstances. Looking at the study on judge's sentences before and after meals, you wouldn't expect to see a giant effect, but seeing some kind of an effect is a priori plausible, since we know many people are irritable when hungry. When you see an unusually large effect size, it's often the case that there is a smaller effect, but you happened to get some extreme data. You definitely shouldn't jump to the conclusion that the effect size is large if you think it is a priori unlikely to be so large (unless you have very high statistical power). As you say, the results shouldn't be discarded, though. edit: If an effect is incredibly improbable (like a relation between country music and suicides), evidence can still sway the belief against it. However, extraordinary claims require extraordinary evidence.
JackFr 8y ago
> The beneficial effects of penicillin are "impossibly large" - would we reject the evidence on those grounds? I think you miss the point -- the efficacy of penicillin is obvious to anyone, whether they've taken statistics or not. The authors point, as I understand it, is that if the effects are so very large they should be obvious to a=even a casual observer. If the effect is so large it's impossible for them to be surprising.
klodolph 8y ago
“The effect size is too large” is not the heuristic, and casting the argument that way does it a disservice. The article gives concrete examples of effect sizes that are impossibly large. Examples of penicillin and insurance payouts are not remotely germane to the discussion… the point of “impossibly large” is that the effect is so large that it would almost certainly have other effects that we would notice. This is the difference between “impossibly large” and “surprisingly large”, which your argument ignores. The exhortation to look at the mechanism is also critical, and ignoring the mechanism is exactly how you end up with the worst inferences… if something is “impossible” or even just “surprising” according to your model you should probably reëvaluate your model.
Bartweiss 8y ago
The point being made isn't that large effects are impossible, or that high-d results should be dismissed outright. Rather, it's that for sufficiently large d, subtlety becomes impossible. Results will either be overwhelmingly obvious to any observer, or false. It's worth giving that list of sample effects in the article another look. In particular, the studies showing that juries usually make decisions matching what the majority of jurors started out believing is d=1.62. This study claimed that hunger was having a bigger effect on trial outcomes than whether people believed the charges were true. It was still worth looking for the confounding effect, certainly, but when people start making claims that disproportionate to comparable claims, simply refusing to accept a causative claim is an entirely reasonable decision. The effects of penicillin are overwhelmingly obvious: for some set of diseases, penicillin produces rapid and complete recovery even in otherwise-terminal patients. I don't know the d for "penicillin treats staph", but in pharmacology d=5 isn't considered unreasonable for precisely these reasons. The average d in psychology, though, is around 0.4 - and that was before many of the 'strongest' results fell apart in recent years. This study's effect size wasn't inherently insane, but it was claiming confidence in the realm of "antibiotics kill bacteria" for a subtle and previously unobserved force on judge's decisionmaking. It absolutely should have raised questions like "if this effect holds, why aren't trial lawyers acutely aware of it and working to manipulate it?"

cultus 8y ago

It's not entirely just a heuristic, but does have a Bayesian interpretation in many circumstances. Looking at the study on judge's sentences before and after meals, you wouldn't expect to see a giant effect, but seeing some kind of an effect is a priori plausible, since we know many people are irritable when hungry. When you see an unusually large effect size, it's often the case that there is a smaller effect, but you happened to get some extreme data. You definitely shouldn't jump to the conclusion that the effect size is large if you think it is a priori unlikely to be so large (unless you have very high statistical power). As you say, the results shouldn't be discarded, though. edit: If an effect is incredibly improbable (like a relation between country music and suicides), evidence can still sway the belief against it. However, extraordinary claims require extraordinary evidence.

JackFr 8y ago

> The beneficial effects of penicillin are "impossibly large" - would we reject the evidence on those grounds? I think you miss the point -- the efficacy of penicillin is obvious to anyone, whether they've taken statistics or not. The authors point, as I understand it, is that if the effects are so very large they should be obvious to a=even a casual observer. If the effect is so large it's impossible for them to be surprising.

klodolph 8y ago

“The effect size is too large” is not the heuristic, and casting the argument that way does it a disservice. The article gives concrete examples of effect sizes that are impossibly large. Examples of penicillin and insurance payouts are not remotely germane to the discussion… the point of “impossibly large” is that the effect is so large that it would almost certainly have other effects that we would notice. This is the difference between “impossibly large” and “surprisingly large”, which your argument ignores. The exhortation to look at the mechanism is also critical, and ignoring the mechanism is exactly how you end up with the worst inferences… if something is “impossible” or even just “surprising” according to your model you should probably reëvaluate your model.

Bartweiss 8y ago

The point being made isn't that large effects are impossible, or that high-d results should be dismissed outright. Rather, it's that for sufficiently large d, subtlety becomes impossible. Results will either be overwhelmingly obvious to any observer, or false. It's worth giving that list of sample effects in the article another look. In particular, the studies showing that juries usually make decisions matching what the majority of jurors started out believing is d=1.62. This study claimed that hunger was having a bigger effect on trial outcomes than whether people believed the charges were true. It was still worth looking for the confounding effect, certainly, but when people start making claims that disproportionate to comparable claims, simply refusing to accept a causative claim is an entirely reasonable decision. The effects of penicillin are overwhelmingly obvious: for some set of diseases, penicillin produces rapid and complete recovery even in otherwise-terminal patients. I don't know the d for "penicillin treats staph", but in pharmacology d=5 isn't considered unreasonable for precisely these reasons. The average d in psychology, though, is around 0.4 - and that was before many of the 'strongest' results fell apart in recent years. This study's effect size wasn't inherently insane, but it was claiming confidence in the realm of "antibiotics kill bacteria" for a subtle and previously unobserved force on judge's decisionmaking. It absolutely should have raised questions like "if this effect holds, why aren't trial lawyers acutely aware of it and working to manipulate it?"