10 ms·
Impossibly Hungry Judges (2017)
- thesausageking 8y agoHere's a good explanation of the flaws in the original study: http://journal.sjdm.org/16/16823/jdm16823.html http://journal.sjdm.org/16/16823/jdm16823.html It's primarily due to "a statistical artifact resulting from favorable rulings taking longer than unfavorable ones".
- baybal2 8y agoNot much of an artifact - during my BLAW101 I was told that in North America judges really don't like people intentionally dragging on cases, but can't do much about them other than give the innuendo of them raising white flag - something along lines "you babbled for 3 hours, for sure we have enough argument from you that the worst case scenario ruling is now out of the question"
- olooney 8y ago"The effect size is too large" seems like a weak heuristic too me. Feynmann talks about the kind of problems such a heuristic can cause: > It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. > Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that... https://hsm.stackexchange.com/questions/264/timeline-of-measurements-of-the-electrons-charge https://hsm.stackexchange.com/questions/264/timeline-of-meas... A more theoretical argument is as follows. Suppose we systematically remove from analysis any data point which are more than 1.5 times the IQR below the first quartile or about the third quartile. (https://en.wikipedia.org/wiki/Outlier#Tukey's_fences https://en.wikipedia.org/wiki/Outlier#Tukey's_fences) Points censured by such a rule are often called "outliers," but because we do not yet know the true underlying probability distribution this terminology is misleading. Then our estimates for skew, kurtosis, and all higher moments will be biased downward, and furthermore the censured sample is more likely to "pass" normality tests such as Shapiro-Wilks which can lead to us applying inappropriate models. However, what is the interpretation of a model when applied to a new observation? If the data point is well within the Tukey fences, then we can use the model to make a valid prediction. However, if it is outside the fences, we can say nothing. What then is the population mean of the predicted value? It is the weighted mean of the values for those data points for which our model is valid and those which lie outside of it. But those that lie outside of it can be enormously influential. An example can make this concrete. Suppose an actuary estimates the amount of an insurance payouts for house insurance. 99% of the time, these effects are small, a few hundred or a few thousand dollars. But 1% of the time they are much larger, perhaps $500,000 or more. If the actuary applies Tukey's fences to his data and fits a model, he may infer that 1% of claims payout each year, and those payouts average $1,000 dollars. So he sets the price of the insurance at $20/year and makes a 100% profit margin. Until the first house burns down, and his company goes bankrupt. The point of this is merely to illustrate the dangers of ignoring any effect which simply seems "too large." The beneficial effects of penicillin are "impossibly large" - would we reject the evidence on those grounds? The change in resistance of a semiconductor in the presence of an electric field seem "impossibly large" until the solid state physics is carefully analyzed - would we reject this clear and highly reproducible evidence of a qualitative change in behavior because it is too evident? A much better approach is to scrutinize the methodology for, say, omitted variables. This is what the more detailed criticisms he cites in the third paragraph do. Or to attempt to reproduce the study, which is perhaps the strongest approach.
- cultus 8y agoIt's not entirely just a heuristic, but does have a Bayesian interpretation in many circumstances. Looking at the study on judge's sentences before and after meals, you wouldn't expect to see a giant effect, but seeing some kind of an effect is a priori plausible, since we know many people are irritable when hungry. When you see an unusually large effect size, it's often the case that there is a smaller effect, but you happened to get some extreme data. You definitely shouldn't jump to the conclusion that the effect size is large if you think it is a priori unlikely to be so large (unless you have very high statistical power). As you say, the results shouldn't be discarded, though. edit: If an effect is incredibly improbable (like a relation between country music and suicides), evidence can still sway the belief against it. However, extraordinary claims require extraordinary evidence.
- JackFr 8y ago> The beneficial effects of penicillin are "impossibly large" - would we reject the evidence on those grounds? I think you miss the point -- the efficacy of penicillin is obvious to anyone, whether they've taken statistics or not. The authors point, as I understand it, is that if the effects are so very large they should be obvious to a=even a casual observer. If the effect is so large it's impossible for them to be surprising.
- klodolph 8y ago“The effect size is too large” is not the heuristic, and casting the argument that way does it a disservice. The article gives concrete examples of effect sizes that are impossibly large. Examples of penicillin and insurance payouts are not remotely germane to the discussion… the point of “impossibly large” is that the effect is so large that it would almost certainly have other effects that we would notice. This is the difference between “impossibly large” and “surprisingly large”, which your argument ignores. The exhortation to look at the mechanism is also critical, and ignoring the mechanism is exactly how you end up with the worst inferences… if something is “impossible” or even just “surprising” according to your model you should probably reëvaluate your model.
- Bartweiss 8y agoThe point being made isn't that large effects are impossible, or that high-d results should be dismissed outright. Rather, it's that for sufficiently large d, subtlety becomes impossible. Results will either be overwhelmingly obvious to any observer, or false. It's worth giving that list of sample effects in the article another look. In particular, the studies showing that juries usually make decisions matching what the majority of jurors started out believing is d=1.62. This study claimed that hunger was having a bigger effect on trial outcomes than whether people believed the charges were true. It was still worth looking for the confounding effect, certainly, but when people start making claims that disproportionate to comparable claims, simply refusing to accept a causative claim is an entirely reasonable decision. The effects of penicillin are overwhelmingly obvious: for some set of diseases, penicillin produces rapid and complete recovery even in otherwise-terminal patients. I don't know the d for "penicillin treats staph", but in pharmacology d=5 isn't considered unreasonable for precisely these reasons. The average d in psychology, though, is around 0.4 - and that was before many of the 'strongest' results fell apart in recent years. This study's effect size wasn't inherently insane, but it was claiming confidence in the realm of "antibiotics kill bacteria" for a subtle and previously unobserved force on judge's decisionmaking. It absolutely should have raised questions like "if this effect holds, why aren't trial lawyers acutely aware of it and working to manipulate it?"
- tantalor 8y ago[2017] Previous comments: https://news.ycombinator.com/item?id=14701328 https://news.ycombinator.com/item?id=14701328
- Waterluvian 8y agoI did a bit of science as an undergrad and a bit more in grad school. I learned a few things, chiefly, that I suck at statistics and won't make a good formal scientist. But the other main takeaway was that when results look strikingly obvious and "beautiful" they're usually just wrong. I did GIS so the statistical data was usually in the form of big colourful maps. And when those maps told such an obvious story, they were really telling me that I messed something up in my processing. Not that this is universally true, but it's what I found to be pretty common over the years.
- nonbel 8y ago>"I suck at statistics and won't make a good formal scientist" Most really good science got done before statistics was even a thing (starting in the 1930s-1940s). Stats has very little to do with being a good scientist.
- PurpleBoxDragon 8y agoA great deal of older science was done by explaining what was already invented. We seem to have reached a point where rarely does a new invention throw off previous science without it being based on new science that was already in disagreement with old science. Part of it is because over the last century we had so many people with the freedom to invent that the easy inventions have already been invented, at least where the inventions don't pose significant moral/legal implications (such as someone inventing a new drug and getting people to try it without going through the proper channels).
- nonbel 8y ago>"We seem to have reached a point where rarely does a new invention throw off previous science without it being based on new science that was already in disagreement with old science." Or perhaps science has become so institutionalized that any truly novel ideas are suppressed either directly or indirectly (eg the funding for that type of thing is already maxed out). Or maybe the truly novel ideas are getting classified: https://slate.com/technology/2018/05/the-thousands-of-secret-patents-that-the-u-s-government-refuses-to-make-public.html https://slate.com/technology/2018/05/the-thousands-of-secret...
- quotemstr 8y agoAm I the only one who feels sharper and more adapt while somewhat hungry? Food makes me lethargic. I've never been sympathetic to ideas that "you should always eat breakfast!". If possible, I'd prefer to skip breakfast and lunch. My idea: our ancestors were opportunistic savanna hunters. Why would we be unable to maintain homeostasis with respect to cognition even without food? Our ancestors would have needed their best cognition when they lacked food!
- girvo 8y agoI haven’t eaten breakfast in 15 years, and have lunch about 2pm. It genuinely makes me think clearer.
- sp332 8y agoMaybe you have a food sensitivity, and your body is having e.g. a mild allergic reaction. On the other hand your sympathetic and parasympathetic nervous systems are at odds, so if you try to get amped up right after eating you will get a stomach ache as your body moves blood away from your stomach and you basically stop digesting food for a while.
- pazimzadeh 8y agoI often feel more clear headed and direct in my reasoning when I am slightly hungry. Once I eat, I'm more likely to entertain implausible ideas, play "what if..." and it feels easier to argue for both sides of any idea.
- gowld 8y agoOptimal alertness/health is at a point between "paralyzed with starvation" and "paralyzed by overstuffedness". What you are calling "somewhat hungry" may be better stated as "full, but not overstuffed". To maintain a consistent energy level, 5+ small snacks is better than 3 large meals.
- air7 8y agoI'v always found it amusing that there is no word for English meaning "not hungry". There's only "full". It exists it many languages alongside "full" and there's a clear distinction between them.
- jellicle 8y agoThis article, which boils down to "if the data seems unlikely from what we know beforehand, throw it the hell out and never look at it again", is the exact opposite of what scientists are taught, and what they are taught - not this article - is the correct approach.
- claytonjy 8y agoThis is a very poor and incorrect reading of the article here. The point is not that the data were unlikely and should be thrown out; it's that if the effect size reported was accurate, that would affect so many other things so strongly that the current state of civilization refutes the estimated effect size from the study. It's not that the data was poor, but that it and the analysis was incomplete and ill considered. In psychology in particular, large effects will seep into everyday life well beyond the phenomena being studied, which gives a very useful check on astronomical effects like those originally reported on judges and time of day.
- MrPrinter 8y agoGP says, and I concur, that the scientific method would not to be just say "this must be impossible, let's ignore it", but try to disprove the study (maybe by trying and failing to replicate the study, or designing better studies).
- claytonjy 8y agoI think we all agree with that position; the issue is the author of TFA did not make that position, so GGP was arguing against a strawman. The original study was disproved, and TFA includes multiple such links near the top.
- Retric 8y agoThe study was not disproved, the conclusion was. The critical part of science is the actual data and methodology. If you do X you will see Y is what constrains new theories. Conclusions on the other hand have very little weight. PS: Sorry pet peeve.
- mark212 8y agoThis is a ridiculous study because it implies that the order of when cases are heard is randomly distributed. That is, a "hard" case is just as likely to be first in the morning session as it is to be fifth in an afternoon session, or vice versa. The data is easily explained as judges taking clear case of leniency first, then doing the tough ones (that could go either way), and ruling on the clear cases of parole denials later in the session. The judges have hopefully done their homework in advance and have read the case files and petitions, so they come in with at least a rough sense for which cases are in each bucket. There's likely a significant degree of agreement on that metric among the members of the panel, so it's not surprising that cases are heard in that order: clear grants, ones that require discussion, and clear denials. The lesson learned seems to me to be when the evidence shows a result this large, take a really hard look at your study before you publish it.
- claytonjy 8y agoThis is exactly what the criticism linked in TFA explores: http://www.pnas.org/content/108/42/E833.long http://www.pnas.org/content/108/42/E833.long A related problem here is that even though the original study was bad for many reasons, it continues to be cited much more than the refutation.
- mark212 8y agoI missed that one. Read the other refutation.
- 99052882514569 8y agoAnd mentioned in podcasts by people[1] who really, seriously, obviously should know much better (based on how famous and otherwise competent they seem to be). [1] https://en.wikipedia.org/wiki/Robert_Sapolsky https://en.wikipedia.org/wiki/Robert_Sapolsky
- oh_sigh 8y agoIf we could stop quoting this, and the baloney implicit bias study, we would all be so much better off.
- calimac 8y agoTHROW THE STUDY DATA OUT! The retained outside counsel cases that have been previously negotiated in chambers are the first to go at the opening of every session. The court takes the outside counsel (attorney) cases out of the gate and the PD pro se go subsequent. If anything this study proves that getting a good attorney pays off. Calling hackers out there... would be great to track private attorney win loss ratio above or below par sentence and publish it on criminal defense leads site to help people select great attorneys and stop paying for schmucks.
- chicob 8y ago«If a psychological effect is this big, we don’t need to discover it and publish it in a scientific journal—you would already know it exists.» On the one hand, shoudn't we? There are a lot of supposed facts that need to be checked. Also, the size of the supposed effect everybody knows still needs to be measured, if, for instance, comparisons between different populations can be made at all. On the other hand, and personally, I find it to be common sense never to have difficult conversations on an empty stomach.