13 ms·

The Flawed Reasoning Behind the Replication Crisis

185 points by dnetesn 7y ago

cortesoft 7y ago
My favorite way to show the base rate fallacy is to take the most extreme example. Suppose I have an invisible dragon detector that is accurate 99.9% of the time... if it tells me there is an invisible dragon in the room, it doesn't mean there is only a 0.1% chance there is not a dragon... there is a 100% chance there is not an invisible dragon in the room, because they don't exist.
- kortilla 7y ago
  Or just link to the xkcd comic about the sun explosion detector. https://www.explainxkcd.com/wiki/index.php/1132:_Frequentists_vs._Bayesians https://www.explainxkcd.com/wiki/index.php/1132:_Frequentist...
  - reallydude 7y ago
    I always interpreted that comic to be synergistically thoughtful. If the sun exploded, it doesn't matter who has the $50, so it's a good bet from a pragmatic standpoint.
- TheOtherHobbes 7y ago
  That only works if you already know something is/isn't true with absolute certainty. It's useless for fundamental research, because by definition you're exploring what you don't know yet. The real problem is more one of labelling. "True according to science" isn't a binary, but it's treated as if it is - especially by marketers. Science is more like a set of concentric circles of decreasing confidence. You can be very confident indeed about the contents of the centre circle which includes undergraduate physics and engineering. You can also be confident that there are commonly agreed edge cases, areas of inaccuracy, and extreme circumstances where the science stops being reliable. As you get further away from the centre confidence decreases. A lot of the debate about replication is about research that is a long way from the centre, where uncertainty is high. But neither researchers nor the science press nor the mainstream media will report this. Studies are usually presented as "Science says...", as if you're supposed to be just as confident of the results of a psychological study that asks a population of 30 undergrads from the same college and the same year some poorly designed questions as you are in Special Relativity.
  - cortesoft 7y ago
    It is just to show why base rate matters by using the most extreme example... it doesn't imply that it is easy to calculate.
- deleted 7y ago
  [deleted]
BoiledCabbage 7y ago
It's incredible it's held up this long. It shows how much science actually is dogma. Human nature wins out over rationality.
- xamuel 7y ago
  Not so incredible if you consider how the sausage is made. People who hate math pursue social science PhDs and are taught "p-values = truth". These same people are pressured to produce lots of research, and not just boring research but preferably shocking research. Journals are incentivized to publish such shocking research because that kind of research gets cited. When you think about it, this stuff will probably hold up for a long time to come.
  - SantalBlush 7y ago
    It has little to do with stereotypes about social scientists hating math or misunderstanding p-values. The current system incentivizes p-hacking. Nobody wants to throw away their work if it doesn't meet the p < 0.05 criterion, especially when their career is on the line.
    - marcus_holmes 7y ago
      so...why are we calling this science if it doesn't meet the basics of the Scientific method?
    - SkyBelow 7y ago
      While it isn't the most concrete sample size, when I was at college the mathematics requirements for the soft sciences were far inferior to the mathematics requirements for hard sciences. To the extent that even the easier statistics course in the math department wasn't require and instead was replaced by a statistics class in the social science department that was only valid for social sciences.
chrisco255 7y ago
"The point is that we have good reason to be skeptical, and we should follow the mantra of the mathematician (and Bayesian) Pierre-Simon Laplace, that extraordinary claims require extraordinary evidence. By ignoring the necessity of priors, significance testing opens the door to false positive results." Such an important quote. I believe the replication crisis is especially present in nutrition studies. We live in an age where headlines go viral and new wave diets are taken on in rapid succession. Take all nutrition studies with a grain of salt. (assuming, of course, that salt is good for you...or is it???)
dwheeler 7y ago
The article says: > The main reason scientists have historically been resistant to using Bayesian inference instead is that they are afraid of being accused of subjectivity. The prior probabilities required for Bayes’ rule feel like an unseemly breach of scientific ethics. Where do these priors come from? It's not just "being afraid"; the problem is that random guessing (of priors) is not a reasonable replacement for science. Bayes' rule is great if you can find a reasonable justification for a prior. Bayes is widely used for decision-making (for example), where you need an answer quickly & you aren't trying to make general scientific claims. But if you can't find a justifiable prior in a scientific work, using Bayes' rule just replaces one statistical fallacy for another one. After all, Bayes' rule will give nonsense answers if you give it a nonsense prior! Bayes' rule is a great tool in many circumstances! But it has a great weakness: it requires a prior. That doesn't make it useless; few tools are useful in all circumstances. But requiring "everyone to use Bayes' rule, even though we have no reasonable way to find a good estimate of the priors," is unlikely to ever happen (and rightly so). The article rightly points out a serious problem with the typical application of statistics, but there needs to be a better justification for priors than is suggested in this article. I could imagine systemic worldwide ways to deal with this. For example, perhaps the scientific community could allow people to propose initial priors, and then allow multiple different papers to improve the estimation of the probability over time. But that would require much more than articles repeatedly saying "there's no serious problem with priors"; having a justifiable way to estimate and update priors is the fundamental problem with Bayesian analysis in the scientific community.
- cwyers 7y ago
  > Bayes' rule is a great tool in many circumstances! But it has a great weakness: it requires a prior. That doesn't make it useless; few tools are useful in all circumstances. But requiring "everyone to use Bayes' rule, even though we have no reasonable way to find a good estimate of the priors," is unlikely to ever happen (and rightly so). The article rightly points out a serious problem with the typical application of statistics, but there needs to be a better justification for priors than is suggested in this article. If you don't have good information on what your prior should be, you can just use a really weak prior. And you probably know more about your data than you think you do -- if you are doing counts, that leads you to a different prior than if you are doing things that can be in increments smaller than a single integer.
fromthestart 7y ago
So how does one go about choosing reasonable bayesian priors for an experiment?
- afthonos 7y ago
  It becomes part of your experimental design. Just like people can quibble with your setup, with your questions, with your procedures, they can quibble with your priors. The difference is that it's out there, explicit. Note that a big weakness of Bayes rule is that you can look at any data and specify a prior that will make it look good. To continue with the mammogram example, suppose the doctor says "We really don't know if you are likely to have cancer or not. So we're just going to give 50-50 odds, and see what the test comes back with." That's a very different prior from the known base rate. The results would be, where C means "Cancer" and "R" means "Positive Result": P(C|R) = P(R|C) * P_prior(C) / P(R) = 1.0 * 0.5 / (1.0 * 0.5 + 0.05 * 0.5) = 0.5 / 0.55 = 0.9 A much higher probability. As you can imagine, you can do that in a paper as well: you know the data you have, and you come up with a "plausible" prior to make the data seem important. In my opinion, in any switch to using Bayesian analysis in scientific work, pre-registering priors will be essential.
  - mike_ivanov 7y ago
    Taking 50/50 as a prior indicates that you don't know the true prevalence of C. Note that here 0.9 is not a probability, but rather a degree of belief. In this concrete example it tells you how much you should be worried upon observing R. If the prevalence of C is unknown (even if C is actually rare!), then given a positive R one should be worried a lot, which is exactly what 0.9 says. Also note the language change and its implications for policy making, etc: Instead of saying "there is [not enough] evidence that .." we say "given the observations and these assumptions we should [not] believe/expect that .."
    - afthonos 7y ago
      All of this is true. I was writing to mainly illustrate that Bayesian analysis is still open to abuse—though as the sibling commenter says, it's hopefully clearer when it's being abused.
master_yoda_1 7y ago
The author teaches a basic course on statistics on harvard extension school and charges ~$2800 for that. He wrote a clickbait random article and I can see it on hacker news front page. And we say only facebook can publish fake news ;)
- marcus_holmes 7y ago
  sooo... what's fake? You made a ton of ad-hominem there, but what's the actual criticism of the article?
  - reallydude 7y ago
    Just from a casual look, the cancer diagnosis modeling is flawed. > the doctor would need to consider the overall incidence rate of cancer among similar women with similar symptoms, not including the result of the mammogram That's a bad assumption. Mammogram is a radiology tool to investigate tissue. It's not a randomizing element as it's fundamental to arriving at the thesis, which is then correlated from MULTIPLE vectors. > a similar patient finds a lump it turns out to be benign A manual inspection is not the same. In good faith, let's assume they are the same for no reason but to argue about how not to do medicine as some precept for "Base rates are effectively random". Turns out, Base rates are not random guesses. What's interesting about Bayesian Theory is we use it all the time and then observe and collect statistical data about the outcomes, after making assumptions (like a specific Base rate) and use back propagation to correct until models fit measurable events. This is why, tests often have caveats about efficacy. Base rate is sometimes, reasonably, unknown because there is no additional correlation. This doesn't indict Base rates wherein the vast majority of cases, there are multiple vectors (or new vectors are generated) that show this process is reliable (beyond a few dice rolls). There have also been cases where there is no corroboration from new measures and the deduction is the that original measure and Base rate were random. It's a lot of hand waving from a classic troll. Why? Probably for students who want to feel like they have "discovered" how the establishment is ignorant.
    - DanBC 7y ago
bambax 7y ago
> we should follow the mantra of the mathematician (and Bayesian) Pierre-Simon [de]* Laplace, that extraordinary claims require extraordinary evidence* We should, but we don't, because we crave extraordinary results and are ready to give up reason to get them. But if it's too good to be true (or too spectacular), it probably is.
mltony 7y ago
I see there's one more problem with thinker sculpture study, that has nothing to do how we analyze the data. There was another study a while ago, where a dime was placed (or not placed for control group) in the public copying machine. And then researchers would ask those people who were using that copying machine a few questions, one of them was "on a scale from 0 to 10 how happy you are?" People who found their dimes reported significantly higher happiness scores, but I don't remember by how much. Assuming this experiment was true, and not some journalists twisting the data wihtout understanding statistics, I see there's some analogy with the sculpture study. In particular, the conclusion I would make is that asking people questions about happiness or religion or anything whatsoever doesn't tell you much about their actual believes, since their answers are heavily affected by their mood at the moment. I asume this statement can be turned into another study verifying its correctness, and then after we collect the results, we'd have to start arguing about Bayes vs. Fischer all over again...
tunesmith 7y ago
One other example of the flaw (I think) that came out in pop culture was a few years ago during the Serial podcast about Adnan Syed. Near the end when the hosts were wrestling about whether they should think he was guilty, the interviewer's friend (the producer? Dana?) came on and shared why she thought he was guilty, and it was this convoluted argument about if Syed was innocent, he'd have to be the unluckiest guy in the world, so therefore he was probably guilty. Her point was that it was just too strong a coincidence that it was his ex-girlfriend that got killed, on that particular day during some period of time when he didn't have an alibi, etc etc. That seemed to me like it could be a bad Bayesian argument - like it would have been a good argument had Serial selected a random citizen to interview, but they selected someone they already knew was connected to the story. Murders happen and by definition the circumstances are always highly unlikely because murders are so rare, and innocent people closely connected to the circumstances of the crime are very, very unlucky by definition. You can't point to that unluckiness as evidence that they're probably guilty.
- deleted 7y ago
  [deleted]
- roywiggins 7y ago
  Add to that, highly unlikely events occur all the time, but most of the time they're not anywhere near a murder. Once you start digging around for unlikely things, you'll find them. Heck, even if you dig into a random person associated with a murder, you'll find some unlikely things, but if you dig into a person who has already been accused of the crime (rightly or wrongly), of course there will be loads of suspicious circumstances. That's why they were accused in the first place.
  - richk449 7y ago
    > highly unlikely events occur all the time They do?
    - reallydude 7y ago
      This is fundamental to Chaos theory (how small initial states can lead to rich systems that contain both expected and unexpected outcomes), which isn't a theory as much as an area of mathematical modeling of the most complex systems. Me waving off a butterfly is just as likely to influence a hurricane as the unmolested butterfly flapping. Any given reader's brain existing to comprehend this comment is highly unlikely, yet it occurs and continues within layers of chaotic systems, that exist turbulently, in perceived reality.
CodiePetersen 7y ago
Wow, I had read an argument criticising statistcal significance before but it made nowhere near as good of a case as this article has.
mcguire 7y ago
"Harvard Business School professor Amy Cuddy’s 2010 study of “power posing:” the idea that adopting a powerful posture for a couple of minutes can change your life for the better by affecting your hormone levels and risk tolerances." Hey, now. I like standing with my hands on my hips, imagining I have a cape flowing in the breeze behind me.
madhadron 7y ago
<sigh> Okay, let's do this again. Anyone arguing over frequentist versus Bayesian statistics is missing the foundations of their statistical training. Both are subsumed by the framework of decision theory. This isn't new. It goes back to Wald's work in the 1950's. And the examples he is talking about wouldn't be saved by different statistical methods. No fiddling with techniques at the end can save a failed design of the trial.
- skygazer 7y ago
  I get that you're probably trying to leave it as an exercise to the reader, but the word "frequentist" doesn't even appear on the wikipedia article for Decision Theory, nor the Stanford Encyclopedia of Philosophy. I can't tell if you're being hyperbolic, or if it requires deep study to grasp how bayesian and frequentist statistics are rendered irrelevant by decision theory. Anyone care to take a stab at a layman's summary for the benefit of the under-educated folks around here?
  - James_Henry 7y ago
    Googling <statistical decision theory> might help you out. My opinion is the Wikipedia page for decision theory could use a good rewrite. Someday I may get around to it. Also, madhadron didn't quite say that Bayesian and frequentist statistics are rendered irrelevant by decision theory. Rather, statistical decision theory includes Bayesian and frequentist statistics as possible statistical decision rules. You might want to not use Bayesian or frequentist rules and instead use minimax regret or something else. I do think that it is fine that people argue about Bayes vs frequentist. I wish they'd consider everything else though.
    - jbay808 7y ago
      >> You might want to not use Bayesian or frequentist rules and instead use minimax regret or something else. ... What? Any such decision algorithm at least in theory still takes a probability distribution as an input. You still need to approximately follow the proper rules of probability to get that distribution, there's no way around it.
6gvONxR4sf7o 7y ago
It's easy to challenge the subjectivity of a prior, pretending that frequentist testing is objective... until you design your own experiment, rather than just analyze one. Turns out this is a more important part. It'd be illustrative to explain that process. A p-value of 0.05 (or a test with 95% significance) will, if there's actually nothing there, tell you'll think there's something there 5% of the time (wrongly). It says nothing about error rates if there is something there, which is presumably what we're interested in. So for that, you'll do what's called a power analysis. Power of the conventional 80% tells you, if there's actually an effect size of X and you know all about your noise, you'll think there's nothing there 20% of the time (wrongly). 5% comes out of nowhere. 80% comes out of nowhere. A priori knowing your noise is often feasible, but not always. X, the presumed effect size you have to magically know before you run your experiment, comes out of nowhere. Getting this right is crucial to a reliable experiment, and it needs to really reflect your prior belief, even though it's frequentist. It's totally subjective. What's worse is that X is often taken to be whatever leads to a feasible/fundable experiment that'll be done fast enough, not anything scientific. It's waaay harder to understand the impact of the subjective 5%/80%/X decisions than it is to understand the impact of your prior. A prior takes way less training. Better yet, you can report your results assuming many different priors and let your reader subjectively decide what to think, so you don't have to commit as hard. tl;dr The "gold standard" way of doing science is already really subjective and that's okay. Equally subjective alternatives can still be better science.
YeGoblynQueenne 7y ago
I was reminded of this article, particulraly the bit about Sally Clark's case, when I read this today: https://www.theguardian.com/uk-news/2019/aug/02/louise-porton-jailed-for-life-for-murdering-young-daughters https://www.theguardian.com/uk-news/2019/aug/02/louise-porto... It's about a woman who was jailed for killing all three of her kids. The prosecutors alleged she wanted to sleep around and she couldn't because of the kids. I'm struct by the fact that there was no direct evidence that the mother was responsible for the deaths. The only "evidence" seems to have been that it was not clear why the girls died: Both deaths were consistent with deliberate airway obstruction, and doctors could not find “any natural reason why either, let alone both, should have died”, prosecutors said. I would have thought that if there was uncertainty about the cause of death (and that is exactly what "the doctors could not find any natural reason" states: uncertainty about the cause of death) then there is no sufficient evidence to convict. But, I'm going by only what's in The Guardian article and I don't know the details of the case.
miles7 7y ago
Great article overall, but I have a serious beef with it and others I’ve seen like it: the writer quickly lapses into saying there’s a problem in “science” and with “scientists”. But p-value testing is not widely used in fields such as chemistry, condensed matter physics, materials science etc. (And I imagine many other scientific fields, such as sub fields of biology, although the ones I just listed are ones I know well.) So this kind of overgeneralization is a serious error and could contribute to more mistrust of science by the general public.
pmisans 7y ago
Has anyone heard of a scientific journal adopting a peer-replicated approach(only accept papers that peers can replicate)? With the replication rates so low, i'd imagine some of these prestigious journals would want to vet their papers more strictly, but a cursory google search didn't dig anything up for me.

185 points by dnetesn 7y ago

My favorite way to show the base rate fallacy is to take the most extreme example. Suppose I have an invisible dragon detector that is accurate 99.9% of the time... if it tells me there is an invisible dragon in the room, it doesn't mean there is only a 0.1% chance there is not a dragon... there is a 100% chance there is not an invisible dragon in the room, because they don't exist.

kortilla 7y ago
Or just link to the xkcd comic about the sun explosion detector. https://www.explainxkcd.com/wiki/index.php/1132:_Frequentists_vs._Bayesians https://www.explainxkcd.com/wiki/index.php/1132:_Frequentist...
- reallydude 7y ago
  I always interpreted that comic to be synergistically thoughtful. If the sun exploded, it doesn't matter who has the $50, so it's a good bet from a pragmatic standpoint.
TheOtherHobbes 7y ago
That only works if you already know something is/isn't true with absolute certainty. It's useless for fundamental research, because by definition you're exploring what you don't know yet. The real problem is more one of labelling. "True according to science" isn't a binary, but it's treated as if it is - especially by marketers. Science is more like a set of concentric circles of decreasing confidence. You can be very confident indeed about the contents of the centre circle which includes undergraduate physics and engineering. You can also be confident that there are commonly agreed edge cases, areas of inaccuracy, and extreme circumstances where the science stops being reliable. As you get further away from the centre confidence decreases. A lot of the debate about replication is about research that is a long way from the centre, where uncertainty is high. But neither researchers nor the science press nor the mainstream media will report this. Studies are usually presented as "Science says...", as if you're supposed to be just as confident of the results of a psychological study that asks a population of 30 undergrads from the same college and the same year some poorly designed questions as you are in Special Relativity.
- cortesoft 7y ago
  It is just to show why base rate matters by using the most extreme example... it doesn't imply that it is easy to calculate.
deleted 7y ago
[deleted]

BoiledCabbage 7y ago

It's incredible it's held up this long. It shows how much science actually is dogma. Human nature wins out over rationality.

xamuel 7y ago
Not so incredible if you consider how the sausage is made. People who hate math pursue social science PhDs and are taught "p-values = truth". These same people are pressured to produce lots of research, and not just boring research but preferably shocking research. Journals are incentivized to publish such shocking research because that kind of research gets cited. When you think about it, this stuff will probably hold up for a long time to come.
- SantalBlush 7y ago
  It has little to do with stereotypes about social scientists hating math or misunderstanding p-values. The current system incentivizes p-hacking. Nobody wants to throw away their work if it doesn't meet the p < 0.05 criterion, especially when their career is on the line.
  - marcus_holmes 7y ago
    so...why are we calling this science if it doesn't meet the basics of the Scientific method?
  - SkyBelow 7y ago
    While it isn't the most concrete sample size, when I was at college the mathematics requirements for the soft sciences were far inferior to the mathematics requirements for hard sciences. To the extent that even the easier statistics course in the math department wasn't require and instead was replaced by a statistics class in the social science department that was only valid for social sciences.

chrisco255 7y ago

"The point is that we have good reason to be skeptical, and we should follow the mantra of the mathematician (and Bayesian) Pierre-Simon Laplace, that extraordinary claims require extraordinary evidence. By ignoring the necessity of priors, significance testing opens the door to false positive results." Such an important quote. I believe the replication crisis is especially present in nutrition studies. We live in an age where headlines go viral and new wave diets are taken on in rapid succession. Take all nutrition studies with a grain of salt. (assuming, of course, that salt is good for you...or is it???)

dwheeler 7y ago

The article says: > The main reason scientists have historically been resistant to using Bayesian inference instead is that they are afraid of being accused of subjectivity. The prior probabilities required for Bayes’ rule feel like an unseemly breach of scientific ethics. Where do these priors come from? It's not just "being afraid"; the problem is that random guessing (of priors) is not a reasonable replacement for science. Bayes' rule is great if you can find a reasonable justification for a prior. Bayes is widely used for decision-making (for example), where you need an answer quickly & you aren't trying to make general scientific claims. But if you can't find a justifiable prior in a scientific work, using Bayes' rule just replaces one statistical fallacy for another one. After all, Bayes' rule will give nonsense answers if you give it a nonsense prior! Bayes' rule is a great tool in many circumstances! But it has a great weakness: it requires a prior. That doesn't make it useless; few tools are useful in all circumstances. But requiring "everyone to use Bayes' rule, even though we have no reasonable way to find a good estimate of the priors," is unlikely to ever happen (and rightly so). The article rightly points out a serious problem with the typical application of statistics, but there needs to be a better justification for priors than is suggested in this article. I could imagine systemic worldwide ways to deal with this. For example, perhaps the scientific community could allow people to propose initial priors, and then allow multiple different papers to improve the estimation of the probability over time. But that would require much more than articles repeatedly saying "there's no serious problem with priors"; having a justifiable way to estimate and update priors is the fundamental problem with Bayesian analysis in the scientific community.

cwyers 7y ago
> Bayes' rule is a great tool in many circumstances! But it has a great weakness: it requires a prior. That doesn't make it useless; few tools are useful in all circumstances. But requiring "everyone to use Bayes' rule, even though we have no reasonable way to find a good estimate of the priors," is unlikely to ever happen (and rightly so). The article rightly points out a serious problem with the typical application of statistics, but there needs to be a better justification for priors than is suggested in this article. If you don't have good information on what your prior should be, you can just use a really weak prior. And you probably know more about your data than you think you do -- if you are doing counts, that leads you to a different prior than if you are doing things that can be in increments smaller than a single integer.

fromthestart 7y ago

So how does one go about choosing reasonable bayesian priors for an experiment?

afthonos 7y ago
It becomes part of your experimental design. Just like people can quibble with your setup, with your questions, with your procedures, they can quibble with your priors. The difference is that it's out there, explicit. Note that a big weakness of Bayes rule is that you can look at any data and specify a prior that will make it look good. To continue with the mammogram example, suppose the doctor says "We really don't know if you are likely to have cancer or not. So we're just going to give 50-50 odds, and see what the test comes back with." That's a very different prior from the known base rate. The results would be, where C means "Cancer" and "R" means "Positive Result": P(C|R) = P(R|C) * P_prior(C) / P(R) = 1.0 * 0.5 / (1.0 * 0.5 + 0.05 * 0.5) = 0.5 / 0.55 = 0.9 A much higher probability. As you can imagine, you can do that in a paper as well: you know the data you have, and you come up with a "plausible" prior to make the data seem important. In my opinion, in any switch to using Bayesian analysis in scientific work, pre-registering priors will be essential.
- mike_ivanov 7y ago
  Taking 50/50 as a prior indicates that you don't know the true prevalence of C. Note that here 0.9 is not a probability, but rather a degree of belief. In this concrete example it tells you how much you should be worried upon observing R. If the prevalence of C is unknown (even if C is actually rare!), then given a positive R one should be worried a lot, which is exactly what 0.9 says. Also note the language change and its implications for policy making, etc: Instead of saying "there is [not enough] evidence that .." we say "given the observations and these assumptions we should [not] believe/expect that .."
  - afthonos 7y ago
    All of this is true. I was writing to mainly illustrate that Bayesian analysis is still open to abuse—though as the sibling commenter says, it's hopefully clearer when it's being abused.

master_yoda_1 7y ago

The author teaches a basic course on statistics on harvard extension school and charges ~$2800 for that. He wrote a clickbait random article and I can see it on hacker news front page. And we say only facebook can publish fake news ;)

marcus_holmes 7y ago
sooo... what's fake? You made a ton of ad-hominem there, but what's the actual criticism of the article?
- reallydude 7y ago
  Just from a casual look, the cancer diagnosis modeling is flawed. > the doctor would need to consider the overall incidence rate of cancer among similar women with similar symptoms, not including the result of the mammogram That's a bad assumption. Mammogram is a radiology tool to investigate tissue. It's not a randomizing element as it's fundamental to arriving at the thesis, which is then correlated from MULTIPLE vectors. > a similar patient finds a lump it turns out to be benign A manual inspection is not the same. In good faith, let's assume they are the same for no reason but to argue about how not to do medicine as some precept for "Base rates are effectively random". Turns out, Base rates are not random guesses. What's interesting about Bayesian Theory is we use it all the time and then observe and collect statistical data about the outcomes, after making assumptions (like a specific Base rate) and use back propagation to correct until models fit measurable events. This is why, tests often have caveats about efficacy. Base rate is sometimes, reasonably, unknown because there is no additional correlation. This doesn't indict Base rates wherein the vast majority of cases, there are multiple vectors (or new vectors are generated) that show this process is reliable (beyond a few dice rolls). There have also been cases where there is no corroboration from new measures and the deduction is the that original measure and Base rate were random. It's a lot of hand waving from a classic troll. Why? Probably for students who want to feel like they have "discovered" how the establishment is ignorant.
  - DanBC 7y ago

bambax 7y ago

> we should follow the mantra of the mathematician (and Bayesian) Pierre-Simon [de]* Laplace, that extraordinary claims require extraordinary evidence* We should, but we don't, because we crave extraordinary results and are ready to give up reason to get them. But if it's too good to be true (or too spectacular), it probably is.

mltony 7y ago

I see there's one more problem with thinker sculpture study, that has nothing to do how we analyze the data. There was another study a while ago, where a dime was placed (or not placed for control group) in the public copying machine. And then researchers would ask those people who were using that copying machine a few questions, one of them was "on a scale from 0 to 10 how happy you are?" People who found their dimes reported significantly higher happiness scores, but I don't remember by how much. Assuming this experiment was true, and not some journalists twisting the data wihtout understanding statistics, I see there's some analogy with the sculpture study. In particular, the conclusion I would make is that asking people questions about happiness or religion or anything whatsoever doesn't tell you much about their actual believes, since their answers are heavily affected by their mood at the moment. I asume this statement can be turned into another study verifying its correctness, and then after we collect the results, we'd have to start arguing about Bayes vs. Fischer all over again...

tunesmith 7y ago

One other example of the flaw (I think) that came out in pop culture was a few years ago during the Serial podcast about Adnan Syed. Near the end when the hosts were wrestling about whether they should think he was guilty, the interviewer's friend (the producer? Dana?) came on and shared why she thought he was guilty, and it was this convoluted argument about if Syed was innocent, he'd have to be the unluckiest guy in the world, so therefore he was probably guilty. Her point was that it was just too strong a coincidence that it was his ex-girlfriend that got killed, on that particular day during some period of time when he didn't have an alibi, etc etc. That seemed to me like it could be a bad Bayesian argument - like it would have been a good argument had Serial selected a random citizen to interview, but they selected someone they already knew was connected to the story. Murders happen and by definition the circumstances are always highly unlikely because murders are so rare, and innocent people closely connected to the circumstances of the crime are very, very unlucky by definition. You can't point to that unluckiness as evidence that they're probably guilty.

deleted 7y ago
[deleted]
roywiggins 7y ago
Add to that, highly unlikely events occur all the time, but most of the time they're not anywhere near a murder. Once you start digging around for unlikely things, you'll find them. Heck, even if you dig into a random person associated with a murder, you'll find some unlikely things, but if you dig into a person who has already been accused of the crime (rightly or wrongly), of course there will be loads of suspicious circumstances. That's why they were accused in the first place.
- richk449 7y ago
  > highly unlikely events occur all the time They do?
  - reallydude 7y ago
    This is fundamental to Chaos theory (how small initial states can lead to rich systems that contain both expected and unexpected outcomes), which isn't a theory as much as an area of mathematical modeling of the most complex systems. Me waving off a butterfly is just as likely to influence a hurricane as the unmolested butterfly flapping. Any given reader's brain existing to comprehend this comment is highly unlikely, yet it occurs and continues within layers of chaotic systems, that exist turbulently, in perceived reality.

CodiePetersen 7y ago

Wow, I had read an argument criticising statistcal significance before but it made nowhere near as good of a case as this article has.

mcguire 7y ago

"Harvard Business School professor Amy Cuddy’s 2010 study of “power posing:” the idea that adopting a powerful posture for a couple of minutes can change your life for the better by affecting your hormone levels and risk tolerances." Hey, now. I like standing with my hands on my hips, imagining I have a cape flowing in the breeze behind me.

madhadron 7y ago

<sigh> Okay, let's do this again. Anyone arguing over frequentist versus Bayesian statistics is missing the foundations of their statistical training. Both are subsumed by the framework of decision theory. This isn't new. It goes back to Wald's work in the 1950's. And the examples he is talking about wouldn't be saved by different statistical methods. No fiddling with techniques at the end can save a failed design of the trial.

skygazer 7y ago
I get that you're probably trying to leave it as an exercise to the reader, but the word "frequentist" doesn't even appear on the wikipedia article for Decision Theory, nor the Stanford Encyclopedia of Philosophy. I can't tell if you're being hyperbolic, or if it requires deep study to grasp how bayesian and frequentist statistics are rendered irrelevant by decision theory. Anyone care to take a stab at a layman's summary for the benefit of the under-educated folks around here?
- James_Henry 7y ago
  Googling <statistical decision theory> might help you out. My opinion is the Wikipedia page for decision theory could use a good rewrite. Someday I may get around to it. Also, madhadron didn't quite say that Bayesian and frequentist statistics are rendered irrelevant by decision theory. Rather, statistical decision theory includes Bayesian and frequentist statistics as possible statistical decision rules. You might want to not use Bayesian or frequentist rules and instead use minimax regret or something else. I do think that it is fine that people argue about Bayes vs frequentist. I wish they'd consider everything else though.
  - jbay808 7y ago
    >> You might want to not use Bayesian or frequentist rules and instead use minimax regret or something else. ... What? Any such decision algorithm at least in theory still takes a probability distribution as an input. You still need to approximately follow the proper rules of probability to get that distribution, there's no way around it.

6gvONxR4sf7o 7y ago

It's easy to challenge the subjectivity of a prior, pretending that frequentist testing is objective... until you design your own experiment, rather than just analyze one. Turns out this is a more important part. It'd be illustrative to explain that process. A p-value of 0.05 (or a test with 95% significance) will, if there's actually nothing there, tell you'll think there's something there 5% of the time (wrongly). It says nothing about error rates if there is something there, which is presumably what we're interested in. So for that, you'll do what's called a power analysis. Power of the conventional 80% tells you, if there's actually an effect size of X and you know all about your noise, you'll think there's nothing there 20% of the time (wrongly). 5% comes out of nowhere. 80% comes out of nowhere. A priori knowing your noise is often feasible, but not always. X, the presumed effect size you have to magically know before you run your experiment, comes out of nowhere. Getting this right is crucial to a reliable experiment, and it needs to really reflect your prior belief, even though it's frequentist. It's totally subjective. What's worse is that X is often taken to be whatever leads to a feasible/fundable experiment that'll be done fast enough, not anything scientific. It's waaay harder to understand the impact of the subjective 5%/80%/X decisions than it is to understand the impact of your prior. A prior takes way less training. Better yet, you can report your results assuming many different priors and let your reader subjectively decide what to think, so you don't have to commit as hard. tl;dr The "gold standard" way of doing science is already really subjective and that's okay. Equally subjective alternatives can still be better science.

YeGoblynQueenne 7y ago

I was reminded of this article, particulraly the bit about Sally Clark's case, when I read this today: https://www.theguardian.com/uk-news/2019/aug/02/louise-porton-jailed-for-life-for-murdering-young-daughters https://www.theguardian.com/uk-news/2019/aug/02/louise-porto... It's about a woman who was jailed for killing all three of her kids. The prosecutors alleged she wanted to sleep around and she couldn't because of the kids. I'm struct by the fact that there was no direct evidence that the mother was responsible for the deaths. The only "evidence" seems to have been that it was not clear why the girls died: Both deaths were consistent with deliberate airway obstruction, and doctors could not find “any natural reason why either, let alone both, should have died”, prosecutors said. I would have thought that if there was uncertainty about the cause of death (and that is exactly what "the doctors could not find any natural reason" states: uncertainty about the cause of death) then there is no sufficient evidence to convict. But, I'm going by only what's in The Guardian article and I don't know the details of the case.

miles7 7y ago

Great article overall, but I have a serious beef with it and others I’ve seen like it: the writer quickly lapses into saying there’s a problem in “science” and with “scientists”. But p-value testing is not widely used in fields such as chemistry, condensed matter physics, materials science etc. (And I imagine many other scientific fields, such as sub fields of biology, although the ones I just listed are ones I know well.) So this kind of overgeneralization is a serious error and could contribute to more mistrust of science by the general public.

pmisans 7y ago

Has anyone heard of a scientific journal adopting a peer-replicated approach(only accept papers that peers can replicate)? With the replication rates so low, i'd imagine some of these prestigious journals would want to vet their papers more strictly, but a cursory google search didn't dig anything up for me.