3 ms·

My go-to phrase: "Even if there were no real difference, we would see a result like this X% of the time anyway."

llasram 11y ago
I agree that makes intuitive sense when applied to explicitly constructed experiments we could attempt to exactly repeat, but the intuition begins to break down in more complicated circumstances, such as retrospective or longitutinal studies. Saying something would happen "X% of the time anyway" becomes a philosophical issue.
- omginternets 11y ago
  Yes, well, we could talk about frequentist vs bayesian statistics, but that's hardly the point of the article. That is what a p-value means, feasibility aside. There are also more intuitive ways to think about it, "false-positive rate" being one.
omginternets 11y ago
Mine is remarkably similar, though less general: "if p=.05, then running a bogus experiment 20 times ensures you'll publish." Suddenly, the abysmal state of nutritional epidemiology makes sense. How many labs do you suspect are running "red meat = death" studies?
- TeMPOraL 11y ago
  One is reminded of an XKCD classic - https://xkcd.com/882/ https://xkcd.com/882/. Sadly, it only goes to point out the sad state of contemporary science. Especially in terms of nutrition, the only way to stay sane is to stick to the good old street-fighting Bayes - if popularly consumed product X was leading to cancer in any way relevant to your life, you'd see people dropping like flies all over the place.
  - omginternets 11y ago
    >contemporary science I suspect you already know this, but it bears repetition, if only because it keeps us optimistic: science is a process. Even nutritional epidemiology will eventually find its way out of the abyss. :) And yes: the ol' jellybean XKCD is quite a propos.
    - TeMPOraL 11y ago
      I agree - but sadly, the key word here is "eventually". It's little of a consolation if you'd like to use some new knowledge today. So as it stands now, I classify every new and contemporary (younger than 10 years) study in psychology, sociology and nutrition as "not science" and "quackery" by default, until proven otherwise. I also assume that any science news reporting in any field is an outright lie with an agenda. Rarely do I see a counterexample. Seriously, science reporting does way more harm than bad research itself.
      - omginternets 11y ago
        > psychology I'd encourage you to be careful in dismissing the entirety of experimental psychology (full disclosure: I'm a cognitive neuroscientist, which is what a psychologist calls himself when he wants to distance himself from social psych). The reproduction crisis doesn't touch all fields of psych in equivalent ways. When you reject the field as a whole, you're also rejecting stuff like this: http://cavlab.net/?lang=en http://cavlab.net/?lang=en
nonbel 11y ago
>'My go-to phrase: "Even if there were no real difference, we would see a result like this X% of the time anyway."' Hmm, p-values are data-dependent random variables. Do this simple experiment in R where the null hypothesis of no difference is as true as can be: replicate(2,t.test(rnorm(10),rnorm(10))$p.value) For example, I got: [1] 0.8446991 0.3935238 So, the p-value can vary wildly (actually will follow a uniform(0,1) distribution) even if there were no real difference. Experimenter A would tell you that "we would see a result like this 84% of the time anyway" and experimenter B would say 39%. If enough experiments are performed, eventually someone will calculate a very small p-value. So which experimenter should we listen to? Which experimenters should bother sharing their results? All are correct. In other words, what should be done with that information about "even if there were no difference"? How should it affect our behavior, decisions, and beliefs?

I agree that makes intuitive sense when applied to explicitly constructed experiments we could attempt to exactly repeat, but the intuition begins to break down in more complicated circumstances, such as retrospective or longitutinal studies. Saying something would happen "X% of the time anyway" becomes a philosophical issue.

omginternets 11y ago
Yes, well, we could talk about frequentist vs bayesian statistics, but that's hardly the point of the article. That is what a p-value means, feasibility aside. There are also more intuitive ways to think about it, "false-positive rate" being one.

omginternets 11y ago

Mine is remarkably similar, though less general: "if p=.05, then running a bogus experiment 20 times ensures you'll publish." Suddenly, the abysmal state of nutritional epidemiology makes sense. How many labs do you suspect are running "red meat = death" studies?

TeMPOraL 11y ago
One is reminded of an XKCD classic - https://xkcd.com/882/ https://xkcd.com/882/. Sadly, it only goes to point out the sad state of contemporary science. Especially in terms of nutrition, the only way to stay sane is to stick to the good old street-fighting Bayes - if popularly consumed product X was leading to cancer in any way relevant to your life, you'd see people dropping like flies all over the place.
- omginternets 11y ago
  >contemporary science I suspect you already know this, but it bears repetition, if only because it keeps us optimistic: science is a process. Even nutritional epidemiology will eventually find its way out of the abyss. :) And yes: the ol' jellybean XKCD is quite a propos.
  - TeMPOraL 11y ago
    I agree - but sadly, the key word here is "eventually". It's little of a consolation if you'd like to use some new knowledge today. So as it stands now, I classify every new and contemporary (younger than 10 years) study in psychology, sociology and nutrition as "not science" and "quackery" by default, until proven otherwise. I also assume that any science news reporting in any field is an outright lie with an agenda. Rarely do I see a counterexample. Seriously, science reporting does way more harm than bad research itself.
    - omginternets 11y ago
      > psychology I'd encourage you to be careful in dismissing the entirety of experimental psychology (full disclosure: I'm a cognitive neuroscientist, which is what a psychologist calls himself when he wants to distance himself from social psych). The reproduction crisis doesn't touch all fields of psych in equivalent ways. When you reject the field as a whole, you're also rejecting stuff like this: http://cavlab.net/?lang=en http://cavlab.net/?lang=en

nonbel 11y ago

>'My go-to phrase: "Even if there were no real difference, we would see a result like this X% of the time anyway."' Hmm, p-values are data-dependent random variables. Do this simple experiment in R where the null hypothesis of no difference is as true as can be: replicate(2,t.test(rnorm(10),rnorm(10))$p.value) For example, I got: [1] 0.8446991 0.3935238 So, the p-value can vary wildly (actually will follow a uniform(0,1) distribution) even if there were no real difference. Experimenter A would tell you that "we would see a result like this 84% of the time anyway" and experimenter B would say 39%. If enough experiments are performed, eventually someone will calculate a very small p-value. So which experimenter should we listen to? Which experimenters should bother sharing their results? All are correct. In other words, what should be done with that information about "even if there were no difference"? How should it affect our behavior, decisions, and beliefs?