4 ms·

> having the misfortune of the one-in-twenty chance of getting a really interesting false positive at p = 0.05 means you either have to double down on the study

> having the misfortune of the one-in-twenty chance of getting a really interesting false positive at p = 0.05 means you either have to double down on the study Or you could get a large sample size to bring the p down to 0.01 or whatever. You know, do science.

sliverstorm 9y ago
You say that like it's free. It's entirely reasonable, IMO, to do more "exploratory" studies at p = 0.05, and then do a follow-up on promising studies at p = 0.01
geofft 9y ago
Well, the smaller your p, the higher your chance of false negatives, too; maybe there was an actual effect there that was worth more study, and you dismissed it. The problem is that a study that's significant at p = 0.05 but not p = 0.01 is worth something to science. It indicates something worth doing several more studies on, preferably by different teams from different institutes with non-overlapping confounding factors. But the practice of science gives you only two options once you find such a study, as a young scientist: promote it immediately and become visible for having discovered something, or aggressively ignore it and go find something else that you think is likely to be immediately promising. "Doing science," as you put it, is the worst option for you because it ensures you spend a lot more time on something for no reward -- and the rational self-interest here is perfectly rational, because if you don't get funding or tenure, you're not going to do any science any more at all. (As seems to have happened with Dr. Cuddy.) To be clear, I have no well-formed opinion on whether Dr. Cuddy is a good scientist, having only read this article. I assume she is, or at least that she's honest in her desires. But I think talented, honest people are nonetheless pressured by incentives, and fixing incentives is better for society (assuming we as a larger society wish our scientists to continue to do science) than assuming that people will be self-sacrificial and altruistic in pursuit of a tiny chance at accomplishing the greater good.
- killjoywashere 9y ago
  Here's the structural problem I see: there are acres, literally, acres of computer science types in SF and certain other enclaves, slaving away on ads when they would rather be curing cancer. But they don't know the first thing about cancer. Meanwhile, there are enclaves where biologists are slaving away capturing data and not even realizing it's data. "Bookkeeping data" they might call it. There should be mixers between the biologists and the computer scientists. Especially non-academic physicians and computer scientists. Look at Kaiser-Permanente, Sutter Health, or Defense Health Agency. Huge orgs, very little research. Data falling off the shelves and they don't know what to do with it.
  - nonbel 9y ago
    Please no mixers. The AI/ML/CS people should start their own different, better approach.
  - slavik81 9y ago
    Mozilla Science runs the global science sprint each year to bring scientists of all kinds together with developers. I bet there's lots of projects that could still use help. https://github.com/mozilla/global-sprint/issues https://github.com/mozilla/global-sprint/issues
  - amazingman 9y ago
    You’re not wrong. Especially not about the ads. What we need, unfortunately, are (at least) 2 very difficult things to achieve legislatively: 1. A legal framework for health-related products & services that meaningfully protects individual privacy, and yet also preserves the conditions for the kind of velocity and rate of change that brings technological progress to the masses reliably and quickly. 2. Massive academic funding for more “pure” research in hardware & software design, engineering, the philosophy of what’s possible, the phenomenon and ethics of what we choose to build, and so on. TLDR Pay more people more money to do important research in the public domain, which I can’t help but think would attract many of the CS types you mentioned.
    - 9y ago
Obi_Juan_Kenobi 9y ago
It's a guideline, not a rule. The more you treat it like a rule, the worse the situation is. There will never be an objective measure for conclusion 'correctness'. It will always be the case that you actually have to think about the data and statistics critically and decide whether the conclusions are appropriate. In certain well-defined situations, you can start to use harder metrics, but that's rarely where the controversy is in the first place. So if you want to, you know, do science, I urge you to use that big beautiful brain of yours. As a bonus, you don't even need to commit mass rat genocide to do it.

sliverstorm 9y ago

You say that like it's free. It's entirely reasonable, IMO, to do more "exploratory" studies at p = 0.05, and then do a follow-up on promising studies at p = 0.01

geofft 9y ago

Well, the smaller your p, the higher your chance of false negatives, too; maybe there was an actual effect there that was worth more study, and you dismissed it. The problem is that a study that's significant at p = 0.05 but not p = 0.01 is worth something to science. It indicates something worth doing several more studies on, preferably by different teams from different institutes with non-overlapping confounding factors. But the practice of science gives you only two options once you find such a study, as a young scientist: promote it immediately and become visible for having discovered something, or aggressively ignore it and go find something else that you think is likely to be immediately promising. "Doing science," as you put it, is the worst option for you because it ensures you spend a lot more time on something for no reward -- and the rational self-interest here is perfectly rational, because if you don't get funding or tenure, you're not going to do any science any more at all. (As seems to have happened with Dr. Cuddy.) To be clear, I have no well-formed opinion on whether Dr. Cuddy is a good scientist, having only read this article. I assume she is, or at least that she's honest in her desires. But I think talented, honest people are nonetheless pressured by incentives, and fixing incentives is better for society (assuming we as a larger society wish our scientists to continue to do science) than assuming that people will be self-sacrificial and altruistic in pursuit of a tiny chance at accomplishing the greater good.

killjoywashere 9y ago
Here's the structural problem I see: there are acres, literally, acres of computer science types in SF and certain other enclaves, slaving away on ads when they would rather be curing cancer. But they don't know the first thing about cancer. Meanwhile, there are enclaves where biologists are slaving away capturing data and not even realizing it's data. "Bookkeeping data" they might call it. There should be mixers between the biologists and the computer scientists. Especially non-academic physicians and computer scientists. Look at Kaiser-Permanente, Sutter Health, or Defense Health Agency. Huge orgs, very little research. Data falling off the shelves and they don't know what to do with it.
- nonbel 9y ago
  Please no mixers. The AI/ML/CS people should start their own different, better approach.
- slavik81 9y ago
  Mozilla Science runs the global science sprint each year to bring scientists of all kinds together with developers. I bet there's lots of projects that could still use help. https://github.com/mozilla/global-sprint/issues https://github.com/mozilla/global-sprint/issues
- amazingman 9y ago
  You’re not wrong. Especially not about the ads. What we need, unfortunately, are (at least) 2 very difficult things to achieve legislatively: 1. A legal framework for health-related products & services that meaningfully protects individual privacy, and yet also preserves the conditions for the kind of velocity and rate of change that brings technological progress to the masses reliably and quickly. 2. Massive academic funding for more “pure” research in hardware & software design, engineering, the philosophy of what’s possible, the phenomenon and ethics of what we choose to build, and so on. TLDR Pay more people more money to do important research in the public domain, which I can’t help but think would attract many of the CS types you mentioned.
  - 9y ago

Obi_Juan_Kenobi 9y ago

It's a guideline, not a rule. The more you treat it like a rule, the worse the situation is. There will never be an objective measure for conclusion 'correctness'. It will always be the case that you actually have to think about the data and statistics critically and decide whether the conclusions are appropriate. In certain well-defined situations, you can start to use harder metrics, but that's rarely where the controversy is in the first place. So if you want to, you know, do science, I urge you to use that big beautiful brain of yours. As a bonus, you don't even need to commit mass rat genocide to do it.