8 ms·

Any line that starts with "Although not statistically significant" should be automatically deleted.

• sparkie 6y ago
  Not if you want to pump and dump a stock.
• downerending 6y ago
  In general, yes. In an emergency when there are no other obvious better alternatives at hand, looking at results with relatively poor p-values might still be a pretty reasonable thing to do.
  • seemslegit 6y ago
    Not really.
    • QuesnayJr 6y ago
      But it doesn't mean that drug doesn't work -- it means that the sample was too small to say that it works. If the sample were 100,000, then that would mean that it doesn't work. This was around 225, which isn't tiny, but given that we are well-endowed with people dying of COVID who would rather not, it makes sense to do more testing. Which they call for.
      • seemslegit 6y ago
        Didn't say it doesn't make sense to do more testing, but relaxing your reasoning standard in emergencies is a bad idea.
        • tarboreus 6y ago
          They said "looking at more results," not "relax our reasoning standards."
          • seemslegit 6y ago
            Yes, you should not look at any result in an emergency you wouldn't otherwise - in fact you should probably only increase scrutiny given how much incentive there is to merely get a headline with your drug mentioned as a potential covid19 cure. The OP is basically arguing for a "something must be done and this is something" reasoning.
            • downerending 6y ago
              There's always "We've tried nothing, and we're all out of ideas." :-)
              • seemslegit 6y ago
                Yes there is. Not having effective options against a new virus is the null hypothesis. So by all means do whatever research has promising leads but don't lower burden of proof for accepting something as clinically effective.
        • downerending 6y ago
          I'm picturing the passengers on the Titanic standing around, refusing to move until presented with an option proven valid to p < 0.95.
          • cy_hauser 6y ago
            I'm picturing the Feds and States shelling out desperately needed money to purchase an almost useless drug because the people on the Titanic are desperate for anything rather than the drowning.
            • downerending 6y ago
              If it does indeed turn out to be useless (which is far from obvious at the moment), it'll be dropped, and not much money will have been spent. It's unfortunate, but a lot of what we're trying right now will turn out to be useless. One or two recent analyses suggest that the (far more expensive) lockdowns might turn out to have been almost useless. That doesn't mean that trying them was the wrong thing to do. We're living in unusually uncertain times.
          • gus_massa 6y ago
            It is like the passengers discussing if they must carry a box with 10 pounds of oranges in each life boat, find some paddles, a plastic cover, or paint the Eyes of Horus in them before launching them. The first three look useful, but I'm not sure if they improve the chance of survival too much. You need some time to find them and enough room in the boat, so it is a trade off if they are useful at all.
      • c06n 6y ago
        > it means that the sample was too small to say that it works. If it works! If it does not, in fact, work, no sample of any size will show that it works.
  • jakeogh 6y ago
    also https://news.ycombinator.com/item?id=23013667 https://news.ycombinator.com/item?id=23013667
• im3w1l 6y ago
  Depends on how much power the study has.
• klodolph 6y ago
  No, that go against the current standards of scientific ethics. “Not statistically significant” does not mean the effect does not exist. It means that the study provides weak evidence, but weak evidence ≠ no evidence. It tags things for further study. This is one of those cases where people get too hung up on P values. You have to look at the bigger picture; look at the entire scientific process.
  • gfodor 6y ago
    I mean, it does provide no evidence for the specific hypothesis. It may provide evidence of other hypotheses. So you're right, but you're also wrong, without making yourself more clear. Poor precision in the use of language around clinical data has been a major problem in all of this, in the press especially, where for the most part they clearly don't give a damn and just have an agenda to spin in most cases. Because of this, scientists publishing ought to try to use language that ensures their work cannot be misinterpreted. The notes in the results section here about non-statistically significant aspects to the data fail that test, since an average reader would likely walk away with a false mental model of what the data is showing. edit: my comment is wrong, i'm tired and haven't had enough coffee. the criticisms below are valid. i do have a point, but won't jam it in here.
    • klodolph 6y ago
      > I mean, it does provide no evidence for the specific hypothesis. Again, this is incorrect. It provides weak evidence, not no evidence. You talk about precise language here but you are conflating “not statistically significant” with “no evidence”. You can’t just handwave the difference between these two things.
    • jefftk 6y ago
      > it does provide no evidence for the specific hypothesis It does provide some evidence: we are more likely to see this result in a world where the hypothesis is true than a world where it is false.
      • gus_massa 6y ago
• pps43 6y ago
  What if there are 10 independent studies, all showing positive, but not statistically significant effect?
  • im3w1l 6y ago
    That's when you do a metastudy I think.
    • pps43 6y ago
      Either that, or pool the data from all studies and analyze it together, almost certainly finding a statistically significant effect.
• scythe 6y ago
  Statistical significance is not a bright-line test. The p value reported is the probability that the observed effects occur if there are actually no effects. Notably, p is always greater than zero. In general, when less data is available, larger p values are considered more interesting, while when more data is available, p must be smaller to be interesting. In particle physics, p < 0.001 means almost nothing. In the particular quoted line, p = 0.07ish which is above the human-defined, conventional threshold of 0.05 but in this case there are almost no other useful data and so the phenomenon is interesting.
• achenatx 6y ago
  The issue with statistical significance is that it might have a strong effect in some people and no effect in most people and statistical significance cant detect that. For example, statistically chocolate doesnt cause acne. Yet I 100% get acne from chocolate. There is enough variance in people that many things that work anecdotally cannot be proven statistically.