4 ms·

I like the idea but I don't really get why they accept "statistically insignificant" results. What I expect from a scientific paper is to prove (even emprically

I like the idea but I don't really get why they accept "statistically insignificant" results. What I expect from a scientific paper is to prove (even emprically) something. For example, if a paper claims something like "we show that using method X instead of Y doesn't improve the results", it won't get published on most journals... except this one, which is awesome, but the paper still has to prove that claim.

acchow 6y ago
If the only goal is p 0.05 and there's many groups around the world trying the same experiment, we should hope that they all publish their non-results otherwise by chance someone is going to have a significant result just by luck.
bboy13 6y ago
When you open the field to reward research itself rather than p-hacking, everyone benefits. It's like a flea market. Bargain bin discounts, junk to many, valuable to some, but it shifts the economics of the whole system and lets specialty niche researchers not have to reinvent the wheel. Also, knowing that a method or study was statistically insignificant is statistically significant for future study design. Thanks!
walty 6y ago
If you only publish the positive results, you miss the context of the unsurprising results. For example, in https://xkcd.com/882/ https://xkcd.com/882/ , if the results from all the experiments in the comic were published, the public would know that the "Green Jellybeans cause acne" result is probably a fluke.
progval 6y ago
Doesn't "statistically insignificant" mean papers which conclude something like "we find no correlation between X and Y"?
- kelnos 6y ago
  I think information like that is perhaps useful, at least to avoid wasting time repeating the same research, but publishing that does not prove that there's no correlation between X and Y, only that the things tested didn't show one.
- BenoitEssiambre 6y ago
  No! It means the data is too noisy to make any conclusion.
  - ellis-bell 6y ago
    not necessarily. it could. but if x and y are truly uncorrelated variables that are observed with unlimited precision, then you still would not reject the null that they are uncorelated
    - BenoitEssiambre 6y ago
      "Truly uncorrelated" is a problematic concept. Most of the time, any two things within each other's light cones, have at least tiny direct or indirect effects on each other. These effects are usually not that tiny in experimental settings because of imperfect experimental tools and procedures.
      - gus_massa 6y ago
        Like the speed of sound and the height of the table where the experiment is. Note that the pressure of the air changes with height. For a few inches the change is negligible and very difficult (impossible?) to measure, but the effect must be out there. I had once a problem with the temperature of the room, you usually ignore it, but it has a bigger effect like a 5% variation.
dguest 6y ago
You can't ever prove that switching to X doesn't change the result. All you can really hope for is putting smaller limits on the amount by which something could be improved. For one dumb example: if you wanted to see if rats can fly, you might observe 100 rats and see that none of them fly away. But this doesn't prove anything: in reality even if 3% of rats could fly you'd get this result about 5% of the time. You've really shown with 95% confidence is that less than 3% of rats fly.
collyw 6y ago
It should stop other groups wasting their time on the same research, no?

acchow 6y ago

If the only goal is p 0.05 and there's many groups around the world trying the same experiment, we should hope that they all publish their non-results otherwise by chance someone is going to have a significant result just by luck.

bboy13 6y ago

When you open the field to reward research itself rather than p-hacking, everyone benefits. It's like a flea market. Bargain bin discounts, junk to many, valuable to some, but it shifts the economics of the whole system and lets specialty niche researchers not have to reinvent the wheel. Also, knowing that a method or study was statistically insignificant is statistically significant for future study design. Thanks!

walty 6y ago

If you only publish the positive results, you miss the context of the unsurprising results. For example, in https://xkcd.com/882/ https://xkcd.com/882/ , if the results from all the experiments in the comic were published, the public would know that the "Green Jellybeans cause acne" result is probably a fluke.

progval 6y ago

Doesn't "statistically insignificant" mean papers which conclude something like "we find no correlation between X and Y"?

kelnos 6y ago
I think information like that is perhaps useful, at least to avoid wasting time repeating the same research, but publishing that does not prove that there's no correlation between X and Y, only that the things tested didn't show one.
BenoitEssiambre 6y ago
No! It means the data is too noisy to make any conclusion.
- ellis-bell 6y ago
  not necessarily. it could. but if x and y are truly uncorrelated variables that are observed with unlimited precision, then you still would not reject the null that they are uncorelated
  - BenoitEssiambre 6y ago
    "Truly uncorrelated" is a problematic concept. Most of the time, any two things within each other's light cones, have at least tiny direct or indirect effects on each other. These effects are usually not that tiny in experimental settings because of imperfect experimental tools and procedures.
    - gus_massa 6y ago
      Like the speed of sound and the height of the table where the experiment is. Note that the pressure of the air changes with height. For a few inches the change is negligible and very difficult (impossible?) to measure, but the effect must be out there. I had once a problem with the temperature of the room, you usually ignore it, but it has a bigger effect like a 5% variation.

dguest 6y ago

You can't ever prove that switching to X doesn't change the result. All you can really hope for is putting smaller limits on the amount by which something could be improved. For one dumb example: if you wanted to see if rats can fly, you might observe 100 rats and see that none of them fly away. But this doesn't prove anything: in reality even if 3% of rats could fly you'd get this result about 5% of the time. You've really shown with 95% confidence is that less than 3% of rats fly.

collyw 6y ago

It should stop other groups wasting their time on the same research, no?