14 ms·

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "signifi

by jerf 5y ago

Speaking not to this study in particular necessarily, I strongly agree with the general point. Science has really been held back by an over-focusing on "significance". But I'm not really interested in a pile of hundreds of thousands of studies that establish a tiny effect with suspiciously-just-barely-significant results. I'm interested in studies that reveal robust results that are reliable enough to be built on to produce other results. Results of 3% variations with p=0.046 aren't. They're dead ends, because you can't put very many of those into the foundations of future papers before the probability of one of your foundations being incorrect is too large. To the extent that those are hard to come by... Yeah! They are! Science is hard. Nobody promised this would be easy. Science shouldn't be something where labs are cranking out easy 3%/p=0.046 papers all the time just to keep funding. It's just a waste of money and time of our smartest people. It should be harder than it is now. Too many proposals are obviously only going to be capable of turning up that result (insufficient statistical power is often obvious right in the proposal, if you take the time to work the math). I'd rather see more wood behind fewer arrows, and see fewer proposals chasing much more statistical power, than the chaff of garbage we get now. If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. "But jerf, isn't it still valuable to map out all the little things like that?" No, it really isn't. We already have every reason in the world to believe the world is drenched in 1%/p=0.05 effects. "Everything's correlated to everything", so that's not some sort of amazing find, it's the totally expected output of living in our reality. Really, this sort of stuff is still just below the noise floor. Plus, the idea that we can remove such small, noisy confounding factors is just silly. We need to look for the things that stand out from that noise floor, not spending billions of dollars doing the equivalent of listening to our spirit guides communicate to us over white noise from the radio.

bluGill 5y ago
> Plus, the idea that we can remove such small, noisy confounding factors is just silly. We need to look for the things that stand out from that noise floor We have found most of them, and all the easy ones. Today the interesting things are near the noise floor. 3000 years ago atoms were well below the noise floor, now we know a lot about them - most of it seems useless in daily life yet a large part of the things we use daily depend on our knowledge of the atom. Science needs to keep separating things from the noise floor. Some of them become important once we understand it.
- exporectomy 5y ago
  Doesn't it make a difference if it's near the noise floor because it's hard to measure (atoms) or if it's near the noise floor because it's hardly there (masks)? Maybe if these "hardly there" results led to further research that isolated some underlying "very there" phenomena, they would be important, but until that happens, who cares if thinking about money makes you slightly less generous than thinking about flowers? If they're not building on previous research to discover more and more important things, then it doesn't seem like useful progress.
- jerf 5y ago
  I don't think we have found most of them. I think we make it look like we've found most of them because we keep throwing money at these crap studies. Bear in mind that my criteria are two-dimensional, and I'll accept either. By all means, go back and establish your 3% effect to a p-value of 0.0001. Or 0.000000001. That makes that 3% much more interesting and useful. It'll especially be interesting and valuable when you fail to do so. But we do not, generally, do that. We just keep piling up small effects with small p-values and thinking we're getting somewhere. Further, if there is a branch of some "science" that we've exhaused so thoroughly that we can't find anything that isn't a 3%/p=0.047 effect anymore... pack it in, we're done here. Move on. However, part of the reason I so blithely say that is that I suspect if we did in fact raise the standards as I propose here, it would realign incentives such that more sciences would start finding more useful results. I suspect, for instance, that a great deal of the soft sciences probably could find some much more significant results if they studied larger groups of people. Or spent more time creating theories that aren't about whether priming people with some sensitive word makes them 3% more racist for the next twelve minutes, or some other thing that even if true really isn't that interesting or useful as a building block for future work.
  - EvgeniyZh 5y ago
    So 3% is not interesting but the difference between 10^-7 and 10^-8 probability that there is no effect is interesting somehow?
    - 5y ago
tommiegannert 5y ago
A few years ago, HN comments complained about the censorship that only leaves successful studies. We need to report on everything we've tried, so we don't walk around on donuts. What's missing in my mind is admitting that results were negative. I'm reading up on financial literacy, and many studies end with some metrics being "great" at p 5%, but then some other metrics are also "great" at p 10%, without the author ever explaining what they would have classified as bad. They're just reported without explanation of what significance they would expect (in their field).
- nix0n 5y ago
  > ...so we don't walk around on donuts I agree with what you're saying, but I don't understand this phrase.
  - Imnimo 5y ago
    The phrase "walk around on donuts" has one Google result and it's this thread.
  - potatoman22 5y ago
    You know how sometimes you'll accidentally step on a donut and you'll have to call your dog over to lick all the jelly off your toes? That.
  - rootusrootus 5y ago
    I don't know where that turn of phrase comes from, but I imagine it's synonymous with 'walking around in circles'.
shakezula 5y ago
I blame most of this on pop science. It's absolutely ruined the average public's respect for the behind the scenes work doing interesting stuff in every field. What's worse is the attitude it breeds. Anti-intellectualism runs rampant amongst even well educated members of my social circle. It's frustrating to say the least.
- shawn-butler 5y ago
  Some say that it is not anti-intellectualism to realize the emperor has no clothes but enlightenment. Either way it’s dangerous.
  - shakezula 5y ago
    It can be both, but you're absolutely right.
- BeFlatXIII 5y ago
  "Believe the science" vs. "understand the process". The former merely uses the language of science to gain legitimacy.
phreeza 5y ago
This is clearly a cost/benefit tradeoff, and the sweet spot will depend entirely on the field. If you are studying the behavior of heads of state, getting an additional N is extremely costly, and having a p=0.05 study is maybe more valuable than having no published study at all, because the stakes are very high and even a 1% chance of (for example) preventing nuclear war is worth a lot. On the other hand, if you are studying fruit flies, an additional N may be much cheaper, and the benefit of yet another low effect size study may be small, so I could see a good argument being made for more stringent standards. In fact I know that in particle physics the bar for discovery is much higher than p=0.05.
- bongoman37 5y ago
  What if it's the other way round and a p<0.05 study says that the best way to make sure a rival country does not do a nuclear strike on you first is to do a massive nuclear strike on them first?
Robotbeat 5y ago
The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. But I definitely agree it’d be nice to go back and show something is true to p=.0001 or whatever. Overwhelmingly solid evidence is truly a wonderful thing, and as you say, it’s really the only way to build a solid foundation. When you engineer stuff, it needs to work 99.99-99.999% of the time or more. Otherwise you’re severely limited to how far your machine can go (in terms of complexity, levels of abstraction and organization) before it spends most of its time in a broken state. I’ve been thinking about this while playing Factorio: so much of our discussion and mental modeling of automation works under the assumption of perfect reliability. If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. Likewise with mathematical proofs or computer transistors or self driving cars or any other kind of automation. The reliability needs to be insanely good. You need to add a bunch of nines to whatever you’re making. A counterpoint to this is when you’re in an emergency and inaction means people die. In that case, you need to accept some uncertainty early on.
- twoslide 5y ago
  > when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. That's not necessarily true in social sciences. When you're working with large survey datasets, many variables are significantly related. That doesn't mean these relationships are meaningful or causal, they could be due to underlying common causes, etc. (Maybe social sciences weren't included in "real science" - but there's where a lot of stats discussions focus)
- mumblemumble 5y ago
  Fine. Do it like the experimental physicists do: if you think you're on to something, refine and repeat the experiment in order to get a more robust, repeatable result. The original sin of the medical and social sciences is failing to recognize a distinction between exploratory research and confirmatory research and behave accordingly.
  - deleted 5y ago
modeless 5y ago
Not only is it not valuable to publish tons of studies with p=.04999 and small effect size, in fact it's harmful. With so many questionable results published in supposedly reputable places it becomes possible to "prove" all sorts of crackpot theories by selectively citing real research. And if you try to dispute the studies you can get accused of being anti-science.
- exporectomy 5y ago
  Only a problem for people who are trying hard not to think. You can just ignore those people. They're not doing any harm believing their beliefs.
  - robbedpeter 5y ago
    The USDA food pyramid and nutrition education would suggest that there's an inherent danger in just letting people believe irrational things after a correction is known. It depends on the belief - flat earth people aren't likely to cause any harm. Bad nutrition information can wreak havoc at scale.
    - vkou 5y ago
      Flat earth beliefs doesn't cause harm, but flat earth believers have largely upgraded to believing more dangerous nonsense.
      - exporectomy 5y ago
        Data or it didn't happen. This really sounds like you're inventing a caricature of your enemy and assigning them "dangerous" qualities so you can hate them more.
        vkou 5y ago
        Nobody needs to caricature the insane beliefs surrounding COVID (or flat earth), people holding them are doing a good enough job of that themselves. I do have a few favorites. "COVID tests give you COVID, so I won't go get tested" is certainly up there. I can't say I give two figs about your opinion on the Earth's topology, but this one is a public health problem, that's crippling hospitals around the country.
sanxiyn 5y ago
I agree we shouldn't listen to noise, but small effect size is not necessarily noise. (I agree it is highly correlated.) I mean, QED's effect size on g factor is 1.001. QED was very much worth finding out.
hyperbovine 5y ago
Come into Bayesian land, the water is fine. The whole NHST edifice starts to seem really shaky once you stop and wonder if "True" and "False" are really the only two possible states of a scientific hypothesis. Andrew Gelman has written about this in many places, e.g. http://www.stat.columbia.edu/~gelman/research/published/abandon.pdf http://www.stat.columbia.edu/~gelman/research/published/aban....
- Retric 5y ago
  Bayesian reasoning has even worse underpinnings. You don’t actually know any of the things the equations want. For example suppose a robot is counting Red and Blue balls from a bin, the count is 400Red and 637Blue, it just classified a Red ball. Now what’s the count, wait what’s the likelihood it misclassified a ball? How accurate are those estimates, and those estimates of those ... For a real world example someone using Bayesian reasoning when counting cards should consider the possibility that the deck doesn’t have the correct cards. And the possibility that the decks cards have been changed over the course of the game.
  - kadoban 5y ago
    Can't you just add that to your equation? Seems like for anything real, this will not go many levels deep at all before it's irrelevent.
  - Karrot_Kream 5y ago
    Huh? You can derive all of those from Bayesian models. If you're counting balls from a bin with replacement, and your bot has counted 400Red with 637Blue, you have a Beta/Binomial model. That means you p_blue | data ~ Beta(401, 638) assuming a Uniform prior. The probability of observing a red ball given the above p_blue | data is P(red_obs | p_blue) = 1 - P(blue_obs | p_blue), which is calculable from p_blue | data. In fact in this simple example you can even analytically derive all of these values, so you don't even need a simulation!
    - tfehring 5y ago
      And if misclassification is a concern (as the parent mentioned) you can put a prior on that rate too!
naasking 5y ago
> If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. And study preregistration to avoid p-hacking and incentivize publishing negative results. And full availability of data, aka "open science".
- DiabloD3 5y ago
  Preregistration, requirement to publish negative or null results, and full data is, arguably, the three legs of modern science. If we collectively don't enforce this, nobody is doing science, they're just fucking around and writing it down.
  - romwell 5y ago
    Also replication studies for negative or null results in addition to positive ones (we don't have either).
  - candiodari 5y ago
    You do realize there is a million negative results for every one positive result? This is equally easy to game, maybe easier.
    - naasking 5y ago
      Yes, and knowing what's been tried and what has failed is important.
      - TOMDM 5y ago
        I think what's being pointed out is that "researchers" could pump out hundreds of easy to test negatives every day if a negative result was just as incentivised. I do agree though, negatives are just as important when the intent is to prove/disprove a meaningful hypothosis.
BenoitEssiambre 5y ago
p = 0.0001 doesn't help much. You can get to an arbitrarily small p by just using more data. The problem is trying to reject a zero width null hypothesis. Scientists should always reject something bigger than infinitesimally small so that they are not catching tiny systematic biases in their experiments. There are always small biases. Gwern's page "Everything Is Correlated" is worth reading: https://www.gwern.net/Everything https://www.gwern.net/Everything
- bpodgursky 5y ago
  It would at least filter out the social science experiments where results on 30 college students is "significant" at p=.04 (and it's too expensive to recruit 3000 of them to force significance).
Tycho 5y ago
What do you (or anyone else) think about the statistical conclusions in this paper? Particularly the adjusted r-squared values reported. https://www.cambridge.org/core/journals/american-political-science-review/article/tabloid-media-campaigns-and-public-opinion-quasiexperimental-evidence-on-euroscepticism-in-england/F530F8AB25994AD7C4BC1D0CAFAD75CF https://www.cambridge.org/core/journals/american-political-s...
rscho 5y ago
If you were the king of science, I'd kindly ask you to think about replacing grant financing and all other financial incentives that go along with publishing. Now that would be efficient. 'Cause I currently make .05-barely-significant-results but if you force me to up my game I will provide .0001-barely-significant-results no problem, even with 'preregistration' or whatever hoop you hold in front of me. As an aside, could you also please make medicine a real science, so I can finally scientifically demonstrate that my boss is wrong?
chuckcode 5y ago
Don't get distracted by the click bait title. Effect size should be captured by statistical significance (larger effects are less likely to happen by chance). Author is really complaining that the original study didn't report enough data to check their analysis or do alternative analysis methods. Better title for article would be "Hard to peer review when you don't share the data"
- jrk 5y ago
  Note the point in the essay that statistical significance is meaningless if the model does not correspond to reality — which, in this case as in many, they very much do not.
setgree 5y ago
> or a p value of something like p = 0.0001 This has been proposed [0], albeit for a threshold of p < 0.005. Here's Andy Gelman and others arguing otherwise [1]. They also got like 800 scientists to sign on to the general idea of no longer using statistical significance at all [2]. [0] https://www.nature.com/articles/s41562-017-0189-z https://www.nature.com/articles/s41562-017-0189-z [1] http://www.stat.columbia.edu/~gelman/research/unpublished/abandon.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/ab... [2] https://www.nature.com/articles/d41586-019-00857-9 https://www.nature.com/articles/d41586-019-00857-9
- LeonB 5y ago
  Given the (estimated) number of scientists in the world and their general propensity to sign on to something… is 800 scientists a significant amount?
  - setgree 5y ago
    I don't know, and it's a fair point; I think I should have just summarised the follow-up article as "here's some follow-up by the authors for context"
kuhewa 5y ago
Nothing is wrong with publishing small effect size results. Setting a P threshold lower or a a higher bar for effect sizes for journal acceptance will just increase the positivity bias and also encourage more dodgy practices. Null results are important. Understanding effect size is as important as significance can manifest by requiring effect size or variance explained to be reported every time the result of a statistical test is presented, e.g. rather than simply "a significant increase was observed (p = 0.01)" and also making that kind of parsing the standard in scientific journalism.
raxxorrax 5y ago
The current science economy around publishing is partially responsible, although it should also be said that finding no correlation is still a gain of knowledge that is valuable to build upon for people in the same field, even if it might not generate the most exciting read for others.
davidmanheim 5y ago
Maybe all studies should be preregistered, including their methods... like this one was? https://osf.io/vzdh6/ https://osf.io/vzdh6/

by jerf 5y ago

bluGill 5y ago

> Plus, the idea that we can remove such small, noisy confounding factors is just silly. We need to look for the things that stand out from that noise floor We have found most of them, and all the easy ones. Today the interesting things are near the noise floor. 3000 years ago atoms were well below the noise floor, now we know a lot about them - most of it seems useless in daily life yet a large part of the things we use daily depend on our knowledge of the atom. Science needs to keep separating things from the noise floor. Some of them become important once we understand it.

exporectomy 5y ago
Doesn't it make a difference if it's near the noise floor because it's hard to measure (atoms) or if it's near the noise floor because it's hardly there (masks)? Maybe if these "hardly there" results led to further research that isolated some underlying "very there" phenomena, they would be important, but until that happens, who cares if thinking about money makes you slightly less generous than thinking about flowers? If they're not building on previous research to discover more and more important things, then it doesn't seem like useful progress.
jerf 5y ago
I don't think we have found most of them. I think we make it look like we've found most of them because we keep throwing money at these crap studies. Bear in mind that my criteria are two-dimensional, and I'll accept either. By all means, go back and establish your 3% effect to a p-value of 0.0001. Or 0.000000001. That makes that 3% much more interesting and useful. It'll especially be interesting and valuable when you fail to do so. But we do not, generally, do that. We just keep piling up small effects with small p-values and thinking we're getting somewhere. Further, if there is a branch of some "science" that we've exhaused so thoroughly that we can't find anything that isn't a 3%/p=0.047 effect anymore... pack it in, we're done here. Move on. However, part of the reason I so blithely say that is that I suspect if we did in fact raise the standards as I propose here, it would realign incentives such that more sciences would start finding more useful results. I suspect, for instance, that a great deal of the soft sciences probably could find some much more significant results if they studied larger groups of people. Or spent more time creating theories that aren't about whether priming people with some sensitive word makes them 3% more racist for the next twelve minutes, or some other thing that even if true really isn't that interesting or useful as a building block for future work.
- EvgeniyZh 5y ago
  So 3% is not interesting but the difference between 10^-7 and 10^-8 probability that there is no effect is interesting somehow?
  - 5y ago

tommiegannert 5y ago

A few years ago, HN comments complained about the censorship that only leaves successful studies. We need to report on everything we've tried, so we don't walk around on donuts. What's missing in my mind is admitting that results were negative. I'm reading up on financial literacy, and many studies end with some metrics being "great" at p 5%, but then some other metrics are also "great" at p 10%, without the author ever explaining what they would have classified as bad. They're just reported without explanation of what significance they would expect (in their field).

nix0n 5y ago
> ...so we don't walk around on donuts I agree with what you're saying, but I don't understand this phrase.
- Imnimo 5y ago
  The phrase "walk around on donuts" has one Google result and it's this thread.
- potatoman22 5y ago
  You know how sometimes you'll accidentally step on a donut and you'll have to call your dog over to lick all the jelly off your toes? That.
- rootusrootus 5y ago
  I don't know where that turn of phrase comes from, but I imagine it's synonymous with 'walking around in circles'.

shakezula 5y ago

I blame most of this on pop science. It's absolutely ruined the average public's respect for the behind the scenes work doing interesting stuff in every field. What's worse is the attitude it breeds. Anti-intellectualism runs rampant amongst even well educated members of my social circle. It's frustrating to say the least.

shawn-butler 5y ago
Some say that it is not anti-intellectualism to realize the emperor has no clothes but enlightenment. Either way it’s dangerous.
- shakezula 5y ago
  It can be both, but you're absolutely right.
BeFlatXIII 5y ago
"Believe the science" vs. "understand the process". The former merely uses the language of science to gain legitimacy.

phreeza 5y ago

This is clearly a cost/benefit tradeoff, and the sweet spot will depend entirely on the field. If you are studying the behavior of heads of state, getting an additional N is extremely costly, and having a p=0.05 study is maybe more valuable than having no published study at all, because the stakes are very high and even a 1% chance of (for example) preventing nuclear war is worth a lot. On the other hand, if you are studying fruit flies, an additional N may be much cheaper, and the benefit of yet another low effect size study may be small, so I could see a good argument being made for more stringent standards. In fact I know that in particle physics the bar for discovery is much higher than p=0.05.

bongoman37 5y ago
What if it's the other way round and a p<0.05 study says that the best way to make sure a rival country does not do a nuclear strike on you first is to do a massive nuclear strike on them first?

Robotbeat 5y ago

The problem is that when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. But I definitely agree it’d be nice to go back and show something is true to p=.0001 or whatever. Overwhelmingly solid evidence is truly a wonderful thing, and as you say, it’s really the only way to build a solid foundation. When you engineer stuff, it needs to work 99.99-99.999% of the time or more. Otherwise you’re severely limited to how far your machine can go (in terms of complexity, levels of abstraction and organization) before it spends most of its time in a broken state. I’ve been thinking about this while playing Factorio: so much of our discussion and mental modeling of automation works under the assumption of perfect reliability. If you had SLIGHTLY below 100% reliability in Factorio, the game would be a terrible grind limited to small factories. Likewise with mathematical proofs or computer transistors or self driving cars or any other kind of automation. The reliability needs to be insanely good. You need to add a bunch of nines to whatever you’re making. A counterpoint to this is when you’re in an emergency and inaction means people die. In that case, you need to accept some uncertainty early on.

twoslide 5y ago
> when you’re on the cusp of a new thing, unless you’re super lucky, the result will necessarily be near the noise floor. Real science is like that. That's not necessarily true in social sciences. When you're working with large survey datasets, many variables are significantly related. That doesn't mean these relationships are meaningful or causal, they could be due to underlying common causes, etc. (Maybe social sciences weren't included in "real science" - but there's where a lot of stats discussions focus)
mumblemumble 5y ago
Fine. Do it like the experimental physicists do: if you think you're on to something, refine and repeat the experiment in order to get a more robust, repeatable result. The original sin of the medical and social sciences is failing to recognize a distinction between exploratory research and confirmatory research and behave accordingly.
- deleted 5y ago

modeless 5y ago

Not only is it not valuable to publish tons of studies with p=.04999 and small effect size, in fact it's harmful. With so many questionable results published in supposedly reputable places it becomes possible to "prove" all sorts of crackpot theories by selectively citing real research. And if you try to dispute the studies you can get accused of being anti-science.

exporectomy 5y ago
Only a problem for people who are trying hard not to think. You can just ignore those people. They're not doing any harm believing their beliefs.
- robbedpeter 5y ago
  The USDA food pyramid and nutrition education would suggest that there's an inherent danger in just letting people believe irrational things after a correction is known. It depends on the belief - flat earth people aren't likely to cause any harm. Bad nutrition information can wreak havoc at scale.
  - vkou 5y ago
    Flat earth beliefs doesn't cause harm, but flat earth believers have largely upgraded to believing more dangerous nonsense.
    - exporectomy 5y ago
      Data or it didn't happen. This really sounds like you're inventing a caricature of your enemy and assigning them "dangerous" qualities so you can hate them more.
      - vkou 5y ago
        Nobody needs to caricature the insane beliefs surrounding COVID (or flat earth), people holding them are doing a good enough job of that themselves. I do have a few favorites. "COVID tests give you COVID, so I won't go get tested" is certainly up there. I can't say I give two figs about your opinion on the Earth's topology, but this one is a public health problem, that's crippling hospitals around the country.

sanxiyn 5y ago

I agree we shouldn't listen to noise, but small effect size is not necessarily noise. (I agree it is highly correlated.) I mean, QED's effect size on g factor is 1.001. QED was very much worth finding out.

hyperbovine 5y ago

Come into Bayesian land, the water is fine. The whole NHST edifice starts to seem really shaky once you stop and wonder if "True" and "False" are really the only two possible states of a scientific hypothesis. Andrew Gelman has written about this in many places, e.g. http://www.stat.columbia.edu/~gelman/research/published/abandon.pdf http://www.stat.columbia.edu/~gelman/research/published/aban....

Retric 5y ago
Bayesian reasoning has even worse underpinnings. You don’t actually know any of the things the equations want. For example suppose a robot is counting Red and Blue balls from a bin, the count is 400Red and 637Blue, it just classified a Red ball. Now what’s the count, wait what’s the likelihood it misclassified a ball? How accurate are those estimates, and those estimates of those ... For a real world example someone using Bayesian reasoning when counting cards should consider the possibility that the deck doesn’t have the correct cards. And the possibility that the decks cards have been changed over the course of the game.
- kadoban 5y ago
  Can't you just add that to your equation? Seems like for anything real, this will not go many levels deep at all before it's irrelevent.
- Karrot_Kream 5y ago
  Huh? You can derive all of those from Bayesian models. If you're counting balls from a bin with replacement, and your bot has counted 400Red with 637Blue, you have a Beta/Binomial model. That means you p_blue | data ~ Beta(401, 638) assuming a Uniform prior. The probability of observing a red ball given the above p_blue | data is P(red_obs | p_blue) = 1 - P(blue_obs | p_blue), which is calculable from p_blue | data. In fact in this simple example you can even analytically derive all of these values, so you don't even need a simulation!
  - tfehring 5y ago
    And if misclassification is a concern (as the parent mentioned) you can put a prior on that rate too!

naasking 5y ago

> If I were King of Science, or at least, editor of a prestigious journal, I'd want to put word out that I'm looking for papers with at least one of some sort of significant effect, or a p value of something like p = 0.0001. Yeah. That's a high bar. I know. That's the point. And study preregistration to avoid p-hacking and incentivize publishing negative results. And full availability of data, aka "open science".

DiabloD3 5y ago
Preregistration, requirement to publish negative or null results, and full data is, arguably, the three legs of modern science. If we collectively don't enforce this, nobody is doing science, they're just fucking around and writing it down.
- romwell 5y ago
  Also replication studies for negative or null results in addition to positive ones (we don't have either).
- candiodari 5y ago
  You do realize there is a million negative results for every one positive result? This is equally easy to game, maybe easier.
  - naasking 5y ago
    Yes, and knowing what's been tried and what has failed is important.
    - TOMDM 5y ago
      I think what's being pointed out is that "researchers" could pump out hundreds of easy to test negatives every day if a negative result was just as incentivised. I do agree though, negatives are just as important when the intent is to prove/disprove a meaningful hypothosis.

BenoitEssiambre 5y ago

p = 0.0001 doesn't help much. You can get to an arbitrarily small p by just using more data. The problem is trying to reject a zero width null hypothesis. Scientists should always reject something bigger than infinitesimally small so that they are not catching tiny systematic biases in their experiments. There are always small biases. Gwern's page "Everything Is Correlated" is worth reading: https://www.gwern.net/Everything https://www.gwern.net/Everything

bpodgursky 5y ago
It would at least filter out the social science experiments where results on 30 college students is "significant" at p=.04 (and it's too expensive to recruit 3000 of them to force significance).

Tycho 5y ago

What do you (or anyone else) think about the statistical conclusions in this paper? Particularly the adjusted r-squared values reported. https://www.cambridge.org/core/journals/american-political-science-review/article/tabloid-media-campaigns-and-public-opinion-quasiexperimental-evidence-on-euroscepticism-in-england/F530F8AB25994AD7C4BC1D0CAFAD75CF https://www.cambridge.org/core/journals/american-political-s...

rscho 5y ago

If you were the king of science, I'd kindly ask you to think about replacing grant financing and all other financial incentives that go along with publishing. Now that would be efficient. 'Cause I currently make .05-barely-significant-results but if you force me to up my game I will provide .0001-barely-significant-results no problem, even with 'preregistration' or whatever hoop you hold in front of me. As an aside, could you also please make medicine a real science, so I can finally scientifically demonstrate that my boss is wrong?

chuckcode 5y ago

Don't get distracted by the click bait title. Effect size should be captured by statistical significance (larger effects are less likely to happen by chance). Author is really complaining that the original study didn't report enough data to check their analysis or do alternative analysis methods. Better title for article would be "Hard to peer review when you don't share the data"

jrk 5y ago
Note the point in the essay that statistical significance is meaningless if the model does not correspond to reality — which, in this case as in many, they very much do not.

setgree 5y ago

> or a p value of something like p = 0.0001 This has been proposed [0], albeit for a threshold of p < 0.005. Here's Andy Gelman and others arguing otherwise [1]. They also got like 800 scientists to sign on to the general idea of no longer using statistical significance at all [2]. [0] https://www.nature.com/articles/s41562-017-0189-z https://www.nature.com/articles/s41562-017-0189-z [1] http://www.stat.columbia.edu/~gelman/research/unpublished/abandon.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/ab... [2] https://www.nature.com/articles/d41586-019-00857-9 https://www.nature.com/articles/d41586-019-00857-9

LeonB 5y ago
Given the (estimated) number of scientists in the world and their general propensity to sign on to something… is 800 scientists a significant amount?
- setgree 5y ago
  I don't know, and it's a fair point; I think I should have just summarised the follow-up article as "here's some follow-up by the authors for context"

kuhewa 5y ago

Nothing is wrong with publishing small effect size results. Setting a P threshold lower or a a higher bar for effect sizes for journal acceptance will just increase the positivity bias and also encourage more dodgy practices. Null results are important. Understanding effect size is as important as significance can manifest by requiring effect size or variance explained to be reported every time the result of a statistical test is presented, e.g. rather than simply "a significant increase was observed (p = 0.01)" and also making that kind of parsing the standard in scientific journalism.

raxxorrax 5y ago

The current science economy around publishing is partially responsible, although it should also be said that finding no correlation is still a gain of knowledge that is valuable to build upon for people in the same field, even if it might not generate the most exciting read for others.

davidmanheim 5y ago

Maybe all studies should be preregistered, including their methods... like this one was? https://osf.io/vzdh6/ https://osf.io/vzdh6/