9 ms·

It will be much harder to call findings ‘significant’ if a team gets its way

128 points by nonbel 9y ago

nonbel 9y ago
Imagine if p <= 1.0 was considered "significant", wouldn't this cut down on publication bias? Shouldn't every study worth funding be worth publishing? They should all be considered significant results. So I think this proposal is: 1) Not really addressing the issues behind the replication crisis 2) Doing the exact opposite of what should be done (raise the significance level)
- jimrandomh 9y ago
  "Significance" is a social concept currently serving two roles: for determining what is considered publishable, and for determining what is considered true. This is meant to modify the second one, not the first. (Publishability should depend on quality of methodology, and be determined at the pre-registration phase before results are in.)
  - nonbel 9y ago
    First, I agree that publishability should be independent of the results. Do you know anyone who disagrees with this though? That seems to be the standard opinion, yet the situation of filtering by p-value persists. Also, in the usual case where the null hypothesis is "no difference", no one really cares about what the p-value is telling them anyway. They care "is my theory accurate/true", when their theory predicts something other than the null hypothesis. I figure we may as well set the significance level in these cases so that everyone "wins", and go on to use analytic techniques that actually address the question of interest.
- tpeo 9y ago
  What's the point of fitting arbitrary functions to noise? Because that's what high p-values would lead to. Undue emphasis on p-values is bad, but what you're suggesting is bonkers.
  - nonbel 9y ago
    Stop funding people to fit arbitrary functions to noise?
    - tpeo 9y ago
      Don't be disingenuous. That would require someone to actually read their papers, which would already be a net loss and a massive waste of time for everyone involved. Not to mention the amount of department politics which could be involved in this issue. Look, not every paper is worth publishing. There are many ways to write noise that can pass for signal given the reader's level of understanding (e.g. crack pottery and "fashionable nonsense"). The usage of p-values implements self-filtering to some degree, and makes everyone better off by increasing the likelihood that at least some papers aren't complete nonsense. And that's good. The only issue is that it is a much weaker constraint on publishing than it was previously thought.
      - JoshTriplett 9y ago
        > Look, not every paper is worth publishing. Many more results should be documented than currently are, though, including negative results. But I agree that changing p-value thresholds is unrelated to that.
        tpeo 9y ago
        I agree, publication bias is a thing. There's no questioning that.
      - Houshalter 9y ago
        And not publishing wastes everyone's time. When researches perform the same experiment again and again until someone gets a significant result to publish. There are entire fields of research supported by nothing but publication bias (e.g. Parapsychology and possibly most soft sciences.)
leeoniya 9y ago
sounds expensive. 0.01 may be more reasonable.
- mrob 9y ago
  Choosing the correct value is a social problem. Eg. The Journal of Wacky Speculation publishes "The Moon is Made of Green Cheese". Using a highly sophisticated cheese detector, the authors measure results consistent with this hypothesis. They calculate it would happen only 4% of the time if the moon were not in fact made of green cheese, and therefore the result is significant. The cheese detector works perfectly, the calculations were accurate, but this does not mean the moon is made of green cheese! We've been there, and we're very certain that it's made of rock. Because the prior probability of the hypothesis was so low, the "significant" result was most likely a fluke. The Journal of Wacky Speculation specializes in this type of hypothesis, so they should use a much lower threshold for p. Meanwhile, the Sensible and Boring Society publishes a study showing that round wheels roll better on ordinary roads than square wheels. We already had good reason to believe this, so p<0.05 shows it's most likely true. The Sensible and Boring Society publishes tests of hypotheses with higher prior probably, so they can use a higher p threshold.
shmageggy 9y ago
Totally agree with Timothy Bates here: > [Bates] called the proposal “a risky distraction”... [It] wouldn’t address many other practices linked to irreproducible results: poor study design, a bias toward publishing positive results, and the practice of “p-value hacking”—fishing for significant-looking results from a huge number of hypotheses. Since the reproducibilty crisis started getting attention, I've noticed that these problems are really ubiquitous. Tweaking study designs and modifying hypotheses after collecting data are very commonplace and much more subtly dangerous than a potential lack of statistical power. Gelman says "Valid p-values cannot be drawn without knowing, not just what was done with the existing data, but what the choices in data coding, exclusion,and analysis would have been, had the data been different." [1] This proposal also says nothing about effect sizes, which also should play a role in how we interpret scientific results. We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05. [1]http://www.stat.columbia.edu/~gelman/research/published/asa_pvalues.pdf http://www.stat.columbia.edu/~gelman/research/published/asa_...
- LoSboccacc 9y ago
  P-values, reproducibility and the focus on statistics before understanding is, of itself, a problem: https://xkcd.com/882/ https://xkcd.com/882/
- petters 9y ago
  > We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05 I'm not sure that "large, obvious effect" and "p=0.05" are compatible, though.
  - shmageggy 9y ago
    Sure it is, of course depending on your domain-dependent definition of "large, obvious". You can have different effect sizes at any given p-value.
- mabbo 9y ago
  I disagree. > a bias toward publishing positive results ... fishing for significant-looking results from a huge number of hypotheses If we accept a lower p-value as significant, there will be fewer hypotheses that have positive results, and therefore less to publish. Should we really do nothing at all until we can come up with a perfect solution to all aspects of the problem? This won't fix the real problem, but it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week.
elsherbini 9y ago
There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't change the effect size, so this would lead to larger necessary samples sizes to study smaller effects. I think in general though, at least in biology, people are waking up to the fact that p-values aren't magical, and that having a really small p-value isn't a goal in and of itself. It is, however, necessary to do some statistics on your data to get it published, but the p-value is just checking a box more than being used as a tool for discovery. Yesterday a cool dataset was released from Jeff Leek which has over 3.6 million p-values from scientific literature[1]. The distribution is fun to look at by discipline[2]. [0] http://rpsychologist.com/d3/NHST/ http://rpsychologist.com/d3/NHST/ [1] https://github.com/jtleek/tidypvals https://github.com/jtleek/tidypvals [2] https://twitter.com/drob/status/890260541876338690 https://twitter.com/drob/status/890260541876338690
- nonbel 9y ago
  Thanks, direct link to the pic: https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg And here it is zoomed in: https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg These figures look very damning to p-values. This basically looks like they are made up anyway. I haven't checked the methods though so who knows. Edit: Well there are 33 "pvalues" > 1 and 20820 that equal zero, so this dataset probably could use more cleaning.
  - Mathnerd314 9y ago
    Most of them are of the form "p > 0.05" or p < 0.01", hence the peaks. You really need to filter down to the red "equals" samples, but apparently they're pretty uncommon in papers to begin with. The only real info from that graph seems to be that geography/business/economics/psychology/sociology allow "p > 0.10" relatively more often, and that there's no data for the economics category alone (from Twitter, apparently the data source is not as good). This is a better pic: https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large Can't really see any difference in fields, besides some artifact in plant biology. Which makes sense since it's just a statistical test.
pcrh 9y ago
The term "significant" is itself part of the problem. In statistics it is often used to describe an arbitrary threshold where a difference between datasets is assumed to be real. In common parlance this term implies "important" or "meaningful", but the statistical tests are not designed to draw such forward-looking conclusions. It would be far easier, and more honest, to simply report the P-value obtained by performing a statistical test without adding the term "significant".
- jaggederest 9y ago
  I'd be interested in a compound metric composed of the likelihood ratio and the p-value. Essentially the more likely your hypothesis, the higher your P value can be, and conversely unlikely hypotheses require stronger evidence. I don't think this solves the problem but maybe it makes that assumption about the hypothesis likelihood more specific.
Tloewald 9y ago
I think that a very stringent threshold for p values (it's going to be a lot harder to get to p < 0.005 than p < 0.05 and it's going to encourage a lot more gaming of the system) is not the solution. At best, it will lead to smaller numbers of much bigger studies with more authors and fewer interesting results. I'd suggest that a better, broader, and more practical solution is to provide a public clearing house for studies and associated data -- let's assume it's a website. a) When you start a study you must describe your study, methodology, and hypothesis, and publish these things on the site. It should be possible to find the study the same way you find any paper that results from it. b) All data must be stored in the same place (but not made public) as it becomes available, and this must be demonstrable. This data should be available to anyone trying to replicate, review, etc. c) Final data sets and analyses must also be also be available, ideally with the code used to do the analyses. All of this should be prerequisite for review and publication, but also for anyone working in the field should no papers result. This solves a lot of problems, including making it easier to conduct meta-analyses when looking for effects that aren't significant in smaller studies, and also address the lack of availability of negative results, and even help combat theft of methodology (e.g. where someone sees work in progress and is able to publish using the methodology). It would make meta-analysis easier, and allow researchers to better take advantage of existing, unpublished results (e.g. to hone their study methods, etc.)
- nerfhammer 9y ago
  Register the hypothesis first, results later. If you get null results, you have to report that.
  - Tloewald 9y ago
    Right. This is a direct consequence of my proposal.
- brownbat 9y ago
  If we were designing things from scratch, I'd even want t odisaggregate two career tracks for scientists: The first, theorists, design interesting studies with open ended questions, explain how the study would provide useful scientific results, or theorize about the implications of existing published results. The second, practitioners, are hired to actually run the studies according to the specifications provided. The practitioners would publish null results of course. Practitioners would also have to acknowledge how many of their results are significant, so it'd be transparent if they were just confirming everything. A bit like barristers and solicitors in many common law countries, a distinction that isn't strictly necessary for the system to work, but prevents some conflicts of interest.
SurrealSoul 9y ago
>You won't believe what this team did to the word significant! >10 significant things that WILL shock you, #7 is this team >THIS crazy trick that will make significant much harder [teams hate him!!] >This finding prank gone significant [gone sexual!?] Sorry, it was such a click-baity title for no reason.
currymj 9y ago
if i were the King of All Science and were feeling despotic, I would do the opposite, I think -- put a moratorium for a few years on any publication of a p-value. Scientists would still be free to make use of hypothesis testing to guide their research. In publications, one could use Bayesian methods and report a full posterior. One could also just give enough information about the null hypothesis and whatever distributions are involved to allow the computation of a p-value. However, any attempt to explicitly publish the p-value itself would result in creative and embarrassing punishments.
- BeetleB 9y ago
  >In publications, one could use Bayesian methods and report a full posterior. I fail to see how this is superior. I'd never advocate banning a legitimate tool just because of abuse.
  - currymj 9y ago
    I mean, it was a "modest proposal" type of suggestion. There are a lot of researchers who have a limited knowledge of statistics and who basically treat the p-value as a magic number. I don't have any sort of philosophical problem with p-values or the tools of hypothesis testing, but I think it would be good for science if the magic number were less salient, if that makes sense.
BeetleB 9y ago
This is just silly. Any golden value for p is bad. If I go back to my statistics text book (and I assume all of them say something to this effect): The appropriate value of p depends on the particular topic of study. Not on the whole discipline. And definitely not across all studies and across all disciplines. You cannot pick a value of p and say "this is good". Pick too low and you'll likely not be able to reproduce many legitimate results.
SubiculumCode 9y ago
Fishing for findings is a problem, but also overly stringent multiple comparison correction requirements. Those who preach Bonferroni should consider a life-long proposal. Let us propose that correction for multiple comparisons be a life long endeavor. At the start of a scientist's career as principle investigator, p<.10 is fine. But by their career's 10th comparison, they have to meet at Bonferroni p<.01, and at their career's 100th comparison, now they have to meet at Bonferroni p<.001. When they are old and gray, or doing genomics, at their 1000th comparison they have to meet at p<0.0001.
- SubiculumCode 9y ago
  Young investigators tend to get smaller grants and sample size can be an issue. Relaxed threshold for the beginning of a career helps them, career wise. Older investigators frequently start heading really large, ambitious projects with large sample sizes, but they would have to face more stringent p-values. Interesting dynamic.
- carbocation 9y ago
  Genomics generally uses 5e-8 (p<0.00000005) as its significance threshold. That aside, your point is clever. How to handle coauthors of different ages is left as an exercise to the reader.
  - SubiculumCode 9y ago
    I figured it as the p-value rate goes with the PI, the funded investigator.
SubiculumCode 9y ago
What is funny asbout p<.05 is that while we frequently have a specific hypothesis in mind (e.g. Mean1<Mean2), convention still dictates that we should use a two-sided t-test instead of a 1-sided t-test. If I am asked to go to .001, then I will use 1-sided t-tests when I have apriori one-sided hypotheses.
sna1l 9y ago
https://www.youtube.com/watch?v=8qrfSh07rT0 https://www.youtube.com/watch?v=8qrfSh07rT0 -- Taleb on p-values, technical, but pretty interesting.

128 points by nonbel 9y ago

Imagine if p <= 1.0 was considered "significant", wouldn't this cut down on publication bias? Shouldn't every study worth funding be worth publishing? They should all be considered significant results. So I think this proposal is: 1) Not really addressing the issues behind the replication crisis 2) Doing the exact opposite of what should be done (raise the significance level)

jimrandomh 9y ago
"Significance" is a social concept currently serving two roles: for determining what is considered publishable, and for determining what is considered true. This is meant to modify the second one, not the first. (Publishability should depend on quality of methodology, and be determined at the pre-registration phase before results are in.)
- nonbel 9y ago
  First, I agree that publishability should be independent of the results. Do you know anyone who disagrees with this though? That seems to be the standard opinion, yet the situation of filtering by p-value persists. Also, in the usual case where the null hypothesis is "no difference", no one really cares about what the p-value is telling them anyway. They care "is my theory accurate/true", when their theory predicts something other than the null hypothesis. I figure we may as well set the significance level in these cases so that everyone "wins", and go on to use analytic techniques that actually address the question of interest.
tpeo 9y ago
What's the point of fitting arbitrary functions to noise? Because that's what high p-values would lead to. Undue emphasis on p-values is bad, but what you're suggesting is bonkers.
- nonbel 9y ago
  Stop funding people to fit arbitrary functions to noise?
  - tpeo 9y ago
    Don't be disingenuous. That would require someone to actually read their papers, which would already be a net loss and a massive waste of time for everyone involved. Not to mention the amount of department politics which could be involved in this issue. Look, not every paper is worth publishing. There are many ways to write noise that can pass for signal given the reader's level of understanding (e.g. crack pottery and "fashionable nonsense"). The usage of p-values implements self-filtering to some degree, and makes everyone better off by increasing the likelihood that at least some papers aren't complete nonsense. And that's good. The only issue is that it is a much weaker constraint on publishing than it was previously thought.
    - JoshTriplett 9y ago
      > Look, not every paper is worth publishing. Many more results should be documented than currently are, though, including negative results. But I agree that changing p-value thresholds is unrelated to that.
      - tpeo 9y ago
        I agree, publication bias is a thing. There's no questioning that.
    - Houshalter 9y ago
      And not publishing wastes everyone's time. When researches perform the same experiment again and again until someone gets a significant result to publish. There are entire fields of research supported by nothing but publication bias (e.g. Parapsychology and possibly most soft sciences.)

leeoniya 9y ago

sounds expensive. 0.01 may be more reasonable.

mrob 9y ago
Choosing the correct value is a social problem. Eg. The Journal of Wacky Speculation publishes "The Moon is Made of Green Cheese". Using a highly sophisticated cheese detector, the authors measure results consistent with this hypothesis. They calculate it would happen only 4% of the time if the moon were not in fact made of green cheese, and therefore the result is significant. The cheese detector works perfectly, the calculations were accurate, but this does not mean the moon is made of green cheese! We've been there, and we're very certain that it's made of rock. Because the prior probability of the hypothesis was so low, the "significant" result was most likely a fluke. The Journal of Wacky Speculation specializes in this type of hypothesis, so they should use a much lower threshold for p. Meanwhile, the Sensible and Boring Society publishes a study showing that round wheels roll better on ordinary roads than square wheels. We already had good reason to believe this, so p<0.05 shows it's most likely true. The Sensible and Boring Society publishes tests of hypotheses with higher prior probably, so they can use a higher p threshold.

shmageggy 9y ago

Totally agree with Timothy Bates here: > [Bates] called the proposal “a risky distraction”... [It] wouldn’t address many other practices linked to irreproducible results: poor study design, a bias toward publishing positive results, and the practice of “p-value hacking”—fishing for significant-looking results from a huge number of hypotheses. Since the reproducibilty crisis started getting attention, I've noticed that these problems are really ubiquitous. Tweaking study designs and modifying hypotheses after collecting data are very commonplace and much more subtly dangerous than a potential lack of statistical power. Gelman says "Valid p-values cannot be drawn without knowing, not just what was done with the existing data, but what the choices in data coding, exclusion,and analysis would have been, had the data been different." [1] This proposal also says nothing about effect sizes, which also should play a role in how we interpret scientific results. We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05. [1]http://www.stat.columbia.edu/~gelman/research/published/asa_pvalues.pdf http://www.stat.columbia.edu/~gelman/research/published/asa_...

LoSboccacc 9y ago
P-values, reproducibility and the focus on statistics before understanding is, of itself, a problem: https://xkcd.com/882/ https://xkcd.com/882/
petters 9y ago
> We should probably draw different conclusions given a tiny effect at p=0.05 versus a large, obvious effect at p=0.05 I'm not sure that "large, obvious effect" and "p=0.05" are compatible, though.
- shmageggy 9y ago
  Sure it is, of course depending on your domain-dependent definition of "large, obvious". You can have different effect sizes at any given p-value.
mabbo 9y ago
I disagree. > a bias toward publishing positive results ... fishing for significant-looking results from a huge number of hypotheses If we accept a lower p-value as significant, there will be fewer hypotheses that have positive results, and therefore less to publish. Should we really do nothing at all until we can come up with a perfect solution to all aspects of the problem? This won't fix the real problem, but it will reduce the number of pop-sci articles spouting bullshit results that don't reproduce next week.

elsherbini 9y ago

There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can't change the effect size, so this would lead to larger necessary samples sizes to study smaller effects. I think in general though, at least in biology, people are waking up to the fact that p-values aren't magical, and that having a really small p-value isn't a goal in and of itself. It is, however, necessary to do some statistics on your data to get it published, but the p-value is just checking a box more than being used as a tool for discovery. Yesterday a cool dataset was released from Jeff Leek which has over 3.6 million p-values from scientific literature[1]. The distribution is fun to look at by discipline[2]. [0] http://rpsychologist.com/d3/NHST/ http://rpsychologist.com/d3/NHST/ [1] https://github.com/jtleek/tidypvals https://github.com/jtleek/tidypvals [2] https://twitter.com/drob/status/890260541876338690 https://twitter.com/drob/status/890260541876338690

nonbel 9y ago
Thanks, direct link to the pic: https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg https://pbs.twimg.com/media/DFqN_vqXkAAMGB_.jpg And here it is zoomed in: https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg https://pbs.twimg.com/media/DFrXCYTW0AEONA9.jpg These figures look very damning to p-values. This basically looks like they are made up anyway. I haven't checked the methods though so who knows. Edit: Well there are 33 "pvalues" > 1 and 20820 that equal zero, so this dataset probably could use more cleaning.
- Mathnerd314 9y ago
  Most of them are of the form "p > 0.05" or p < 0.01", hence the peaks. You really need to filter down to the red "equals" samples, but apparently they're pretty uncommon in papers to begin with. The only real info from that graph seems to be that geography/business/economics/psychology/sociology allow "p > 0.10" relatively more often, and that there's no data for the economics category alone (from Twitter, apparently the data source is not as good). This is a better pic: https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large https://pbs.twimg.com/media/DFqssedWsAA3h4Q.jpg:large Can't really see any difference in fields, besides some artifact in plant biology. Which makes sense since it's just a statistical test.

pcrh 9y ago

The term "significant" is itself part of the problem. In statistics it is often used to describe an arbitrary threshold where a difference between datasets is assumed to be real. In common parlance this term implies "important" or "meaningful", but the statistical tests are not designed to draw such forward-looking conclusions. It would be far easier, and more honest, to simply report the P-value obtained by performing a statistical test without adding the term "significant".

jaggederest 9y ago
I'd be interested in a compound metric composed of the likelihood ratio and the p-value. Essentially the more likely your hypothesis, the higher your P value can be, and conversely unlikely hypotheses require stronger evidence. I don't think this solves the problem but maybe it makes that assumption about the hypothesis likelihood more specific.

Tloewald 9y ago

I think that a very stringent threshold for p values (it's going to be a lot harder to get to p < 0.005 than p < 0.05 and it's going to encourage a lot more gaming of the system) is not the solution. At best, it will lead to smaller numbers of much bigger studies with more authors and fewer interesting results. I'd suggest that a better, broader, and more practical solution is to provide a public clearing house for studies and associated data -- let's assume it's a website. a) When you start a study you must describe your study, methodology, and hypothesis, and publish these things on the site. It should be possible to find the study the same way you find any paper that results from it. b) All data must be stored in the same place (but not made public) as it becomes available, and this must be demonstrable. This data should be available to anyone trying to replicate, review, etc. c) Final data sets and analyses must also be also be available, ideally with the code used to do the analyses. All of this should be prerequisite for review and publication, but also for anyone working in the field should no papers result. This solves a lot of problems, including making it easier to conduct meta-analyses when looking for effects that aren't significant in smaller studies, and also address the lack of availability of negative results, and even help combat theft of methodology (e.g. where someone sees work in progress and is able to publish using the methodology). It would make meta-analysis easier, and allow researchers to better take advantage of existing, unpublished results (e.g. to hone their study methods, etc.)

nerfhammer 9y ago
Register the hypothesis first, results later. If you get null results, you have to report that.
- Tloewald 9y ago
  Right. This is a direct consequence of my proposal.
brownbat 9y ago
If we were designing things from scratch, I'd even want t odisaggregate two career tracks for scientists: The first, theorists, design interesting studies with open ended questions, explain how the study would provide useful scientific results, or theorize about the implications of existing published results. The second, practitioners, are hired to actually run the studies according to the specifications provided. The practitioners would publish null results of course. Practitioners would also have to acknowledge how many of their results are significant, so it'd be transparent if they were just confirming everything. A bit like barristers and solicitors in many common law countries, a distinction that isn't strictly necessary for the system to work, but prevents some conflicts of interest.

SurrealSoul 9y ago

>You won't believe what this team did to the word significant! >10 significant things that WILL shock you, #7 is this team >THIS crazy trick that will make significant much harder [teams hate him!!] >This finding prank gone significant [gone sexual!?] Sorry, it was such a click-baity title for no reason.

currymj 9y ago

if i were the King of All Science and were feeling despotic, I would do the opposite, I think -- put a moratorium for a few years on any publication of a p-value. Scientists would still be free to make use of hypothesis testing to guide their research. In publications, one could use Bayesian methods and report a full posterior. One could also just give enough information about the null hypothesis and whatever distributions are involved to allow the computation of a p-value. However, any attempt to explicitly publish the p-value itself would result in creative and embarrassing punishments.

BeetleB 9y ago
>In publications, one could use Bayesian methods and report a full posterior. I fail to see how this is superior. I'd never advocate banning a legitimate tool just because of abuse.
- currymj 9y ago
  I mean, it was a "modest proposal" type of suggestion. There are a lot of researchers who have a limited knowledge of statistics and who basically treat the p-value as a magic number. I don't have any sort of philosophical problem with p-values or the tools of hypothesis testing, but I think it would be good for science if the magic number were less salient, if that makes sense.

BeetleB 9y ago

This is just silly. Any golden value for p is bad. If I go back to my statistics text book (and I assume all of them say something to this effect): The appropriate value of p depends on the particular topic of study. Not on the whole discipline. And definitely not across all studies and across all disciplines. You cannot pick a value of p and say "this is good". Pick too low and you'll likely not be able to reproduce many legitimate results.

SubiculumCode 9y ago

Fishing for findings is a problem, but also overly stringent multiple comparison correction requirements. Those who preach Bonferroni should consider a life-long proposal. Let us propose that correction for multiple comparisons be a life long endeavor. At the start of a scientist's career as principle investigator, p<.10 is fine. But by their career's 10th comparison, they have to meet at Bonferroni p<.01, and at their career's 100th comparison, now they have to meet at Bonferroni p<.001. When they are old and gray, or doing genomics, at their 1000th comparison they have to meet at p<0.0001.

SubiculumCode 9y ago
Young investigators tend to get smaller grants and sample size can be an issue. Relaxed threshold for the beginning of a career helps them, career wise. Older investigators frequently start heading really large, ambitious projects with large sample sizes, but they would have to face more stringent p-values. Interesting dynamic.
carbocation 9y ago
Genomics generally uses 5e-8 (p<0.00000005) as its significance threshold. That aside, your point is clever. How to handle coauthors of different ages is left as an exercise to the reader.
- SubiculumCode 9y ago
  I figured it as the p-value rate goes with the PI, the funded investigator.

SubiculumCode 9y ago

What is funny asbout p<.05 is that while we frequently have a specific hypothesis in mind (e.g. Mean1<Mean2), convention still dictates that we should use a two-sided t-test instead of a 1-sided t-test. If I am asked to go to .001, then I will use 1-sided t-tests when I have apriori one-sided hypotheses.

sna1l 9y ago

https://www.youtube.com/watch?v=8qrfSh07rT0 https://www.youtube.com/watch?v=8qrfSh07rT0 -- Taleb on p-values, technical, but pretty interesting.