3 ms·

The sample size was also 31 users, 40 controls. This really isn't a meaningful result.

by MillenialMan 5y ago

kilbuz 5y ago
I did not read the results, but this definitely depends on the effect size and the variance of whatever they are studying. If 31 people were raised from the dead, for example, it's plenty of sample size.
- ramraj07 5y ago
  The issue is that the sample might be highly biased. If it's just 31 white males from Lubbock Texas then what does it even say? Stoners in that town speak different?
- MillenialMan 5y ago
  With such a small sample size, formal methods break down. The sample could be biased. In fact, it's almost guaranteed. The ability to p-hack (even inadvertently) is extremely high. Publication bias is also practically guaranteed. These kinds of issues void significance to a reasonably high degree. At these kinds of sample sizes, the biggest issues are not going to be included or visible in the study. Unless the reported significance is extremely high, it's not meaningful.
jcims 5y ago
I don't really participate in scientific literature or journals, but I see a lot of notes in HN around papers that appear to be overblowing the significance of their findings. Is there a way for researchers to publish 'interesting observations' as just that, something that they measured that may or may not be noteworthy but felt worth sharing with the broader community?
- thebooktocome 5y ago
  The traditional solution to this problem is to insert a paragraph in the conclusion saying something along the lines of "this merits further study". However, that particular phrase has been overused in journal articles, to the point that it doesn't mean anything anymore.
- MillenialMan 5y ago
  It's not really something researchers can control. I was commenting on the news release, and it being posted on Hacker News, because that's more of an issue.

3 ms·

The sample size was also 31 users, 40 controls. This really isn't a meaningful result.

by MillenialMan 5y ago

The sample size was also 31 users, 40 controls. This really isn't a meaningful result.

kilbuz 5y ago
I did not read the results, but this definitely depends on the effect size and the variance of whatever they are studying. If 31 people were raised from the dead, for example, it's plenty of sample size.
- ramraj07 5y ago
  The issue is that the sample might be highly biased. If it's just 31 white males from Lubbock Texas then what does it even say? Stoners in that town speak different?
- MillenialMan 5y ago
  With such a small sample size, formal methods break down. The sample could be biased. In fact, it's almost guaranteed. The ability to p-hack (even inadvertently) is extremely high. Publication bias is also practically guaranteed. These kinds of issues void significance to a reasonably high degree. At these kinds of sample sizes, the biggest issues are not going to be included or visible in the study. Unless the reported significance is extremely high, it's not meaningful.
jcims 5y ago
I don't really participate in scientific literature or journals, but I see a lot of notes in HN around papers that appear to be overblowing the significance of their findings. Is there a way for researchers to publish 'interesting observations' as just that, something that they measured that may or may not be noteworthy but felt worth sharing with the broader community?
- thebooktocome 5y ago
  The traditional solution to this problem is to insert a paragraph in the conclusion saying something along the lines of "this merits further study". However, that particular phrase has been overused in journal articles, to the point that it doesn't mean anything anymore.
- MillenialMan 5y ago
  It's not really something researchers can control. I was commenting on the news release, and it being posted on Hacker News, because that's more of an issue.