4 ms·

Suggest HN: Studies based on small sample sizes shouldn't be on HN

31 points by syllogism 10y ago

Reading about studies based on small sample sizes has in total misinformed me. I've been convinced of many surprising things based on a trust that the research was well conducted, and that published results are on average true. The building replication crisis has shown that this faith was misplaced, particularly in certain fields. For instance, it's now clear that I would have believed more true things and fewer false things if I had not heard about a single result from social psychology. I'm still rooting out beliefs that I came to based on bad research. (Unfortunately my mind doesn't maintain an index of beliefs by source. I wish I could request this feature.) I'm rightly bitter about this, and I see the problem continuing. For instance, this submission is worse than worthless: https://news.ycombinator.com/item?id=13144483 . It's impossible to tell what experiment was done without buying the paper, and on average experiments of this type have been shown to be of negative value: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ . We're all worse off for reading and talking about these articles. I'd like to see HN add a heavy link penalty on all reports of research based on small samples, with an exception for good replication studies. Off the top of my head, n<10,000 might be a good threshold.

sz4kerto 10y ago
Good idea, but a hard limit like 10000 is not that good. Statistical methods don't require specific sample sizes. Cherry picking, incorrectly chosen methods are the problems, not the sample sizes, usually. A hard limit would suggest that anything above this sample size is credible, while that's very far from the truth. The reproducibility crisis is very much unrelated to sample sizes.
- syllogism 10y ago
  Sure, you can't design a study badly and make it up on volume. — but there's a point at which the sample is so small we really don't need to look much closer. For instance, I don't want to spend time figuring out whether an n<100 study is any good. It isn't.
  - stenl 10y ago
    Surely the required sample size depends on the effect size. You won't need n>100 to learn that elephants are larger than ants. The problem is in experiment design and interpretation generally, not sample size specifically. That said, all of us who know how to read a scientific paper (and have institutional access to the journals) should take 5 mins and point out flaws when we see them.
  - jdietrich 10y ago
    You don't need a large sample to establish that beheading is fatal. Sample size is only one factor in statistical significance; often it's a fairly unimportant factor.
  - aaron695 10y ago
    > For instance, I don't want to spend time figuring out whether an n<100 study is any good. It isn't. So an AIDs cure that cured 80 out of 90 men wouldn't be worth reporting on HN? Studies that have >10,000 people I'd think are close to non existent, these are hugely expensive and I think most are well known. I think it's a job for the scientific community to work out, I can't see moderators on HN doing it.
    - syllogism 10y ago
      Personally I wouldn't click on a link that was reporting on a potential AIDS cure that was tested on 80 or 90 people.
      - davelnewton 10y ago
        I'd click on a link on a report that cured 80 of 90 AIDS patients. I think you've misunderstood the nature of preliminary clinical trials.
golemotron 10y ago
That's a great idea for a journal aggregator. This is a news site.
- syllogism 10y ago
  It's a news site that has editorial policies about which stories are worth paying attention to and which ones aren't.
vivekd 10y ago
I disagree with this idea. For starters, sample sizes alone don't give you reliable results, even with large sample sizes you might have other issues that affect the results such as poor design, it may not be reproducible or (as is often the case) it may be that no one has tried to reproduce it yet. Scientific research is often imperfect, as a practical matter due to lack of resources and many other reasons. But over time, as scientists peer review and reproduce each other's research, we come closer to scientific consensus regarding the matters of study. Scientific studies, by their very nature, are often designed to inquire into matters of uncertainty. And no one study can completely claim to banish uncertainty and provide concrete and certain answers. The path to certainty requires the toil of reviewing and reproducing the results. So I think anyone who puts forward a single scientific study to bolster a claim or point of view is making a very weak and unconvincing argument to anyone who understands science. Rather, we like scientific studies, not because they give us concrete neat solutions but because they offer interesting insights that may or not be conclusive. I don't think it should be the job of moderators to determine the weight or conclusivity of scientific studies. I think that should be done by the community. I also think the community would lose something if were to get rid of articles on interesting research just because that research was inconclusive. Research can be interesting and open up interesting possibilities even when it doesn't give us concrete truths to hold to.
- syllogism 10y ago
  It sounds to me like you reject the premise that an intelligent reader who reads studies like this will have their beliefs diverge from the truth over time. I read your comment as saying, "Sure you shouldn't be convinced by a single study. But if you read enough of these, you'll converge towards the truth". I used to believe that, but I don't now. Let's say some study is an increment of evidence about some hypothesis H. We start off with a prior on H, and we want to know how to update it after reading about a study. What I've taken away from the replication crisis is that my average update was an order of magnitude too large. The appropriate evidentiary weighting of a study in social psychology is so small that I can't usefully consider it as any evidence at all. I'm better off ignoring it. A little example for you. Let's say you read some study about the act of physically smiling causing a positive change of mood, due to a feedback effect. Before reading this study, you would've given this claim p=0.1. Possible, and you can make sense of the explanation, but no particular reason to believe it. After the study, you give it p=0.4. By no means settled, and you don't even necessarily believe it. But you're giving it much more credibility than before. Later it turns out that the study was actually on very few people and was p-hacked. It therefore fails to replicate. The appropriate post-reading belief level turns out to be around p=0.101. You were tricked. If you repeat this pattern, you end up carrying around a bunch of surprising beliefs that have a quite random relationship to the truth. Some of the studies will be correct, but on average you'll be more wrong. Reading these articles might be "interesting", on an entertainment-based definition, but they're the opposite of information. If you read enough of these, you'll make yourself superstitious.
Kenji 10y ago
If you take the results of soft 'sciences' seriously you've already lost. Recognize submissions like the one you linked as noise and move on. Common sense goes a long way.
- aaron695 10y ago
  I think this comment actually nails it. Most of the current issues in published science are around the 'soft sciences' They are notoriously hard to disprove I think because they are so wishy washy. Maths, Physics, Chemistry can get you disproven in a hard way. There's a lot more incentive to make sure your study is valid.
  - gus_massa 10y ago
    There is a lot of dubious Physics that is very popular here. My antifavorite is the Em-drive: Most popular article: "EmDrive study officially published" https://news.ycombinator.com/item?id=12995125 https://news.ycombinator.com/item?id=12995125 (595 points, 21 days ago, 432 comments) (My second choice is Cold fusion / LENR.) This has slightly less votes that the most popular stories about colossal experiments: "Second Gravitational Wave Detected at LIGO" https://news.ycombinator.com/item?id=11910700 https://news.ycombinator.com/item?id=11910700 (621 points, 178 days ago, 175 comments) "Higgs Boson Explained by Cartoon" https://news.ycombinator.com/item?id=4193590 https://news.ycombinator.com/item?id=4193590 (646 points, 1622 days ago, 128 comments) The problem is that most people like nice stories, and a massless thruster that can make space travel much faster is interesting. And it's much easier to explain than why we need the Higgs's boson or gravitational waves. The two later have no possible application in sight. They are interesting results, but it's not clear how they will change your life. (I'm optimistic, let's see what happens in 30 years.) The Em-drive looks more promising. Cheap and fast space travel soon, for some definition of soon. It's easier to explain, in particular because the theoretical explanation is worse than the experimental setup. No one tries to explain something about gauge invariance of curvature tensors.
    - Kenji 10y ago
      As far as I know, the EmDrive is still considered controversial and it is not certain whether and how it works. It is interesting because it could cause a revolution to how we view impulse, so it might be a colossal thing just as much as it might be complete nonsense. So I think the comparison is disingenous.
ebbv 10y ago
Seems to me like you haven't learned the right lesson. Small sample sizes can still yield useful results. It depends on the nature of the study. Instead of closing yourself (and the rest of us) based on a fairly arbitrary criteria, you're going to have to use critical thinking. That should be applied to everything you read not just studies.
dbg31415 10y ago
If we are promoting censorship can we start with: 1) Politics 2) Links to Reddit; any discussion about Reddit 3) Stories about the dinosaur tail found in amber https://hn.algolia.com/?query=dinosaur%20tail&sort=byDate&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=dinosaur%20tail&sort=byDate&pr... (posted 26 times in the last week) To be clear I'm not for censorship, but if I was the above is where I'd start.
davelnewton 10y ago
Disagree, somewhat loudly. I don't blame other sources for my own beliefs. If I choose to believe something based on a news article that's squarely on me. In any case, the paper you cite is available here: http://fulltext.study/preview/pdf/917856.pdf http://fulltext.study/preview/pdf/917856.pdf
brudgers 10y ago
Generally, I think scientific studies are uninformative as news because: 1. Most people are not close enough to the science to form an informed technical opinion. 2. The scientific conclusions of most studies are impractical to implement. John Oliver's analysis is both sophisticated and entertaining: https://www.youtube.com/watch?v=0Rnq1NpHdmw https://www.youtube.com/watch?v=0Rnq1NpHdmw
gus_massa 10y ago
In an ideal world I agree, but it's very difficult to implement. I'd ask to not upvote stories with a small sample size. In many articles consider writing a small rant about the methodological problems. Lucky most of the times I just found a comment by tokenadult that is better than my draft and has link to support the claims. It's a pity that he is not writing them recently. Try writing a civil comment explaining that the study is dubious due to the small sample size. Most press coverage skip this detail, so you have to look at the research article to copy it. You'll get at least 1 upvote. Don't worry too much. Just eat some chocolate https://news.ycombinator.com/item?id=9714985 https://news.ycombinator.com/item?id=9714985
drbigglesworth 10y ago
The focus on sample size is misplaced: small sample sizes frequently have sufficient power to be statistically significant. The error you are making is in placing too much faith in a single study, and possibly in the journalists/bloggers who report on a single study. I have lost count of the number of times I've read a press release or news article that completely misrepresents the findings of a study (i.e., the error is in the reporting, not the article). It's also frequently forgotten that journal articles are written to communicate findings to peers, not the larger public. There are nuances that are understood within a community that are unknown when one is coming from a different background. Until there is a body of work on a phenomenon, perhaps framed by a review article or two, it's best to regard single reports (regardless of perceived quality) as interesting curios but nothing more.

31 points by syllogism 10y ago

sz4kerto 10y ago

Good idea, but a hard limit like 10000 is not that good. Statistical methods don't require specific sample sizes. Cherry picking, incorrectly chosen methods are the problems, not the sample sizes, usually. A hard limit would suggest that anything above this sample size is credible, while that's very far from the truth. The reproducibility crisis is very much unrelated to sample sizes.

syllogism 10y ago
Sure, you can't design a study badly and make it up on volume. — but there's a point at which the sample is so small we really don't need to look much closer. For instance, I don't want to spend time figuring out whether an n<100 study is any good. It isn't.
- stenl 10y ago
  Surely the required sample size depends on the effect size. You won't need n>100 to learn that elephants are larger than ants. The problem is in experiment design and interpretation generally, not sample size specifically. That said, all of us who know how to read a scientific paper (and have institutional access to the journals) should take 5 mins and point out flaws when we see them.
- jdietrich 10y ago
  You don't need a large sample to establish that beheading is fatal. Sample size is only one factor in statistical significance; often it's a fairly unimportant factor.
- aaron695 10y ago
  > For instance, I don't want to spend time figuring out whether an n<100 study is any good. It isn't. So an AIDs cure that cured 80 out of 90 men wouldn't be worth reporting on HN? Studies that have >10,000 people I'd think are close to non existent, these are hugely expensive and I think most are well known. I think it's a job for the scientific community to work out, I can't see moderators on HN doing it.
  - syllogism 10y ago
    Personally I wouldn't click on a link that was reporting on a potential AIDS cure that was tested on 80 or 90 people.
    - davelnewton 10y ago
      I'd click on a link on a report that cured 80 of 90 AIDS patients. I think you've misunderstood the nature of preliminary clinical trials.

golemotron 10y ago

That's a great idea for a journal aggregator. This is a news site.

syllogism 10y ago
It's a news site that has editorial policies about which stories are worth paying attention to and which ones aren't.

vivekd 10y ago

I disagree with this idea. For starters, sample sizes alone don't give you reliable results, even with large sample sizes you might have other issues that affect the results such as poor design, it may not be reproducible or (as is often the case) it may be that no one has tried to reproduce it yet. Scientific research is often imperfect, as a practical matter due to lack of resources and many other reasons. But over time, as scientists peer review and reproduce each other's research, we come closer to scientific consensus regarding the matters of study. Scientific studies, by their very nature, are often designed to inquire into matters of uncertainty. And no one study can completely claim to banish uncertainty and provide concrete and certain answers. The path to certainty requires the toil of reviewing and reproducing the results. So I think anyone who puts forward a single scientific study to bolster a claim or point of view is making a very weak and unconvincing argument to anyone who understands science. Rather, we like scientific studies, not because they give us concrete neat solutions but because they offer interesting insights that may or not be conclusive. I don't think it should be the job of moderators to determine the weight or conclusivity of scientific studies. I think that should be done by the community. I also think the community would lose something if were to get rid of articles on interesting research just because that research was inconclusive. Research can be interesting and open up interesting possibilities even when it doesn't give us concrete truths to hold to.

syllogism 10y ago
It sounds to me like you reject the premise that an intelligent reader who reads studies like this will have their beliefs diverge from the truth over time. I read your comment as saying, "Sure you shouldn't be convinced by a single study. But if you read enough of these, you'll converge towards the truth". I used to believe that, but I don't now. Let's say some study is an increment of evidence about some hypothesis H. We start off with a prior on H, and we want to know how to update it after reading about a study. What I've taken away from the replication crisis is that my average update was an order of magnitude too large. The appropriate evidentiary weighting of a study in social psychology is so small that I can't usefully consider it as any evidence at all. I'm better off ignoring it. A little example for you. Let's say you read some study about the act of physically smiling causing a positive change of mood, due to a feedback effect. Before reading this study, you would've given this claim p=0.1. Possible, and you can make sense of the explanation, but no particular reason to believe it. After the study, you give it p=0.4. By no means settled, and you don't even necessarily believe it. But you're giving it much more credibility than before. Later it turns out that the study was actually on very few people and was p-hacked. It therefore fails to replicate. The appropriate post-reading belief level turns out to be around p=0.101. You were tricked. If you repeat this pattern, you end up carrying around a bunch of surprising beliefs that have a quite random relationship to the truth. Some of the studies will be correct, but on average you'll be more wrong. Reading these articles might be "interesting", on an entertainment-based definition, but they're the opposite of information. If you read enough of these, you'll make yourself superstitious.

Kenji 10y ago

If you take the results of soft 'sciences' seriously you've already lost. Recognize submissions like the one you linked as noise and move on. Common sense goes a long way.

aaron695 10y ago
I think this comment actually nails it. Most of the current issues in published science are around the 'soft sciences' They are notoriously hard to disprove I think because they are so wishy washy. Maths, Physics, Chemistry can get you disproven in a hard way. There's a lot more incentive to make sure your study is valid.
- gus_massa 10y ago
  There is a lot of dubious Physics that is very popular here. My antifavorite is the Em-drive: Most popular article: "EmDrive study officially published" https://news.ycombinator.com/item?id=12995125 https://news.ycombinator.com/item?id=12995125 (595 points, 21 days ago, 432 comments) (My second choice is Cold fusion / LENR.) This has slightly less votes that the most popular stories about colossal experiments: "Second Gravitational Wave Detected at LIGO" https://news.ycombinator.com/item?id=11910700 https://news.ycombinator.com/item?id=11910700 (621 points, 178 days ago, 175 comments) "Higgs Boson Explained by Cartoon" https://news.ycombinator.com/item?id=4193590 https://news.ycombinator.com/item?id=4193590 (646 points, 1622 days ago, 128 comments) The problem is that most people like nice stories, and a massless thruster that can make space travel much faster is interesting. And it's much easier to explain than why we need the Higgs's boson or gravitational waves. The two later have no possible application in sight. They are interesting results, but it's not clear how they will change your life. (I'm optimistic, let's see what happens in 30 years.) The Em-drive looks more promising. Cheap and fast space travel soon, for some definition of soon. It's easier to explain, in particular because the theoretical explanation is worse than the experimental setup. No one tries to explain something about gauge invariance of curvature tensors.
  - Kenji 10y ago
    As far as I know, the EmDrive is still considered controversial and it is not certain whether and how it works. It is interesting because it could cause a revolution to how we view impulse, so it might be a colossal thing just as much as it might be complete nonsense. So I think the comparison is disingenous.

ebbv 10y ago

Seems to me like you haven't learned the right lesson. Small sample sizes can still yield useful results. It depends on the nature of the study. Instead of closing yourself (and the rest of us) based on a fairly arbitrary criteria, you're going to have to use critical thinking. That should be applied to everything you read not just studies.

dbg31415 10y ago

If we are promoting censorship can we start with: 1) Politics 2) Links to Reddit; any discussion about Reddit 3) Stories about the dinosaur tail found in amber https://hn.algolia.com/?query=dinosaur%20tail&sort=byDate&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=dinosaur%20tail&sort=byDate&pr... (posted 26 times in the last week) To be clear I'm not for censorship, but if I was the above is where I'd start.

davelnewton 10y ago

Disagree, somewhat loudly. I don't blame other sources for my own beliefs. If I choose to believe something based on a news article that's squarely on me. In any case, the paper you cite is available here: http://fulltext.study/preview/pdf/917856.pdf http://fulltext.study/preview/pdf/917856.pdf

brudgers 10y ago

Generally, I think scientific studies are uninformative as news because: 1. Most people are not close enough to the science to form an informed technical opinion. 2. The scientific conclusions of most studies are impractical to implement. John Oliver's analysis is both sophisticated and entertaining: https://www.youtube.com/watch?v=0Rnq1NpHdmw https://www.youtube.com/watch?v=0Rnq1NpHdmw

gus_massa 10y ago

In an ideal world I agree, but it's very difficult to implement. I'd ask to not upvote stories with a small sample size. In many articles consider writing a small rant about the methodological problems. Lucky most of the times I just found a comment by tokenadult that is better than my draft and has link to support the claims. It's a pity that he is not writing them recently. Try writing a civil comment explaining that the study is dubious due to the small sample size. Most press coverage skip this detail, so you have to look at the research article to copy it. You'll get at least 1 upvote. Don't worry too much. Just eat some chocolate https://news.ycombinator.com/item?id=9714985 https://news.ycombinator.com/item?id=9714985

drbigglesworth 10y ago

The focus on sample size is misplaced: small sample sizes frequently have sufficient power to be statistically significant. The error you are making is in placing too much faith in a single study, and possibly in the journalists/bloggers who report on a single study. I have lost count of the number of times I've read a press release or news article that completely misrepresents the findings of a study (i.e., the error is in the reporting, not the article). It's also frequently forgotten that journal articles are written to communicate findings to peers, not the larger public. There are nuances that are understood within a community that are unknown when one is coming from a different background. Until there is a body of work on a phenomenon, perhaps framed by a review article or two, it's best to regard single reports (regardless of perceived quality) as interesting curios but nothing more.