3 ms·

I disagree with this idea. For starters, sample sizes alone don't give you reliable results, even with large sample sizes you might have other issues that affec

by vivekd 10y ago

I disagree with this idea. For starters, sample sizes alone don't give you reliable results, even with large sample sizes you might have other issues that affect the results such as poor design, it may not be reproducible or (as is often the case) it may be that no one has tried to reproduce it yet. Scientific research is often imperfect, as a practical matter due to lack of resources and many other reasons. But over time, as scientists peer review and reproduce each other's research, we come closer to scientific consensus regarding the matters of study. Scientific studies, by their very nature, are often designed to inquire into matters of uncertainty. And no one study can completely claim to banish uncertainty and provide concrete and certain answers. The path to certainty requires the toil of reviewing and reproducing the results. So I think anyone who puts forward a single scientific study to bolster a claim or point of view is making a very weak and unconvincing argument to anyone who understands science. Rather, we like scientific studies, not because they give us concrete neat solutions but because they offer interesting insights that may or not be conclusive. I don't think it should be the job of moderators to determine the weight or conclusivity of scientific studies. I think that should be done by the community. I also think the community would lose something if were to get rid of articles on interesting research just because that research was inconclusive. Research can be interesting and open up interesting possibilities even when it doesn't give us concrete truths to hold to.

syllogism 10y ago
It sounds to me like you reject the premise that an intelligent reader who reads studies like this will have their beliefs diverge from the truth over time. I read your comment as saying, "Sure you shouldn't be convinced by a single study. But if you read enough of these, you'll converge towards the truth". I used to believe that, but I don't now. Let's say some study is an increment of evidence about some hypothesis H. We start off with a prior on H, and we want to know how to update it after reading about a study. What I've taken away from the replication crisis is that my average update was an order of magnitude too large. The appropriate evidentiary weighting of a study in social psychology is so small that I can't usefully consider it as any evidence at all. I'm better off ignoring it. A little example for you. Let's say you read some study about the act of physically smiling causing a positive change of mood, due to a feedback effect. Before reading this study, you would've given this claim p=0.1. Possible, and you can make sense of the explanation, but no particular reason to believe it. After the study, you give it p=0.4. By no means settled, and you don't even necessarily believe it. But you're giving it much more credibility than before. Later it turns out that the study was actually on very few people and was p-hacked. It therefore fails to replicate. The appropriate post-reading belief level turns out to be around p=0.101. You were tricked. If you repeat this pattern, you end up carrying around a bunch of surprising beliefs that have a quite random relationship to the truth. Some of the studies will be correct, but on average you'll be more wrong. Reading these articles might be "interesting", on an entertainment-based definition, but they're the opposite of information. If you read enough of these, you'll make yourself superstitious.
- davelnewton 10y ago
  Information isn't the opposite of information. You more or less sum up my reaction to your suggestion with this (rather telling) phrase: "you'll make yourself superstitious" (Emphasis mine.)
  - syllogism 10y ago
    I don't think I understand your comment. What I'm saying is that if you read these studies, you'll become less informed over time. You will believe fewer true things and more false things. I think it's reasonable to call that the opposite of information
    - davelnewton 10y ago
      I will? No, no I won't. And no, you don't become less-informed by taking in information. You become ill-informed if you choose to believe incomplete, partial, or misleading information. But that's your choice. Some people read a fairy tale and believe it--not the fairy tale's fault. To me it seems like you're absolving yourself of responsibility. If I don't understand the nature of a study then I take it for what it is: a report about results of a study I don't understand. I choose whether or not to believe it based on partial data. It doesn't make me believe it. I may believe it, or parts of it, or none of it. I make that decision.
DanBC 10y ago
> But over time, as scientists peer review and reproduce each other's research, we come closer to scientific consensus regarding the matters of study. That's the problem though. These small sometimes poorly run studies are conducted, then published, then a press release is issued. The press release sometimes over hypes the paper. Then newspapers sometimes over hype or misinterpret the press release, and rarely if ever link to the actual paper. And then that news paper link ends up on HN, where people talk about their interpretations of a misinterpreted press release that hyped up the original paper. It sucks.

by vivekd 10y ago

syllogism 10y ago

It sounds to me like you reject the premise that an intelligent reader who reads studies like this will have their beliefs diverge from the truth over time. I read your comment as saying, "Sure you shouldn't be convinced by a single study. But if you read enough of these, you'll converge towards the truth". I used to believe that, but I don't now. Let's say some study is an increment of evidence about some hypothesis H. We start off with a prior on H, and we want to know how to update it after reading about a study. What I've taken away from the replication crisis is that my average update was an order of magnitude too large. The appropriate evidentiary weighting of a study in social psychology is so small that I can't usefully consider it as any evidence at all. I'm better off ignoring it. A little example for you. Let's say you read some study about the act of physically smiling causing a positive change of mood, due to a feedback effect. Before reading this study, you would've given this claim p=0.1. Possible, and you can make sense of the explanation, but no particular reason to believe it. After the study, you give it p=0.4. By no means settled, and you don't even necessarily believe it. But you're giving it much more credibility than before. Later it turns out that the study was actually on very few people and was p-hacked. It therefore fails to replicate. The appropriate post-reading belief level turns out to be around p=0.101. You were tricked. If you repeat this pattern, you end up carrying around a bunch of surprising beliefs that have a quite random relationship to the truth. Some of the studies will be correct, but on average you'll be more wrong. Reading these articles might be "interesting", on an entertainment-based definition, but they're the opposite of information. If you read enough of these, you'll make yourself superstitious.

davelnewton 10y ago
Information isn't the opposite of information. You more or less sum up my reaction to your suggestion with this (rather telling) phrase: "you'll make yourself superstitious" (Emphasis mine.)
- syllogism 10y ago
  I don't think I understand your comment. What I'm saying is that if you read these studies, you'll become less informed over time. You will believe fewer true things and more false things. I think it's reasonable to call that the opposite of information
  - davelnewton 10y ago
    I will? No, no I won't. And no, you don't become less-informed by taking in information. You become ill-informed if you choose to believe incomplete, partial, or misleading information. But that's your choice. Some people read a fairy tale and believe it--not the fairy tale's fault. To me it seems like you're absolving yourself of responsibility. If I don't understand the nature of a study then I take it for what it is: a report about results of a study I don't understand. I choose whether or not to believe it based on partial data. It doesn't make me believe it. I may believe it, or parts of it, or none of it. I make that decision.

DanBC 10y ago

> But over time, as scientists peer review and reproduce each other's research, we come closer to scientific consensus regarding the matters of study. That's the problem though. These small sometimes poorly run studies are conducted, then published, then a press release is issued. The press release sometimes over hypes the paper. Then newspapers sometimes over hype or misinterpret the press release, and rarely if ever link to the actual paper. And then that news paper link ends up on HN, where people talk about their interpretations of a misinterpreted press release that hyped up the original paper. It sucks.