3 ms·

I guess I can see how you got that impression from my reply to sarreph. My emphasis was on the "presumptive analysis" part, not the "self-reported anecdotes" pa

I guess I can see how you got that impression from my reply to sarreph. My emphasis was on the "presumptive analysis" part, not the "self-reported anecdotes" part, but there is no way for anyone to know that since I didn't indicate it. I hope my first reply to you made my intended meaning clear because it's too late to edit now. In case not: reproducibility is a major concern with psychological studies in general. Drawing broad/subjective conclusions will make the results easier reproduce, but it also leaves more room for confirmation bias, so it's not a valid solution to the problem.

hyperpape 11y ago
Interesting. After your last post, I thought we weren't disagreeing about anything, we'd just miscommunicated, but now I'm not sure. "Drawing broad/subjective conclusions will make the results easier reproduce." Not sure I can say anything more useful beyond what I said before. I just don't think this is true.
- dbbolton 11y ago
  Well, I mean it can make reproducibility easier, not that it definitely will. Say- just for argument's sake- that another group of researchers did this same experiment but came to a different conclusion such as: "members of this demographic were more likely to describe a certain type of self-promotion as being positively-received, despite the fact that this type of self-promotion was consistently described as negative by the recipient group". Whenever anyone else repeated the experiment, it would seemingly be much easier to align their conclusion with that of the original experiment than with this imaginary example just by virtue of the fact that the original conclusion was so broad.
  - hyperpape 11y ago
    Ok, there is a real disagreement. As I understand it, reproducibility means "you can run this same experiment and expect to get the same effect." Reproducibility is an issue because of study designs that are unusually to get accidental results. So you have your p value < .05, but the worry is that it's just an accident. If it's not an accident, then you should be able to run the exact same study again and get a similar effect. The problem of reproducibility is that even _that_ may not possible for some notable studies. Because of bad experimental design, we were just reporting meaningless results (the 5% chance that you get a nice p-value for an effect that isn't real). So in the case of this experiment, reproducibility is "if you survey people about bragging, will coders report their comments in the same way?" not "is bragging perceived negatively?" I think what you're describing is sometimes called a "conceptual reproduction" where you run a different study that you can argue supports the same conclusion. For instance, you've run an experiment where people seem to show a cognitive bias. Then someone else runs a different experiment, with a different design, and says "my results show the same cognitive bias that dbbolton showed in his study". In this case, the experimental design is different, and the second study doesn't directly demonstrate that you results could be reproduced. But in theory, it does suggest that there's a real phenomenon, because the same explanation seems natural for both experiments. There has been an overuse of conceptual reproductions in psychology, but what really has people worried is that major studies may literally not be reproducible. P.S. all of this is based on my understanding of how the terms are used, and I'm not 100% confident I'm right about it. I think I am, but I'm not a psychologist, take my opinion with a grain of salt.

hyperpape 11y ago

Interesting. After your last post, I thought we weren't disagreeing about anything, we'd just miscommunicated, but now I'm not sure. "Drawing broad/subjective conclusions will make the results easier reproduce." Not sure I can say anything more useful beyond what I said before. I just don't think this is true.

dbbolton 11y ago
Well, I mean it can make reproducibility easier, not that it definitely will. Say- just for argument's sake- that another group of researchers did this same experiment but came to a different conclusion such as: "members of this demographic were more likely to describe a certain type of self-promotion as being positively-received, despite the fact that this type of self-promotion was consistently described as negative by the recipient group". Whenever anyone else repeated the experiment, it would seemingly be much easier to align their conclusion with that of the original experiment than with this imaginary example just by virtue of the fact that the original conclusion was so broad.
- hyperpape 11y ago
  Ok, there is a real disagreement. As I understand it, reproducibility means "you can run this same experiment and expect to get the same effect." Reproducibility is an issue because of study designs that are unusually to get accidental results. So you have your p value < .05, but the worry is that it's just an accident. If it's not an accident, then you should be able to run the exact same study again and get a similar effect. The problem of reproducibility is that even _that_ may not possible for some notable studies. Because of bad experimental design, we were just reporting meaningless results (the 5% chance that you get a nice p-value for an effect that isn't real). So in the case of this experiment, reproducibility is "if you survey people about bragging, will coders report their comments in the same way?" not "is bragging perceived negatively?" I think what you're describing is sometimes called a "conceptual reproduction" where you run a different study that you can argue supports the same conclusion. For instance, you've run an experiment where people seem to show a cognitive bias. Then someone else runs a different experiment, with a different design, and says "my results show the same cognitive bias that dbbolton showed in his study". In this case, the experimental design is different, and the second study doesn't directly demonstrate that you results could be reproduced. But in theory, it does suggest that there's a real phenomenon, because the same explanation seems natural for both experiments. There has been an overuse of conceptual reproductions in psychology, but what really has people worried is that major studies may literally not be reproducible. P.S. all of this is based on my understanding of how the terms are used, and I'm not 100% confident I'm right about it. I think I am, but I'm not a psychologist, take my opinion with a grain of salt.