4 ms·

What misconception, exactly? I understand that reproducibility is not the only factor in evaluating a study's weight, but it's still important, and still proble

What misconception, exactly? I understand that reproducibility is not the only factor in evaluating a study's weight, but it's still important, and still problematic in psychology. >Bad evidence may be perfectly reproducible. That was part of my point, namely that a weakly supported interpretation of subjective reports is probably a lot easier to reproduce than more empiric studies, but it does nothing to further the art. >finding out what people say when you ask them to remember X is a perfectly legitimate thing to collect data on, as long you recognize its limitations. I'm actually not arguing that subjective reports aren't legitimate data though. I think this study's methodology was also flawed due to the way it was structured. Group A was asked to describe an act of self-promotion of their choosing, and in a separate survey, Group B was asked to describe an instance of their choosing when someone had self-promoted to them. It's disjointed and lacking context. There is a distinct possibility that people in Group A chose to report a time when they felt that their bragging was socially acceptable and perhaps the people they bragged to were actually happy for them (e.g. health of their child). Likewise, I think the people in Group B were more likely to report a particularly obnoxious brag simply because that would be more memorable. A better (albeit admittedly more difficult to execute) study would involve investigating real-world and online instances of humblebragging, where there is direct relationship between promoter and recipient, and the subjects don't get to decide which experience they describe.

hyperpape 11y ago
The misconception that reproducibility has anything to do with the things that we can criticize about this study. That's the only way I can figure out to read your original post. If that's not your point, then we don't have anything in particular to disagree about, and my only advice would be to be more explicit about what point you're trying to make.
- dbbolton 11y ago
  I guess I can see how you got that impression from my reply to sarreph. My emphasis was on the "presumptive analysis" part, not the "self-reported anecdotes" part, but there is no way for anyone to know that since I didn't indicate it. I hope my first reply to you made my intended meaning clear because it's too late to edit now. In case not: reproducibility is a major concern with psychological studies in general. Drawing broad/subjective conclusions will make the results easier reproduce, but it also leaves more room for confirmation bias, so it's not a valid solution to the problem.
  - hyperpape 11y ago
    Interesting. After your last post, I thought we weren't disagreeing about anything, we'd just miscommunicated, but now I'm not sure. "Drawing broad/subjective conclusions will make the results easier reproduce." Not sure I can say anything more useful beyond what I said before. I just don't think this is true.
    - dbbolton 11y ago
      Well, I mean it can make reproducibility easier, not that it definitely will. Say- just for argument's sake- that another group of researchers did this same experiment but came to a different conclusion such as: "members of this demographic were more likely to describe a certain type of self-promotion as being positively-received, despite the fact that this type of self-promotion was consistently described as negative by the recipient group". Whenever anyone else repeated the experiment, it would seemingly be much easier to align their conclusion with that of the original experiment than with this imaginary example just by virtue of the fact that the original conclusion was so broad.
      - hyperpape 11y ago
        Ok, there is a real disagreement. As I understand it, reproducibility means "you can run this same experiment and expect to get the same effect." Reproducibility is an issue because of study designs that are unusually to get accidental results. So you have your p value < .05, but the worry is that it's just an accident. If it's not an accident, then you should be able to run the exact same study again and get a similar effect. The problem of reproducibility is that even _that_ may not possible for some notable studies. Because of bad experimental design, we were just reporting meaningless results (the 5% chance that you get a nice p-value for an effect that isn't real). So in the case of this experiment, reproducibility is "if you survey people about bragging, will coders report their comments in the same way?" not "is bragging perceived negatively?" I think what you're describing is sometimes called a "conceptual reproduction" where you run a different study that you can argue supports the same conclusion. For instance, you've run an experiment where people seem to show a cognitive bias. Then someone else runs a different experiment, with a different design, and says "my results show the same cognitive bias that dbbolton showed in his study". In this case, the experimental design is different, and the second study doesn't directly demonstrate that you results could be reproduced. But in theory, it does suggest that there's a real phenomenon, because the same explanation seems natural for both experiments. There has been an overuse of conceptual reproductions in psychology, but what really has people worried is that major studies may literally not be reproducible. P.S. all of this is based on my understanding of how the terms are used, and I'm not 100% confident I'm right about it. I think I am, but I'm not a psychologist, take my opinion with a grain of salt.

hyperpape 11y ago

The misconception that reproducibility has anything to do with the things that we can criticize about this study. That's the only way I can figure out to read your original post. If that's not your point, then we don't have anything in particular to disagree about, and my only advice would be to be more explicit about what point you're trying to make.

dbbolton 11y ago
I guess I can see how you got that impression from my reply to sarreph. My emphasis was on the "presumptive analysis" part, not the "self-reported anecdotes" part, but there is no way for anyone to know that since I didn't indicate it. I hope my first reply to you made my intended meaning clear because it's too late to edit now. In case not: reproducibility is a major concern with psychological studies in general. Drawing broad/subjective conclusions will make the results easier reproduce, but it also leaves more room for confirmation bias, so it's not a valid solution to the problem.
- hyperpape 11y ago
  Interesting. After your last post, I thought we weren't disagreeing about anything, we'd just miscommunicated, but now I'm not sure. "Drawing broad/subjective conclusions will make the results easier reproduce." Not sure I can say anything more useful beyond what I said before. I just don't think this is true.
  - dbbolton 11y ago
    Well, I mean it can make reproducibility easier, not that it definitely will. Say- just for argument's sake- that another group of researchers did this same experiment but came to a different conclusion such as: "members of this demographic were more likely to describe a certain type of self-promotion as being positively-received, despite the fact that this type of self-promotion was consistently described as negative by the recipient group". Whenever anyone else repeated the experiment, it would seemingly be much easier to align their conclusion with that of the original experiment than with this imaginary example just by virtue of the fact that the original conclusion was so broad.
    - hyperpape 11y ago
      Ok, there is a real disagreement. As I understand it, reproducibility means "you can run this same experiment and expect to get the same effect." Reproducibility is an issue because of study designs that are unusually to get accidental results. So you have your p value < .05, but the worry is that it's just an accident. If it's not an accident, then you should be able to run the exact same study again and get a similar effect. The problem of reproducibility is that even _that_ may not possible for some notable studies. Because of bad experimental design, we were just reporting meaningless results (the 5% chance that you get a nice p-value for an effect that isn't real). So in the case of this experiment, reproducibility is "if you survey people about bragging, will coders report their comments in the same way?" not "is bragging perceived negatively?" I think what you're describing is sometimes called a "conceptual reproduction" where you run a different study that you can argue supports the same conclusion. For instance, you've run an experiment where people seem to show a cognitive bias. Then someone else runs a different experiment, with a different design, and says "my results show the same cognitive bias that dbbolton showed in his study". In this case, the experimental design is different, and the second study doesn't directly demonstrate that you results could be reproduced. But in theory, it does suggest that there's a real phenomenon, because the same explanation seems natural for both experiments. There has been an overuse of conceptual reproductions in psychology, but what really has people worried is that major studies may literally not be reproducible. P.S. all of this is based on my understanding of how the terms are used, and I'm not 100% confident I'm right about it. I think I am, but I'm not a psychologist, take my opinion with a grain of salt.