4 ms·

Reproducibility is already a huge issue in modern psychological research, and this type of presumptive analysis of self-reported anecdotes isn't helping.

hyperpape 11y ago
This shows a misconception. Bad evidence may be perfectly reproducible. Being reproducible seems to mostly be an issue about statistics and experimental methodology, but your reproducible findings may not be good evidence for any deeper finding. Moreover, while I'm not impressed with the evidence of this study, finding out what people say when you ask them to remember X is a perfectly legitimate thing to collect data on, as long you recognize its limitations.
- dbbolton 11y ago
  What misconception, exactly? I understand that reproducibility is not the only factor in evaluating a study's weight, but it's still important, and still problematic in psychology. >Bad evidence may be perfectly reproducible. That was part of my point, namely that a weakly supported interpretation of subjective reports is probably a lot easier to reproduce than more empiric studies, but it does nothing to further the art. >finding out what people say when you ask them to remember X is a perfectly legitimate thing to collect data on, as long you recognize its limitations. I'm actually not arguing that subjective reports aren't legitimate data though. I think this study's methodology was also flawed due to the way it was structured. Group A was asked to describe an act of self-promotion of their choosing, and in a separate survey, Group B was asked to describe an instance of their choosing when someone had self-promoted to them. It's disjointed and lacking context. There is a distinct possibility that people in Group A chose to report a time when they felt that their bragging was socially acceptable and perhaps the people they bragged to were actually happy for them (e.g. health of their child). Likewise, I think the people in Group B were more likely to report a particularly obnoxious brag simply because that would be more memorable. A better (albeit admittedly more difficult to execute) study would involve investigating real-world and online instances of humblebragging, where there is direct relationship between promoter and recipient, and the subjects don't get to decide which experience they describe.
  - hyperpape 11y ago
    The misconception that reproducibility has anything to do with the things that we can criticize about this study. That's the only way I can figure out to read your original post. If that's not your point, then we don't have anything in particular to disagree about, and my only advice would be to be more explicit about what point you're trying to make.
    - dbbolton 11y ago
      I guess I can see how you got that impression from my reply to sarreph. My emphasis was on the "presumptive analysis" part, not the "self-reported anecdotes" part, but there is no way for anyone to know that since I didn't indicate it. I hope my first reply to you made my intended meaning clear because it's too late to edit now. In case not: reproducibility is a major concern with psychological studies in general. Drawing broad/subjective conclusions will make the results easier reproduce, but it also leaves more room for confirmation bias, so it's not a valid solution to the problem.
      - hyperpape 11y ago
        Interesting. After your last post, I thought we weren't disagreeing about anything, we'd just miscommunicated, but now I'm not sure. "Drawing broad/subjective conclusions will make the results easier reproduce." Not sure I can say anything more useful beyond what I said before. I just don't think this is true.
        dbbolton 11y ago
        Well, I mean it can make reproducibility easier, not that it definitely will. Say- just for argument's sake- that another group of researchers did this same experiment but came to a different conclusion such as: "members of this demographic were more likely to describe a certain type of self-promotion as being positively-received, despite the fact that this type of self-promotion was consistently described as negative by the recipient group". Whenever anyone else repeated the experiment, it would seemingly be much easier to align their conclusion with that of the original experiment than with this imaginary example just by virtue of the fact that the original conclusion was so broad.

hyperpape 11y ago

This shows a misconception. Bad evidence may be perfectly reproducible. Being reproducible seems to mostly be an issue about statistics and experimental methodology, but your reproducible findings may not be good evidence for any deeper finding. Moreover, while I'm not impressed with the evidence of this study, finding out what people say when you ask them to remember X is a perfectly legitimate thing to collect data on, as long you recognize its limitations.

dbbolton 11y ago
What misconception, exactly? I understand that reproducibility is not the only factor in evaluating a study's weight, but it's still important, and still problematic in psychology. >Bad evidence may be perfectly reproducible. That was part of my point, namely that a weakly supported interpretation of subjective reports is probably a lot easier to reproduce than more empiric studies, but it does nothing to further the art. >finding out what people say when you ask them to remember X is a perfectly legitimate thing to collect data on, as long you recognize its limitations. I'm actually not arguing that subjective reports aren't legitimate data though. I think this study's methodology was also flawed due to the way it was structured. Group A was asked to describe an act of self-promotion of their choosing, and in a separate survey, Group B was asked to describe an instance of their choosing when someone had self-promoted to them. It's disjointed and lacking context. There is a distinct possibility that people in Group A chose to report a time when they felt that their bragging was socially acceptable and perhaps the people they bragged to were actually happy for them (e.g. health of their child). Likewise, I think the people in Group B were more likely to report a particularly obnoxious brag simply because that would be more memorable. A better (albeit admittedly more difficult to execute) study would involve investigating real-world and online instances of humblebragging, where there is direct relationship between promoter and recipient, and the subjects don't get to decide which experience they describe.
- hyperpape 11y ago
  The misconception that reproducibility has anything to do with the things that we can criticize about this study. That's the only way I can figure out to read your original post. If that's not your point, then we don't have anything in particular to disagree about, and my only advice would be to be more explicit about what point you're trying to make.
  - dbbolton 11y ago
    I guess I can see how you got that impression from my reply to sarreph. My emphasis was on the "presumptive analysis" part, not the "self-reported anecdotes" part, but there is no way for anyone to know that since I didn't indicate it. I hope my first reply to you made my intended meaning clear because it's too late to edit now. In case not: reproducibility is a major concern with psychological studies in general. Drawing broad/subjective conclusions will make the results easier reproduce, but it also leaves more room for confirmation bias, so it's not a valid solution to the problem.
    - hyperpape 11y ago
      Interesting. After your last post, I thought we weren't disagreeing about anything, we'd just miscommunicated, but now I'm not sure. "Drawing broad/subjective conclusions will make the results easier reproduce." Not sure I can say anything more useful beyond what I said before. I just don't think this is true.
      - dbbolton 11y ago
        Well, I mean it can make reproducibility easier, not that it definitely will. Say- just for argument's sake- that another group of researchers did this same experiment but came to a different conclusion such as: "members of this demographic were more likely to describe a certain type of self-promotion as being positively-received, despite the fact that this type of self-promotion was consistently described as negative by the recipient group". Whenever anyone else repeated the experiment, it would seemingly be much easier to align their conclusion with that of the original experiment than with this imaginary example just by virtue of the fact that the original conclusion was so broad.