6 ms·

He is a scientist; unless you're also a domain expert in his field with at least as much expertise as him, or a newcomer with radically impressive work, you can

by tedks 12y ago

He is a scientist; unless you're also a domain expert in his field with at least as much expertise as him, or a newcomer with radically impressive work, you can't just not call him a scientist because he disagrees with you. If a domain expert disagrees with you, it is probably because you are wrong. That's just the base rate and it's probably true here as well. Traditionally, replication hasn't been done in psychology because the experiments have to be set up in really clever complicated ways in order to tease out effects. Even the most dedicated replicator can't ever fully replicate a psychology experiment. You'd have to ship the actors used as confederates across the country to do that and of course nobody does that. It's foolish to think any psych study has ever been replicated. The way psychologists deal with this is the concept of converging operations. This has to do more with being clever and considering implications of a thing you're trying to study with respect to other theories you have much more evidence for and in different situations. If you think stereotype threat is a thing, for example, it would make sense for stereotype boost to exist if the stereotype is positive. If a priming effect speeds up the recognition of "doctor" after a subject has been shown "nurse", maybe recognition for "warmth" will be slower after a subject has been shown "frozen". If people only encode based on schema variables, they'll be unable to remember information unrelated to a schema even if they're provided with congruent encoding cues. Converging operations is just studying the same concept from different angles and seeing if you keep seeing consistent results. If the results aren't consistent the hypothesis you're investigating will, in fact, die. Psychology is an advancing science as of right now. If people were just publishing random noise there'd be no reason for concepts to fall out of favor. I think people on hacker news tend to wildly underestimate the degree of skill involved in experimental psychology. So much of statistics, even, has come out of psychology because of the need to refine and clarify what effect you're seeing. Alternatively, if you think because of this existing academic psychology is lacking, you're in a fantastic place to be. Do literally anything to get some seed money, just working in a standard SDE job would be fine, and then run some small experiments (and replicate them, of course) on learning or addiction. Then write the next Farmville or build the next Facebook. If there's really that much low-hanging fruit, go out and pluck it. Put up or shut up.

yummyfajitas 12y ago
Traditionally, replication hasn't been done in psychology because the experiments have to be set up in really clever complicated ways in order to tease out effects. Even the most dedicated replicator can't ever fully replicate a psychology experiment. You'd have to ship the actors used as confederates across the country to do that and of course nobody does that. Now wait a second. When a psychology study comes out, it claims "This experiment shows stereotype thread reduces performance of Honduran men on a Math test." Such studies rarely claim "This experiment shows stereotype threat reduces the performance of Honduran men on a Math test when Jill the skinny experimenter repeats the code words." If using different experimenters yields a different result, it means the effect being described is probably not as robust as the experimenters claim. The cause might be Jill's shifty eyes rather than stereotype threat. That's a replication failure, in the sense that it shows the claimed effect is far weaker (or causally different) than the original study claimed.
- tedks 12y ago
  I would say that it's typically assumed that there are confounding factors in any experiment. An experimenter has to try to minimize that, but you're still not going to be able to replicate a full experiment. If you have more than one researcher administering a test in a stereotype threat experiment, that'll reduce the likelihood of that being the confound. If other studies (not replications, but their own experiments, with their own, different approaches to study the problem) agree with the first experiment, the effect is likely real. I would also say that you're viewing the entire system far too antagonistically. Nobody goes into academic psychology to get rich off of it.
  - droopyEyelids 12y ago
    But if you're not minimizing the confounding factors enough that the result can be repeated, your experiment failed. Your hypothesis has neither been provably confirmed nor denied. At that point you are, at best, still gathering evidence, and publishing results would be an error.
pbhjpbhj 12y ago

by tedks 12y ago

yummyfajitas 12y ago

Traditionally, replication hasn't been done in psychology because the experiments have to be set up in really clever complicated ways in order to tease out effects. Even the most dedicated replicator can't ever fully replicate a psychology experiment. You'd have to ship the actors used as confederates across the country to do that and of course nobody does that. Now wait a second. When a psychology study comes out, it claims "This experiment shows stereotype thread reduces performance of Honduran men on a Math test." Such studies rarely claim "This experiment shows stereotype threat reduces the performance of Honduran men on a Math test when Jill the skinny experimenter repeats the code words." If using different experimenters yields a different result, it means the effect being described is probably not as robust as the experimenters claim. The cause might be Jill's shifty eyes rather than stereotype threat. That's a replication failure, in the sense that it shows the claimed effect is far weaker (or causally different) than the original study claimed.

tedks 12y ago
I would say that it's typically assumed that there are confounding factors in any experiment. An experimenter has to try to minimize that, but you're still not going to be able to replicate a full experiment. If you have more than one researcher administering a test in a stereotype threat experiment, that'll reduce the likelihood of that being the confound. If other studies (not replications, but their own experiments, with their own, different approaches to study the problem) agree with the first experiment, the effect is likely real. I would also say that you're viewing the entire system far too antagonistically. Nobody goes into academic psychology to get rich off of it.
- droopyEyelids 12y ago
  But if you're not minimizing the confounding factors enough that the result can be repeated, your experiment failed. Your hypothesis has neither been provably confirmed nor denied. At that point you are, at best, still gathering evidence, and publishing results would be an error.

pbhjpbhj 12y ago