4 ms·

Hard to produce is not the same as hard to reproduce. Genuinely new results require developing new insights, new techniques or just making new guesses that hap

Hard to produce is not the same as hard to reproduce. Genuinely new results require developing new insights, new techniques or just making new guesses that happen to be right. All are rare enough. But once that is done, the information gets out and the experiments should be easily reproducible. Some dispciplines get a lot of reproduction as a side effect of using last year's breakthrough research as "mere" technology for implementing next year's expriment. But I guess that sort of thing is less natural pyschology and sociology.

PeterisP 9y ago
For many experiments, the cost (required effort) of reproducing them is the same as producing them, simply because you need to do all the same major activities once more. If they needed to spend an hour each with a hundred subjects, you need to do that too. If they needed to carefully run a long process on some materials, you need to do that to. If they needed to build a particular apparatus specific to that experiment, then you need to build or buy one. And it's possibly more costly than the original since the initial lab likely was able to re-use some of their existing resources, trained people, infrastructure, lab animals, equipment (whatever matters for the discipline) which you don't have; they likely chose this particular experiment because it fits well with the stuff that they have that's not available everywhere. If they developed a particular experimental setup / gear / network of volunteers doing interviews in the subpopulation you're studying / cell or animal line / etc, then they're using that for many publications over the years, but you need to spend the whole cost to reproduce any one of them.
- adrianratnapala 9y ago
  Good, point. I was once an experimental physicist and never psychologist. In physics there is a lot of trial and error before you can get the experiment to do anything at all. Anyone who reads your paper needs fewer trials. So I was going to agree with you that in fields where experiments work differently, reproduction can cost as much as the original study. But on second thought, that's wrong: and indeed it is the whole point of the thread. There is always trial-and-error. You simply don't know if your theory is correct. And there is always risk of a methodolical weakness that you only understand after seeing the data. The trouble is after you have already spent all your time and grant money on your study, you have every incentive to paper over these issues. Indeed even if you are scrupulously honest, you will have to shove study in the proverbial file drawer - because you don't have the resources to try another iteration. So I think it's still true that a new experiment costs more than reproduction, and many of the problems with studies arise from trying to pretend that cost doesn't exist.
- tome 9y ago
  The difference is that with a reproduction you know you are expected to succeed. It's hard to find a Roman gold coin in farmer's field but once you've found your first one you can be more confident that the second will be easier to come by.

PeterisP 9y ago

For many experiments, the cost (required effort) of reproducing them is the same as producing them, simply because you need to do all the same major activities once more. If they needed to spend an hour each with a hundred subjects, you need to do that too. If they needed to carefully run a long process on some materials, you need to do that to. If they needed to build a particular apparatus specific to that experiment, then you need to build or buy one. And it's possibly more costly than the original since the initial lab likely was able to re-use some of their existing resources, trained people, infrastructure, lab animals, equipment (whatever matters for the discipline) which you don't have; they likely chose this particular experiment because it fits well with the stuff that they have that's not available everywhere. If they developed a particular experimental setup / gear / network of volunteers doing interviews in the subpopulation you're studying / cell or animal line / etc, then they're using that for many publications over the years, but you need to spend the whole cost to reproduce any one of them.

adrianratnapala 9y ago
Good, point. I was once an experimental physicist and never psychologist. In physics there is a lot of trial and error before you can get the experiment to do anything at all. Anyone who reads your paper needs fewer trials. So I was going to agree with you that in fields where experiments work differently, reproduction can cost as much as the original study. But on second thought, that's wrong: and indeed it is the whole point of the thread. There is always trial-and-error. You simply don't know if your theory is correct. And there is always risk of a methodolical weakness that you only understand after seeing the data. The trouble is after you have already spent all your time and grant money on your study, you have every incentive to paper over these issues. Indeed even if you are scrupulously honest, you will have to shove study in the proverbial file drawer - because you don't have the resources to try another iteration. So I think it's still true that a new experiment costs more than reproduction, and many of the problems with studies arise from trying to pretend that cost doesn't exist.
tome 9y ago
The difference is that with a reproduction you know you are expected to succeed. It's hard to find a Roman gold coin in farmer's field but once you've found your first one you can be more confident that the second will be easier to come by.