3 ms·

I've seen discussions about replication a lot. Isn't it more often than not the case that replication is made expensive because the paper describing it simply l

I've seen discussions about replication a lot. Isn't it more often than not the case that replication is made expensive because the paper describing it simply leaves out too many details, making replication more of a guessing exercise than actual, well, replication?

ChuckMcM 10y ago
"...the paper describing it simply leaves out too many details," ... and that would be because the patent hasn't been issued yet.
semi-extrinsic 10y ago
Well, yes and no. Yes, to a certain extent papers don't have enough details for replication, but this is mainly because you just can't fit enough details into the scope of a journal paper. In a PhD thesis, you should be able to. And no, it's not that replication becomes a guessing exercise, but it's the fact that the time spent setting up, debugging and then running a replication of even a perfectly described benchtop-style experiment costs much much more than the actual equipment involved. And the ROI is poor, since replication studies are hard to publish.
- btilly 10y ago
  Should, sure. But there can be critical details needed to replicate which the experimenter didn't realize were important and didn't think to record. And possibly didn't notice. One of the reasons why replication matters is that it is a way of discovering these factors which matter but weren't obvious.
epistasis 10y ago
There's also skill. Its not as simple as running the same shell script twice, this time on your lab's cluster.
FooBarWidget 10y ago
So ask the original researchers for more details? What is the problem here?
- IshKebab 10y ago
  The problem is that they probably don't want people to be able to easily replicate it, if it really works. It's the same reason compsci papers rarely include source code.
  - FooBarWidget 10y ago
    If they don't want people to easily replicate then why do they publish at all? Doesn't it defeat the scientific process? If it's about protection of the invention then why not patent it?

ChuckMcM 10y ago

"...the paper describing it simply leaves out too many details," ... and that would be because the patent hasn't been issued yet.

semi-extrinsic 10y ago

Well, yes and no. Yes, to a certain extent papers don't have enough details for replication, but this is mainly because you just can't fit enough details into the scope of a journal paper. In a PhD thesis, you should be able to. And no, it's not that replication becomes a guessing exercise, but it's the fact that the time spent setting up, debugging and then running a replication of even a perfectly described benchtop-style experiment costs much much more than the actual equipment involved. And the ROI is poor, since replication studies are hard to publish.

btilly 10y ago
Should, sure. But there can be critical details needed to replicate which the experimenter didn't realize were important and didn't think to record. And possibly didn't notice. One of the reasons why replication matters is that it is a way of discovering these factors which matter but weren't obvious.

epistasis 10y ago

There's also skill. Its not as simple as running the same shell script twice, this time on your lab's cluster.

FooBarWidget 10y ago

So ask the original researchers for more details? What is the problem here?

IshKebab 10y ago
The problem is that they probably don't want people to be able to easily replicate it, if it really works. It's the same reason compsci papers rarely include source code.
- FooBarWidget 10y ago
  If they don't want people to easily replicate then why do they publish at all? Doesn't it defeat the scientific process? If it's about protection of the invention then why not patent it?