7 ms·

Every PhD student should, in order to obtain PhD, not only do his original research but reproduce another published result. This will make PhD harder to get, so

Every PhD student should, in order to obtain PhD, not only do his original research but reproduce another published result. This will make PhD harder to get, so PhD’s will become more valuable (and more vetted), and we will have people verify published studies.

metacritic12 4y ago
Great idea! Reproducing is also a great way to understand the field, and can actually help you a lot with your own research and techniques. This also ensures the most famous results are the most replicated. Hugely valuable to society since the most famous results also drive the most thinking / decisions / policies.
eganist 4y ago
Wow, I can't think of anything practically wrong with this. I can think of plenty of reasons why it won't happen (notably the downward pressure on the candidate pipeline, which for a variety of reasons isn't an outcome research institutions don't want), but I'm wondering if the best place to apply the pressure would be the accreditation institutions and the Department of Education.
- MonkeyMalarky 4y ago
  Might want a restriction like research from an unaffiliated institution / research group. I don't think reproducing results from someone in the same lab has much value.
  - eganist 4y ago
    Ah, that's a good idea. And it pushes a more adversarial dynamic that might motivate people to debunk eyebrow-raising conclusions from competing institutions.
- watwut 4y ago
  One problem is that PhDs are already overworked and effectively taken advantage of. So, something would need to be removed from stuff they already do.
maze-le 4y ago
We (as a society) might start with assigning more value to reproduction in general. There is an implicit assumption that reproduction and verification (or falsification) of published results is somehow not worthy of time and attention.
logifail 4y ago
> Every PhD student should, in order to obtain PhD, not only do his original research but reproduce another published result Speaking for my own PhD (now around 25 years ago): it's hard enough getting one's own damned work to reproduce, never mind the work of a complete stranger.
- CodeSgt 4y ago
  I think that's the point
  - lemursage 4y ago
    I think the problem here is that articles and publications in general do not provide enough detail to reproduce the results. Even though, that is the exact reason you write an article -- to make research reproducible. Peer-reviewed journals should, and they progressively try to be more strict about it, enforce the necessary level of detail and open-sourcing of methods. Reproduction in science is a though problem -- not only it requires specialist lab equipment, methods or procedures, but sometimes "a trick" is needed. Inconveniently, it may not be mentioned in the publication for whatever reason. Therefore, giving an unexperienced PhD a task of reproducing somebody's work may end up to be a futile effort. But the idea is neat in general, I think.
    - utucuro 4y ago
      Today, I'd say reproduction of results is far more important than getting more research published. I would even go so far as to say that peer review is worthless, it may well be time to replace it with peer reproduction as a prerequisite for having an article published.
- taylorius 4y ago
  If you can't reproduce it, that is also a valuable result.
_flux 4y ago
I'll one-up this idea! So for each publication, you'd need to provide two serious attempts of reproducing other published results. You would make the reproductions yourself, but also anyone else could do them for you (here's where PhD students come in), but once used for a publication, they'd be "spent tokens". (No, I'm not going to even mention NFT. ..wait a minute, crap..) From here it would follow that eventually all results would be reproduced (or failed to do so), until we eventually run out of results to reproduce and would need to shift the burden from two reproductions down to one—probably go from 2 to 1 in some stochastic manner. (Maybe also similar number of serious peer reviews of such reproductions would be required to keep the system running.) Maybe we could make a new currency out of this. Actually, it sort of automatically does that by itself. I foresee no issues in this whatsover.
bongoman37 4y ago
blackbear_ 4y ago
Why make it a PhD requirement? You'd end up with half-assed attempts to reproduce easy results and little mafia rings of my-reproduction-for-your-reproduction. Much better to make reproducibility a valuable research contribution in and of its own and let "the market" take care of it. Create better incentives for all authors. For example, make it possible to submit the code for analysis alongside an article and highlight articles who do (similar to the open-access movement), create journals dedicated to reproducibility, measure successful reproduction attempts alongside citations of each article, etc.
- Supermancho 4y ago
  > Much better to make reproducibility a valuable research contribution in and of its own and let "the market" take care of it. "the market" is the same force that has led to the perverse incentives of constantly pushing new "discoveries" above validations. The small variations you have prescribed have been tried (to various degrees) and have not had an effect on aggregate behavior. > Why make it a PhD requirement? You'd end up with half-assed attempts to reproduce easy results and little mafia rings of my-reproduction-for-your-reproduction. The why is obvious. The purpose of a PHD is for the furthering of knowledge, so it's a perpetual source of motivation. > You'd end up with half-assed attempts to reproduce easy results and little mafia rings of my-reproduction-for-your-reproduction. That's how it might end up (largely based on the rigor of the PHD program), but it's better than what we have, which is no good incentive to reproduce results.
RC_ITR 4y ago
There is no replication problem here. It's a matter of opinion, mostly. I'm assuming since the article is paywalled, most people aren't reading and instead applying their own biases. The challenging researchers are simply implying that the original study did not account for the positive effects of red meat (vaguely 'vitamins and minerals).
BiteCode_dev 4y ago
"attempt to reproduce" then. Because a lot of those studies cannot be reproduced, that's the point.
suyula 4y ago
You've just passed your viva-- now all that's left to do is eat this marshmallow, or wait 10 minutes and get two marshmallows.

metacritic12 4y ago

Great idea! Reproducing is also a great way to understand the field, and can actually help you a lot with your own research and techniques. This also ensures the most famous results are the most replicated. Hugely valuable to society since the most famous results also drive the most thinking / decisions / policies.

eganist 4y ago

Wow, I can't think of anything practically wrong with this. I can think of plenty of reasons why it won't happen (notably the downward pressure on the candidate pipeline, which for a variety of reasons isn't an outcome research institutions don't want), but I'm wondering if the best place to apply the pressure would be the accreditation institutions and the Department of Education.

MonkeyMalarky 4y ago
Might want a restriction like research from an unaffiliated institution / research group. I don't think reproducing results from someone in the same lab has much value.
- eganist 4y ago
  Ah, that's a good idea. And it pushes a more adversarial dynamic that might motivate people to debunk eyebrow-raising conclusions from competing institutions.
watwut 4y ago
One problem is that PhDs are already overworked and effectively taken advantage of. So, something would need to be removed from stuff they already do.

maze-le 4y ago

We (as a society) might start with assigning more value to reproduction in general. There is an implicit assumption that reproduction and verification (or falsification) of published results is somehow not worthy of time and attention.

logifail 4y ago

> Every PhD student should, in order to obtain PhD, not only do his original research but reproduce another published result Speaking for my own PhD (now around 25 years ago): it's hard enough getting one's own damned work to reproduce, never mind the work of a complete stranger.

CodeSgt 4y ago
I think that's the point
- lemursage 4y ago
  I think the problem here is that articles and publications in general do not provide enough detail to reproduce the results. Even though, that is the exact reason you write an article -- to make research reproducible. Peer-reviewed journals should, and they progressively try to be more strict about it, enforce the necessary level of detail and open-sourcing of methods. Reproduction in science is a though problem -- not only it requires specialist lab equipment, methods or procedures, but sometimes "a trick" is needed. Inconveniently, it may not be mentioned in the publication for whatever reason. Therefore, giving an unexperienced PhD a task of reproducing somebody's work may end up to be a futile effort. But the idea is neat in general, I think.
  - utucuro 4y ago
    Today, I'd say reproduction of results is far more important than getting more research published. I would even go so far as to say that peer review is worthless, it may well be time to replace it with peer reproduction as a prerequisite for having an article published.
taylorius 4y ago
If you can't reproduce it, that is also a valuable result.

_flux 4y ago

I'll one-up this idea! So for each publication, you'd need to provide two serious attempts of reproducing other published results. You would make the reproductions yourself, but also anyone else could do them for you (here's where PhD students come in), but once used for a publication, they'd be "spent tokens". (No, I'm not going to even mention NFT. ..wait a minute, crap..) From here it would follow that eventually all results would be reproduced (or failed to do so), until we eventually run out of results to reproduce and would need to shift the burden from two reproductions down to one—probably go from 2 to 1 in some stochastic manner. (Maybe also similar number of serious peer reviews of such reproductions would be required to keep the system running.) Maybe we could make a new currency out of this. Actually, it sort of automatically does that by itself. I foresee no issues in this whatsover.

bongoman37 4y ago

blackbear_ 4y ago

Why make it a PhD requirement? You'd end up with half-assed attempts to reproduce easy results and little mafia rings of my-reproduction-for-your-reproduction. Much better to make reproducibility a valuable research contribution in and of its own and let "the market" take care of it. Create better incentives for all authors. For example, make it possible to submit the code for analysis alongside an article and highlight articles who do (similar to the open-access movement), create journals dedicated to reproducibility, measure successful reproduction attempts alongside citations of each article, etc.

Supermancho 4y ago
> Much better to make reproducibility a valuable research contribution in and of its own and let "the market" take care of it. "the market" is the same force that has led to the perverse incentives of constantly pushing new "discoveries" above validations. The small variations you have prescribed have been tried (to various degrees) and have not had an effect on aggregate behavior. > Why make it a PhD requirement? You'd end up with half-assed attempts to reproduce easy results and little mafia rings of my-reproduction-for-your-reproduction. The why is obvious. The purpose of a PHD is for the furthering of knowledge, so it's a perpetual source of motivation. > You'd end up with half-assed attempts to reproduce easy results and little mafia rings of my-reproduction-for-your-reproduction. That's how it might end up (largely based on the rigor of the PHD program), but it's better than what we have, which is no good incentive to reproduce results.

RC_ITR 4y ago

There is no replication problem here. It's a matter of opinion, mostly. I'm assuming since the article is paywalled, most people aren't reading and instead applying their own biases. The challenging researchers are simply implying that the original study did not account for the positive effects of red meat (vaguely 'vitamins and minerals).

BiteCode_dev 4y ago

"attempt to reproduce" then. Because a lot of those studies cannot be reproduced, that's the point.

suyula 4y ago

You've just passed your viva-- now all that's left to do is eat this marshmallow, or wait 10 minutes and get two marshmallows.