5 ms·

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every profe

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those absolutely stunning results were just intelligently crafted experiments that are not applicable elsewhere. Basically, my whole thesis is just to show that some paper was wrong and the idea is not applicable in another experiment.

chrisfinazzo 8y ago
Not surprised to hear this at all. I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'. In social sciences, this is an astoundingly common problem. Then, of course, when I have to defend my comp. exam (for a second time) in front of the faculty, I spent the last 5 minutes lamenting the very phenomenon you described. Research can be deeply engrossing, but it doesn't mean shit when you go out into the world looking to land your next job. Of my 3 reviewers, only 1 had spent significant time in industry before teaching so, think of this as a wake up call for people to get outside the bubble and broaden their horizons once in a while. Tl;dr The program I was (Communications) in changed their requirements right after I left to eliminate this distinction and now students are required to develop and implement an idea (Marketing campaign, Training seminar, Comms. Audit w/ recommendations, etc.). You might get, 'that's nice' or even 'interesting', but this is usually followed up with 'how can you help me solve problems x and y right now on deadline?'
- nonbel 8y ago
  >"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).
  - chrisfinazzo 8y ago
    If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong. @epistasis' using the term 'p-hack' is incredibly apt in this case. The ability for a study to be deemed "correct" in proving a hypothesis might be true of small samples if you are sufficiently rigorous in designing the experiment, but it could still fall apart at larger sizes - which only proves that part of your hypothesis was not statistically meaningful regardless of the conclusion you reached. Again, a thorough discussion of limitations around any research should address this. E.g, "Our findings indicate x, but there is no way to know without further study if y is actually true instead"
    - nonbel 8y ago
      >"If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect?" Not quite sure what you are trying to say but this doesnt sound like any statistical model I've seen used in social science. Usually they assume some distribution (eg normal, or "t") and then assume both samples are taken from that distribution. From that they derive the proportion of times you would see the observed difference in means between samples.
      - chrisfinazzo 8y ago
        It's really the exceptional case I'm thinking of here. Basically, you screw up such that the data you've collected is meaningless. Sure, you could do the statistical analysis correctly and get a result, but it ends up being the opposite not because the relationship is actually different, but because some part of your design is flawed. In that case, you would never end up with a correct result. (This is of course the worst possible outcome, but I would be remiss if I didn't mention it here.) Although I have no background in it, I suppose the same could happen in the other sciences if there are side effects that you don't anticipate and don't measure accurately enough. ^ This again is why reproducibility is important. Earlier in this thread, @majidazmi mentioned that a major focus of their thesis was to prove that a prior experiment was incorrect in its findings.
wil3 8y ago
To be fair, the professors are like this because the funding agencies---ie, the government and a handful of private groups---essentially refuse to fund duplication research. I thus think that the issue has broader origins than the attitudes of individual faculty.
- jpeloquin 8y ago
  The review panels that score grants at the National Institutes of Health (NIH) are comprised of professors. (Mostly; being a professor isn't technically required.) If a proposal is scored well by the review panel, it is almost always funded. The National Science Foundation (NSF) works similarly. If (a) the review panels were on board with duplication and (b) people started submitting proposals for duplication research, those proposals would probably get funded. Edit to clarify: The point is that professors, as a group, already have the power to allocate funds to duplication research if they wished to do so. Blame for a lack of duplication research cannot be pinned solely on the funding agencies.
chriskanan 8y ago
I often have my PhD students re-implement ideas to gain skills and to verify a method works. It is possible to publish these efforts, but it isn't easy. It typically involves comparing multiple methods on datasets they haven't been tested on before to see how well the results generalize beyond the original paper. It is hard to publish in prestigious venues with this approach, but we have had some success. Replication and comarison makes for a good MS or early PhD project, but later PhD students have to showcase an ability to create new algorithms. In my lab, we try very hard to not cherry pick and to do good work that generalizes.
- edhu2017 8y ago
  Agreed, reimplementation is pretty good for students to learn. Our lab requires everybody to work on an implementation project
- gus_massa 8y ago
  From your profile, you are working in Deep Learning. In other fields it's much more difficult to be sure that the replication is accurate. Do you have the correct variety of rats? Are they receiving the same kind of food? Does they get the same illumination during the day? ... Theoretically al the details should be clear from the published paper, but most of the times the paper is full of "underspecified"[0] parts. Also, in many fields even if the paper is only about calculations in a computer, the programs are not published (or are a mess (or an unpublished mess)) and the data are not published (or are a mess (or an unpublished mess)). So it's more difficult to even make a direct copy of the results of the paper. Also, there is a lot of informal replication, essentially what you do but without the final publication step. Just replicate somewhat similar to the original paper, but then publish a version with some extension or tweak. [0] aka "missing"
- PinkMilkshake 8y ago
  > Replication and comparison makes for a good MS or early PhD project Do you think that might be the solution? That is, to get an MS your final project has to be an attempt at replication, and then a PhD has to be a new contribution. If that became the standard, would it solve a large part of this whole issue?
tomrod 8y ago
Why not arxiv?
- chrisseaton 8y ago
  Why not arXiv... what? Why not arXiv pay people to do reproductions? They don't have anything like the funding required to do that.
  - tomrod 8y ago
    Why not host reproductions in arXiv to build up a name, similar to how devs use github to build a profile.
    - chrisseaton 8y ago
      The problem isn't literally hosting the reproduction study result PDF files when you're done - you could do that on GitHub or anywhere else nobody cares. The problem is funding the time and resources needed to do the reproduction studies themselves. > The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct.
baxtr 8y ago
Yes, that’s true. It’s also because the view is: published == valid. An idea I had once was to say: the group that repeats or falsifies the results of a paper will be published in the same journal. So this motivate people to piggy back on high profile publications. Unsure if it’ll work though
- chrisfinazzo 8y ago
  The idea that just because it was published makes it valid is insane on so many levels. Usually, it is required - or at least good sense - to address limitations and draw conclusions based on your findings. This assertion flies in the face of that practice.
stinos 8y ago
The first paragraph reads a bit like a possibly faulty generalization based on your personal experience as covered by the rest of the post. I might be wrong, but I don't think it's that bad. Depends on the field maybe, and I don't read enough papers myself to have a proper complete view on this, but I definitely recall reading papers which where about recreating another experiment, figuring out that it was correct and adding some more steps to gain additional insight.
raarts 8y ago
And me thinking that science isn't about trying to confirm findings, but trying to disprove them and if nobody can do that, the original hypothesis must be true.
chrisseaton 8y ago
I think it's pretty common to ask PhD students to reproduce some results as part of their initial literature review.

chrisfinazzo 8y ago

Not surprised to hear this at all. I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'. In social sciences, this is an astoundingly common problem. Then, of course, when I have to defend my comp. exam (for a second time) in front of the faculty, I spent the last 5 minutes lamenting the very phenomenon you described. Research can be deeply engrossing, but it doesn't mean shit when you go out into the world looking to land your next job. Of my 3 reviewers, only 1 had spent significant time in industry before teaching so, think of this as a wake up call for people to get outside the bubble and broaden their horizons once in a while. Tl;dr The program I was (Communications) in changed their requirements right after I left to eliminate this distinction and now students are required to develop and implement an idea (Marketing campaign, Training seminar, Comms. Audit w/ recommendations, etc.). You might get, 'that's nice' or even 'interesting', but this is usually followed up with 'how can you help me solve problems x and y right now on deadline?'

nonbel 8y ago
>"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).
- chrisfinazzo 8y ago
  If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong. @epistasis' using the term 'p-hack' is incredibly apt in this case. The ability for a study to be deemed "correct" in proving a hypothesis might be true of small samples if you are sufficiently rigorous in designing the experiment, but it could still fall apart at larger sizes - which only proves that part of your hypothesis was not statistically meaningful regardless of the conclusion you reached. Again, a thorough discussion of limitations around any research should address this. E.g, "Our findings indicate x, but there is no way to know without further study if y is actually true instead"
  - nonbel 8y ago
    >"If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect?" Not quite sure what you are trying to say but this doesnt sound like any statistical model I've seen used in social science. Usually they assume some distribution (eg normal, or "t") and then assume both samples are taken from that distribution. From that they derive the proportion of times you would see the observed difference in means between samples.
    - chrisfinazzo 8y ago
      It's really the exceptional case I'm thinking of here. Basically, you screw up such that the data you've collected is meaningless. Sure, you could do the statistical analysis correctly and get a result, but it ends up being the opposite not because the relationship is actually different, but because some part of your design is flawed. In that case, you would never end up with a correct result. (This is of course the worst possible outcome, but I would be remiss if I didn't mention it here.) Although I have no background in it, I suppose the same could happen in the other sciences if there are side effects that you don't anticipate and don't measure accurately enough. ^ This again is why reproducibility is important. Earlier in this thread, @majidazmi mentioned that a major focus of their thesis was to prove that a prior experiment was incorrect in its findings.

wil3 8y ago

To be fair, the professors are like this because the funding agencies---ie, the government and a handful of private groups---essentially refuse to fund duplication research. I thus think that the issue has broader origins than the attitudes of individual faculty.

jpeloquin 8y ago
The review panels that score grants at the National Institutes of Health (NIH) are comprised of professors. (Mostly; being a professor isn't technically required.) If a proposal is scored well by the review panel, it is almost always funded. The National Science Foundation (NSF) works similarly. If (a) the review panels were on board with duplication and (b) people started submitting proposals for duplication research, those proposals would probably get funded. Edit to clarify: The point is that professors, as a group, already have the power to allocate funds to duplication research if they wished to do so. Blame for a lack of duplication research cannot be pinned solely on the funding agencies.

chriskanan 8y ago

I often have my PhD students re-implement ideas to gain skills and to verify a method works. It is possible to publish these efforts, but it isn't easy. It typically involves comparing multiple methods on datasets they haven't been tested on before to see how well the results generalize beyond the original paper. It is hard to publish in prestigious venues with this approach, but we have had some success. Replication and comarison makes for a good MS or early PhD project, but later PhD students have to showcase an ability to create new algorithms. In my lab, we try very hard to not cherry pick and to do good work that generalizes.

edhu2017 8y ago
Agreed, reimplementation is pretty good for students to learn. Our lab requires everybody to work on an implementation project
gus_massa 8y ago
From your profile, you are working in Deep Learning. In other fields it's much more difficult to be sure that the replication is accurate. Do you have the correct variety of rats? Are they receiving the same kind of food? Does they get the same illumination during the day? ... Theoretically al the details should be clear from the published paper, but most of the times the paper is full of "underspecified"[0] parts. Also, in many fields even if the paper is only about calculations in a computer, the programs are not published (or are a mess (or an unpublished mess)) and the data are not published (or are a mess (or an unpublished mess)). So it's more difficult to even make a direct copy of the results of the paper. Also, there is a lot of informal replication, essentially what you do but without the final publication step. Just replicate somewhat similar to the original paper, but then publish a version with some extension or tweak. [0] aka "missing"
PinkMilkshake 8y ago
> Replication and comparison makes for a good MS or early PhD project Do you think that might be the solution? That is, to get an MS your final project has to be an attempt at replication, and then a PhD has to be a new contribution. If that became the standard, would it solve a large part of this whole issue?

tomrod 8y ago

Why not arxiv?

chrisseaton 8y ago
Why not arXiv... what? Why not arXiv pay people to do reproductions? They don't have anything like the funding required to do that.
- tomrod 8y ago
  Why not host reproductions in arXiv to build up a name, similar to how devs use github to build a profile.
  - chrisseaton 8y ago
    The problem isn't literally hosting the reproduction study result PDF files when you're done - you could do that on GitHub or anywhere else nobody cares. The problem is funding the time and resources needed to do the reproduction studies themselves. > The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct.

baxtr 8y ago

Yes, that’s true. It’s also because the view is: published == valid. An idea I had once was to say: the group that repeats or falsifies the results of a paper will be published in the same journal. So this motivate people to piggy back on high profile publications. Unsure if it’ll work though

chrisfinazzo 8y ago
The idea that just because it was published makes it valid is insane on so many levels. Usually, it is required - or at least good sense - to address limitations and draw conclusions based on your findings. This assertion flies in the face of that practice.

stinos 8y ago

The first paragraph reads a bit like a possibly faulty generalization based on your personal experience as covered by the rest of the post. I might be wrong, but I don't think it's that bad. Depends on the field maybe, and I don't read enough papers myself to have a proper complete view on this, but I definitely recall reading papers which where about recreating another experiment, figuring out that it was correct and adding some more steps to gain additional insight.

raarts 8y ago

And me thinking that science isn't about trying to confirm findings, but trying to disprove them and if nobody can do that, the original hypothesis must be true.

chrisseaton 8y ago

I think it's pretty common to ask PhD students to reproduce some results as part of their initial literature review.