7 ms·

> One major problem is that there's not much real incentive to make your work reproducible. Who is paying for non-reproducible papers? What’s the point?

by drb91 8y ago

roel_v 8y ago
No, the question is: who is paying for reproducible ones? What does it even mean, apart from in some abstract numerical disciplines?
- ajuc 8y ago
  > What does it even mean https://en.wikipedia.org/wiki/Reproducibility#Reproducible_research https://en.wikipedia.org/wiki/Reproducibility#Reproducible_r... The term reproducible research refers to the idea that the ultimate product of academic research is the paper along with the laboratory notebooks [12] and full computational environment used to produce the results in the paper such as the code, data, etc. that can be used to reproduce the results and create new work based on the research.[13][14][15][16][17] Typical examples of reproducible research comprise compendia of data, code and text files, often organised around an R Markdown source document[18] or a Jupyter notebook.[19] > abstract numerical disciplines Could you give an example of a non-numerical science?
  - esrauch 8y ago
    I think you too his question too literally: I think he meant "what does it even mean pragmatically and why do you expect funding to go to less novel but more easily reproducible results instead of more novel and harder to reproduce papers?"
    - thecatspaw 8y ago
      funding should go to more reproducible studies because those are the ones that are usefull. What use is a study saying that X will cure cancer Y if noone can reproduce that and verify?
    - drb91 8y ago
      Well, reproduced research is inherently more valuable than novel but unsupported research. Otherwise why fund research at all? Just hire alchemists or poets, at least you’ll be entertained.
      - esrauch 8y ago
        The current state of research seems to reflect that there is a balance: almost none of the most cited papers go really far out of theirs way to be trivially reproducible, even in computing where doing so is unusually much easier. If a paper is very novel and impactful but difficult to reproduce, it doesn't seem to matter much for citation counts as long as people just believe it to be true. And yet there's no revolution among funding sources to send money away from those and instead towards researchers who spend more time on it.
  - roel_v 8y ago
    Yes of course I know what the Wikipedia definition is. Have you ever tried to 'replicate' a paper? Let's say someone did a survey of something. How are you going to 'replicate' that? Or used a particular piece of equipment, or did some measurements in a field? Sure, it's easy to restrict yourself to the data files that go into some software, and say 'oh I used the same data set and software version, my research is 'replicatable'.' My point is that there are a lot of aspects to most research that aren't easy (or even at all) to replicate, even if one is put in all the effort required to document every minute circumstance. There is more to 'science' than CS algos and lab bench biomed. "Could you give an example of a non-numerical science?" This is the point you're probably going to argue about what is or is not 'science', so let me use 'scholarship' instead from the fields I work in: law, environmental science and geography. There is a lot of research that is not purely numerical in nature. Note that this doesn't mean that it doesn't use numbers; you misread what I wrote - I didn't say 'numerical disciplines', I said 'abstract numerical disciplines', by which I meant disciplines that require at least some judgement of qualitative properties or inexact measurements of things along the way, somewhere.
    - ajuc 8y ago
      > Have you ever tried to 'replicate' a paper? No, but I had spent a whole year at university trying to replicate experiments in physics lab. I was studying CS but they had too many physics teachers so we had many of the same courses :) The equipment was abused by decades of students, you had 45 minutes to do the experiment, and the results usually weren't even the correct order of magnitude :) It was stuff like measuring speed of sound, atmospheric pressure, gravity constant, etc. Still - at least I knew it's the problem with equipment or with me, not with some obscure details that weren't mentioned and that I can't replicate. > My point is that there are a lot of aspects to most research that aren't easy (or even at all) to replicate And my point is that it makes publishing everything that can be adjusted that much more important. Another question - when there's economic and proffesional incentive to fudge the numbers, and no way to check if the numbers were fudged - how do you trust the results? I wouldn't.
    - deleted 8y ago
      [deleted]
    - 8y ago
- drb91 8y ago
  Well, it means you have a higher confidence level, which is pretty much just standard induction. I’d assume people are paying for it who want high quality research.
perl4ever 8y ago
The Journal of Irreproducible Results[1], of course! [1]http://www.jir.com/ http://www.jir.com/

> One major problem is that there's not much real incentive to make your work reproducible. Who is paying for non-reproducible papers? What’s the point?

by drb91 8y ago

> One major problem is that there's not much real incentive to make your work reproducible. Who is paying for non-reproducible papers? What’s the point?

roel_v 8y ago

No, the question is: who is paying for reproducible ones? What does it even mean, apart from in some abstract numerical disciplines?

ajuc 8y ago
> What does it even mean https://en.wikipedia.org/wiki/Reproducibility#Reproducible_research https://en.wikipedia.org/wiki/Reproducibility#Reproducible_r... The term reproducible research refers to the idea that the ultimate product of academic research is the paper along with the laboratory notebooks [12] and full computational environment used to produce the results in the paper such as the code, data, etc. that can be used to reproduce the results and create new work based on the research.[13][14][15][16][17] Typical examples of reproducible research comprise compendia of data, code and text files, often organised around an R Markdown source document[18] or a Jupyter notebook.[19] > abstract numerical disciplines Could you give an example of a non-numerical science?
- esrauch 8y ago
  I think you too his question too literally: I think he meant "what does it even mean pragmatically and why do you expect funding to go to less novel but more easily reproducible results instead of more novel and harder to reproduce papers?"
  - thecatspaw 8y ago
    funding should go to more reproducible studies because those are the ones that are usefull. What use is a study saying that X will cure cancer Y if noone can reproduce that and verify?
  - drb91 8y ago
    Well, reproduced research is inherently more valuable than novel but unsupported research. Otherwise why fund research at all? Just hire alchemists or poets, at least you’ll be entertained.
    - esrauch 8y ago
      The current state of research seems to reflect that there is a balance: almost none of the most cited papers go really far out of theirs way to be trivially reproducible, even in computing where doing so is unusually much easier. If a paper is very novel and impactful but difficult to reproduce, it doesn't seem to matter much for citation counts as long as people just believe it to be true. And yet there's no revolution among funding sources to send money away from those and instead towards researchers who spend more time on it.
- roel_v 8y ago
  Yes of course I know what the Wikipedia definition is. Have you ever tried to 'replicate' a paper? Let's say someone did a survey of something. How are you going to 'replicate' that? Or used a particular piece of equipment, or did some measurements in a field? Sure, it's easy to restrict yourself to the data files that go into some software, and say 'oh I used the same data set and software version, my research is 'replicatable'.' My point is that there are a lot of aspects to most research that aren't easy (or even at all) to replicate, even if one is put in all the effort required to document every minute circumstance. There is more to 'science' than CS algos and lab bench biomed. "Could you give an example of a non-numerical science?" This is the point you're probably going to argue about what is or is not 'science', so let me use 'scholarship' instead from the fields I work in: law, environmental science and geography. There is a lot of research that is not purely numerical in nature. Note that this doesn't mean that it doesn't use numbers; you misread what I wrote - I didn't say 'numerical disciplines', I said 'abstract numerical disciplines', by which I meant disciplines that require at least some judgement of qualitative properties or inexact measurements of things along the way, somewhere.
  - ajuc 8y ago
    > Have you ever tried to 'replicate' a paper? No, but I had spent a whole year at university trying to replicate experiments in physics lab. I was studying CS but they had too many physics teachers so we had many of the same courses :) The equipment was abused by decades of students, you had 45 minutes to do the experiment, and the results usually weren't even the correct order of magnitude :) It was stuff like measuring speed of sound, atmospheric pressure, gravity constant, etc. Still - at least I knew it's the problem with equipment or with me, not with some obscure details that weren't mentioned and that I can't replicate. > My point is that there are a lot of aspects to most research that aren't easy (or even at all) to replicate And my point is that it makes publishing everything that can be adjusted that much more important. Another question - when there's economic and proffesional incentive to fudge the numbers, and no way to check if the numbers were fudged - how do you trust the results? I wouldn't.
  - deleted 8y ago
    [deleted]
  - 8y ago
drb91 8y ago
Well, it means you have a higher confidence level, which is pretty much just standard induction. I’d assume people are paying for it who want high quality research.

perl4ever 8y ago

The Journal of Irreproducible Results[1], of course! [1]http://www.jir.com/ http://www.jir.com/