4 ms·

> If someone cares enough about the work to build on it, they will replicate it anyway. Does it really deserve to be called work if it doesn't include the a fu

> If someone cares enough about the work to build on it, they will replicate it anyway. Does it really deserve to be called work if it doesn't include the a full, working set of instructions that if followed to a T allow it to be replicated? To me that's more like pollution, making it someone else's problem. I certainly don't see how "we did this, just trust us" can even be considered science, and that's not because I don't understand the scientific method, that's because I don't make a living with it, and have no incentive to not rock the boat.

davidktr 3y ago
You just described the majority of scientific papers. A "working set of instructions" is not really feasible in most cases. You can't include every piece of hard- and software required to replicate your own setup.
- johngladtj 3y ago
  You should.
- lliamander 3y ago
  Sounds like a problem worth solving.
- johnnyworker 3y ago
  Then don't call it science, since it doesn't contribute anything to the body of human knowledge. I think it's fascinating that we can at the same time hold things like "one is none" to be true, or that you should write tests first, but with science we already got so used to a lack of discipline that we just declare it fine. It's not hard to not climb a tower you can't get down from. It's the default, actually. You start with something small where you can describe everything that goes into replicating it. Then you replicate it yourself, based on your own instructions. Before that, you don't bother anyone else with it. Once that is done, and others can replicate as well, it "actually exists". And if that means the majority of stuff has to be thrown out, I'd suggest doing that sooner rather than later, instead of just accumulating scientific debt.
  - davidktr 3y ago
    Imagine two scientists, Bob and Alice. Bob has spent the last 5 years examining a theory thoroughly. Now he can explain down to the last detail why the theory does not hold water, and why generations of researchers have been wrong about the issue. Unfortunately, he cannot offer an alternative, and nobody else can follow his long winded arguments anyway. Meanwhile, Alice has spent the last 5 years making the best possible use of the flawed theory, and published a lot of original research. Sure, many of her publications are rubbish, but a few contain interesting results. Contrary to Bob, Alice can show actual results and has publications. Who do you believe will remain in academia? And, according to public perception, will seem more like an actual scientist?
    - johnnyworker 3y ago
      I disagree that Bob doesn't produce actual results, or that something that is mostly rubbish, but partly "interesting" is an actual result. We know the current incentives are all sorts of broken, across the board. Goodhart's law and all that. To me the question isn't who remains in academia given the current broken model, but who would remain in academia in one that isn't as broken. To put a point on it, if public distrust of science becomes big enough, it all can go away before you can say "cultural revolution" or "fascist strongman". Then there'd be no more academia, and its shell would be inhabited by party members, so to speak. I'd gladly sacrifice the ability of Alice and others like her to live off producing "mostly rubbish" to at least have a chance to save science itself.
    - tnecniv 3y ago
      Then Bob has failed. Academic science isn’t just the doing science part but the articulation and presentation of your work to the broader community. If Bob knows this space so well, he should be able to clearly communicate the issue and, ideally, present an easily understandable counter example to the existing theory. Technical folks undervalue presentation when writing articles and presenting at conferences. The burden of proof is on the presenter, and, unless there’s some incredible demonstration at the end, most researchers won’t have the time or attention to slog through your mess of a paper to decipher it. There’s only so much time in the day and too many papers to read. In my experience, the best researchers are also the best presenters. I’ve been to great talks out of my domain that I left feeling like I understood the importance of their work despite not understanding the details. I’ve also seen many talks in my field that I thought were awful because the presentation was convoluted or they didn’t motivate the importance of their problem / why their work addressed it
MrJohz 3y ago
I work with code, which is about as reproducible as it is possible to get - the artifacts I produce are literally just instructions on how to reproduce the work I've done again, and again, and again. And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine, despite the two environments being as identical as I can possibly make them. I agree that reproduction in scientific work is important, but it is also apparently impossible in the best possible circumstances. When dealing with physical materials, inexact measurements, margins of error, etc, I think we have to accept that there is no set of instructions that, if followed to a T, will ever ensure perfect replication.
- johnnyworker 3y ago
  > And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine But this is the other way around. Have you ever written a program that doesn't run anywhere except a single machine of yours? Would you release it and advertise it and encourage other people to use it as dependency in their software? If it only runs on one machine of yours, you don't even know if your code is doing something, or something else in the machine/OS. Or in terms of science, whether the research says something about the world, or just about the research setup.
  - MrJohz 3y ago
    I think you misunderstand the point of scientific publication here (at least in theory, perhaps less so in practice). The purpose of a paper is typically to say "I have achieved these results in this environment (as far as I can tell)", and encourages reproduction. But the original result is useful in its own right - it tells us that there may be something worth exploring. Yes, it may just be a measurement error (I remember the magic faster than light neutrinos), but if it is exciting enough, and lots of eyes end up looking, then flaws are typically found fairly quickly. And yes, there are often overly excited press releases that accompany it - the "advertise it and encourage others to us it as a dependency" part of it analogy - but this is typically just noise in the context of scientific research. If that is your main problem with scientific publishing, you may want to be more critical of science journalism instead. Fwiw, yes of course I've written code that only runs on my machine. I imagine everyone has, typically accidentally. You do it, you realise your mistake, you learn something from it. Which is exactly what we expect from scientific papers that can't be reproduced.

davidktr 3y ago

You just described the majority of scientific papers. A "working set of instructions" is not really feasible in most cases. You can't include every piece of hard- and software required to replicate your own setup.

johngladtj 3y ago
You should.
lliamander 3y ago
Sounds like a problem worth solving.
johnnyworker 3y ago
Then don't call it science, since it doesn't contribute anything to the body of human knowledge. I think it's fascinating that we can at the same time hold things like "one is none" to be true, or that you should write tests first, but with science we already got so used to a lack of discipline that we just declare it fine. It's not hard to not climb a tower you can't get down from. It's the default, actually. You start with something small where you can describe everything that goes into replicating it. Then you replicate it yourself, based on your own instructions. Before that, you don't bother anyone else with it. Once that is done, and others can replicate as well, it "actually exists". And if that means the majority of stuff has to be thrown out, I'd suggest doing that sooner rather than later, instead of just accumulating scientific debt.
- davidktr 3y ago
  Imagine two scientists, Bob and Alice. Bob has spent the last 5 years examining a theory thoroughly. Now he can explain down to the last detail why the theory does not hold water, and why generations of researchers have been wrong about the issue. Unfortunately, he cannot offer an alternative, and nobody else can follow his long winded arguments anyway. Meanwhile, Alice has spent the last 5 years making the best possible use of the flawed theory, and published a lot of original research. Sure, many of her publications are rubbish, but a few contain interesting results. Contrary to Bob, Alice can show actual results and has publications. Who do you believe will remain in academia? And, according to public perception, will seem more like an actual scientist?
  - johnnyworker 3y ago
    I disagree that Bob doesn't produce actual results, or that something that is mostly rubbish, but partly "interesting" is an actual result. We know the current incentives are all sorts of broken, across the board. Goodhart's law and all that. To me the question isn't who remains in academia given the current broken model, but who would remain in academia in one that isn't as broken. To put a point on it, if public distrust of science becomes big enough, it all can go away before you can say "cultural revolution" or "fascist strongman". Then there'd be no more academia, and its shell would be inhabited by party members, so to speak. I'd gladly sacrifice the ability of Alice and others like her to live off producing "mostly rubbish" to at least have a chance to save science itself.
  - tnecniv 3y ago
    Then Bob has failed. Academic science isn’t just the doing science part but the articulation and presentation of your work to the broader community. If Bob knows this space so well, he should be able to clearly communicate the issue and, ideally, present an easily understandable counter example to the existing theory. Technical folks undervalue presentation when writing articles and presenting at conferences. The burden of proof is on the presenter, and, unless there’s some incredible demonstration at the end, most researchers won’t have the time or attention to slog through your mess of a paper to decipher it. There’s only so much time in the day and too many papers to read. In my experience, the best researchers are also the best presenters. I’ve been to great talks out of my domain that I left feeling like I understood the importance of their work despite not understanding the details. I’ve also seen many talks in my field that I thought were awful because the presentation was convoluted or they didn’t motivate the importance of their problem / why their work addressed it

MrJohz 3y ago

I work with code, which is about as reproducible as it is possible to get - the artifacts I produce are literally just instructions on how to reproduce the work I've done again, and again, and again. And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine, despite the two environments being as identical as I can possibly make them. I agree that reproduction in scientific work is important, but it is also apparently impossible in the best possible circumstances. When dealing with physical materials, inexact measurements, margins of error, etc, I think we have to accept that there is no set of instructions that, if followed to a T, will ever ensure perfect replication.

johnnyworker 3y ago
> And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine But this is the other way around. Have you ever written a program that doesn't run anywhere except a single machine of yours? Would you release it and advertise it and encourage other people to use it as dependency in their software? If it only runs on one machine of yours, you don't even know if your code is doing something, or something else in the machine/OS. Or in terms of science, whether the research says something about the world, or just about the research setup.
- MrJohz 3y ago
  I think you misunderstand the point of scientific publication here (at least in theory, perhaps less so in practice). The purpose of a paper is typically to say "I have achieved these results in this environment (as far as I can tell)", and encourages reproduction. But the original result is useful in its own right - it tells us that there may be something worth exploring. Yes, it may just be a measurement error (I remember the magic faster than light neutrinos), but if it is exciting enough, and lots of eyes end up looking, then flaws are typically found fairly quickly. And yes, there are often overly excited press releases that accompany it - the "advertise it and encourage others to us it as a dependency" part of it analogy - but this is typically just noise in the context of scientific research. If that is your main problem with scientific publishing, you may want to be more critical of science journalism instead. Fwiw, yes of course I've written code that only runs on my machine. I imagine everyone has, typically accidentally. You do it, you realise your mistake, you learn something from it. Which is exactly what we expect from scientific papers that can't be reproduced.