4 ms·

One of the areas paid journals could make the case for their value and relevance would be in demonstrating reproducability.

qudat 8y ago
Requiring reproducibility would radically change the academic landscape for scientists. I'm sure in the end it would be for the greater good but I think most career researchers in the US would hate this kind of change.
- roel_v 8y ago
  It would require either all researchers to be 25% software engineers, or to add 30-50% headcount by adding SE's. It would be a radical change in methods and funding. There is massive institutional pushback to moving in this direction, as people already feel too much is done in/by computers. To make any research to be publishable depend on software engineering technicalities... OTOH, it would be awesome for me, as I've been doing consulting on exactly this for 15+ years :)
  - ajuc 8y ago
    Is it better to get the correct answers 25% slower, or incorrect answers at 100% speed? How do you do science if you can't repeat experiments?
    - roel_v 8y ago
      shrug What does 'the correct answer' even mean for any real world problem? There is so much we don't know. I'm not saying we should just make up papers and call it good, but there is a gradient between 'making stuff up' and 'doing everything according to a hypothetical ideal methodology all the time'. It's a matter of opinion how much not having a 100% repeatable experiment for each paper moves us towards the left of that gradient. It would be an improvement, yes, but I would put it more in the 'moves us from 75 to 80' ballpark than in the 'moves us from 20 to 100' that most people in this thread seem to believe. "How do you do science if you can't repeat experiments?" Well, the way we've been doing it for 300 years? It's not like the flawed processes of the past didn't yield anything. To put it in operations research terms: science doesn't work like hill climbing, it's more like a stochastic optimization process without a (well defined) halting condition. Again, I'm not saying we can't and shouldn't strive to improve, but the first year grad student level huff puffery in this thread is just completely detached from reality. 'Science' as you were explained the concept in high school is only a high-level abstraction of the concept, it's not how actual things are or get done in the real world.
      - your-nanny 8y ago
        "first year grad student level huff puffery" personally I find the worriers overstating the problem of reproducibility. lots of results are reproduced all the time, but in the course of testing other hypotheses. if a result depends on exact replication of methods it isn't that robust an effect, and just might be a trivial one.
  - ahartmetz 8y ago
    I have worked on experimental high-energy physics with computers and had the same idea - physicists should just hire software engineers. Large amounts of physicist time are spent cobbling together, and suffering from, shoddy software. At least we had public repositories with version numbers, and datasets were shared and uniquely named.
    - your-nanny 8y ago
      that's my job!
deleted 8y ago
[deleted]

qudat 8y ago

Requiring reproducibility would radically change the academic landscape for scientists. I'm sure in the end it would be for the greater good but I think most career researchers in the US would hate this kind of change.

roel_v 8y ago
It would require either all researchers to be 25% software engineers, or to add 30-50% headcount by adding SE's. It would be a radical change in methods and funding. There is massive institutional pushback to moving in this direction, as people already feel too much is done in/by computers. To make any research to be publishable depend on software engineering technicalities... OTOH, it would be awesome for me, as I've been doing consulting on exactly this for 15+ years :)
- ajuc 8y ago
  Is it better to get the correct answers 25% slower, or incorrect answers at 100% speed? How do you do science if you can't repeat experiments?
  - roel_v 8y ago
    shrug What does 'the correct answer' even mean for any real world problem? There is so much we don't know. I'm not saying we should just make up papers and call it good, but there is a gradient between 'making stuff up' and 'doing everything according to a hypothetical ideal methodology all the time'. It's a matter of opinion how much not having a 100% repeatable experiment for each paper moves us towards the left of that gradient. It would be an improvement, yes, but I would put it more in the 'moves us from 75 to 80' ballpark than in the 'moves us from 20 to 100' that most people in this thread seem to believe. "How do you do science if you can't repeat experiments?" Well, the way we've been doing it for 300 years? It's not like the flawed processes of the past didn't yield anything. To put it in operations research terms: science doesn't work like hill climbing, it's more like a stochastic optimization process without a (well defined) halting condition. Again, I'm not saying we can't and shouldn't strive to improve, but the first year grad student level huff puffery in this thread is just completely detached from reality. 'Science' as you were explained the concept in high school is only a high-level abstraction of the concept, it's not how actual things are or get done in the real world.
    - your-nanny 8y ago
      "first year grad student level huff puffery" personally I find the worriers overstating the problem of reproducibility. lots of results are reproduced all the time, but in the course of testing other hypotheses. if a result depends on exact replication of methods it isn't that robust an effect, and just might be a trivial one.
- ahartmetz 8y ago
  I have worked on experimental high-energy physics with computers and had the same idea - physicists should just hire software engineers. Large amounts of physicist time are spent cobbling together, and suffering from, shoddy software. At least we had public repositories with version numbers, and datasets were shared and uniquely named.
  - your-nanny 8y ago
    that's my job!

deleted 8y ago

[deleted]