5 ms·

Arxiv, github, and a self-publishing style platform that supports reproducible digital artifacts, i.e., the published paper. IIRC, many flawed papers were the r

Arxiv, github, and a self-publishing style platform that supports reproducible digital artifacts, i.e., the published paper. IIRC, many flawed papers were the result of data errors saved in a spreadsheet.

adamnemecek 6y ago
And better citations. I want to be able to link to a particular sentence of a particular version of a paper. Also pull requests.
- mattkrause 6y ago
  Could you explain what the sentence-level citation adds? I've heard several people ask for this, but never understood why. Most citation formats let you include page numbers; you can usually work in other location information ("See Foo et al. (2020)'s Figure 3A") too.
  - adamnemecek 6y ago
    Maybe but most people dont do it.
- dijksterhuis 6y ago
  > I want to be able to link to a particular sentence of a particular version of a paper. Probably best that you read through whole papers instead of looking for one sentence.
beagle3 6y ago
Would help in stuff that doesn't require physical measurements - e.g. computer science, neural network results; would not help in medicine, psychology, biology or chemistry, where reactions are reported by (often indirect) observation, and it is often the data that is fudged (or just made up) in retracted papers.
- hobofan 6y ago
  It would also help with those to some degree, though I feel like a lot of them could do a lot better with what's currently available. I'm currently studying biochemistry and have a few years of experience as a software engineer. In trying to dive into the papers in the field and just trying to replicate the data analysis, I came to see how bad the state of data and code availability is. It varies a lot between subfields, but overall the current state seems pretty abysmal.
- abdullahkhalids 6y ago
  Iirc, some fake data papers have been identified because the random errors on the data were not correct. Besides reporting is good because others can do alternate analysis on the data.
jhrmnn 6y ago
I fully agree and try to do that in my research, but it's very hard to do 100%. I just wrapped up a 1.5-year computational research project, and we have all the raw and processed data, the main code, the processing scripts, the notebook that generates the figures, etc. But it's still not a fully automated pipeline. The missing pieces: • Some older calculations were run with somewhat older versions of the code. Of course we believe that the results wouldn't change, and we recalculated some, but not all. We didn't keep track of exactly which version was used for which calculations, because that's simply very demanding in the middle of a complex research project. • Some data in the text and tables of the paper are still extracted manually from the code. We don't have a full templating system where the data could be automatically inserted into the paper. You could use something like Jinja to do it, but then every coauthor needs to have high technical skills and it's just time-consuming to maintain in general.
- haihaibye 6y ago
  > We didn't keep track of exactly which version was used for which calculations, because that's simply very demanding in the middle of a complex research project. Doing the bare necessities of science (recording methods for results) can be almost automated: Put all code in Git then in your script add something like: git rev-parse HEAD >> run.log
dijksterhuis 6y ago
ML researcher here -- many people do this already: https://github.com/carlini/adv-eval-paper https://github.com/carlini/adv-eval-paper Personally I try to put as much on GitHub as possible.

adamnemecek 6y ago

And better citations. I want to be able to link to a particular sentence of a particular version of a paper. Also pull requests.

mattkrause 6y ago
Could you explain what the sentence-level citation adds? I've heard several people ask for this, but never understood why. Most citation formats let you include page numbers; you can usually work in other location information ("See Foo et al. (2020)'s Figure 3A") too.
- adamnemecek 6y ago
  Maybe but most people dont do it.
dijksterhuis 6y ago
> I want to be able to link to a particular sentence of a particular version of a paper. Probably best that you read through whole papers instead of looking for one sentence.

beagle3 6y ago

Would help in stuff that doesn't require physical measurements - e.g. computer science, neural network results; would not help in medicine, psychology, biology or chemistry, where reactions are reported by (often indirect) observation, and it is often the data that is fudged (or just made up) in retracted papers.

hobofan 6y ago
It would also help with those to some degree, though I feel like a lot of them could do a lot better with what's currently available. I'm currently studying biochemistry and have a few years of experience as a software engineer. In trying to dive into the papers in the field and just trying to replicate the data analysis, I came to see how bad the state of data and code availability is. It varies a lot between subfields, but overall the current state seems pretty abysmal.
abdullahkhalids 6y ago
Iirc, some fake data papers have been identified because the random errors on the data were not correct. Besides reporting is good because others can do alternate analysis on the data.

jhrmnn 6y ago

I fully agree and try to do that in my research, but it's very hard to do 100%. I just wrapped up a 1.5-year computational research project, and we have all the raw and processed data, the main code, the processing scripts, the notebook that generates the figures, etc. But it's still not a fully automated pipeline. The missing pieces: • Some older calculations were run with somewhat older versions of the code. Of course we believe that the results wouldn't change, and we recalculated some, but not all. We didn't keep track of exactly which version was used for which calculations, because that's simply very demanding in the middle of a complex research project. • Some data in the text and tables of the paper are still extracted manually from the code. We don't have a full templating system where the data could be automatically inserted into the paper. You could use something like Jinja to do it, but then every coauthor needs to have high technical skills and it's just time-consuming to maintain in general.

haihaibye 6y ago
> We didn't keep track of exactly which version was used for which calculations, because that's simply very demanding in the middle of a complex research project. Doing the bare necessities of science (recording methods for results) can be almost automated: Put all code in Git then in your script add something like: git rev-parse HEAD >> run.log

dijksterhuis 6y ago

ML researcher here -- many people do this already: https://github.com/carlini/adv-eval-paper https://github.com/carlini/adv-eval-paper Personally I try to put as much on GitHub as possible.