4 ms·

i had this problem many years ago (15?). at the time i was working as a postdoc, calculating the evolution of the ionizing background with redhshift from the i

i had this problem many years ago (15?). at the time i was working as a postdoc, calculating the evolution of the ionizing background with redhshift from the inverse effect (lyman alpha clouds near quasars get fried by the quasar; the extent of this gives an indirect way to measure the ionizing background at that redshift). i had a bunch of perl scripts (ah, those were the days) that mangled various files before feeding them into fortran least-squares stats code that took a day or so to run. by the end, it was pretty much chaos. i was a self-taught programmer, these were probably the second or third "significant" programs i had ever written. nothing was documented, everything took so long that i couldn't check much... i had bugs, of course. in the end i published. maybe 6 months later i got an email from someone in the states. they were trying to reproduce my results. in the end, they did (as far as i know). so the system worked. incidentally (perhaps the only useful point here) they must have used different data. that's something worth explaining in more detail - "my" data came from years of painstaking work by a bunch of people working for my thesis supervisor. yet 6 months later the results could be duplicated from a week or so of data from a much more powerful telescope (the keck). so data in research often aren't as critical as you might think. things progress at such a rate that even if you don't share data, it's trivial to reproduce just a short time later... (and i am pretty sure that this is true in gene sequencing, for example)

frossie 17y ago
But this is a problem that can be tackled, and those who take it seriouslt already do so. For example, in our pipelines we use an infrastructure that always adds every command executed on the file, with every exact parameter, to the metadata of the file, starting from one canonical archived file - and hence, one can indeed reproduce manually the result of the pipeline given sufficient time and dedication. [Edit: we also write the git shar1] The same way that say, biology labs have processes they engage in in order to convince us that their samples are not contaminated, we can have processes that lead to the reproducibility of data.
- andrewcooke 17y ago
  [is that frossie frossie? hi!] i'm not sure what problem you're talking about - someone reproduced the results with separate code and data, so what's to worry about? (i don't mean that because it was confirmed it was ok, but rather that if it had been wrong, we would have known in the end... after all, people make mistakes all the time - science is a collective enterprise that relies on many overlapping, interlocking pieces)
  - frossie 17y ago
    [Yeah, didn't I share an office with you 15 years ago? :-) ] But data can't always be reproduced - Shoemaker-Levy won't hit Jupiter again any time soon, and while big results will cause a rush for verification, the little steps are usually believed as is. Moreover astronomy is a special case where - let's face it - if we go down the wrong path for a decade or so, nobody's bleeding. Take on the other hand the processing of climate data - if crap engineering (not malice) causes garbage to come out of the data, this is a problem for everybody. So I think the OP is right in that proper auditing of computer-processed science data should be possible, I was just pointing out that it's not as hard (technically) as people think to achieve.

frossie 17y ago

But this is a problem that can be tackled, and those who take it seriouslt already do so. For example, in our pipelines we use an infrastructure that always adds every command executed on the file, with every exact parameter, to the metadata of the file, starting from one canonical archived file - and hence, one can indeed reproduce manually the result of the pipeline given sufficient time and dedication. [Edit: we also write the git shar1] The same way that say, biology labs have processes they engage in in order to convince us that their samples are not contaminated, we can have processes that lead to the reproducibility of data.

andrewcooke 17y ago
[is that frossie frossie? hi!] i'm not sure what problem you're talking about - someone reproduced the results with separate code and data, so what's to worry about? (i don't mean that because it was confirmed it was ok, but rather that if it had been wrong, we would have known in the end... after all, people make mistakes all the time - science is a collective enterprise that relies on many overlapping, interlocking pieces)
- frossie 17y ago
  [Yeah, didn't I share an office with you 15 years ago? :-) ] But data can't always be reproduced - Shoemaker-Levy won't hit Jupiter again any time soon, and while big results will cause a rush for verification, the little steps are usually believed as is. Moreover astronomy is a special case where - let's face it - if we go down the wrong path for a decade or so, nobody's bleeding. Take on the other hand the processing of climate data - if crap engineering (not malice) causes garbage to come out of the data, this is a problem for everybody. So I think the OP is right in that proper auditing of computer-processed science data should be possible, I was just pointing out that it's not as hard (technically) as people think to achieve.