3 ms·

"And the only reason it's coming out is because some people -- for whatever reason -- have a bug in their shorts about climate science" On the other hand, scie

"And the only reason it's coming out is because some people -- for whatever reason -- have a bug in their shorts about climate science" On the other hand, scientists have spent decades fully aware that they'll have to build their own labs, obtain their own equipment, get their own animals if the work requires them, and follow the described procedures. They can't use the other guy's mice or zebrafish or monkey. They probably can't use the other guy's lab. They might be able to use the same telescope for astronomy, but it's better to use a different telescope to control for some fluke of the original. So given all that, it hardly seems a huge deal to not be able to use someone else's code, as long as you know what the code was supposed to be doing.

jerf 16y ago
"So given all that, it hardly seems a huge deal to not be able to use someone else's code, as long as you know what the code was supposed to be doing." But that's a vacuous statement. The only description of what the code is doing is the code. The description you get in a paper is the goal of the code, but goals aren't results. (Mistaking stated goals for results is a surprisingly common systematic error, programmers make it all the time when they believe the hype a project puts out before the project actually has any results. Once you start looking for it it's hard to see a day go by without someone doing this.) I can make a simple information theoretic argument that it is blindingly obvious that any significant code base will have many more bits in it than any journal article could, so it is literally mathematically impossible for a journal article to accurately describe the code. If goals were the same as results all of our jobs would look very different. Moreover, the reason why you can't use the other guy's monkeys is purely physical. If they could use the other guy's monkeys, they would. They try as it is; the entire purpose of controlled breeding gene lines is to try to erase variations. If they could use guaranteed-atomically-identical lab hardware, they would. (The story of cold fusion would probably be very different if this were possible, for instance; instead of an effect that nobody could reliably replicate, there would be one set of results that everybody could replicate and probably rapidly explain. Note I'm not making a claim about cold fusion itself, just the way history turned out.) Don't elevate an accidental physical limitation to an essential component of science. Very few things hit the true ideal of science, and in general the closer you can get, the closer you should get.
- jonhendry 16y ago
  "If they could use the other guy's monkeys, they would. " No, they wouldn't, any more than a scientist would use only one monkey. Because animals vary. Hell, my Mom had two uteruses. If you don't look at different specimens, you don't know if what you observe is peculiar to the one specimen, or generally applicable. Using atomically identical hardware would be a bad idea, because then you wouldn't know for sure if your results are replicated because the experiment was good, or because of some quirk in the apparatus.
  - jerf 16y ago
    I only say it is desirable to be able to replicate results. Nowhere do I say that science must consist ONLY of replicating results. It actually is a bit weird to me that so many people leap to this conclusion when it is obviously falsified by other existing sciences where results absolutely can be exactly replicated and yet scientists do not deliberately discard that ability. Replication of results is merely a step in the process. Once you establish replicated results, you proceed from there. Maybe you observe that the results are irrelevant because your monkey is weird in some way, and you test it on another monkey to establish that point; others can then examine their copies and decide whether you've got a point. Maybe you build on the experiment with a standard test bed. Maybe you take apart the apparatus to demonstrate how a bit of impurity corrupted the results, and then everybody else can do the same disassembly. Maybe you shuffle in some new monkeys and hardware and run the test again to be sure. Not being able to precisely replicate results is a handicap that the various sciences proceeds through anyhow because it has no choice, not a desirable part of the process. Sure, there's a small amount of danger that you might overfit your results, but there's no guarantees anywhere; it's less than the danger that you face from non-replicable results. Not having replicability only throws away options, options that smart people could use to further their science, option that when missing can only slow progress down. That dumb people might misuse it really isn't a very interesting point. And again, I reiterate that to the extent possible animal researchers do in fact try their best to get animals identical as possible, so I give you not only the theory I outline above, but the practice of biology as well, where carefully controlled standardized gene lines are used.

jerf 16y ago

"So given all that, it hardly seems a huge deal to not be able to use someone else's code, as long as you know what the code was supposed to be doing." But that's a vacuous statement. The only description of what the code is doing is the code. The description you get in a paper is the goal of the code, but goals aren't results. (Mistaking stated goals for results is a surprisingly common systematic error, programmers make it all the time when they believe the hype a project puts out before the project actually has any results. Once you start looking for it it's hard to see a day go by without someone doing this.) I can make a simple information theoretic argument that it is blindingly obvious that any significant code base will have many more bits in it than any journal article could, so it is literally mathematically impossible for a journal article to accurately describe the code. If goals were the same as results all of our jobs would look very different. Moreover, the reason why you can't use the other guy's monkeys is purely physical. If they could use the other guy's monkeys, they would. They try as it is; the entire purpose of controlled breeding gene lines is to try to erase variations. If they could use guaranteed-atomically-identical lab hardware, they would. (The story of cold fusion would probably be very different if this were possible, for instance; instead of an effect that nobody could reliably replicate, there would be one set of results that everybody could replicate and probably rapidly explain. Note I'm not making a claim about cold fusion itself, just the way history turned out.) Don't elevate an accidental physical limitation to an essential component of science. Very few things hit the true ideal of science, and in general the closer you can get, the closer you should get.

jonhendry 16y ago
"If they could use the other guy's monkeys, they would. " No, they wouldn't, any more than a scientist would use only one monkey. Because animals vary. Hell, my Mom had two uteruses. If you don't look at different specimens, you don't know if what you observe is peculiar to the one specimen, or generally applicable. Using atomically identical hardware would be a bad idea, because then you wouldn't know for sure if your results are replicated because the experiment was good, or because of some quirk in the apparatus.
- jerf 16y ago
  I only say it is desirable to be able to replicate results. Nowhere do I say that science must consist ONLY of replicating results. It actually is a bit weird to me that so many people leap to this conclusion when it is obviously falsified by other existing sciences where results absolutely can be exactly replicated and yet scientists do not deliberately discard that ability. Replication of results is merely a step in the process. Once you establish replicated results, you proceed from there. Maybe you observe that the results are irrelevant because your monkey is weird in some way, and you test it on another monkey to establish that point; others can then examine their copies and decide whether you've got a point. Maybe you build on the experiment with a standard test bed. Maybe you take apart the apparatus to demonstrate how a bit of impurity corrupted the results, and then everybody else can do the same disassembly. Maybe you shuffle in some new monkeys and hardware and run the test again to be sure. Not being able to precisely replicate results is a handicap that the various sciences proceeds through anyhow because it has no choice, not a desirable part of the process. Sure, there's a small amount of danger that you might overfit your results, but there's no guarantees anywhere; it's less than the danger that you face from non-replicable results. Not having replicability only throws away options, options that smart people could use to further their science, option that when missing can only slow progress down. That dumb people might misuse it really isn't a very interesting point. And again, I reiterate that to the extent possible animal researchers do in fact try their best to get animals identical as possible, so I give you not only the theory I outline above, but the practice of biology as well, where carefully controlled standardized gene lines are used.