5 ms·
I did my Ph.D. in a field requiring complex computer simulations. This article, like most climate "skeptic" critiques, is a logical fallacy writ large. A kern
by timr 17y ago
I did my Ph.D. in a field requiring complex computer simulations. This article, like most climate "skeptic" critiques, is a logical fallacy writ large. A kernel of truth, surrounded by massive layers of exaggeration and misinterpretation of fact.
First, the notion that anyone could "peer-review" software for correctness is obviously absurd. We know this is an impossible standard. We accept that software is a (probably buggy) model, and (in my experience) reviewers are therefore extremely skeptical of results from computer models -- to the point that they're actually biased against publication. And since reviewers know that they can't verify correctness, they tend to look for theoretical papers that very closely reproduce experimental results.
Second, this type of critique is pushing hard on a straw man. The implicit assumption is that each paper is a stand-alone morsel of truth, and that the peer-review process somehow guarantees a paper's claims. This is not the case, and no scientist believes it to be true. It doesn't matter that peer review fails sometimes, because the system doesn't depend on it being flawless.
Science is about process, not people. Peer review is an important part of that process, but it's openly acknowledged that review has flaws. Thus, ask a scientist about any particular paper, and while s/he may be enthusiastic about the result or the theory, you'll almost never hear a scientist take the paper for granted as fact. More often than not, discussion of an interesting paper will center around designing other tests that can independently reproduce the paper's result. Ultimately, you have a system where a lot of genuinely smart, skeptical people are attacking ideas from all sides until either a consensus emerges, or the idea is destroyed.
The "skeptics" have it wrong, not because they're incorrect about human nature, but because they're attacking a fictional vision of science that exists only in their minds. They imagine a world revolving around a theory that is supported by one piece of data, extrapolated by buggy computer models, and presented by corrupt individuals who (for some unknown reason) have an incentive to doctor the results in the same way. In reality, science is massively redundant and competitive, computer models are treated as ancillary data (at best), and individual corruption is exposed by the redundancy of the system. If the "skeptics" put as much effort into understanding the science they critique as they do trying to find "smoking guns" for individual researchers, they would understand this dynamic.
- jeremyw 17y agoMembers of all fields of science are calling for open data and source code, so I read a political message in your opposition (and focus on climatology). As for correctness, the article doesn't use that CS term and I haven't seen anybody ask for that impossible standard (in the strict CS sense.) Find flaws earlier. That's the goal.
- gjm11 17y agoso I read a political message in your opposition (and focus on climatology): Well, here's the very first sentence of the article he's commenting on: "Recent revelations that the peer review system in climatology might have been compromised by the biases of corrupt reviewers miss a much bigger problem." If focusing on climatology is evidence of a "political message", then I think you've fingered the wrong suspect.
- jeremyw 17y agoAgain, the calls for open access come from scientific fields broadly (where it already isn't happening voluntarily). If you're going to refute that on HN, where the audience isn't pitchfork-wielding, science-averse troglodytes, then you're probably making a narrow argument targeting a specific opposition, i.e. political.
- gjm11 17y agoI don't understand. The person you replied to didn't object to open access. He objected to this particular article, which (as it happens) says other things besides "open access would be a good thing". The article suggests that part of the peer review process should be an examination of the software used, which is about as practical as saying that an experimental paper's reviewers should visit the lab and examine all the apparatus. (Actually, considerably less practical, as anyone who's ever done any sort of review on any nontrivial piece of software knows.) This is not the same as saying that there should be "open access". The article opens by citing an alleged integrity problem in climatology and saying that lack of access to the relevant software is an even bigger problem. This is not the same as saying that there should be "open access". The article has a very clear political agenda, as you can note from its beginning (climate-specific stuff, which you yourself have suggested taking as diagnostic for political agenda) and its end (approving links to Eric "famous computer scientist" Raymond and TCS Daily). timr is (1) pointing out the political agenda, for which you accuse him of being political, and (2) describing some specific problems he has with the article, which you characterize as "refuting calls for open access" even though his complaints do not at all take that form. (I think it would be wonderful if every scientific publication were accompanied by every byte of code and data used in the work it describes, or at least the nearest thing to that that's legally and commercially feasible. But it wouldn't cure the problems that the article professes to find in the peer review process, and the only difference it would make to arguments over climate science is that the no-anthropogenic-global-warming people would find something else to complain about.)