7 ms·

Eric Raymond on "Hiding the Decline, Part 1"

72 points by skmurphy 17y ago

skmurphy 17y ago
opening paragraph comments on a code fragment: From the CRU code file osborn-tree6/briffa_sep98_d.pro , used to prepare a graph purported to be of Northern Hemisphere temperatures and reconstructions. ; ; Apply a VERY ARTIFICAL correction for decline!! ; yrloc=[1400,findgen(19)*5.+1904] valadj= [0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,$ 2.5,2.6,2.6,2.6,2.6,2.6]*0.75 ; fudge factor if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’ ; yearlyadj=interpol(valadj,yrloc,timey) This, people, is blatant data-cooking, with no pretense otherwise. It flattens a period of warm temperatures in the 1930s--see those negative coefficients? Then, later on, it applies a positive multiplier so you get a nice dramatic hockey stick at the end of the century.
- indy 17y ago
  jgrahamc mentioned in an earlier comment (and also on his blog: http://www.jgc.org/blog/2009/11/about-that-cru-hack.html http://www.jgc.org/blog/2009/11/about-that-cru-hack.html) that these variables aren't used.
  - waterlesscloud 17y ago
    Great, so now all we need is to run this in the released data and verify if the graph was produced with the code commented in or out. If there was released data, that is. The point being we only know how the code looks in that file, we don't know when the commented out sections were commented out. So nothing is proven here either way.
  - defen 17y ago
    Check out harris-tree/briffa_sep98_e.pro - http://di2.nu/foia/harris-tree/briffa_sep98_e.pro http://di2.nu/foia/harris-tree/briffa_sep98_e.pro You'll see that it is used in that version of the program.
  - skmurphy 17y ago
    They are used in the next version, see http://di2.nu/foia/harris-tree/briffa_sep98_e.pro http://di2.nu/foia/harris-tree/briffa_sep98_e.pro (promoted from a comment by jgrahamc on his blog which I would think should get added to his main blog post).
- etal 17y ago
  Notes: 1. The fudge factor is flat after 1970. The hockey stick we care about is after 1980 or so, sharply increasing in the 1990s and 2000s. 2. The fudge corresponds to problems with recent tree-ring data that were discussed in previously published literature (I don't have the citation). The phrase "very artificial" almost certainly means it's a crude estimate based on some model that's simpler than the author would prefer -- hence, the flat line for recent values. 3. If this was an open-source project, you'd see a reference to the article that explains where the fudge came from, but it's not. Natural scientists tend to not really believe in software engineering, they just write a procedure and munge it until it works, leaving dead code and mysterious comments in their wake. (Source control? That sounds like fluffy industry stuff...) As JCG pointed out, yearlyadj isn't used in the copy of this script that we have, although it might have been used in an earlier draft of the script.
  - yummyfajitas 17y ago
    Natural scientists tend to not really believe in software engineering, they just write a procedure and munge it until it works, leaving dead code and mysterious comments in their wake. (Source control? That sounds like fluffy industry stuff...) Not the ones I've worked with. While they may be a little behind the times (CVS instead of hg, fortran instead of c), most academics aren't stupid. Even by academic standards, this code is fugly.
    - etal 17y ago
      Physics and math software is usually fine, sure, since the performance and reliability of the code matters in the experiments. I mean traditional biology and the less computer-oriented sciences, where they tend to just use scripts to filter data into a usable form, crunch some numbers, and generate visualizations. There, unless the software is the point of the research, writing code doesn't get the same priority (or respect) as traditional experiments. A lot of those professors just don't care about writing maintainable code; I'm guessing Dr. Briffa is one of those.
martincmartin 17y ago
As The Economist points out: "The 'hiding' concerned the decision to leave out a set of tree-ring-growth data that had stopped reflecting local temperature changes. That alteration in growth pattern is strange, and unexplained, but eliminating it is not sinister." and: " 'I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) and from 1961 for Keith’s to hide the decline.' Trickery associated with Dr Mann was catnip to the sceptics. But Dr Jones has clarified that 'The word trick was used here colloquially as in a clever thing to do. It is ludicrous to suggest that it refers to anything untoward.' " http://www.economist.com/opinion/displaystory.cfm?story_id=14960149 http://www.economist.com/opinion/displaystory.cfm?story_id=1...
- greendestiny 17y ago
  If the tree-ring growth data doesn't match directly measured temperature at significant intervals of time (necessitating what amounts to arbitrarily changing data sources mid graph) why on earth would you trust it for periods where its the only data source. I think the evidence for AGW extends beyond this particular incident, but I think its highly unethical and very poor science.
  - jsz0 17y ago
    Was it the only data source in other time periods? or was it collaborated by other data?
    - greendestiny 17y ago
      No idea, reasonable questions, but I don't see much redeeming about simply swapping data sets mid-graph.
      - tensor 17y ago
        That's a fair criticism, but it would be worth seeing the figure caption for it as well. If the caption had within it that the source was switched part way due to a known problem, that would be fine. Out of curiosity, anyone know a journal reference for that particular graph?
jacoblyles 17y ago
While I'm glad people are digging through the leaked source code, I doubt they find anything so blatant as outright fraud.
j_baker 17y ago
I'm not necessarily attempting to justify any data-cooking whatsoever, but why is it such a big deal? I mean, the climate "skeptics" do it all the time and it hardly makes the news. We're never going to have a decent public discourse on this issue until both sides are honest with the public. Maybe I'm just a pessimist, but I don't see this happening any time soon.
- DomesticMouse 17y ago
  Science that relies on cooked data isn't science. It's snake oil sales.
- rick888 17y ago
  "I'm not necessarily attempting to justify any data-cooking whatsoever, but why is it such a big deal? I mean, the climate "skeptics" do it all the time and it hardly makes the news." Are you serious? Laws are going to be passed (cap and trade) in the US that heavily regulate businesses based on data from these organizations. If congress doesn't take a step back and look into this, they are even more corrupt than I thought. Al Gore got a noble peace prize based on this and he has made millions through his carbon credit programs (and speeches, etc). The truth needs to be known. If global warming is not a result of man-made pollution (or if temperatures are not actually increasing or decreasing at a dangerous level), we need to know.
alex_c 17y ago
The code is pretty damning, but before I get really outraged, a bit of context is always useful. Has anyone figured out where this graph has been used?
- jgrahamc 17y ago
  No. And, also, the interpolated value yearlyadj isn't used. It's only referenced in commented out code. http://www.jgc.org/blog/2009/11/about-that-cru-hack.html http://www.jgc.org/blog/2009/11/about-that-cru-hack.html
hop 17y ago
How do all the "best" climate prediction models have squiggly lines after 10 years from now and not a broad range or something realistic. Their algorithims smell of BS and way too many variables go into it to make them have anywhere near their plotted accuracy for 100 year projections. From wikpedia: http://en.wikipedia.org/wiki/File:Global_Warming_Predictions.png http://en.wikipedia.org/wiki/File:Global_Warming_Predictions...
- petercooper 17y ago
  I'd be interested in that too. I wonder if they include things like predictable levels of solar activity, volcano eruptions, weather cycles and the like to get this. Or, worse, it might even be stochastic or "random walk"-like which would make the variations meaningless (see diagram on http://en.wikipedia.org/wiki/Random_walk http://en.wikipedia.org/wiki/Random_walk )..
- camccann 17y ago
  Well, the graph you link to doesn't include error bars or any other indication of uncertainty. Leaving out error margins and plotting only the mean is unfortunately common when presenting scientific results to general audiences; but that doesn't mean the actual scientists don't have the full data.
- azgolfer 17y ago
  Anyone who says they have a good computer model of the climate is either ignorant, stupid, corrupt or some combination thereof.
ieure 17y ago
I don't know what ESR's politics looked like before, but he's been in deep teabagger territory for a while now. I would take anything he says on the subject of politics with… Well, I'd just not take it. He can pretend to be objective, but the use of winger slang (“AGW” and “MSM” are right-wing dog whistles in the same way as “Democrat party” or “socialism”) is a giveaway. Here are some of his prognostications from the election last year. They're delightfully wingnutty, with a crunchy glibertarian shell: “The Obama campaign smells of defeat”: http://esr.ibiblio.org/?p=449 http://esr.ibiblio.org/?p=449 “Democrats’ polling league in a generic contest has collapsed. Control of Congress is in play. Wow. Just wow. Considering how hard the Republicans have been fucking up and alienating their base, this is astonishing. EPIC FAIL.” “Sarah Palin, American Centrist”: http://esr.ibiblio.org/?p=481 http://esr.ibiblio.org/?p=481 “So when I tell you that I think Sarah Palin’s religious position is pretty near dead center in the American spectrum, you can be pretty sure I’m not fudging to make that position look good.” “The Obama campaign smells of defeat: II”: http://esr.ibiblio.org/?p=490 http://esr.ibiblio.org/?p=490 “The attempt to smear, discredit and delegitimize Palin has steadily become more intense and more damaging — to the Democrats.” When it comes to politics, dude simply does not know what the fuck.
- javert 17y ago
  You're just attacking ESR's character. It's fallacious to discount his blog post just because you don't like his character. Please, can't we have a rational discussion on the topic?
  - ieure 17y ago
    No, what I’m doing is pointing out that his opinions are a) wrong and b) delivered with right-wing spin. As such, you should not take what he says about matters political at face value.
    - davidmathers 17y ago
      I don't know what ESR's politics looked like before Before what? I don't remember there ever being a time when esr wasn't a deranged wingnut.
yters 17y ago
So maybe we haven't found a smoking gun yet, but the general gist of the documents and emails in that package is not very encouraging. You can find reviews of IPCC's book that are full of very strident critiques. You can find emails discussing the inherent inaccuracy of their work. You can find internal dismissal of the tree ring data, stating how useless it is. You can find a scientist openly glad that he's not being held accountable so he can push his climate interests. After spending a couple hours pouring through the repository, I haven't found much that makes this work seem stable enough to base global lifechanging legislation on.
Joeboy 17y ago
Can anybody tell me what, if any, published work this data has made it into?

72 points by skmurphy 17y ago

opening paragraph comments on a code fragment: From the CRU code file osborn-tree6/briffa_sep98_d.pro , used to prepare a graph purported to be of Northern Hemisphere temperatures and reconstructions. ; ; Apply a VERY ARTIFICAL correction for decline!! ; yrloc=[1400,findgen(19)*5.+1904] valadj= [0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,$ 2.5,2.6,2.6,2.6,2.6,2.6]*0.75 ; fudge factor if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’ ; yearlyadj=interpol(valadj,yrloc,timey) This, people, is blatant data-cooking, with no pretense otherwise. It flattens a period of warm temperatures in the 1930s--see those negative coefficients? Then, later on, it applies a positive multiplier so you get a nice dramatic hockey stick at the end of the century.

indy 17y ago
jgrahamc mentioned in an earlier comment (and also on his blog: http://www.jgc.org/blog/2009/11/about-that-cru-hack.html http://www.jgc.org/blog/2009/11/about-that-cru-hack.html) that these variables aren't used.
- waterlesscloud 17y ago
  Great, so now all we need is to run this in the released data and verify if the graph was produced with the code commented in or out. If there was released data, that is. The point being we only know how the code looks in that file, we don't know when the commented out sections were commented out. So nothing is proven here either way.
- defen 17y ago
  Check out harris-tree/briffa_sep98_e.pro - http://di2.nu/foia/harris-tree/briffa_sep98_e.pro http://di2.nu/foia/harris-tree/briffa_sep98_e.pro You'll see that it is used in that version of the program.
- skmurphy 17y ago
  They are used in the next version, see http://di2.nu/foia/harris-tree/briffa_sep98_e.pro http://di2.nu/foia/harris-tree/briffa_sep98_e.pro (promoted from a comment by jgrahamc on his blog which I would think should get added to his main blog post).
etal 17y ago
Notes: 1. The fudge factor is flat after 1970. The hockey stick we care about is after 1980 or so, sharply increasing in the 1990s and 2000s. 2. The fudge corresponds to problems with recent tree-ring data that were discussed in previously published literature (I don't have the citation). The phrase "very artificial" almost certainly means it's a crude estimate based on some model that's simpler than the author would prefer -- hence, the flat line for recent values. 3. If this was an open-source project, you'd see a reference to the article that explains where the fudge came from, but it's not. Natural scientists tend to not really believe in software engineering, they just write a procedure and munge it until it works, leaving dead code and mysterious comments in their wake. (Source control? That sounds like fluffy industry stuff...) As JCG pointed out, yearlyadj isn't used in the copy of this script that we have, although it might have been used in an earlier draft of the script.
- yummyfajitas 17y ago
  Natural scientists tend to not really believe in software engineering, they just write a procedure and munge it until it works, leaving dead code and mysterious comments in their wake. (Source control? That sounds like fluffy industry stuff...) Not the ones I've worked with. While they may be a little behind the times (CVS instead of hg, fortran instead of c), most academics aren't stupid. Even by academic standards, this code is fugly.
  - etal 17y ago
    Physics and math software is usually fine, sure, since the performance and reliability of the code matters in the experiments. I mean traditional biology and the less computer-oriented sciences, where they tend to just use scripts to filter data into a usable form, crunch some numbers, and generate visualizations. There, unless the software is the point of the research, writing code doesn't get the same priority (or respect) as traditional experiments. A lot of those professors just don't care about writing maintainable code; I'm guessing Dr. Briffa is one of those.

martincmartin 17y ago

As The Economist points out: "The 'hiding' concerned the decision to leave out a set of tree-ring-growth data that had stopped reflecting local temperature changes. That alteration in growth pattern is strange, and unexplained, but eliminating it is not sinister." and: " 'I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) and from 1961 for Keith’s to hide the decline.' Trickery associated with Dr Mann was catnip to the sceptics. But Dr Jones has clarified that 'The word trick was used here colloquially as in a clever thing to do. It is ludicrous to suggest that it refers to anything untoward.' " http://www.economist.com/opinion/displaystory.cfm?story_id=14960149 http://www.economist.com/opinion/displaystory.cfm?story_id=1...

greendestiny 17y ago
If the tree-ring growth data doesn't match directly measured temperature at significant intervals of time (necessitating what amounts to arbitrarily changing data sources mid graph) why on earth would you trust it for periods where its the only data source. I think the evidence for AGW extends beyond this particular incident, but I think its highly unethical and very poor science.
- jsz0 17y ago
  Was it the only data source in other time periods? or was it collaborated by other data?
  - greendestiny 17y ago
    No idea, reasonable questions, but I don't see much redeeming about simply swapping data sets mid-graph.
    - tensor 17y ago
      That's a fair criticism, but it would be worth seeing the figure caption for it as well. If the caption had within it that the source was switched part way due to a known problem, that would be fine. Out of curiosity, anyone know a journal reference for that particular graph?

jacoblyles 17y ago

While I'm glad people are digging through the leaked source code, I doubt they find anything so blatant as outright fraud.

j_baker 17y ago

I'm not necessarily attempting to justify any data-cooking whatsoever, but why is it such a big deal? I mean, the climate "skeptics" do it all the time and it hardly makes the news. We're never going to have a decent public discourse on this issue until both sides are honest with the public. Maybe I'm just a pessimist, but I don't see this happening any time soon.

DomesticMouse 17y ago
Science that relies on cooked data isn't science. It's snake oil sales.
rick888 17y ago
"I'm not necessarily attempting to justify any data-cooking whatsoever, but why is it such a big deal? I mean, the climate "skeptics" do it all the time and it hardly makes the news." Are you serious? Laws are going to be passed (cap and trade) in the US that heavily regulate businesses based on data from these organizations. If congress doesn't take a step back and look into this, they are even more corrupt than I thought. Al Gore got a noble peace prize based on this and he has made millions through his carbon credit programs (and speeches, etc). The truth needs to be known. If global warming is not a result of man-made pollution (or if temperatures are not actually increasing or decreasing at a dangerous level), we need to know.

alex_c 17y ago

The code is pretty damning, but before I get really outraged, a bit of context is always useful. Has anyone figured out where this graph has been used?

jgrahamc 17y ago
No. And, also, the interpolated value yearlyadj isn't used. It's only referenced in commented out code. http://www.jgc.org/blog/2009/11/about-that-cru-hack.html http://www.jgc.org/blog/2009/11/about-that-cru-hack.html

hop 17y ago

How do all the "best" climate prediction models have squiggly lines after 10 years from now and not a broad range or something realistic. Their algorithims smell of BS and way too many variables go into it to make them have anywhere near their plotted accuracy for 100 year projections. From wikpedia: http://en.wikipedia.org/wiki/File:Global_Warming_Predictions.png http://en.wikipedia.org/wiki/File:Global_Warming_Predictions...

petercooper 17y ago
I'd be interested in that too. I wonder if they include things like predictable levels of solar activity, volcano eruptions, weather cycles and the like to get this. Or, worse, it might even be stochastic or "random walk"-like which would make the variations meaningless (see diagram on http://en.wikipedia.org/wiki/Random_walk http://en.wikipedia.org/wiki/Random_walk )..
camccann 17y ago
Well, the graph you link to doesn't include error bars or any other indication of uncertainty. Leaving out error margins and plotting only the mean is unfortunately common when presenting scientific results to general audiences; but that doesn't mean the actual scientists don't have the full data.
azgolfer 17y ago
Anyone who says they have a good computer model of the climate is either ignorant, stupid, corrupt or some combination thereof.

ieure 17y ago

I don't know what ESR's politics looked like before, but he's been in deep teabagger territory for a while now. I would take anything he says on the subject of politics with… Well, I'd just not take it. He can pretend to be objective, but the use of winger slang (“AGW” and “MSM” are right-wing dog whistles in the same way as “Democrat party” or “socialism”) is a giveaway. Here are some of his prognostications from the election last year. They're delightfully wingnutty, with a crunchy glibertarian shell: “The Obama campaign smells of defeat”: http://esr.ibiblio.org/?p=449 http://esr.ibiblio.org/?p=449 “Democrats’ polling league in a generic contest has collapsed. Control of Congress is in play. Wow. Just wow. Considering how hard the Republicans have been fucking up and alienating their base, this is astonishing. EPIC FAIL.” “Sarah Palin, American Centrist”: http://esr.ibiblio.org/?p=481 http://esr.ibiblio.org/?p=481 “So when I tell you that I think Sarah Palin’s religious position is pretty near dead center in the American spectrum, you can be pretty sure I’m not fudging to make that position look good.” “The Obama campaign smells of defeat: II”: http://esr.ibiblio.org/?p=490 http://esr.ibiblio.org/?p=490 “The attempt to smear, discredit and delegitimize Palin has steadily become more intense and more damaging — to the Democrats.” When it comes to politics, dude simply does not know what the fuck.

javert 17y ago
You're just attacking ESR's character. It's fallacious to discount his blog post just because you don't like his character. Please, can't we have a rational discussion on the topic?
- ieure 17y ago
  No, what I’m doing is pointing out that his opinions are a) wrong and b) delivered with right-wing spin. As such, you should not take what he says about matters political at face value.
  - davidmathers 17y ago
    I don't know what ESR's politics looked like before Before what? I don't remember there ever being a time when esr wasn't a deranged wingnut.

yters 17y ago

So maybe we haven't found a smoking gun yet, but the general gist of the documents and emails in that package is not very encouraging. You can find reviews of IPCC's book that are full of very strident critiques. You can find emails discussing the inherent inaccuracy of their work. You can find internal dismissal of the tree ring data, stating how useless it is. You can find a scientist openly glad that he's not being held accountable so he can push his climate interests. After spending a couple hours pouring through the repository, I haven't found much that makes this work seem stable enough to base global lifechanging legislation on.

Joeboy 17y ago

Can anybody tell me what, if any, published work this data has made it into?