8 ms·

Not remotely what is happening, is it? Actually, I disagree. There's a lot of that happening. You just aren't exposed to it. You say you'd be genuinely fascin

Not remotely what is happening, is it? Actually, I disagree. There's a lot of that happening. You just aren't exposed to it. You say you'd be genuinely fascinated by a debate about how scientists could be mistaken. OK, let's find out if you mean it. The last debate we had here on this got derailed into a discussion of Conservative party policy. Let's keep it on science this time. To start, let me say that I'm not a climate skeptic ... yet. I'm not actually sure what I am at the moment. Let's say I'm actually a science skeptic. I think groupthink, bias, corruption and outright incompetence in science is far more widespread than people intuit, and it can distort entire subfields of science. People are starting to realise this in fields like psychology due to the replication crisis but there are problems in many other fields. Now, I wasn't surprised to discover in this comments section that the article was fake propaganda (i.e. "all spending on conservative think tanks for any reason" is "funding denial"). In the past year or so I've dug into climate change skepticism to figure out what they think and why. Here's what modern climate change skepticism looks like: lots and lots of blogs, with very long entries, consisting of large quantities of data analysis and graphs. If it's professionally funded it sure doesn't look like that. It looks like smart people who know maths and science writing about a field they understand but are outsiders in. They reveal a lot of things from the raw data that are deeply troubling and which I've never read about in the media. I feel strongly I should have read about them there because the concerns are reasonable. I've been able to check some of these claims for myself and found them to be true. Most troubling to me has been the massive extent to which the global temperature record is now synthetic. Timeseries that once showed the world getting cooler have been fed into ever-more complex algorithms that radically change the entire datasets and any conclusions drawn from them. This article on a skeptic website is one that I double checked for myself: https://realclimatescience.com/61-fake-data/ https://realclimatescience.com/61-fake-data/ It makes a checkable claim - that one of the very few global temperature datasets, the one published by NOAA, comes in "raw" and "adjusted" forms, and that in the adjusted form nearly 60% of all claimed measurements from weather stations are in fact not real measurements at all but the output of a computer model. Additionally the quantity of simulated measurements has shot up massively over time: older temperature data is much less synthetic than the rest of algorithms. I think here on HN we all know how slippery software can be, so learning this was quite a shock. I couldn't quite believe it in fact but the datasets are public files anyone can download, so I downloaded them and wrote some quick Python scripts to check. The scientists aren't hiding anything: the measurements that are "estimated" using their algorithms are marked with an E in the dataset, just as the skeptic blog claimed, and sure enough, calculating the ratios showed that 1970 for instance only 9% of values were the result of a computer model, but in 2018 it was over 40% and this year is even higher. Look at this graph to get a sense of the growth of the issue: https://realclimatescience.com/wp-content/uploads/2019/02/PercentOfUSHCNMonthlyTemperatureDataWhichIsFabricated_shadow.jpg https://realclimatescience.com/wp-content/uploads/2019/02/Pe... And worse, if you take away the output of the models, global warming disappears entirely. https://realclimatescience.com/wp-content/uploads/2019/02/USHCNTemperatureTrendsSince1990_shadow-2.jpg https://realclimatescience.com/wp-content/uploads/2019/02/US... Now, NOAA have an explanation for this sort of thing: this sort of heavy adjustment is required because weather stations move around, come and go, readings are taken at different times of day, etc. So the data needs processing and cleaning before it's used. That's fine. I can buy that, at least to some extent. But this did make me curious about the code that's doing this adjustment, not least because I kept encountering climate blogs written by apparently rigorous authors who said certain artifacts made them suspect a software bug was introduced at some point. So I downloaded NOAA's code to assess the quality of the software - software that appears to be required for the world to show any warming trend at all. I'm a software developer and always have been so I'm confident I can take part in this area of climatology, at least in a trivial way. I'd assumed that given the huge sums spent on research and incredible costs being thrown around for tackling climate change, the software in question would be under heavy development and written to the highest possible standards. Unfortunately it's not. Global temperature data is the output of programs that look like every programmers worst nightmare: large piles of FORTRAN that trace directly back to 1985, that's been patched every few years in unauditable ways. All variables are global. As you would expect, they all have cryptic names. Functions start with explanations of which global variables are used and which aren't. Version control doesn't exist. There are occasional comments at the top of a function summarising that a change was made, so that's at least something, but are they comprehensive? There are no unit tests. None of it was written by professional software engineers. The algorithms and code are both extremely complicated. Much of it is undocumented. Not surprisingly, there is evidence in the comments of bugs being fixed only after going undetected for decades. The program can be found here if you're curious: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/software/52i/ This is NOAAs code. I mentioned there are only a few global datasets. One of the others comes from the notorious CRU at the University of East Anglia. When they got hacked their code leaked, and it's even worse. The CRU datasets are totally unreproducible: they appear to have asked a scientist to spend a few months on it at some point, and he wrote extensive notes about how frustrated he got: http://di2.nu/foia/HARRY_READ_ME-20.html http://di2.nu/foia/HARRY_READ_ME-20.html Their code was much the same as above but apparently, much worse documented, and it seems to have suffered from undetected integer overflow bugs. That's about where I've got up to. So I'd say the skeptics have got me listening in at least one basic way: all global warming theories are rooted in data that comes from a tiny number of people, processed using code that can't possibly have been reviewed by professionals, and those scientists have adjusted the raw data so much that temperature datasets that used to show average cooling (in the USA) now show rapid warming. That needs addressing just on the grounds of basic scientific integrity.

taharvey 7y ago
Ok, I'm going to take this one. First, are you claiming that weather scientists and forecasters are part of a vast global conspiracy? Seriously? These guys aren't even climate scientists, they are just building more accurate sensor networks and forecasting models. And if you look at the forecasting models running on todays supercomputers they are devastatingly accurate compared to 2 decades ago. So one can guess they are getting it more right, than wrong. It think to biggest issue here is "normal" people don't understand math or stats. I work with weather data sets in building simulation, which involves sensors and sensor fusion. First, there is no sensor that doesn't involve algorithms, statistics, or some sort of "data cleaning". Unless you are talking about a glass thermometer visually inspected... and if thats the only "data" you'll accept then we are talking serious luddite tendencies here. All weather data needs cleaning. There are data drops than need interpolation, sensor stats checks to detect erroneous readings, multi-sensor fusion to make more accurate values. Get real.
NeedMoreTea 7y ago
I think you will have easily inferred last time around, I am in the UK. The Met Office have all the raw UK temperature readings on their site going back to the early 19th century for some stations. France is similar. Stations have come and gone, or moved, and there has been a growth of overall number as the decades have passed. They feed into global readings. If they are being adjusted I would want ot know why, but they are not being adjusted on the public site, or in the historic dataset you can download. I wouldn't assume every adjustment is fraud, as stations have indeed moved. Second, if temperature declines across the continental US, as a fast skim of the first link claims to show, that in no way disproves an increase in global temperature. For decades the prediction has always been some areas will get warmer, some might get colder, just as some will dry and others get wetter. Europe has bloody obviously been warming, unmissably so, across the last 40 years. So neither are necessarily in contradiction with global heating. It might be reasonable for NOAA to bring formal rigour to their software, but MilSpec or Nuclear safety level programming costs. If they're in from the start Unit Tests need not, if they're added after they cost too. That would have budgetary implications. I have no idea how viable funding for a project to improve the software would be - though throughout the software industry rewrites have an ignominious track record. And via a funding source you inexplicably appear to trust least of all. Personally I am much, much more leery of industry research and funding. But that's our previous discussion. :p Still, having found what you believe are questionable adjustments, you can't just dump the lot and say "dada, see nothing to see, the raw graph shows the opposite!" because we haven't established that all the applied adjustments are incorrect, or actually any of them, yet. You yourself gave a couple of valid reasons why adjustment may be necessary for accuracy, there are many others. For unmanned remote stations, calibration curves may need to change or be refitted as sensors age. You do neatly identify the point down these rabbit holes where I see bad faith, or intent to mislead - or very poor practice if I wish to be charitable. It's the throwing out of absolutely everything to prove a point and curve on a graph - despite acknowledging there could be multiple good reasons to adjust, or showing that any are mistaken. It's a very simple - to the layman - presentation of "adjusted == fraud", here's the raw, see I can show black is clearly white. It sounds superficially convincing until you dig, even a little. To close, and expand: + You can't just scream "FAKE DATA" and remove the lot, or MCAS would have resulted in all Boeings having no wings (shite analogy, but hey) + You audit the data going back + you establish which adjustments are questionable and why, and make a credible case as to what the problem is, and the case for discarding. + you accept others - for, as you mention, a station that had to move etc. Or is the premise that no adjustment is ever permissible? + you present a graph with the suspicious entries only, along with the processed entries with the suspicious removed, you also show the combined curve of all adjusted entries - suspicious and not. Now you can get a feel for extent of error. + what you very probably should never do unless your intent is to mislead, is simply present 100% raw data on a graph, as we've already established there are very valid reasons to need to adjust some data. Were they playing it that way I might have much more time for some of these blogs. Mainly I find they are doing, at some point, the sleight of hand of proving 1=0 we used to amuse ourselves with in maths class. That really is all I have time for this evening. :)

taharvey 7y ago

Ok, I'm going to take this one. First, are you claiming that weather scientists and forecasters are part of a vast global conspiracy? Seriously? These guys aren't even climate scientists, they are just building more accurate sensor networks and forecasting models. And if you look at the forecasting models running on todays supercomputers they are devastatingly accurate compared to 2 decades ago. So one can guess they are getting it more right, than wrong. It think to biggest issue here is "normal" people don't understand math or stats. I work with weather data sets in building simulation, which involves sensors and sensor fusion. First, there is no sensor that doesn't involve algorithms, statistics, or some sort of "data cleaning". Unless you are talking about a glass thermometer visually inspected... and if thats the only "data" you'll accept then we are talking serious luddite tendencies here. All weather data needs cleaning. There are data drops than need interpolation, sensor stats checks to detect erroneous readings, multi-sensor fusion to make more accurate values. Get real.

NeedMoreTea 7y ago

I think you will have easily inferred last time around, I am in the UK. The Met Office have all the raw UK temperature readings on their site going back to the early 19th century for some stations. France is similar. Stations have come and gone, or moved, and there has been a growth of overall number as the decades have passed. They feed into global readings. If they are being adjusted I would want ot know why, but they are not being adjusted on the public site, or in the historic dataset you can download. I wouldn't assume every adjustment is fraud, as stations have indeed moved. Second, if temperature declines across the continental US, as a fast skim of the first link claims to show, that in no way disproves an increase in global temperature. For decades the prediction has always been some areas will get warmer, some might get colder, just as some will dry and others get wetter. Europe has bloody obviously been warming, unmissably so, across the last 40 years. So neither are necessarily in contradiction with global heating. It might be reasonable for NOAA to bring formal rigour to their software, but MilSpec or Nuclear safety level programming costs. If they're in from the start Unit Tests need not, if they're added after they cost too. That would have budgetary implications. I have no idea how viable funding for a project to improve the software would be - though throughout the software industry rewrites have an ignominious track record. And via a funding source you inexplicably appear to trust least of all. Personally I am much, much more leery of industry research and funding. But that's our previous discussion. :p Still, having found what you believe are questionable adjustments, you can't just dump the lot and say "dada, see nothing to see, the raw graph shows the opposite!" because we haven't established that all the applied adjustments are incorrect, or actually any of them, yet. You yourself gave a couple of valid reasons why adjustment may be necessary for accuracy, there are many others. For unmanned remote stations, calibration curves may need to change or be refitted as sensors age. You do neatly identify the point down these rabbit holes where I see bad faith, or intent to mislead - or very poor practice if I wish to be charitable. It's the throwing out of absolutely everything to prove a point and curve on a graph - despite acknowledging there could be multiple good reasons to adjust, or showing that any are mistaken. It's a very simple - to the layman - presentation of "adjusted == fraud", here's the raw, see I can show black is clearly white. It sounds superficially convincing until you dig, even a little. To close, and expand: + You can't just scream "FAKE DATA" and remove the lot, or MCAS would have resulted in all Boeings having no wings (shite analogy, but hey) + You audit the data going back + you establish which adjustments are questionable and why, and make a credible case as to what the problem is, and the case for discarding. + you accept others - for, as you mention, a station that had to move etc. Or is the premise that no adjustment is ever permissible? + you present a graph with the suspicious entries only, along with the processed entries with the suspicious removed, you also show the combined curve of all adjusted entries - suspicious and not. Now you can get a feel for extent of error. + what you very probably should never do unless your intent is to mislead, is simply present 100% raw data on a graph, as we've already established there are very valid reasons to need to adjust some data. Were they playing it that way I might have much more time for some of these blogs. Mainly I find they are doing, at some point, the sleight of hand of proving 1=0 we used to amuse ourselves with in maths class. That really is all I have time for this evening. :)