5 ms·

I think you will have easily inferred last time around, I am in the UK. The Met Office have all the raw UK temperature readings on their site going back to the

I think you will have easily inferred last time around, I am in the UK. The Met Office have all the raw UK temperature readings on their site going back to the early 19th century for some stations. France is similar. Stations have come and gone, or moved, and there has been a growth of overall number as the decades have passed. They feed into global readings. If they are being adjusted I would want ot know why, but they are not being adjusted on the public site, or in the historic dataset you can download. I wouldn't assume every adjustment is fraud, as stations have indeed moved. Second, if temperature declines across the continental US, as a fast skim of the first link claims to show, that in no way disproves an increase in global temperature. For decades the prediction has always been some areas will get warmer, some might get colder, just as some will dry and others get wetter. Europe has bloody obviously been warming, unmissably so, across the last 40 years. So neither are necessarily in contradiction with global heating. It might be reasonable for NOAA to bring formal rigour to their software, but MilSpec or Nuclear safety level programming costs. If they're in from the start Unit Tests need not, if they're added after they cost too. That would have budgetary implications. I have no idea how viable funding for a project to improve the software would be - though throughout the software industry rewrites have an ignominious track record. And via a funding source you inexplicably appear to trust least of all. Personally I am much, much more leery of industry research and funding. But that's our previous discussion. :p Still, having found what you believe are questionable adjustments, you can't just dump the lot and say "dada, see nothing to see, the raw graph shows the opposite!" because we haven't established that all the applied adjustments are incorrect, or actually any of them, yet. You yourself gave a couple of valid reasons why adjustment may be necessary for accuracy, there are many others. For unmanned remote stations, calibration curves may need to change or be refitted as sensors age. You do neatly identify the point down these rabbit holes where I see bad faith, or intent to mislead - or very poor practice if I wish to be charitable. It's the throwing out of absolutely everything to prove a point and curve on a graph - despite acknowledging there could be multiple good reasons to adjust, or showing that any are mistaken. It's a very simple - to the layman - presentation of "adjusted == fraud", here's the raw, see I can show black is clearly white. It sounds superficially convincing until you dig, even a little. To close, and expand: + You can't just scream "FAKE DATA" and remove the lot, or MCAS would have resulted in all Boeings having no wings (shite analogy, but hey) + You audit the data going back + you establish which adjustments are questionable and why, and make a credible case as to what the problem is, and the case for discarding. + you accept others - for, as you mention, a station that had to move etc. Or is the premise that no adjustment is ever permissible? + you present a graph with the suspicious entries only, along with the processed entries with the suspicious removed, you also show the combined curve of all adjusted entries - suspicious and not. Now you can get a feel for extent of error. + what you very probably should never do unless your intent is to mislead, is simply present 100% raw data on a graph, as we've already established there are very valid reasons to need to adjust some data. Were they playing it that way I might have much more time for some of these blogs. Mainly I find they are doing, at some point, the sleight of hand of proving 1=0 we used to amuse ourselves with in maths class. That really is all I have time for this evening. :)

repolfx 7y ago
All that's fine, and if you keep clicking through some of the links I posted you'll find that the sorts of analyses you're asking for have been done, because indeed, it's not enough to merely see smoke. You need to find the fire too. I can't sum up every possible argument in a HN post, only show that there are problematic signs worthy of further and deeper inspection. That is, skepticism isn't some weird cranky idea. At least some of it is based, at heart, in the question of whether there are mistakes in algorithms or software. That's a totally normal question to ask about any data analysis. For instance, let's take your audit point. You audit the data going back. In the USA this is somewhat possible. For the HadCRUT dataset it's not, because they lost/destroyed the original raw data, and their computations to clean it up are not reproducible, as the notes about "Harry's" attempts showed quite clearly. That is already a major problem; in effect we're being asked to take the correctness of CRU's data adjustments on faith. So let's focus on the NOAA dataset. A large amount of it is synthetic, more and more over time. You say we can't just scream "fake data" and remove it, and I agree we can't because that would at this point erase most of the official US temperature record. But ... seriously? You don't find it at all concerning that in the past humanity was apparently capable of measuring temperature with thousands of weather stations across the USA, and now we've apparently lost that ability just when we need it most - forced to rely on 'estimates' generated by computer simulation? The foundation of science is fitting theories to match the data, but here, the data is being generated by the theories. I think anyone who cares about science should care at least about that. Now, which adjustments are questionable and why - that's a great question and is pretty much where I'm up to in my own research. There are two main strands of alteration: TOB adjustment, and the PHA. TOB is "time of observation bias". PHA is the pair-wise homogenisation algorithm. There are other adjustments made too: more and more algorithms have been added over time, but those seem to be the two big ones. PHA seems to be involved in the 'estimation' process to fill out or adjust the reported weather station readings. This graph shows the affect of them split out: https://realclimatescience.com/wp-content/uploads/2019/08/US-Historical-Climatology-Network-Average-Monthly-Temperature-4.png https://realclimatescience.com/wp-content/uploads/2019/08/US... As you can see the adjustments have little/no impact today. But they rewrite the past significantly: raw data pre-adjustment doesn't show any warming in the 20th century, the 1940s in particular were as hot as today. TOB adjustment was developed first and pushes down the temperatures in the past, then PHA+extras (what this graph labels "final") was developed later and pushes it down further. This can be interpreted in a few different ways. One way is that climatologists found that in the past people were consistently bad at measuring temperature accurately, so they process the data to remove the failures and the data is now accurate. Another less charitable interpretation is that the algorithms have the effect of fitting data to match the theory, a form of scientific malpractice that is not unknown in history. To figure out what's going on requires much more analysis. Note that nothing here requires an evil conspiracy. Climatologists were faced with a huge problem in the 1970s: basic chemical theory said CO2 is a greenhouse gas, we're burning lots of fossil fuels, so the world should be getting warmer. At least if you assume the climate is quite simple. But the raw data didn't show that. It was also messy, filled with glitches and could obviously benefit from some cleaning up before usage. So scientists started trying out various analyses and adjustments, but ultimately they're not comparing against a ground truth. They don't know what the data "should" be except through climate theory, so it would be very easy to end up trying lots of algorithms and ending up selecting the ones that result in "obviously correct" outcomes. Meanwhile the number of people involved in this is very small, they're all colleagues, their income depends on the belief their understanding of the climate is always growing, and it's unclear how you could ever resolve the apparent contradiction between chemical theory and raw thermometer data in a field like climatology. It's not like you can experiment on the Earth. In other words, there's a big grey area between cleaning up the data and adjusting the data to fit the theory. Did they go into that zone, and if so, did they cross the invisible line? It's subjective, but not obviously impossible. Anyway back to the analysis. So far I only found an analysis of TOB adjustment. It can be found in the second half of this rather long blog post: https://realclimatescience.com/erasing-americas-hot-past/ https://realclimatescience.com/erasing-americas-hot-past/ Start reading from "I have new tools". The quick summary is: some weather stations used to be read in the morning and others the afternoon, and this was suspected to need fixing. The author analyses Minnesota because it has a balanced mix of AM/PM stations, and shows the PM stations are indeed consistently reporting about 0.5 C of difference to the AM stations. But then he analyses their coordinates and finds that this is because the PM stations are on average a degree of latitude further south, and that average temperature increases about 1C per degree of latitude, so you would expect such a difference based on location alone. In other words, correcting for location eliminates any difference that could be caused by TOB. I'm still in the process of checking these claims. I didn't reproduce the graphs yet like I did for the claim of synthetic data. The claim 1 degree of latitude = 1 degree C of warmth appears backed up by this source: https://www.physics.byu.edu/faculty/christensen/Physics%20137/Figures/Temperature/Effects%20of%20Latitude%20on%20Annual%20Temperature%20Range.htm https://www.physics.byu.edu/faculty/christensen/Physics%2013... So it sounds plausible. If no TOB is observable in Minnesota, why did NOAA scientists conclude it was a necessary adjustment to make everywhere? It's also kind of suspect that this would matter, since random noise should get washed out by the averaging and long term trend analysis. BTW, you ask do I accept that data has to be adjusted because stations moved. I certainly did at first, when I started looking into this. The deeper I look the more I'm starting to question this. Station moves and upgrades can certainly introduce artifacts into the reading of a single station, but, scientists only really care about global averages. For this to require correction at all would require the effect of moves to be highly correlated. Final notes on the science. what you very probably should never do unless your intent is to mislead, is simply present 100% raw data on a graph That's not the science I was taught! One of the pillars of science is providing your raw data. If there is uncertainty in the measurement that's what error bars are for, and the various statistical techniques for handling uncertainty. Picking data points and excluding others that don't fit the theory is literally "bad science 101". Also, if climatologists have such severe concerns over the integrity of their raw data that they need to interpolate or synthesise nearly half of all readings, they should be screaming at the top of their voice that nobody should do anything until the global weather station network is in a better shape and that should be their top funding priority. Are they doing this? If so I've never seen any reference to it. We should be seeing more measurements and better data over time, not a decline in the number of trustworthy weather stations (which is what's happening, hence the rise in synthetic data).

repolfx 7y ago

All that's fine, and if you keep clicking through some of the links I posted you'll find that the sorts of analyses you're asking for have been done, because indeed, it's not enough to merely see smoke. You need to find the fire too. I can't sum up every possible argument in a HN post, only show that there are problematic signs worthy of further and deeper inspection. That is, skepticism isn't some weird cranky idea. At least some of it is based, at heart, in the question of whether there are mistakes in algorithms or software. That's a totally normal question to ask about any data analysis. For instance, let's take your audit point. You audit the data going back. In the USA this is somewhat possible. For the HadCRUT dataset it's not, because they lost/destroyed the original raw data, and their computations to clean it up are not reproducible, as the notes about "Harry's" attempts showed quite clearly. That is already a major problem; in effect we're being asked to take the correctness of CRU's data adjustments on faith. So let's focus on the NOAA dataset. A large amount of it is synthetic, more and more over time. You say we can't just scream "fake data" and remove it, and I agree we can't because that would at this point erase most of the official US temperature record. But ... seriously? You don't find it at all concerning that in the past humanity was apparently capable of measuring temperature with thousands of weather stations across the USA, and now we've apparently lost that ability just when we need it most - forced to rely on 'estimates' generated by computer simulation? The foundation of science is fitting theories to match the data, but here, the data is being generated by the theories. I think anyone who cares about science should care at least about that. Now, which adjustments are questionable and why - that's a great question and is pretty much where I'm up to in my own research. There are two main strands of alteration: TOB adjustment, and the PHA. TOB is "time of observation bias". PHA is the pair-wise homogenisation algorithm. There are other adjustments made too: more and more algorithms have been added over time, but those seem to be the two big ones. PHA seems to be involved in the 'estimation' process to fill out or adjust the reported weather station readings. This graph shows the affect of them split out: https://realclimatescience.com/wp-content/uploads/2019/08/US-Historical-Climatology-Network-Average-Monthly-Temperature-4.png https://realclimatescience.com/wp-content/uploads/2019/08/US... As you can see the adjustments have little/no impact today. But they rewrite the past significantly: raw data pre-adjustment doesn't show any warming in the 20th century, the 1940s in particular were as hot as today. TOB adjustment was developed first and pushes down the temperatures in the past, then PHA+extras (what this graph labels "final") was developed later and pushes it down further. This can be interpreted in a few different ways. One way is that climatologists found that in the past people were consistently bad at measuring temperature accurately, so they process the data to remove the failures and the data is now accurate. Another less charitable interpretation is that the algorithms have the effect of fitting data to match the theory, a form of scientific malpractice that is not unknown in history. To figure out what's going on requires much more analysis. Note that nothing here requires an evil conspiracy. Climatologists were faced with a huge problem in the 1970s: basic chemical theory said CO2 is a greenhouse gas, we're burning lots of fossil fuels, so the world should be getting warmer. At least if you assume the climate is quite simple. But the raw data didn't show that. It was also messy, filled with glitches and could obviously benefit from some cleaning up before usage. So scientists started trying out various analyses and adjustments, but ultimately they're not comparing against a ground truth. They don't know what the data "should" be except through climate theory, so it would be very easy to end up trying lots of algorithms and ending up selecting the ones that result in "obviously correct" outcomes. Meanwhile the number of people involved in this is very small, they're all colleagues, their income depends on the belief their understanding of the climate is always growing, and it's unclear how you could ever resolve the apparent contradiction between chemical theory and raw thermometer data in a field like climatology. It's not like you can experiment on the Earth. In other words, there's a big grey area between cleaning up the data and adjusting the data to fit the theory. Did they go into that zone, and if so, did they cross the invisible line? It's subjective, but not obviously impossible. Anyway back to the analysis. So far I only found an analysis of TOB adjustment. It can be found in the second half of this rather long blog post: https://realclimatescience.com/erasing-americas-hot-past/ https://realclimatescience.com/erasing-americas-hot-past/ Start reading from "I have new tools". The quick summary is: some weather stations used to be read in the morning and others the afternoon, and this was suspected to need fixing. The author analyses Minnesota because it has a balanced mix of AM/PM stations, and shows the PM stations are indeed consistently reporting about 0.5 C of difference to the AM stations. But then he analyses their coordinates and finds that this is because the PM stations are on average a degree of latitude further south, and that average temperature increases about 1C per degree of latitude, so you would expect such a difference based on location alone. In other words, correcting for location eliminates any difference that could be caused by TOB. I'm still in the process of checking these claims. I didn't reproduce the graphs yet like I did for the claim of synthetic data. The claim 1 degree of latitude = 1 degree C of warmth appears backed up by this source: https://www.physics.byu.edu/faculty/christensen/Physics%20137/Figures/Temperature/Effects%20of%20Latitude%20on%20Annual%20Temperature%20Range.htm https://www.physics.byu.edu/faculty/christensen/Physics%2013... So it sounds plausible. If no TOB is observable in Minnesota, why did NOAA scientists conclude it was a necessary adjustment to make everywhere? It's also kind of suspect that this would matter, since random noise should get washed out by the averaging and long term trend analysis. BTW, you ask do I accept that data has to be adjusted because stations moved. I certainly did at first, when I started looking into this. The deeper I look the more I'm starting to question this. Station moves and upgrades can certainly introduce artifacts into the reading of a single station, but, scientists only really care about global averages. For this to require correction at all would require the effect of moves to be highly correlated. Final notes on the science. what you very probably should never do unless your intent is to mislead, is simply present 100% raw data on a graph That's not the science I was taught! One of the pillars of science is providing your raw data. If there is uncertainty in the measurement that's what error bars are for, and the various statistical techniques for handling uncertainty. Picking data points and excluding others that don't fit the theory is literally "bad science 101". Also, if climatologists have such severe concerns over the integrity of their raw data that they need to interpolate or synthesise nearly half of all readings, they should be screaming at the top of their voice that nobody should do anything until the global weather station network is in a better shape and that should be their top funding priority. Are they doing this? If so I've never seen any reference to it. We should be seeing more measurements and better data over time, not a decline in the number of trustworthy weather stations (which is what's happening, hence the rise in synthetic data).