5 ms·

Your comment contains hypotheticals ("run random models and wait long enough you will always be able to find at least one model that predicted the current obser

Your comment contains hypotheticals ("run random models and wait long enough you will always be able to find at least one model that predicted the current observations") that are NOT what the present study did. As noted in TFA, they looked at 17 forecasts from 14 models and found that the temperature predictions of 10 of them were consistent with what has happened in the meantime. And if you adjust for forcing (CO2, methane) then the predictions improve further.

mistermann 7y ago
> As noted in TFA, they looked at 17 forecasts from 14 models Pardon a little pedantry: >> The researchers compared annual average surface temperatures across the globe to the surface temperatures predicted in 17 forecasts. Those predictions were drawn from 14 separate computer models released between 1970 and 2001. In some cases, the studies and their computer codes were so old that the team had to extract data published in papers, using special software to gauge the exact numbers represented by points on a printed graph. They compared annual average surface temperatures across the globe to 17 forecasts, from 14 separate models. But....how many models did they look at, before choosing those particular 14 models? TFA doesn't say. Articles written in this style provide rich fodder for conspiracy theorists, particularly because there is no shortage of examples in the past where authoritative, trustworthy organizations have gotten caught in lies. Rare is the climate change article I've read that can't easily have similar holes poked in it. What's the real truth here? Based on the literal content of this article, no one knows. It is speculation vs speculation. I've said it before, and I will say it again: if your persuasion tactics aren't working, it might be worth considering whether you should change your tactics.
- hcknwscommenter 7y ago
  TFA article does say. They looked at 17 forecasts drawn from 14 models. You are the one who implicitly believes they are lying about that and only reporting 17 out of some huge unknown number. The reality is that the vast majority (>99%) of climate change research does not produce the type of forecast, or really any forecast at all, that one could do this analysis on. Therefore, the 17 forecast number is actually very easy to believe.
  - whokal 7y ago
    With sincerity, I think you may misunderstand what GP is pointing out. I understood “forecasts” to be a simulation run using a model. In that case, they took 14 models and ran those each at least once to come up with 17 predictions. What GP wants is transparency around how many possible models were there over the last decades from which they chose the 14. This gives the reader a better understanding of whether 14 is a lot or a little. It is a fair point IMO regardless of their overall stance on the issue.
    - mturmon 7y ago
      They did a literature search and chose what was reproducible. This has been covered in TFA and highlighted by a comment nearby: https://news.ycombinator.com/item?id=21709446 https://news.ycombinator.com/item?id=21709446
  - ekianjo 7y ago
    How many models are currently published for reference?
  - jointpdf 7y ago
    Simulation is one way to produce a numerical forecast, but the prediction of any forecast model would never be the result of a single simulation run. Model predictions are always (or should always) be reported as an expected value, or the average over many simulation runs. The expected value of a random variable (e.g. mode output) is probably the most fundamental concept in all of statistics/probability, and many mind-wrenching (read: wrong+incoherent) internet hot takes are rooted in logical fallacies about the science of prediction. The reason that there are more predictions (17) than models (14) is that several of the models (eg Hansen 1981) consider several distinct “scenarios” based on differing assumptions about future levels of GHG emissions. Secondly, the forecast models evaluated in the study are from the likes of James Hansen, William Nordhaus, and the IPCC (in other words, authoritative sources). So at the very least this adds weight to the hypothesis that the models published by the IPCC are accurate and methodologically sound. Sure, there is some probability that the models and reality corresponded by random chance, but that probability is made ever more diminishingly small as evidence accumulates. So, I don’t think any of the comments in this thread that accuse the authors of retrospective cherry-picking of climate models are anything but embarrassing examples of pseudo-intellectual contrarianism and/or confusion over basic statistics. The models considered by the study, their predictions, and the methodology used are all located here and were easily found after 30s of digging: https://github.com/hausfath/OldModels https://github.com/hausfath/OldModels
- jartelt 7y ago
  Even if they said how many models they looked at before choosing 14 models (and explained why they chose those 14), conspiracy theorists and people skepticalof climate science would still argue that it is not legitimate because of survivorship bias and several other reasons. "You looked at 1000 models, but what about the 1000 models that scientists made and didn't publish. You didn't look at those!" People can always suggest more tests you need to do or poke holes in any analysis. That doesn't mean you just stop ding the work. Overall it is impressive that different models made over 30 years can predict fairly accurate results. In my opinion that shows that the underlying physics of the models is correct and makes me trust the models even more.
  - mistermann 7y ago
    > Even if they said how many models they looked at before choosing 14 models (and explained why they chose those 14), conspiracy theorists and people skeptical of climate science would still argue that it is not legitimate because of survivorship bias and several other reasons. Undoubtedly some would, but are you willing to consider the possibility that the number/percentage of people that would do so might be altered by the language used in articles on climate science? > People can always suggest more tests you need to do or poke holes in any analysis. Some analyses are easier to poke holes in than others. If this article was tightened up in the manner I describe, excluding the possibility of survivorship bias, then an entirely new approach to denial would be required. As written, survivorship bias is a very real possibility. Wikipedia says: "Survivorship bias or survival bias is the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility. This can lead to false conclusions in several different ways. It is a form of selection bias." Survivorship bias is a real phenomenon, not a conspiracy theory. It is neither illogical or unscientific to consider the possibility that is might be in play behind the scenes here, and the article as written makes no attempt to rule it out. > Overall it is impressive that different models made over 30 years can predict fairly accurate results. In my opinion that shows that the underlying physics of the models is correct and makes me trust the models even more. I suspect this is because your mind does not consider it a possibility that the models discussed do not result from survivorship bias. A conspiratorial mind however, would most likely strongly consider that a possibility, and therefore compute a completely different level of trust. Imagine for a moment how the human mind works when reading something. Typically, if the mind is primed with preexisting beliefs, it is going to be actively searching (interpreting) for confirmation of those beliefs, and this behavior will likely be stronger if the topic happens to be one associated with one's identity. So, your mind processes an article such as this optimistically (uncritically), whereas a "denier" or "conspiracy theorist" is going to read it pessimistically (critically). https://en.wikipedia.org/wiki/Confirmation_bias#Biased_interpretation https://en.wikipedia.org/wiki/Confirmation_bias#Biased_inter... https://www.verywellmind.com/what-is-a-confirmation-bias-2795024 https://www.verywellmind.com/what-is-a-confirmation-bias-279... Here is another interesting example. See this reddit thread on the article: https://www.reddit.com/r/science/comments/e63ic5/of_17_climate_models_published_between_the_early/ https://www.reddit.com/r/science/comments/e63ic5/of_17_clima... The top comment is: (/u/aClimateScientist) "Hi all, I'm a coauthor of this paper (Henri Drake) and happy to answer questions." Someone asks the question: "How did you account for survivorship bias of models? IOW how did you select the 17? I mean if I asked 1,000 psychics to predict GDP growth, I could probably find 10-20 who were pretty close and say that that proves they're clairvoyant." As of when I'm writing this, neither /u/aClimateScientist nor /u/avogadros_number (the article submitter, who seems to have considerable background in climate science) have addressed this question. (Or, same thread: a moderator [Removed] comment.) My question to you is: do you think it is possible that the manner in which your mind evaluates this particular situation (an unanswered question, in a thread with 29 total comments) may differ from how the mind of a "denier" or "conspiracy theorist" might evaluate the situation, and how that evaluation may in turn be used (or not) to tune your respective mental models and subconscious heuristics? A followup question is: do you believe this perspective is irrelevant (has no bearing on) the climate change dilemma? ------------------------------ I sometimes wonder if the climate change dilemma was reproduced in a sim-style video game, would the human mind be less resistant to recognizing and reacting to the behavior of their adversary? If the adversary is a machine rather than a human, might the tribalism component of subconscious decision making decrease? This is one of those things that seems so obvious, I wonder if there might be some studies on it. As shocked as people here are at the continued existence of "deniers" (or more accurately, their mental models of deniers), I am similarly shocked at the psychological unwillingness of intelligent people to acknowledge certain unfortunate realities. It's kind of comically/tragically ironic if you think about it.
Akababa 7y ago
But we don't know how those 17 were chosen in the first place. There would be survivorship bias if models which match observations get more attention. With that being said, I doubt that survivorship bias is strong enough to explain their combined result of 10/17 models being significant.
- jointpdf 7y ago
  Since one of authors posted the paper, we do know: > “We conducted a literature search to identify papers published prior to the early-1990s that include climate model outputs containing both a time-series of projected future GMST (with a minimum of two points in time) and future forcings (including both a publication date and future projected atmospheric CO2 concentrations, at a minimum). Eleven papers with fourteen distinct projections were identified that fit these criteria.
  - mistermann 7y ago
    > Eleven papers with fourteen distinct projections were identified that fit these criteria. This brings to mind the old joke: "I used to smoke weed. I still do, but I used to, too." Specifically, the phrase "were identified that fit these criteria" sets off my skepticism spider senses. Kind of like the popular "97% of scientists..." meme, that doesn't quite hold water when you dig under the covers. To be clear, I'm not saying deceit is taking place, I am simply saying that I am skeptical. I wasn't able to find any details on the specifics of the literature search, such that the search could be reproduced. Any idea if that was included in the study and I somehow overlooked it?

mistermann 7y ago

> As noted in TFA, they looked at 17 forecasts from 14 models Pardon a little pedantry: >> The researchers compared annual average surface temperatures across the globe to the surface temperatures predicted in 17 forecasts. Those predictions were drawn from 14 separate computer models released between 1970 and 2001. In some cases, the studies and their computer codes were so old that the team had to extract data published in papers, using special software to gauge the exact numbers represented by points on a printed graph. They compared annual average surface temperatures across the globe to 17 forecasts, from 14 separate models. But....how many models did they look at, before choosing those particular 14 models? TFA doesn't say. Articles written in this style provide rich fodder for conspiracy theorists, particularly because there is no shortage of examples in the past where authoritative, trustworthy organizations have gotten caught in lies. Rare is the climate change article I've read that can't easily have similar holes poked in it. What's the real truth here? Based on the literal content of this article, no one knows. It is speculation vs speculation. I've said it before, and I will say it again: if your persuasion tactics aren't working, it might be worth considering whether you should change your tactics.

hcknwscommenter 7y ago
TFA article does say. They looked at 17 forecasts drawn from 14 models. You are the one who implicitly believes they are lying about that and only reporting 17 out of some huge unknown number. The reality is that the vast majority (>99%) of climate change research does not produce the type of forecast, or really any forecast at all, that one could do this analysis on. Therefore, the 17 forecast number is actually very easy to believe.
- whokal 7y ago
  With sincerity, I think you may misunderstand what GP is pointing out. I understood “forecasts” to be a simulation run using a model. In that case, they took 14 models and ran those each at least once to come up with 17 predictions. What GP wants is transparency around how many possible models were there over the last decades from which they chose the 14. This gives the reader a better understanding of whether 14 is a lot or a little. It is a fair point IMO regardless of their overall stance on the issue.
  - mturmon 7y ago
    They did a literature search and chose what was reproducible. This has been covered in TFA and highlighted by a comment nearby: https://news.ycombinator.com/item?id=21709446 https://news.ycombinator.com/item?id=21709446
- ekianjo 7y ago
  How many models are currently published for reference?
- jointpdf 7y ago
  Simulation is one way to produce a numerical forecast, but the prediction of any forecast model would never be the result of a single simulation run. Model predictions are always (or should always) be reported as an expected value, or the average over many simulation runs. The expected value of a random variable (e.g. mode output) is probably the most fundamental concept in all of statistics/probability, and many mind-wrenching (read: wrong+incoherent) internet hot takes are rooted in logical fallacies about the science of prediction. The reason that there are more predictions (17) than models (14) is that several of the models (eg Hansen 1981) consider several distinct “scenarios” based on differing assumptions about future levels of GHG emissions. Secondly, the forecast models evaluated in the study are from the likes of James Hansen, William Nordhaus, and the IPCC (in other words, authoritative sources). So at the very least this adds weight to the hypothesis that the models published by the IPCC are accurate and methodologically sound. Sure, there is some probability that the models and reality corresponded by random chance, but that probability is made ever more diminishingly small as evidence accumulates. So, I don’t think any of the comments in this thread that accuse the authors of retrospective cherry-picking of climate models are anything but embarrassing examples of pseudo-intellectual contrarianism and/or confusion over basic statistics. The models considered by the study, their predictions, and the methodology used are all located here and were easily found after 30s of digging: https://github.com/hausfath/OldModels https://github.com/hausfath/OldModels
jartelt 7y ago
Even if they said how many models they looked at before choosing 14 models (and explained why they chose those 14), conspiracy theorists and people skepticalof climate science would still argue that it is not legitimate because of survivorship bias and several other reasons. "You looked at 1000 models, but what about the 1000 models that scientists made and didn't publish. You didn't look at those!" People can always suggest more tests you need to do or poke holes in any analysis. That doesn't mean you just stop ding the work. Overall it is impressive that different models made over 30 years can predict fairly accurate results. In my opinion that shows that the underlying physics of the models is correct and makes me trust the models even more.
- mistermann 7y ago
  > Even if they said how many models they looked at before choosing 14 models (and explained why they chose those 14), conspiracy theorists and people skeptical of climate science would still argue that it is not legitimate because of survivorship bias and several other reasons. Undoubtedly some would, but are you willing to consider the possibility that the number/percentage of people that would do so might be altered by the language used in articles on climate science? > People can always suggest more tests you need to do or poke holes in any analysis. Some analyses are easier to poke holes in than others. If this article was tightened up in the manner I describe, excluding the possibility of survivorship bias, then an entirely new approach to denial would be required. As written, survivorship bias is a very real possibility. Wikipedia says: "Survivorship bias or survival bias is the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility. This can lead to false conclusions in several different ways. It is a form of selection bias." Survivorship bias is a real phenomenon, not a conspiracy theory. It is neither illogical or unscientific to consider the possibility that is might be in play behind the scenes here, and the article as written makes no attempt to rule it out. > Overall it is impressive that different models made over 30 years can predict fairly accurate results. In my opinion that shows that the underlying physics of the models is correct and makes me trust the models even more. I suspect this is because your mind does not consider it a possibility that the models discussed do not result from survivorship bias. A conspiratorial mind however, would most likely strongly consider that a possibility, and therefore compute a completely different level of trust. Imagine for a moment how the human mind works when reading something. Typically, if the mind is primed with preexisting beliefs, it is going to be actively searching (interpreting) for confirmation of those beliefs, and this behavior will likely be stronger if the topic happens to be one associated with one's identity. So, your mind processes an article such as this optimistically (uncritically), whereas a "denier" or "conspiracy theorist" is going to read it pessimistically (critically). https://en.wikipedia.org/wiki/Confirmation_bias#Biased_interpretation https://en.wikipedia.org/wiki/Confirmation_bias#Biased_inter... https://www.verywellmind.com/what-is-a-confirmation-bias-2795024 https://www.verywellmind.com/what-is-a-confirmation-bias-279... Here is another interesting example. See this reddit thread on the article: https://www.reddit.com/r/science/comments/e63ic5/of_17_climate_models_published_between_the_early/ https://www.reddit.com/r/science/comments/e63ic5/of_17_clima... The top comment is: (/u/aClimateScientist) "Hi all, I'm a coauthor of this paper (Henri Drake) and happy to answer questions." Someone asks the question: "How did you account for survivorship bias of models? IOW how did you select the 17? I mean if I asked 1,000 psychics to predict GDP growth, I could probably find 10-20 who were pretty close and say that that proves they're clairvoyant." As of when I'm writing this, neither /u/aClimateScientist nor /u/avogadros_number (the article submitter, who seems to have considerable background in climate science) have addressed this question. (Or, same thread: a moderator [Removed] comment.) My question to you is: do you think it is possible that the manner in which your mind evaluates this particular situation (an unanswered question, in a thread with 29 total comments) may differ from how the mind of a "denier" or "conspiracy theorist" might evaluate the situation, and how that evaluation may in turn be used (or not) to tune your respective mental models and subconscious heuristics? A followup question is: do you believe this perspective is irrelevant (has no bearing on) the climate change dilemma? ------------------------------ I sometimes wonder if the climate change dilemma was reproduced in a sim-style video game, would the human mind be less resistant to recognizing and reacting to the behavior of their adversary? If the adversary is a machine rather than a human, might the tribalism component of subconscious decision making decrease? This is one of those things that seems so obvious, I wonder if there might be some studies on it. As shocked as people here are at the continued existence of "deniers" (or more accurately, their mental models of deniers), I am similarly shocked at the psychological unwillingness of intelligent people to acknowledge certain unfortunate realities. It's kind of comically/tragically ironic if you think about it.

Akababa 7y ago

But we don't know how those 17 were chosen in the first place. There would be survivorship bias if models which match observations get more attention. With that being said, I doubt that survivorship bias is strong enough to explain their combined result of 10/17 models being significant.

jointpdf 7y ago
Since one of authors posted the paper, we do know: > “We conducted a literature search to identify papers published prior to the early-1990s that include climate model outputs containing both a time-series of projected future GMST (with a minimum of two points in time) and future forcings (including both a publication date and future projected atmospheric CO2 concentrations, at a minimum). Eleven papers with fourteen distinct projections were identified that fit these criteria.
- mistermann 7y ago
  > Eleven papers with fourteen distinct projections were identified that fit these criteria. This brings to mind the old joke: "I used to smoke weed. I still do, but I used to, too." Specifically, the phrase "were identified that fit these criteria" sets off my skepticism spider senses. Kind of like the popular "97% of scientists..." meme, that doesn't quite hold water when you dig under the covers. To be clear, I'm not saying deceit is taking place, I am simply saying that I am skeptical. I wasn't able to find any details on the specifics of the literature search, such that the search could be reproduced. Any idea if that was included in the study and I somehow overlooked it?