5 ms·

How accurate are models of complex systems? Consider that if highly accurate models were really possible someone could make one that would accurately predict N

by fwungy 4y ago

How accurate are models of complex systems? Consider that if highly accurate models were really possible someone could make one that would accurately predict NFL game outcomes. When that happens we will know it because Sports Books would no longer take bets on NFL games. Sports Books profit because they get a slight statistical advantage which pays off due to bettor volume. So we know they can't make an accurately predicting model for football. How accurate can they possibly get with climate change modeling? It's orders of magnitude more difficult. Football is formally rule bounded and tons of data on a long history of games is readily available. In contrast, the vast majority of climate data is based on theoretical methods. This isn't bad, it's just far less precise because the variance of all those theory based estimators accumulates across the model, widening the confidence interval of its outputs to a point where it may not be useful. Finally, climate scientists are people. They have careers, families, financial responsibilities. What happens if they say the climate is fine? Some other crisis gets the funding and their careers are destroyed. They can't just switch to a different science, they've invested a lot of time and money becoming a specific thing. I found this article very wordy and skimmed parts, but it looks to me like a goalpost moving exercise to explain why their aggressive predictions for the present time didn't come true.

PartiallyTyped 4y ago
I cant tell whether the parent comment is disingenuous though it looks a lot like it, but I will take it at face value. The models created by the oil companies back in the 70s seem to be “breathtakingly accurate” [1] Second, the problem with your analogy is that with football you expect the prediction to be X team will win, and similarly expect the predictions to be X weather event at $place will occur. The problem with your expectations of predictions regarding the latter is the sample space. The sample space is so large that extreme weather events will occur $somewhere. Predicting exactly where and their trajectories is difficult due to chaos and the complexity of the system. There is a bound on our capacity to predict exact events long ahead of time. However, what we can do is give a probability of the event occurring over a large enough region. When you scale said region to the earth, we can ignore very small details and focus on the bigger picture, which is that the variance in weather in creases. The problem with an increasing variance is that extreme events occur more frequently in all directions. Meaning high and low temperatures, droughts and floods and so on. To observe this in action, a year or so ago europe had severe floods that killed many people and simultaneously over the next summer experienced historic droughts. The sample space strikes again and we saw historic droughts in China too. [1] https://www.theguardian.com/business/2023/jan/12/exxon-climate-change-global-warming-research https://www.theguardian.com/business/2023/jan/12/exxon-clima... [2] Firmament, by Dr Simon Clark
strgcmc 4y ago
I'm sorry, but the sports betting analogy is hilariously wrongheaded. In order to set any prices at all, books have reasonably good models for the sports they offer, including NFL. But these models are not and do not need to be perfect, in order for a profitable business model to exist. There are in fact, plenty of "sharps" who can beat the books on certain sports or certain types of markets (e.g. maybe you can model O/U markets very well, but money line not so much). The book does not "stop taking bets" on the entire event in question. The book simply has to limit the "sharp" individually. Of course then we get into the whole cat and mouse game of chasing sharp action through straw bettors and playing games of information theory (i.e. how much money can a sharp win while still plausibly not outing themselves as an obvious sharp). The book only would have a serious problem, if the entire world became sharp and every bettor was betting sharply. But alas there you have a paradox: anyone who is currently a sharp, has a strong self interest to preserve the secrecy of their models and methods, because knowing something that nobody else knows is precisely what creates their edge. And so therefore, as long as sharps have this self-interest, then there is no problem for books to keep operating, and the problem is reduced to one of identifying and limiting sharps occasionally (basically a "cost of doing business" that books acknowledge). The only way for the scenario you describe to happen, is for disinterested actors with no thought of profit for themselves, to not only "solve" a particular market, but also release it to the entire world and make it a commodity. I'm not saying it's impossible, but I'm saying that normal economic incentives work against this outcome.
- fwungy 4y ago
  The book just moves the line so they're covered in either direction. The sharp works on inside knowledge, not advanced mathematic modeling. The sharp is someone with inside information, e.g. star player twisted his ankle recently and no one knows. Bit if any moron could pull up a perfect model of the game outcome they'd all do it and the book would have no line to play. Books are happy to take action on anything where they have the advantage and are always wary of sharps ( won't take their money if known). I really could have picked any prediction model though so we shouldn't get hung up on sports Books. For example, why are banks still lending big money on expensive sea side property? Shouldn't their internal modeling be showing that these places will soon be flooded? Oceanside property should be a bargain right now. It isn't though, because the people on charge don't actually believe in the climate change models they push.
wizzwizz4 4y ago
> How accurate can they possibly get with climate change modeling? At least as accurate as weather reports, which can be pretty reliable days in advance. The outcome of a football match often can't be predicted accurately during the match. This should suggest that the atmosphere and a sport are qualitatively different things. > What happens if they say the climate is fine? They go into some other statistical modelling field, probably to do with meteorology, and we start to get accurate weather reports two weeks in advance. It's not like high-level sports players, who have specialised into that sport, and are unlikely to ever be high-level players of any other sport. (If anything, there are incentives for high-level sports players to throw games to keep them more exciting! Maybe that's why we can't predict the games? It's not, but what if it were?)
IngoBlechschmid 4y ago
1. Climate is a long-term average. Long-time averages are much simpler to predict than individual outcomes. You cannot predict the result of a coin toss, but you can be reasonably confident that among 1000 tosses there will be between 400 and 600 occurrences of heads. 2. Scientists, especially young ones, have an incentive to challenge dominating scientific views because that is how you make yourself known in the scientific community. Being known can be transformed to getting the next academic position.
- fwungy 4y ago
  We have millions of years of climate history of which only a tiny fraction has substantial data. The rest is theoretical data that can only be verified by a web of theoretical means. Variance accumulates across models. The less actual data you have the wider the variance is on your estimators. When your data is mostly projected the confidence interval on your outcome makes it meaningless, e.g. average temperatures will change by plus or minus 20 degrees.

by fwungy 4y ago

PartiallyTyped 4y ago

I cant tell whether the parent comment is disingenuous though it looks a lot like it, but I will take it at face value. The models created by the oil companies back in the 70s seem to be “breathtakingly accurate” [1] Second, the problem with your analogy is that with football you expect the prediction to be X team will win, and similarly expect the predictions to be X weather event at $place will occur. The problem with your expectations of predictions regarding the latter is the sample space. The sample space is so large that extreme weather events will occur $somewhere. Predicting exactly where and their trajectories is difficult due to chaos and the complexity of the system. There is a bound on our capacity to predict exact events long ahead of time. However, what we can do is give a probability of the event occurring over a large enough region. When you scale said region to the earth, we can ignore very small details and focus on the bigger picture, which is that the variance in weather in creases. The problem with an increasing variance is that extreme events occur more frequently in all directions. Meaning high and low temperatures, droughts and floods and so on. To observe this in action, a year or so ago europe had severe floods that killed many people and simultaneously over the next summer experienced historic droughts. The sample space strikes again and we saw historic droughts in China too. [1] https://www.theguardian.com/business/2023/jan/12/exxon-climate-change-global-warming-research https://www.theguardian.com/business/2023/jan/12/exxon-clima... [2] Firmament, by Dr Simon Clark

strgcmc 4y ago

I'm sorry, but the sports betting analogy is hilariously wrongheaded. In order to set any prices at all, books have reasonably good models for the sports they offer, including NFL. But these models are not and do not need to be perfect, in order for a profitable business model to exist. There are in fact, plenty of "sharps" who can beat the books on certain sports or certain types of markets (e.g. maybe you can model O/U markets very well, but money line not so much). The book does not "stop taking bets" on the entire event in question. The book simply has to limit the "sharp" individually. Of course then we get into the whole cat and mouse game of chasing sharp action through straw bettors and playing games of information theory (i.e. how much money can a sharp win while still plausibly not outing themselves as an obvious sharp). The book only would have a serious problem, if the entire world became sharp and every bettor was betting sharply. But alas there you have a paradox: anyone who is currently a sharp, has a strong self interest to preserve the secrecy of their models and methods, because knowing something that nobody else knows is precisely what creates their edge. And so therefore, as long as sharps have this self-interest, then there is no problem for books to keep operating, and the problem is reduced to one of identifying and limiting sharps occasionally (basically a "cost of doing business" that books acknowledge). The only way for the scenario you describe to happen, is for disinterested actors with no thought of profit for themselves, to not only "solve" a particular market, but also release it to the entire world and make it a commodity. I'm not saying it's impossible, but I'm saying that normal economic incentives work against this outcome.

fwungy 4y ago
The book just moves the line so they're covered in either direction. The sharp works on inside knowledge, not advanced mathematic modeling. The sharp is someone with inside information, e.g. star player twisted his ankle recently and no one knows. Bit if any moron could pull up a perfect model of the game outcome they'd all do it and the book would have no line to play. Books are happy to take action on anything where they have the advantage and are always wary of sharps ( won't take their money if known). I really could have picked any prediction model though so we shouldn't get hung up on sports Books. For example, why are banks still lending big money on expensive sea side property? Shouldn't their internal modeling be showing that these places will soon be flooded? Oceanside property should be a bargain right now. It isn't though, because the people on charge don't actually believe in the climate change models they push.

wizzwizz4 4y ago

> How accurate can they possibly get with climate change modeling? At least as accurate as weather reports, which can be pretty reliable days in advance. The outcome of a football match often can't be predicted accurately during the match. This should suggest that the atmosphere and a sport are qualitatively different things. > What happens if they say the climate is fine? They go into some other statistical modelling field, probably to do with meteorology, and we start to get accurate weather reports two weeks in advance. It's not like high-level sports players, who have specialised into that sport, and are unlikely to ever be high-level players of any other sport. (If anything, there are incentives for high-level sports players to throw games to keep them more exciting! Maybe that's why we can't predict the games? It's not, but what if it were?)

IngoBlechschmid 4y ago

1. Climate is a long-term average. Long-time averages are much simpler to predict than individual outcomes. You cannot predict the result of a coin toss, but you can be reasonably confident that among 1000 tosses there will be between 400 and 600 occurrences of heads. 2. Scientists, especially young ones, have an incentive to challenge dominating scientific views because that is how you make yourself known in the scientific community. Being known can be transformed to getting the next academic position.

fwungy 4y ago
We have millions of years of climate history of which only a tiny fraction has substantial data. The rest is theoretical data that can only be verified by a web of theoretical means. Variance accumulates across models. The less actual data you have the wider the variance is on your estimators. When your data is mostly projected the confidence interval on your outcome makes it meaningless, e.g. average temperatures will change by plus or minus 20 degrees.