6 ms·

Leaving modeling up to the professionals is completely the wrong lesson. Time series forecasting from business, epidemiological or economic data forecasting is

Leaving modeling up to the professionals is completely the wrong lesson. Time series forecasting from business, epidemiological or economic data forecasting is hard so you should always take the results with a large grain of salt. The professionals get it wrong too and just like the weather nearby points from the model are more likely to be accurate than distant ones. Smoothing helps. Estimating error helps. Domain knowledge helps. Experience in applying the model helps. There are techniques other than the one demonstrated in the article. Sometimes those help. There are also professionals who create models which are used to support the bias of the modeler or the modelers employer (i.e. cherry picking). Politicians and pundits more often than not take the results of these models and draw conclusions which are unwarranted or at least highly uncertain without mentioning the uncertainty or only giving it lip service.

glofish 7y ago
yes, I do think that there is nothing wrong with fitting the data and asking questions suggesting that only an "expert" can possibly fit the data correctly is the wrong conclusion
- kurthr 7y ago
  It's certainly true that forcasting using exponential fits is bound to have huge error bars, but it also seems foolish to "show" that it can't be accurately done early on a curve without noticing that it is fairly easy to fit (and relatively stable) once it becomes non-exponential (e.g. once rate of change is constant). Of course in many cases there are external bounds that are similarly useful, for example when you know the maxima (e.g. full population size). Then the question to the early data is simply whether a fit to an exponential is "better" than successively higher order polynomials. Currently, this forcasting information seems not particularly helpful at all for fitting epidemiological data for Covid19 since nowhere do we have data for a case where strict distancing hasn't been an effect of high death and hospitalization rates well before 50% of the population would provide any effective immunity.
wenc 7y ago
Yes. On one hand, I tire of numerous hot takes that start with "I'm not an epidemiologist, but..." which is followed by opinion un-anchored by practical experience and are coupled with a disregard for what has come before (community context). There are also a number of people in adjacent fields who are taking advantage of the situation to advance their careers. On the other hand, I realize most experts carry the baggage of unrecognized assumptions in their toolkit, which need to be challenged (at the first principles level) by people outside the domain who don't carry the same baggage. The ideal situation would be to leverage the expertise of both epidemiologists and non-epidemiologists on a common platform (that is not Twitter) and have them check and build on top each other's work.
- stev_stev 7y ago
  This is maybe the only sane answer I have ever seen on these kinds of topics. I wish I could have come up with something this succinct.
Spooky23 7y ago
Uncertainty is part of the business for something like this. Good faith, open communication is essential to public expense. That’s one of the reasons why some of the governors in the US are getting accolades as their approach to communication contrasts against other political figures with a more political focused approach. Capturing and interpreting risk appropriately is more important than being right. The various models scoped the problem, but it’s hard to model things like a complete breakdown of federal response.
khawkins 7y ago
I wish more of society would apply this wisdom to climate change modelling.
diehunde 7y ago
True. Also, if you read Superforecasting[1] you'll learn about several experiments where "non-experts" beat experts by a lot by just being well informed. Even when the experts had access to confidential data. [1] https://www.amazon.com/Superforecasting-Science-Prediction-Philip-Tetlock/dp/0804136718 https://www.amazon.com/Superforecasting-Science-Prediction-P...
- stev_stev 7y ago
  I’m very skeptical of evidentiary patterns laid out in NYT bestseller style books.
  - diehunde 7y ago
    His project[1] was part of program created by IARPA[2] so I wouldn't be that skeptical [1]https://en.wikipedia.org/wiki/The_Good_Judgment_Project https://en.wikipedia.org/wiki/The_Good_Judgment_Project [2]https://en.wikipedia.org/wiki/Intelligence_Advanced_Research_Projects_Activity https://en.wikipedia.org/wiki/Intelligence_Advanced_Research...
    - stev_stev 7y ago
      Are these what the book is based off of? Or are these just the credentials of the author? In the former case I would find that more compelling to be worth looking into than just the link to the amazon book. In any case, I wasn’t trying to specifically dismiss the book (which I have not read) but just pointing out that linking to book of that sort doesn’t give me (personally) the feeling that there is much weight behind the citation. The other links you provided do the opposite!
      - diehunde 7y ago
        Yes the book is based on that project.
        stev_stev 7y ago
eanzenberg 7y ago
Cool so does this mean climate models, with enormous look-ahead windows, are complete garbage?
- gjm11 7y ago
  Nope. (Sorry. I take it you would like them to be complete garbage.) There are two main ways in which a model can give bad results. 1. It can be a fundamentally bad model. E.g., Kelvin had a perfectly reasonable model of the long-term evolution of the temperature of the earth from which he deduced that it couldn't be more than a few hundred million years old, probably less. The problem was that he didn't know about radioactivity, which turns out to make a tremendous difference. 2. It can be a reasonable model but the parameters can be hard to estimate accurately, and this can lead to inaccurate predictions. In some cases, small errors in the parameters lead to large errors in the predictions. For instance, if you try to forecast the weather a year ahead then you will get all the details wrong. This article is about the second kind of problem, and that's the one that shows the most dramatic degradation as you look at longer and longer intervals. Different properties of the model output can show very different sensitivities to the inputs. For instance, suppose you make a model of the solar system and try to predict where the planets will be ten million years from now. I expect (though I haven't actually done the relevant calculations) you will find that tiny errors in your measurements of the planets' present positions and momenta will lead to very large errors in estimating how far around their orbits the planets are, and probably also in estimating the precession of the orbits. But I bet that almost every simulation run will put the planets' distances from the sun at about the same values, which will be about the same as they are now, and I would bet very heavily that those distances will be a pretty good match for reality. (Barring utterly out-of-model events like the human race getting technologically advanced enough to move planets around and finding it useful to do so.) Or suppose you make a model of the weather and try to predict things five years from now. The model will do very poorly at predicting which days will be rainy in (say) New York, and if you run it many times with tiny changes in its parameters it will give completely different predictions. But it will very consistently tell you that in New York it'll be warmer in July than in January, and it will almost certainly be right; it will do pretty well at predicting rainfall in the Sahara Desert, and it will almost certainly be right about that. Some things behave chaotically, some things not so much. So, what about those climate models? Weather is chaotic: small variations quickly lead to huge changes. Climate is much less so: to whatever extent the models are fundamentally accurate, small changes to the parameters tend to produce small changes to outputs like global mean temperature ten years later. How fundamentally accurate are they? They seem to be pretty decent; when you go back and look at the predictions of old models for what are now past years, they do quite well. (To predict the climate you need to predict human behaviour, because human behaviour affects the climate. Some of the predictions of human behaviour that went into the models were wrong. So you can get a sort of halfway house by running an old model but using now-historical information about things like how much CO2 we would put into the atmosphere. Unsurprisingly, this improves their performance relative to just using the old models as they stand.) See e.g. https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019GL085378 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019... for some analysis. So: no, the sort of effects described in the article don't mean that climate models are complete garbage. The other sort of problem (fundamental errors in the model itself as opposed to its parameters) could make them garbage, but the evidence available so far suggests that even decades-old models were pretty good in this respect.
FabHK 7y ago
> Time series forecasting […] is hard […] > Leaving modeling up to the professionals is completely the wrong lesson. I don’t understand what you’re saying. Nobody is advocating that amateurs be precluded from modeling to their heart’s content. So what are you advocating? One should discard models from professionals? One should prefer models from non-experts? One should crowdsource models? One should make policy decisions based on gut instinct?
- ignoramous 7y ago
  > I don’t understand what you’re saying. I think what GP means is, leaving it upto professionals alone is a recipe for disaster because of the inherent corruption, greed, biases, and prejudices at play. > So what are you advocating? Not speaking for GP, but simpler models are more likely to be a good approximation and more than enough, anyway [0]? Simplicity helps because it stops the pundits and politicians from sort of running away with it? [0] Newton's theory of gravitation has been supplanted by Einstein's theory of relativity and yet Newton's theory remains generally "empirically adequate". Indeed, Newton's theory generally has excellent predictive power. Yet Newton's theory is not an approximation of Einstein's theory. For illustration, consider an apple falling down from a tree. Under Newton's theory, the apple falls because Earth exerts a force on the apple—what is called "the force of gravity". Under Einstein's theory, Earth does not exert any force on the apple. Hence, Newton's theory might be regarded as being, in some sense, completely wrong but extremely useful (The usefulness of Newton's theory comes partly from being vastly simpler, both mathematically and computationally, than Einstein's theory). https://en.wikipedia.org/wiki/All_models_are_wrong https://en.wikipedia.org/wiki/All_models_are_wrong
  - FabHK 7y ago
    > because of the inherent corruption, greed, biases, and prejudices at play. You and I must be thinking of different professionals. At any rate, here are two very very very simple models of this epidemic: a) cases will grow linearly, b) cases will grow exponentially. Super simple. extremely different outcome. Now, one still has to make policy decisions. I think the solution is not fewer professionals, but more and better professionals.
deleted 7y ago
[deleted]

glofish 7y ago

yes, I do think that there is nothing wrong with fitting the data and asking questions suggesting that only an "expert" can possibly fit the data correctly is the wrong conclusion

kurthr 7y ago
It's certainly true that forcasting using exponential fits is bound to have huge error bars, but it also seems foolish to "show" that it can't be accurately done early on a curve without noticing that it is fairly easy to fit (and relatively stable) once it becomes non-exponential (e.g. once rate of change is constant). Of course in many cases there are external bounds that are similarly useful, for example when you know the maxima (e.g. full population size). Then the question to the early data is simply whether a fit to an exponential is "better" than successively higher order polynomials. Currently, this forcasting information seems not particularly helpful at all for fitting epidemiological data for Covid19 since nowhere do we have data for a case where strict distancing hasn't been an effect of high death and hospitalization rates well before 50% of the population would provide any effective immunity.

wenc 7y ago

Yes. On one hand, I tire of numerous hot takes that start with "I'm not an epidemiologist, but..." which is followed by opinion un-anchored by practical experience and are coupled with a disregard for what has come before (community context). There are also a number of people in adjacent fields who are taking advantage of the situation to advance their careers. On the other hand, I realize most experts carry the baggage of unrecognized assumptions in their toolkit, which need to be challenged (at the first principles level) by people outside the domain who don't carry the same baggage. The ideal situation would be to leverage the expertise of both epidemiologists and non-epidemiologists on a common platform (that is not Twitter) and have them check and build on top each other's work.

stev_stev 7y ago
This is maybe the only sane answer I have ever seen on these kinds of topics. I wish I could have come up with something this succinct.

Spooky23 7y ago

Uncertainty is part of the business for something like this. Good faith, open communication is essential to public expense. That’s one of the reasons why some of the governors in the US are getting accolades as their approach to communication contrasts against other political figures with a more political focused approach. Capturing and interpreting risk appropriately is more important than being right. The various models scoped the problem, but it’s hard to model things like a complete breakdown of federal response.

khawkins 7y ago

I wish more of society would apply this wisdom to climate change modelling.

diehunde 7y ago

True. Also, if you read Superforecasting[1] you'll learn about several experiments where "non-experts" beat experts by a lot by just being well informed. Even when the experts had access to confidential data. [1] https://www.amazon.com/Superforecasting-Science-Prediction-Philip-Tetlock/dp/0804136718 https://www.amazon.com/Superforecasting-Science-Prediction-P...

stev_stev 7y ago
I’m very skeptical of evidentiary patterns laid out in NYT bestseller style books.
- diehunde 7y ago
  His project[1] was part of program created by IARPA[2] so I wouldn't be that skeptical [1]https://en.wikipedia.org/wiki/The_Good_Judgment_Project https://en.wikipedia.org/wiki/The_Good_Judgment_Project [2]https://en.wikipedia.org/wiki/Intelligence_Advanced_Research_Projects_Activity https://en.wikipedia.org/wiki/Intelligence_Advanced_Research...
  - stev_stev 7y ago
    Are these what the book is based off of? Or are these just the credentials of the author? In the former case I would find that more compelling to be worth looking into than just the link to the amazon book. In any case, I wasn’t trying to specifically dismiss the book (which I have not read) but just pointing out that linking to book of that sort doesn’t give me (personally) the feeling that there is much weight behind the citation. The other links you provided do the opposite!
    - diehunde 7y ago
      Yes the book is based on that project.
      - stev_stev 7y ago

eanzenberg 7y ago

Cool so does this mean climate models, with enormous look-ahead windows, are complete garbage?

gjm11 7y ago
Nope. (Sorry. I take it you would like them to be complete garbage.) There are two main ways in which a model can give bad results. 1. It can be a fundamentally bad model. E.g., Kelvin had a perfectly reasonable model of the long-term evolution of the temperature of the earth from which he deduced that it couldn't be more than a few hundred million years old, probably less. The problem was that he didn't know about radioactivity, which turns out to make a tremendous difference. 2. It can be a reasonable model but the parameters can be hard to estimate accurately, and this can lead to inaccurate predictions. In some cases, small errors in the parameters lead to large errors in the predictions. For instance, if you try to forecast the weather a year ahead then you will get all the details wrong. This article is about the second kind of problem, and that's the one that shows the most dramatic degradation as you look at longer and longer intervals. Different properties of the model output can show very different sensitivities to the inputs. For instance, suppose you make a model of the solar system and try to predict where the planets will be ten million years from now. I expect (though I haven't actually done the relevant calculations) you will find that tiny errors in your measurements of the planets' present positions and momenta will lead to very large errors in estimating how far around their orbits the planets are, and probably also in estimating the precession of the orbits. But I bet that almost every simulation run will put the planets' distances from the sun at about the same values, which will be about the same as they are now, and I would bet very heavily that those distances will be a pretty good match for reality. (Barring utterly out-of-model events like the human race getting technologically advanced enough to move planets around and finding it useful to do so.) Or suppose you make a model of the weather and try to predict things five years from now. The model will do very poorly at predicting which days will be rainy in (say) New York, and if you run it many times with tiny changes in its parameters it will give completely different predictions. But it will very consistently tell you that in New York it'll be warmer in July than in January, and it will almost certainly be right; it will do pretty well at predicting rainfall in the Sahara Desert, and it will almost certainly be right about that. Some things behave chaotically, some things not so much. So, what about those climate models? Weather is chaotic: small variations quickly lead to huge changes. Climate is much less so: to whatever extent the models are fundamentally accurate, small changes to the parameters tend to produce small changes to outputs like global mean temperature ten years later. How fundamentally accurate are they? They seem to be pretty decent; when you go back and look at the predictions of old models for what are now past years, they do quite well. (To predict the climate you need to predict human behaviour, because human behaviour affects the climate. Some of the predictions of human behaviour that went into the models were wrong. So you can get a sort of halfway house by running an old model but using now-historical information about things like how much CO2 we would put into the atmosphere. Unsurprisingly, this improves their performance relative to just using the old models as they stand.) See e.g. https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019GL085378 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019... for some analysis. So: no, the sort of effects described in the article don't mean that climate models are complete garbage. The other sort of problem (fundamental errors in the model itself as opposed to its parameters) could make them garbage, but the evidence available so far suggests that even decades-old models were pretty good in this respect.

FabHK 7y ago

> Time series forecasting […] is hard […] > Leaving modeling up to the professionals is completely the wrong lesson. I don’t understand what you’re saying. Nobody is advocating that amateurs be precluded from modeling to their heart’s content. So what are you advocating? One should discard models from professionals? One should prefer models from non-experts? One should crowdsource models? One should make policy decisions based on gut instinct?

ignoramous 7y ago
> I don’t understand what you’re saying. I think what GP means is, leaving it upto professionals alone is a recipe for disaster because of the inherent corruption, greed, biases, and prejudices at play. > So what are you advocating? Not speaking for GP, but simpler models are more likely to be a good approximation and more than enough, anyway [0]? Simplicity helps because it stops the pundits and politicians from sort of running away with it? [0] Newton's theory of gravitation has been supplanted by Einstein's theory of relativity and yet Newton's theory remains generally "empirically adequate". Indeed, Newton's theory generally has excellent predictive power. Yet Newton's theory is not an approximation of Einstein's theory. For illustration, consider an apple falling down from a tree. Under Newton's theory, the apple falls because Earth exerts a force on the apple—what is called "the force of gravity". Under Einstein's theory, Earth does not exert any force on the apple. Hence, Newton's theory might be regarded as being, in some sense, completely wrong but extremely useful (The usefulness of Newton's theory comes partly from being vastly simpler, both mathematically and computationally, than Einstein's theory). https://en.wikipedia.org/wiki/All_models_are_wrong https://en.wikipedia.org/wiki/All_models_are_wrong
- FabHK 7y ago
  > because of the inherent corruption, greed, biases, and prejudices at play. You and I must be thinking of different professionals. At any rate, here are two very very very simple models of this epidemic: a) cases will grow linearly, b) cases will grow exponentially. Super simple. extremely different outcome. Now, one still has to make policy decisions. I think the solution is not fewer professionals, but more and better professionals.

deleted 7y ago

[deleted]