4 ms·

I know I'm being cynical, but I'm not sure anyone understands these things. If the top scientists from well-funded teams came up with estimates that were so wil

I know I'm being cynical, but I'm not sure anyone understands these things. If the top scientists from well-funded teams came up with estimates that were so wildly off, I'm not sure why we have any hope of getting these things right. We need a better philosophical explanation. If you ask me, the biggest issue is that they assume that it just spreads kind of randomly. That doesn't seem to be what happened with COVID-19. It flourished in some parts and limped along in others. We have theories about subways or air pollution but none are very well correlated. And then there's Farr's law which doesn't match these models very well.

btrettel 6y ago
> If the top scientists from well-funded teams came up with estimates that were so wildly off, I'm not sure why we have any hope of getting these things right. A "top" scientist produces a ton of publications and brings in a lot of grant money. You don't need to be right to be a "top" scientist. Change the incentives and you'll change the outcomes. The failure of blind predictions (i.e., without the opportunity to calibrate the model to the data) says more about the culture than whether this is possible. Weather forecasting does a fairly good job: https://journals.ametsoc.org/doi/full/10.1175/2011MWR3525.1 https://journals.ametsoc.org/doi/full/10.1175/2011MWR3525.1 It's instructive to figure out why. I'd say it's because weather forecasters have a ton of data, forecasts can quickly be proven right or wrong (so they get a lot of feedback), and they have a culture emphasizing accuracy. It also helps that they're modeling physics, though various submodels (turbulence, etc.) may not be any more credible than the models in epidemiology. Blind predictions have been tried in a variety of domains. I'm most familiar with computational fluid dynamics, e.g.: https://www.sciencedirect.com/science/article/abs/pii/S0379711209000034 https://www.sciencedirect.com/science/article/abs/pii/S03797... https://ntrs.nasa.gov/search.jsp?R=19970027380 https://ntrs.nasa.gov/search.jsp?R=19970027380 (There are many more examples.) As I recall, it's not uncommon for different groups even using the same software to come up with radically different predictions. I'd say scientists and engineers need much more practice making blind predictions. Blind prediction exercises should occur regularly. There's a yearly competition to predict the heat release rate of a Christmas tree: https://fpe.umd.edu/events/christmas-fire-safety-demo https://fpe.umd.edu/events/christmas-fire-safety-demo I entered one year. Rather than using a fancy computer model, I merely smoothed last year's data and scaled it by the ratio of the mass of this year's tree to the mass of last year's tree. As I recall, I did better than most!
jandrewrogers 6y ago
The models are insufficiently multidisciplinary and some of the needed expertise is almost entirely outside of academia and scientific circles whence expertise is drawn to create the models. This is really evident when you look at the epidemiological models that were actually used. What you find in practice is that the quality of the myriad sub-models that make up the models is extremely uneven, and reflect the expertise of the people building the model, both good and bad. Some key elements of the models are expertly designed, but other critical elements are so naively designed for lack of domain expertise that it completely destroys any signal one might hope to find from the expertly designed parts. This is unfortunately endemic, I've seen this in every model I've been asked to look at. As for why multidisciplinary expertise is not used to build these models, in my experience there is a lot of gatekeeping against non-academics, even though critical state-of-the-art modeling research and know-how on things like population-scale behavioral dynamics is done almost completely outside academia.
saeranv 6y ago
> If the top scientists from well-funded teams came up with estimates that were so wildly off, I'm not sure why we have any hope of getting these things right. I think there's a lot of confusion from the general public about how to evaluate models. A modeling being "correct" is sometimes not possible, and useful, sometimes a model has to be used instead to reduce uncertainty, not to exactly predict X amount of cases in Y amount of time. There's just so much randomness in systems that prevent accuracy in predictions.

btrettel 6y ago

> If the top scientists from well-funded teams came up with estimates that were so wildly off, I'm not sure why we have any hope of getting these things right. A "top" scientist produces a ton of publications and brings in a lot of grant money. You don't need to be right to be a "top" scientist. Change the incentives and you'll change the outcomes. The failure of blind predictions (i.e., without the opportunity to calibrate the model to the data) says more about the culture than whether this is possible. Weather forecasting does a fairly good job: https://journals.ametsoc.org/doi/full/10.1175/2011MWR3525.1 https://journals.ametsoc.org/doi/full/10.1175/2011MWR3525.1 It's instructive to figure out why. I'd say it's because weather forecasters have a ton of data, forecasts can quickly be proven right or wrong (so they get a lot of feedback), and they have a culture emphasizing accuracy. It also helps that they're modeling physics, though various submodels (turbulence, etc.) may not be any more credible than the models in epidemiology. Blind predictions have been tried in a variety of domains. I'm most familiar with computational fluid dynamics, e.g.: https://www.sciencedirect.com/science/article/abs/pii/S0379711209000034 https://www.sciencedirect.com/science/article/abs/pii/S03797... https://ntrs.nasa.gov/search.jsp?R=19970027380 https://ntrs.nasa.gov/search.jsp?R=19970027380 (There are many more examples.) As I recall, it's not uncommon for different groups even using the same software to come up with radically different predictions. I'd say scientists and engineers need much more practice making blind predictions. Blind prediction exercises should occur regularly. There's a yearly competition to predict the heat release rate of a Christmas tree: https://fpe.umd.edu/events/christmas-fire-safety-demo https://fpe.umd.edu/events/christmas-fire-safety-demo I entered one year. Rather than using a fancy computer model, I merely smoothed last year's data and scaled it by the ratio of the mass of this year's tree to the mass of last year's tree. As I recall, I did better than most!

jandrewrogers 6y ago

The models are insufficiently multidisciplinary and some of the needed expertise is almost entirely outside of academia and scientific circles whence expertise is drawn to create the models. This is really evident when you look at the epidemiological models that were actually used. What you find in practice is that the quality of the myriad sub-models that make up the models is extremely uneven, and reflect the expertise of the people building the model, both good and bad. Some key elements of the models are expertly designed, but other critical elements are so naively designed for lack of domain expertise that it completely destroys any signal one might hope to find from the expertly designed parts. This is unfortunately endemic, I've seen this in every model I've been asked to look at. As for why multidisciplinary expertise is not used to build these models, in my experience there is a lot of gatekeeping against non-academics, even though critical state-of-the-art modeling research and know-how on things like population-scale behavioral dynamics is done almost completely outside academia.

saeranv 6y ago

> If the top scientists from well-funded teams came up with estimates that were so wildly off, I'm not sure why we have any hope of getting these things right. I think there's a lot of confusion from the general public about how to evaluate models. A modeling being "correct" is sometimes not possible, and useful, sometimes a model has to be used instead to reduce uncertainty, not to exactly predict X amount of cases in Y amount of time. There's just so much randomness in systems that prevent accuracy in predictions.