6 ms·

I know it's not a simple answer, but how would you embed checks to flag or highlight potentially incorrect results to the user?

kqr 4y ago
For physical processes, you can sometimes lean on conservation laws. Financial, no-arbitrage "laws". Sometimes it can help to run two different models, or different numerical methods, and compare their results. More generally, it's very, very hard. I am fairly sure there are multiple published results in computational fluid dynamics that are subtly wrong.
- meheleventyone 4y ago
  A lot of computer models are deliberately wrong but useful as well which is another ball game in terms of analysing results.
- badpun 4y ago
  > I am fairly sure there are multiple published results in computational fluid dynamics that are subtly wrong. I wonder what that means for the accuraccy of the climate models...
  - thrown_22 4y ago
    All models are wrong, some are useful.
  - geysersam 4y ago
    It's probably impossible to accurately simulate the climate of the planet decades into the future. That does not meaningfully detract from the evidence for human caused global warming however.
    - halffaday 4y ago
      I have to applaud the subtle elegance of the sarcasm here
      - geysersam 4y ago
        No sarcasm intended. The science of climate change rests on many legs. It was already established long before massive models on the scale we have today were possible. Computer simulations attempts to understand how it will affect us, but does not contribute much evidence.
  - jeffreyrogers 4y ago
    They are all wrong. Historically they've overpredicted warming, and then the models are updated to fit the (new) data better. They perform worse in areas with less data, like polar regions. To forestall knee-jerk downvotes: I'm not saying climate change isn't real or that anthropogenic global warming doesn't exist. I'm saying the models are not yet developed enough to predict very accurately. Early heliocentric models made poor predictions too because they assumed circular rather than elliptical orbits, they were still more "right" than geocentric models.
    - brutusborn 4y ago
      Can you suggest good places to look for more info on this? I have tried a few times to find info on the accuracy of these models and couldn't find much. And most models seem to be closed source.
    - leephillips 4y ago
      In fact, climate models going back over fifty years have performed quite well; surprisingly so: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019GL085378 https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/201... https://www.theguardian.com/environment/climate-consensus-97-per-cent/2016/jul/27/climate-models-are-accurately-predicting-ocean-and-global-warming https://www.theguardian.com/environment/climate-consensus-97...
      - jeffreyrogers 4y ago
        It depends on how you define "quite well". They are definitely directionally correct and in the right ballpark. This is a politically sensitive research area so the researchers are incentivized to make strong claims that you wouldn't make about, say, predicting sports outcomes or the stock market. The first link you posted is kind of punting on the hard part by saying that the reason the models overpredict warming is because CO2 didn't rise as much as they expected so if you put the actual observed CO2 concentration in then the temperature prediction comes out closer to what was observed. But the CO2 concentration is a parameter of the model, so they didn't capture its dynamics properly and then had to retroactively change it to get the observed data. Again, I want to reiterate that I'm not disputing the process of climate change or saying it's not a problem. I'm saying that modeling it is hard and historically the models have overestimated warming.
  - bluenose69 4y ago
    All manner of tests are done in fluid-dynamical modelling. Test suites are developed along with models. Simulations are done with a range of alternative models, or individual models with different conditions are applied, and the differences between the simulations are studied in detail. Models must rely on a fairly large set of parameterizations of processes that cannot be resolved directly, and tests are done to see which parameterizations make the most sense. Models stem from the academic environment, not the business environment. This means that the models are open-source, so everyone can see what everyone else is doing. And 'everyone' is a pretty large set. Note that there is no commercial element to any of this, which means there is no incentive to hide problems. It's the reverse: reputation is earned by finding problems, not by hiding them. A large part of the challenge of climate modelling is the leveraging of increasing computer power to resolve progressively smaller scales of motion, and this uncovers the need to understand those scales in isolation. Modelling is often used at this level also. None of this is to say that climate models are perfect. They obviously are not. But the system of academic science is very good at improving models and, importantly, exposing their limits. An indication of the latter is the pairing of uncertainties with predictions: a hallmark of this scientific community.
simplotek 4y ago
> I know it's not a simple answer, but how would you embed checks to flag or highlight potentially incorrect results to the user? For starters, some problems do have analytical solutions. Yo can compare a FEM model of these problems with the known analytical solution and see if it's close enough or not. One of them is the elasto-plastic plate with a hole. You can also run unit tests at the element level.

kqr 4y ago

For physical processes, you can sometimes lean on conservation laws. Financial, no-arbitrage "laws". Sometimes it can help to run two different models, or different numerical methods, and compare their results. More generally, it's very, very hard. I am fairly sure there are multiple published results in computational fluid dynamics that are subtly wrong.

meheleventyone 4y ago
A lot of computer models are deliberately wrong but useful as well which is another ball game in terms of analysing results.
badpun 4y ago
> I am fairly sure there are multiple published results in computational fluid dynamics that are subtly wrong. I wonder what that means for the accuraccy of the climate models...
- thrown_22 4y ago
  All models are wrong, some are useful.
- geysersam 4y ago
  It's probably impossible to accurately simulate the climate of the planet decades into the future. That does not meaningfully detract from the evidence for human caused global warming however.
  - halffaday 4y ago
    I have to applaud the subtle elegance of the sarcasm here
    - geysersam 4y ago
      No sarcasm intended. The science of climate change rests on many legs. It was already established long before massive models on the scale we have today were possible. Computer simulations attempts to understand how it will affect us, but does not contribute much evidence.
- jeffreyrogers 4y ago
  They are all wrong. Historically they've overpredicted warming, and then the models are updated to fit the (new) data better. They perform worse in areas with less data, like polar regions. To forestall knee-jerk downvotes: I'm not saying climate change isn't real or that anthropogenic global warming doesn't exist. I'm saying the models are not yet developed enough to predict very accurately. Early heliocentric models made poor predictions too because they assumed circular rather than elliptical orbits, they were still more "right" than geocentric models.
  - brutusborn 4y ago
    Can you suggest good places to look for more info on this? I have tried a few times to find info on the accuracy of these models and couldn't find much. And most models seem to be closed source.
  - leephillips 4y ago
    In fact, climate models going back over fifty years have performed quite well; surprisingly so: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019GL085378 https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/201... https://www.theguardian.com/environment/climate-consensus-97-per-cent/2016/jul/27/climate-models-are-accurately-predicting-ocean-and-global-warming https://www.theguardian.com/environment/climate-consensus-97...
    - jeffreyrogers 4y ago
      It depends on how you define "quite well". They are definitely directionally correct and in the right ballpark. This is a politically sensitive research area so the researchers are incentivized to make strong claims that you wouldn't make about, say, predicting sports outcomes or the stock market. The first link you posted is kind of punting on the hard part by saying that the reason the models overpredict warming is because CO2 didn't rise as much as they expected so if you put the actual observed CO2 concentration in then the temperature prediction comes out closer to what was observed. But the CO2 concentration is a parameter of the model, so they didn't capture its dynamics properly and then had to retroactively change it to get the observed data. Again, I want to reiterate that I'm not disputing the process of climate change or saying it's not a problem. I'm saying that modeling it is hard and historically the models have overestimated warming.
- bluenose69 4y ago
  All manner of tests are done in fluid-dynamical modelling. Test suites are developed along with models. Simulations are done with a range of alternative models, or individual models with different conditions are applied, and the differences between the simulations are studied in detail. Models must rely on a fairly large set of parameterizations of processes that cannot be resolved directly, and tests are done to see which parameterizations make the most sense. Models stem from the academic environment, not the business environment. This means that the models are open-source, so everyone can see what everyone else is doing. And 'everyone' is a pretty large set. Note that there is no commercial element to any of this, which means there is no incentive to hide problems. It's the reverse: reputation is earned by finding problems, not by hiding them. A large part of the challenge of climate modelling is the leveraging of increasing computer power to resolve progressively smaller scales of motion, and this uncovers the need to understand those scales in isolation. Modelling is often used at this level also. None of this is to say that climate models are perfect. They obviously are not. But the system of academic science is very good at improving models and, importantly, exposing their limits. An indication of the latter is the pairing of uncertainties with predictions: a hallmark of this scientific community.

simplotek 4y ago

> I know it's not a simple answer, but how would you embed checks to flag or highlight potentially incorrect results to the user? For starters, some problems do have analytical solutions. Yo can compare a FEM model of these problems with the known analytical solution and see if it's close enough or not. One of them is the elasto-plastic plate with a hole. You can also run unit tests at the element level.