4 ms·

Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with th

Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with them, your hypermodel may be inadequate.

bo1024 3y ago
To me, this comes back to the question of skin in the game. If you have skin in the game, then you produce the best uncertainty estimates you can (by any means). If you don't, you just sit back and say "well these are the error bars my model came up with".
- PeterisP 3y ago
  There are ways of scoring forecasts that reward accurate-and-certain forecasts in a manner where it's provably optimal to provide the most accurate estimates for your (un)certainty as you can.
  - bo1024 3y ago
    Yes, of course. I don't see that as very related to my point. For example, consider how 538 or The Economist predict elections. They might claim they'll use squared error or log score, but when it comes down to a big mistake, they'll blame it on factors outside their models.
    - eru 3y ago
      Well, but at least 538 has a reputation to defend as an accurate forecaster. So they have some skin in the game. (Of course, that's not as good as betting money.)
- hgomersall 3y ago
  It's worse than that. Oftentimes the skin in the game provides a motivation to mislead. C.f. most of the economics profession.
  - nequo 3y ago
    This is a pretty sweeping generalization, but if you have concrete examples to offer that support your claim, I’d be curious.
  - eru 3y ago
    How do economists have skin in the game? Many of them eg work in universities and some even have tenure. There's not much skin in the game between any forecasts they might make and their academic prospects. Economists working for companies often have to help them understand micro and macro-economics. Eg (some of) Google's economists help them design the ad auctions. It's relatively easy to figure out for Google how well those ad auctions work. So they certain have skin in the game. But: what motivation to mislead do those economists have?
    - hgomersall 3y ago
      Many economists are so fully bought into their models that they can't think of any alternatives, despite them being essentially useless. I interpreted skin-in-the-game in that way - as professionally committed. Perhaps something different was meant.
      - eru 3y ago
        > Many economists are so fully bought into their models [...] How do you know that? Whenever I interact with economists, mostly online via blogs but also sometimes via email, they always seem painfully aware of the shortcomings of their models, and don't seem to confuse them with reality. Perhaps you have studied a different sub-population of economists than the ones I have anecdotal experience with?
        hgomersall 3y ago
        I'm a sense, that makes my point. Why do they persist with models that don't represent reality despite knowing it? Eventually you must realise that adding epicycles isn't going to cut it, yet still the sage voices echo the standard dogma when economies are dragged into the doldrums by policy posed by useless models. Bought into is not the same as believing.
gyrovagueGeist 3y ago
They can also mean pushed forward uncertainty from input parameters which isn't exactly the same as model error
- hgomersall 3y ago
  I'm not sure I see the distinction. Would you mind clarifying?
  - 8organicbits 3y ago
    Model: water freezes below 0° C. Input: temperature is measured at -1° C. Prediction: water will freeze. Actual: water didn't freeze. Actual temperature: 2° C. The model isn't broken, it gives an incorrect result because of input error.
    - hgomersall 3y ago
      Well I'd say the model is broken because it didn't capture the uncertainty in the measurements.
      - gyrovagueGeist 3y ago
        Taking the example in this comment thread, even if the model takes an arbitrary nonparametric distribution of input temperatures and perfectly returns the posterior distribution of freezing events there is still a difference in model error and forward UQ error. The model itself can perfectly describe the physics, but it only knows what you can give it. This may be limited by measurement uncertainty of your equipment, etc, but it is separate from the model itself. In this area, "the model" is typically considered as the input parameter to quantity of interest map itself. It's not the full problem from gathering data to prediction. Model error would be things like failing to capture the physics (due to approximations, compute limits, etc), intrinsic aleatoric uncertainty in the freezing process itself, etc. Making this distinction helps talk about where the uncertainty comes from, how it can be mitigated, and how to use higher level models and resampling to understand its impact across the full problem.

Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with them, your hypermodel may be inadequate.

bo1024 3y ago

To me, this comes back to the question of skin in the game. If you have skin in the game, then you produce the best uncertainty estimates you can (by any means). If you don't, you just sit back and say "well these are the error bars my model came up with".

PeterisP 3y ago
There are ways of scoring forecasts that reward accurate-and-certain forecasts in a manner where it's provably optimal to provide the most accurate estimates for your (un)certainty as you can.
- bo1024 3y ago
  Yes, of course. I don't see that as very related to my point. For example, consider how 538 or The Economist predict elections. They might claim they'll use squared error or log score, but when it comes down to a big mistake, they'll blame it on factors outside their models.
  - eru 3y ago
    Well, but at least 538 has a reputation to defend as an accurate forecaster. So they have some skin in the game. (Of course, that's not as good as betting money.)
hgomersall 3y ago
It's worse than that. Oftentimes the skin in the game provides a motivation to mislead. C.f. most of the economics profession.
- nequo 3y ago
  This is a pretty sweeping generalization, but if you have concrete examples to offer that support your claim, I’d be curious.
- eru 3y ago
  How do economists have skin in the game? Many of them eg work in universities and some even have tenure. There's not much skin in the game between any forecasts they might make and their academic prospects. Economists working for companies often have to help them understand micro and macro-economics. Eg (some of) Google's economists help them design the ad auctions. It's relatively easy to figure out for Google how well those ad auctions work. So they certain have skin in the game. But: what motivation to mislead do those economists have?
  - hgomersall 3y ago
    Many economists are so fully bought into their models that they can't think of any alternatives, despite them being essentially useless. I interpreted skin-in-the-game in that way - as professionally committed. Perhaps something different was meant.
    - eru 3y ago
      > Many economists are so fully bought into their models [...] How do you know that? Whenever I interact with economists, mostly online via blogs but also sometimes via email, they always seem painfully aware of the shortcomings of their models, and don't seem to confuse them with reality. Perhaps you have studied a different sub-population of economists than the ones I have anecdotal experience with?
      - hgomersall 3y ago
        I'm a sense, that makes my point. Why do they persist with models that don't represent reality despite knowing it? Eventually you must realise that adding epicycles isn't going to cut it, yet still the sage voices echo the standard dogma when economies are dragged into the doldrums by policy posed by useless models. Bought into is not the same as believing.

gyrovagueGeist 3y ago

They can also mean pushed forward uncertainty from input parameters which isn't exactly the same as model error

hgomersall 3y ago
I'm not sure I see the distinction. Would you mind clarifying?
- 8organicbits 3y ago
  Model: water freezes below 0° C. Input: temperature is measured at -1° C. Prediction: water will freeze. Actual: water didn't freeze. Actual temperature: 2° C. The model isn't broken, it gives an incorrect result because of input error.
  - hgomersall 3y ago
    Well I'd say the model is broken because it didn't capture the uncertainty in the measurements.
    - gyrovagueGeist 3y ago
      Taking the example in this comment thread, even if the model takes an arbitrary nonparametric distribution of input temperatures and perfectly returns the posterior distribution of freezing events there is still a difference in model error and forward UQ error. The model itself can perfectly describe the physics, but it only knows what you can give it. This may be limited by measurement uncertainty of your equipment, etc, but it is separate from the model itself. In this area, "the model" is typically considered as the input parameter to quantity of interest map itself. It's not the full problem from gathering data to prediction. Model error would be things like failing to capture the physics (due to approximations, compute limits, etc), intrinsic aleatoric uncertainty in the freezing process itself, etc. Making this distinction helps talk about where the uncertainty comes from, how it can be mitigated, and how to use higher level models and resampling to understand its impact across the full problem.