4 ms·

Much of the discussion here including the linked article fail to make an important distinction between domains. Prediction can be done quite effectively on thin

Much of the discussion here including the linked article fail to make an important distinction between domains. Prediction can be done quite effectively on thin tailed processes. A lot of the counter examples listed in the comments here are physical systems which are thin tailed. I see aircraft autopilot, collision detection, ozone depletion. These are all well understood physical phenomenon in which large deviations do not occur — your car doesn’t get teleported elsewhere in the middle of avoiding a collision. If a large deviation did occur, say a meteor striking between your car and the object it is attempting to avoid, the collision avoidance system would almost certainly fail. These events occur so infrequently that the system can just assume they won’t and boast a high success rate. Meanwhile the examples from the linked article are fat tailed processes. Recessions, GDP, interest rates, exchange rates. These are all subject to large discontinuous jumps. Anyone doing a 5 year rate prediction in July 2019 would have been required to predict the pandemic in order to accurately forecast. This is a single example but predictions in this domain are regularly blown out by being teleported to a completely different world. Unlike the thin tailed domain these events happen frequently enough that they’re the only thing that matters for the forecast. Knowing which class your generating process belongs to is critical to understanding whether forecasting will be effective or not. I’ll take collision detection and leave economic forecasts at the door any day.

kqr 3y ago
It's not that we can't forecast heavy tailed processes -- it's just that the forecasts are used wrong. The appropriate layman's forecast of a recession within the next year is something like a constant 11 %. I'm willing to bet this outperforms most "predictions" out there. But! When people see that number they go, "right, so it's vastly more likely it does not happen" and then completely ignore the possibility. The problem is not in the probability, but in the failure to adequately assign a cost function to the less likely outcomes.
- diab0lic 3y ago
  > The appropriate layman's forecast of a recession within the next year is something like a constant 11 %. I'm willing to bet this outperforms most "predictions" out there. I think we agree here but the unspoken measure is the amount of uncertainty in each forecast. This is still a very inaccurate prediction compared to the collision detection system which nearly always gets it right. You hit the nail on the head in your last paragraph. Doing something useful doesn’t require an accurate prediction. I agree entirely that assigning an appropriate cost function and responding accordingly guides you to useful actions. Edit: Just discovered your blog through your profile. The topics look super relevant to my interests. Thank you for sharing your thoughts online, I’m looking forward to reading them!
  - kqr 3y ago
    De Finetti used a different word to separate "prediction" (object-level, concrete outcome) from "prevision" (probabilistic statement). The first is often nonsensical, the latter useful. Alas, these are not widely understood words.
- roenxi 3y ago
  Thankyou for putting it eloquently. The other subtle problem is that people tend to ask silly questions. Maybe I've just been lucky so far, but the focus on recessions seems like a relic of the pre-1970s time when GDP correlated with wages. I'm not seeing why it matters so much these days. Interest rates are centrally planned, so it is a bit of a philosophical question to ask what it means to "forecast" them. Really it is more a question of politics than any reliable process and the answer is "lower than makes sense". The economy for the last 10 or 20 years looked like it is carrying a lot of value-destructive companies looking for an opportunity to collapse that were almost certainly related to cheap credit. Uber and WeWork spring to mind. US debt is accumulating in a way that would have been making people nervous for decades. The problem here isn't forecasting, it is money being corralled to purposes that consume more resources than get generated. Being unable to predict what the perturbation against trend will be in 12 months time doesn't seem useful, appears to be impossible and it is weird that people love to focus on it. There are much more serious things to keep an eye on.
paulpauper 3y ago
the article was not that good overall. people who do this stuff for a living do not take such a naive or reductionist approach to forecasting.
jmalicki 3y ago
That you're talking about both collision detection and economics is very telling for the issues a lot of people have with things like these. When an economist predicts a heightened probability of a recession, people believe the economist, and adjust their investments to avoid it, this is seen as a failure in prediction. Yet when a collision detection system detects a collision, and the car brakes to avoid the collision, this is also a false positive as no collision happened! It's not clear why people see these as different. When the system takes the prediction into account to avoid the predicted calamity, we shouldn't see that as a failure in prediction.

kqr 3y ago

It's not that we can't forecast heavy tailed processes -- it's just that the forecasts are used wrong. The appropriate layman's forecast of a recession within the next year is something like a constant 11 %. I'm willing to bet this outperforms most "predictions" out there. But! When people see that number they go, "right, so it's vastly more likely it does not happen" and then completely ignore the possibility. The problem is not in the probability, but in the failure to adequately assign a cost function to the less likely outcomes.

diab0lic 3y ago
> The appropriate layman's forecast of a recession within the next year is something like a constant 11 %. I'm willing to bet this outperforms most "predictions" out there. I think we agree here but the unspoken measure is the amount of uncertainty in each forecast. This is still a very inaccurate prediction compared to the collision detection system which nearly always gets it right. You hit the nail on the head in your last paragraph. Doing something useful doesn’t require an accurate prediction. I agree entirely that assigning an appropriate cost function and responding accordingly guides you to useful actions. Edit: Just discovered your blog through your profile. The topics look super relevant to my interests. Thank you for sharing your thoughts online, I’m looking forward to reading them!
- kqr 3y ago
  De Finetti used a different word to separate "prediction" (object-level, concrete outcome) from "prevision" (probabilistic statement). The first is often nonsensical, the latter useful. Alas, these are not widely understood words.
roenxi 3y ago
Thankyou for putting it eloquently. The other subtle problem is that people tend to ask silly questions. Maybe I've just been lucky so far, but the focus on recessions seems like a relic of the pre-1970s time when GDP correlated with wages. I'm not seeing why it matters so much these days. Interest rates are centrally planned, so it is a bit of a philosophical question to ask what it means to "forecast" them. Really it is more a question of politics than any reliable process and the answer is "lower than makes sense". The economy for the last 10 or 20 years looked like it is carrying a lot of value-destructive companies looking for an opportunity to collapse that were almost certainly related to cheap credit. Uber and WeWork spring to mind. US debt is accumulating in a way that would have been making people nervous for decades. The problem here isn't forecasting, it is money being corralled to purposes that consume more resources than get generated. Being unable to predict what the perturbation against trend will be in 12 months time doesn't seem useful, appears to be impossible and it is weird that people love to focus on it. There are much more serious things to keep an eye on.

paulpauper 3y ago

the article was not that good overall. people who do this stuff for a living do not take such a naive or reductionist approach to forecasting.

jmalicki 3y ago

That you're talking about both collision detection and economics is very telling for the issues a lot of people have with things like these. When an economist predicts a heightened probability of a recession, people believe the economist, and adjust their investments to avoid it, this is seen as a failure in prediction. Yet when a collision detection system detects a collision, and the car brakes to avoid the collision, this is also a false positive as no collision happened! It's not clear why people see these as different. When the system takes the prediction into account to avoid the predicted calamity, we shouldn't see that as a failure in prediction.