3 ms·

Daniel Kahneman was once asked how we should respond when we make an analysis mistake. He said: Whenever we are surprised by something, even if we admit that w

Daniel Kahneman was once asked how we should respond when we make an analysis mistake. He said: Whenever we are surprised by something, even if we admit that we made a mistake, we say, ‘Oh I’ll never make that mistake again.’ But, in fact, what you should learn when you make a mistake because you did not anticipate something is that the world is difficult to anticipate. That’s the correct lesson to learn from surprises: that the world is surprising. That last line should be written on forecasters’ walls: The correct lesson to learn from surprises is that the world is surprising.

themodelplumber 7y ago
Isn't this why we run simulations and calculate odds? The world is surprising. Still, if you stick to the odds and something breaks, maybe your model really needs tweaking. We have, wisely, gone back and changed thousands of formal models in fields from physics to economics to social science for this reason: We don't want to make that mistake again. And in many of those cases we didn't make that particular mistake again. So it seems like while it's worth understanding that the world is surprising, this doesn't diminish the value of fixing a broken model or preparing better next time.
- manifestsilence 7y ago
  Sure, but in assessing risk, it pays to not overestimate our ability to predict the world. Some model subjects, such as the stock market and the weather, exhibit chaotic behavior. Those who try to ignore that fact get burned.
- bo1024 7y ago
  I think this shows the difference between a regime like physics where we expect models to generally work, versus another kind of regime like the stock market where it's unclear whether they work. The point of the above quote is that in some regimes, models [that we can currently build] don't work. So the lesson to take away from a bad prediction isn't "this model is bad", it's "any model I build will be bad". There might not be enough signal in the data you have (or even is possible to have). It's exacerbated in cases where it's hard to assess the model's performance, e.g. the stock market mostly goes up so any model that predicts that will look pretty good most of the time.
  - teslaberry 7y ago
    no , the difference between physics and complex human systems, is that complex human systems incorporate human behavior into them. George soros called this 'reflexivity'. Complex human systems possess tipping points and negative and postive feedback loops like greshams law and thiers law. in this sense, the difference, between human behavior is that it is a branch of physics that resembles the study of turbulence, it is not only notoriously unpredictable, but the element driving the unpredictability is human consciousness of it's OWN behavior ( in mass and indvidiually) einstein himself said the most mysterious fact of the human universe is that it has developed the capacity to understand itself. consciousness and the complexity of human systems IS physics at some hyper complex level, that is both self organizing and too complex to ever be distentangled as easily understood as 'deterministic' for the purposes of meaningful hard experimentation. that is why social science and other sciences are such bullshit. thankfully sound progress can be built upon bullshit as long as it is more sensible bullshit than the bullshit that came before. the foundations of our understanding of the science of human behavior are not so much built upon pylons of steel as they are recrafted from a puddle of mud and shit, by adding clay and sugar and fermenting the mass, and then removing some of the shit and leaving behind a less smelly pile . thousands of years of this process leads civilization, here, to hacker news comments containing this philosophy.
poiuyt098 7y ago
>That’s the correct lesson to learn from surprises: that the world is surprising. That's a mistaken analysis. Not surprised.

themodelplumber 7y ago

Isn't this why we run simulations and calculate odds? The world is surprising. Still, if you stick to the odds and something breaks, maybe your model really needs tweaking. We have, wisely, gone back and changed thousands of formal models in fields from physics to economics to social science for this reason: We don't want to make that mistake again. And in many of those cases we didn't make that particular mistake again. So it seems like while it's worth understanding that the world is surprising, this doesn't diminish the value of fixing a broken model or preparing better next time.

manifestsilence 7y ago
Sure, but in assessing risk, it pays to not overestimate our ability to predict the world. Some model subjects, such as the stock market and the weather, exhibit chaotic behavior. Those who try to ignore that fact get burned.
bo1024 7y ago
I think this shows the difference between a regime like physics where we expect models to generally work, versus another kind of regime like the stock market where it's unclear whether they work. The point of the above quote is that in some regimes, models [that we can currently build] don't work. So the lesson to take away from a bad prediction isn't "this model is bad", it's "any model I build will be bad". There might not be enough signal in the data you have (or even is possible to have). It's exacerbated in cases where it's hard to assess the model's performance, e.g. the stock market mostly goes up so any model that predicts that will look pretty good most of the time.
- teslaberry 7y ago
  no , the difference between physics and complex human systems, is that complex human systems incorporate human behavior into them. George soros called this 'reflexivity'. Complex human systems possess tipping points and negative and postive feedback loops like greshams law and thiers law. in this sense, the difference, between human behavior is that it is a branch of physics that resembles the study of turbulence, it is not only notoriously unpredictable, but the element driving the unpredictability is human consciousness of it's OWN behavior ( in mass and indvidiually) einstein himself said the most mysterious fact of the human universe is that it has developed the capacity to understand itself. consciousness and the complexity of human systems IS physics at some hyper complex level, that is both self organizing and too complex to ever be distentangled as easily understood as 'deterministic' for the purposes of meaningful hard experimentation. that is why social science and other sciences are such bullshit. thankfully sound progress can be built upon bullshit as long as it is more sensible bullshit than the bullshit that came before. the foundations of our understanding of the science of human behavior are not so much built upon pylons of steel as they are recrafted from a puddle of mud and shit, by adding clay and sugar and fermenting the mass, and then removing some of the shit and leaving behind a less smelly pile . thousands of years of this process leads civilization, here, to hacker news comments containing this philosophy.

poiuyt098 7y ago

>That’s the correct lesson to learn from surprises: that the world is surprising. That's a mistaken analysis. Not surprised.