16 ms·

The forecasting fallacy (2020)

91 points by rognjen 3y ago

wslh 3y ago
I like the article but not the final conclusion "We can’t predict much at all", I think we can predict more than we think, it is similar to asking if the glass is half empty and half full. For example, sci-fi, scientists, and some economists have predicted a lot of things before but we don't have an accurate time of happening: one thing is to predict an event for next year and another see a trend that will happen in the next 50 years. There is even a futurist science. Regarding AI, and forgetting "AI cults" it is incredible that the neural networks that we are using now are similar to the ones studied decades ago but there was a breakthrough in other aspects such as computing capacity and techniques [1]. [1] https://www.nature.com/articles/323533a0 https://www.nature.com/articles/323533a0
- wslh 3y ago
  Randomly, today's Fred Wilson's "What Happened In 2023" post [1] includes the following content: "...The second is the emergence of a new tech megatrend, AI, which has been developing in front of our very eyes for as long as I have been in tech, so that is over forty years now." [1] https://news.ycombinator.com/item?id=38825073 https://news.ycombinator.com/item?id=38825073
philipswood 3y ago
What?! The article ends with a Alan Kay quote attributed to Cindy Gallup: >Or as Cindy Gallup likes to say: > >“In order to predict the future, you have to invent it". So she does like to say it (see quote below), but it seemed strange to end with a "second hand" quote. From https://www.lbbonline.com/news/5-minutes-with-cindy-gallop https://www.lbbonline.com/news/5-minutes-with-cindy-gallop LBB > ‘In order to predict the future, you have to invent it’, Alan Kay, is reportedly your favourite quote. Why? CG > Because I am all about inventing the future. Too many people feel that the future is something that happens without us, that we have no control over, that simply rolls us over in its wake. I believe in deciding what you want the future to be, and then inventing it.
darkerside 3y ago
Language is important. We can predict anything and everything. There are some things we can't _reliably_ predict. This article ironically suffers from its own thesis. It assumes that because we haven't provided some things successfully in the past, we will never predict anything in the future. A simple counterexample should dispel this silly notion. We used to consider the weather completely unpredictable. Now we have elaborate systems and theories that allow us to predict the weather with at least some accuracy. A more reasonable thesis might be, we can't reliably predict human behavior, because much like the uncertainty principle, each prediction that is published, which it must be to be meaningful, affects the behavior it is trying to predict.
skippyboxedhero 3y ago
You can predict what will occur quite easily, you can't necessarily predict when it will occur. A lot of the "failed" predictions relate to markets...the reason why you can't predict this stuff is because humans are irrational and those irrational humans control outcomes in the short and medium term. For example, you can see that a recession should have occurred. What people didn't expect is fiscal stimulus worth about 50% of GDP, tens of trillions in monetary stimulus, etc. Yes, if the government just deposits hundreds of billions into people's bank accounts then it is going to impact growth. I remember back in 2007, Blackstone RE made insane leveraged bets at the very top of the market, it is very easy to point out rationally "these are absolutely terrible investments, the price is awful, these aren't economic"...today, all these bets got bailed out by the government (after a short period of bankruptcy/restructuring), the person responsible is probably going to be made head of Blackstone, that unit has hundreds of billions in AUM, etc. The assumption that people make with forecasts has to be: the long-term is today. That is it. You will often be wrong but that does not mean that your model is wrong (indeed, the reason why this stuff is so predictable is because people believe that the models have stopped working repeatedly). If you take something as apparently "unpredictable" as the market, you can predict returns to within 10bps very easily over the long-term because the fundamentals do not change (but, again, the current period has been the most unpredictable because of the level of government intervention, it is unprecedented...the government cannot hold back the waves forever though). EDIT: referencing the 2005 interest rate prediction is quite humorous too, 2% against a predicted 5%...this was basically the start of it. Back then, no-one thought the Fed would cut rates to this level for, essentially, no reason...the Fed cut, the result was a financial crash. Turns out those predictions (which were essentially the long-term neutral rate) were right and the Fed was wrong...but the only account you hear about is: those damn forecasters, they failed to predict the Fed torching the economy, so stupid. Lol.
kbrkbr 3y ago
I think this article has two shortcomings that make its sweeping conclusions shaky. First, it identifies forecasting with point forecasting. There are other ways to put forecasting questions, e.g. lower and upper level with a certain probability. Also it mentions Tetlock, but only his negative findings, not his positive ones that lead to Good Judgement Project, which suggest the contrary of the conclusion of this article [1]. Thus I think it is not up to the latest research results. See you over at gjopen.com, if you are interested and have lots of time to waste... [1] https://en.m.wikipedia.org/wiki/The_Good_Judgment_Project https://en.m.wikipedia.org/wiki/The_Good_Judgment_Project
- btilly 3y ago
  Tetlock paints a different picture of his positive findings than you do. Specifically, Tetlock's project opens with key issues of scope about what to even try to forecast. Based on his previous work in expert prediction, he concluded that geopolitics is sufficiently chaotic to be impossible to predict 10 years out. So while he did a lot of work on forecasting, it is generally focused on the next year or to. Which means that Tetlock agrees that we can't predict 10 years out.
  - kbrkbr 3y ago
    I agree with the assessment that there are not many systems we can predict 10 years out with great confidence, specifically geopolitics. But I do not think I painted much of a picture of Tetlock's results. I read the article as concluding: let's stop predicting, it does not work. Let's start building. (After stating we cannot predict this, and we cannot predict that.) And I think Tetlock"s result contradict that, as I said. Sometimes and under certain circumstances we can predict quite well.
    - mcshicks 3y ago
      I have a graph from "Expert Political Judgement" that I've kept on a cork board for over a decade. It's from page 55 in my edition. It charts "Objective Frequency" vs "Subjective Probability" It has three curves, Experts (people in Government, paid to make political assessments), Dilettantes (people who are well read, read NYT, WSJ and the like), and College Undergrads. The Expert and Dilettante lines are more or less on top of each other. The undergrads are observably much worse and farther from the "Perfect Calibration line" that is a 45 degree line between objective frequency and subjective probability. So it's not the case that there is no difference in people's ability to predict political events, it's that so called "experts" are no better than people who follow current events closely. This was for me the main takeaway from the book, is that nobody can predict political events very well, but some groups are measurably worse than others. Tetlock has a brief section that somewhat mirrors your argument on page 186 "Misunderstanding what game is being played" where one expert tells him making predictions is all about getting your sound bite out, not about being correct. In this game, stronger, incorrect predictions might be advantageous in that they can change the narrative.
hardlianotion 3y ago
Everyone that provides a forecast that others depend on should really be on the hook to report on the outcome, and to provide the forecast error distribution, if the forecast is one they make regularly.
- jakderrida 3y ago
  I remember actually importing all the WSJ economic forecast surveys and testing the accuracy of each forecaster to find out which one I should bet my money on in the future. The results were... unimpressive for all of them. What they call forecasts aren't really rigorously modeled predictions of the future. Pretty much all of them come down to a very simple formula of long-term mean reverting growth rates. The forecasters that got awards for best forecast were either always optimistic and got the award during good economic periods or always pessimistic and got it during an unprecedented recession. So disappointing. I really believed in it before I studied the numbers.
sega_sai 3y ago
Any forecasts that don't involve probabilities or confidence intervals are useless. Also, if any forecasters were really serious, they would register their past forecasts and show how good they have been in the past. But I think most of forecasters are probably afraid of showing their true track record.
- qznc 3y ago
  I agree. The article succumbs to a False Binary Fallacy, where forecasts are either correct or wrong. The real question about forecasts is how certain they are. https://en.wikipedia.org/wiki/False_dilemma https://en.wikipedia.org/wiki/False_dilemma
  - hef19898 3y ago
    First rule of forecasts in supply chain management: the forecast is always wrong. And still people ignore that cardinal, and a lotnof smaller, rules all the time.
    - kqr 3y ago
      Right or wrong is not the measure by which to evaluate forecasts, just as it's not the appropriate yardstick for models.
Zolde 3y ago
One should ask economists what a recession is, not how to predict one. Good modelers do not necessarily need (or want) to know what they are predicting and still beat "domain experts". Authority without clear track-record is a net negative to getting good results. It is better to stick to anonymity, and only let the track-record do the talking/weighting. Without a clear track-record it does not even matter if the prediction-maker has skin in the game. If you do have skin in the game, there is no reason to sell your hide cheaply, or even give it away. You instead take the profit others say does and can not exist beyond "luck": If you can't even beat a random walk, you have no business evaluating the limitations of predictive modeling. The big consultancy companies making bold predictions don't even need to be right. Customers read the predictions these consultancy companies peddle, because these customers are not bold enough to make their own predictions. And nobody ever got fired for buying the predictions from big consultancy companies and incorporating them into a business strategy.
- notahacker 3y ago
  > One should ask economists what a recession is, not how to predict one. Most economists would agree. It's everyone else that says "well if you know so much about how shocks and policy changes cause recessions, why can't you tell me if there will be a recession in $country in Q2 2025?". And in economics, "skin in the game" means policy responses to avoid dire forecast outcomes (or lack of them when nobody expect oil prices to change or a major bank to collapse). There's no shortage of opportunity to make money by beating everyone else at the prediction game, but the funds that have consistently profited from spotting the recessions ahead of everyone else don't exist any more than the always-right public expert forecasters.
- hef19898 3y ago
  Consultancies predicting something isn't forecasting, it is marketing. And there or only a rare few thing I disagree more stongly with the statement, that good modellers / data scientist / whatever only need knowledge about how to model stuff to beat domain experts. It takes domain experts to judge whether or not a model correct, to identify the known and unknown unknowns and limitations of these models. Claiming otherwise is deeply arrogant, and it ended in disaster everytime I saw it tried. Good modellers need enough domain knowledge to properly work with, and understand, domain experts. And domain experts need sufficient knowledge about modelling to do the same. Both need the willingness to do so. And every modeller needs to accept that reality beats models, always.
INGELRII 3y ago
A forecasting system in aircraft autopilot that can accurately forecast when the plane hits the mountain is always wrong. Forecasting when the forecast depends on the actions of agents that can be informed by the forecast changes the game. If the Fed model forecasts recession and the Fed takes action to prevent it from happening, it changes everything. Only a forecasting model that is not observed/believed by policy makers can predict without intervention. Layman's idea of forecasting: Predict what happens in the future. Economic forecasting: Forecast is input for actions. Predict what happens in the future, using this model, these variables, and everything else stay the same. You can check afterward if the model is an accurate forecaster by removing the changes caused by variables outside the model.
- Zolde 3y ago
  It is always harder to accurately forecast actual recession, than it is to forecast the predictions of the Fed model. You don't need an information edge there, just information parity. When the Fed takes action, it is usually a very rational action, with a clear-defined goal of long-term economic health. This makes their actions easier to predict than other market participants. So you went the hard route, forecasting the highly complex system directly, but then "variables outside the model" caused the "accurate" model to not perform well? You don't buy anything with that, since you live in a world with outside variables which mess up your predictions. The solution is to make your model actually accurate, by incorporating these "variables outside the model": Predict what others will predict.
- paulpauper 3y ago
  Yeah, if forecasts did work, ppl would change behavior accordingly, rendering the model useless eventually
jxf 3y ago
The author would also conclude: * Collision avoidance systems are terrible at forecasting collisions because they almost never result in a collision. (The point of the system is to help you avoid an upcoming collision.) * The prediction that Y2K would happen was a bad one since it didn't happen. (We spent billions of dollars to make sure it didn't.) * The 1978 prediction that the ozone layer would be depleted by 2010 was a bad one since it didn't happen. (Humans took action to reverse CFCs and the ozone layer began to regenerate.) When you make a forecast about an event wherein agents can change the course of the event, the correct evaluation of the forecast is not "did the event happen?" but "would the event have happened but for intervention?". The author seems to miss this larger point.
- troupe 3y ago
  > The author would also conclude: > The prediction that Y2K would happen was a bad one since it didn't happen. (We spent billions of dollars to make sure it didn't.) I would argue that we don't really know what would have happened had the world not spent all the money on upgrading systems. It appears a very large number of them would have continued to work as expected and it isn't immediately clear if the ones that were replaced would have resulted in a catastrophe.
  - snowwrestler 3y ago
    The people who were working on Y2K did know what would happen in many cases. Their work avoided known huge messes in banking, infrastructure, aviation, and healthcare, among others. What they didn’t do, much, is write or blog about their work. A lot of fixes were to commercial or government systems running on commercial or government hardware. Publicly disclosing problems and fixes was not part of those cultures. So it is very hard, today, for members of the public to go back and reconstruct the problems and solutions to “prove” that there were real issues. Which has led some people to believe, incorrectly, that there were not real issues.
    - troupe 3y ago
      I was part of that remediation effort in the healthcare sector. And yes there were things that were fixed that prevented problems. However, given how many things were not fixed, it is amazing how few problems actually happened. (Someone got charged for 100 years of late library fees...) Is that because we found and patched all the systems that would have actually had a problem? Maybe. I'm guessing it is because many of the things that were fixed, wouldn't have actually caused any signifiant problem--at least not at the scale that was being predicted. But as you point out, if there were any systems that would have failed in a catastrophic way, those were evidently fixed.
cesaref 3y ago
If you want a counter example, go and investigate algo trading hedge funds - you'll find they do a pretty solid job of predicting the future. Sure, some of them predict only a few ms into the future, some a few minutes (the one I worked for was in that category) and others will do interday strategies. I'm pretty sure there are examples which have a track record of decent returns above the markets they trade in with longer term strategies. So, i'd say there are examples of forecasting working, but generally the people who are good at it don't write about it, and instead use their knowledge and ideas to quietly make money from their insights :)
- chadash 3y ago
  I don’t have a background in this but I was under the impression that much of algorithmic trading is that there are trillions of pennies lying around and if you have an algorithm that picks up those pennies faster than anyone else, you make a lot of money. So it’s capitalizing on tiny market inefficiencies rather than directional predictions.
  - WJW 3y ago
    There's a wide variety of strategies available. The type you mention of picking up small inefficiencies certainly exists but there are plenty of other strategies that involve having some sort of informational edge. Some hedge fund managers just read a lot of earnings releases, but there are also more sophisticated approaches: a famous example would be the fund that paid for satellite imagery of the parking lots of certain shops, so that they could count how many cars there were and extrapolate that into whether the chain was growing or not. Another straightforward example would involve using proprietary weather forecasting software to try and predict the global grain/cocoa/coffee/whatever harvest, so that you can then trade accordingly if you can see a bumper crop coming up.
    - altdataseller 3y ago
      Both of those examples have been exploited to death and no longer are profitable
civilized 3y ago
There's a germ of insight here that could use some development and nuancing. > The future is uncertain. You cannot predict it. But you can create it. For millions of years, prediction has been the engine by which humans have created the future. We don't always call it that, but prediction is the engine. Let's start from the most basic facts of life. We know from experience (our own and others') what plants will sustain us and what will kill us, so we can predict what present-day choices of food will create a positive future. We create our positive future by making choices in accordance with those predictions. It seems that we need to figure out what separates the kind of prediction that is the engine of human life and progress from the kind that is just useless blather.
- CharlesW 3y ago
  Also to your point, it makes no sense to say, "You cannot predict it. But you can create it." The reason a person or entity creates something is that they've predicted a desired outcome for that creation. > It seems that we need to figure out what separates the kind of prediction that is the engine of human life and progress from the kind that is just useless blather. For sure. The author treats content marketing by management consultancies — blather — as serious efforts to predict outcomes. But these are stories about potential outcomes created to lure customers to the rim of their sales funnel. In other words, their actual prediction is that publishing thousands of "thought leadership" pieces will improve SEO and sales engagement, which is probably true.
- cheschire 3y ago
  Really? I think rather than prediction as our future creating engine, it's been critical thinking and problem solving. Most inventions are solutions to problems, not solutions in search of problems. Industrialization was in response to a need to scale up production. The internet was in response to a discoverability problem. Smartphones were in response to a need to do personal computing on the go. Monetizing those things was the inflection point for success in all those cases, but even prior to monetization most human ingenuity has been based in problem solving. Which is looking backwards. Not forwards.
  - 3y ago
hamilyon2 3y ago
Isn't inability to accurately predict some economic metrics consequence of efficient market hypothesis? All available and some unavailable information is already reflected in market. So, sum of reasonable guesses of next year GDP more or less is today's market index. Anything over that is some baseless speculation with no skin in the game.
- nimbius 3y ago
  correct me if im wrong but the approximate cycle of boom/bust each decade or so for capitalism is a well documented feature? that it sort of has to "reinvent" itself each time in order for continued existence? couldnt one plan around this in broader strokes that dont involve the sorts of precision quantitative analysis that wallstreet seems so fond of?
- bee_rider 3y ago
  We need a market uncertainty principle or something, haha.
- adrianN 3y ago
  Afaik the efficient market hypothesis says nothing about how long the market takes to optimize after new information is available (and I believe the market needs to solve an np-hard optimization problem). So in principle you could beat the market, by using a better algorithm or more compute.
jaygray0919 3y ago
I just read Hari Seldon to find out what will happen in the future.
hk__2 3y ago
(2020)
diab0lic 3y ago
Much of the discussion here including the linked article fail to make an important distinction between domains. Prediction can be done quite effectively on thin tailed processes. A lot of the counter examples listed in the comments here are physical systems which are thin tailed. I see aircraft autopilot, collision detection, ozone depletion. These are all well understood physical phenomenon in which large deviations do not occur — your car doesn’t get teleported elsewhere in the middle of avoiding a collision. If a large deviation did occur, say a meteor striking between your car and the object it is attempting to avoid, the collision avoidance system would almost certainly fail. These events occur so infrequently that the system can just assume they won’t and boast a high success rate. Meanwhile the examples from the linked article are fat tailed processes. Recessions, GDP, interest rates, exchange rates. These are all subject to large discontinuous jumps. Anyone doing a 5 year rate prediction in July 2019 would have been required to predict the pandemic in order to accurately forecast. This is a single example but predictions in this domain are regularly blown out by being teleported to a completely different world. Unlike the thin tailed domain these events happen frequently enough that they’re the only thing that matters for the forecast. Knowing which class your generating process belongs to is critical to understanding whether forecasting will be effective or not. I’ll take collision detection and leave economic forecasts at the door any day.
- kqr 3y ago
  It's not that we can't forecast heavy tailed processes -- it's just that the forecasts are used wrong. The appropriate layman's forecast of a recession within the next year is something like a constant 11 %. I'm willing to bet this outperforms most "predictions" out there. But! When people see that number they go, "right, so it's vastly more likely it does not happen" and then completely ignore the possibility. The problem is not in the probability, but in the failure to adequately assign a cost function to the less likely outcomes.
  - diab0lic 3y ago
    > The appropriate layman's forecast of a recession within the next year is something like a constant 11 %. I'm willing to bet this outperforms most "predictions" out there. I think we agree here but the unspoken measure is the amount of uncertainty in each forecast. This is still a very inaccurate prediction compared to the collision detection system which nearly always gets it right. You hit the nail on the head in your last paragraph. Doing something useful doesn’t require an accurate prediction. I agree entirely that assigning an appropriate cost function and responding accordingly guides you to useful actions. Edit: Just discovered your blog through your profile. The topics look super relevant to my interests. Thank you for sharing your thoughts online, I’m looking forward to reading them!
FergusArgyll 3y ago
Some questions from an economic analysis standpoint. If someone can predict the future reliably, why don't financial firms hire them? If a firm did hire them, how much money are they getting paid? why so little? Is the economic value of correct predictions lower than you'd think? Does the market believe "past performance is no guarantee of future results"?
jncfhnb 3y ago
Ugh. This annoys me. Can we predict things super accurately? Often no. But you know what’s better than anecdotes about times predictions were bad? Training and testing sets to judge how good we expect models and predictions to be from the beginning. Because a lot of these are not high confidence predictions. And no, it’s not “black swans”. Are those a thing? Sure. It’s ok that models can’t account for things that are not modeled or seen before. But if these things are common enough that they’re systemic and the mode is just not actually accommodating for the world of relevant factors, then it’s not going to have been a good model on the test set to begin with. And we would know that.
paulpauper 3y ago
does this matter if they cannot predict? Doctors cannot predict who gets heart attacks or cancer but they are still needed anyway.
dang 3y ago
Discussed at the time: The Forecasting Fallacy - https://news.ycombinator.com/item?id=24521279 https://news.ycombinator.com/item?id=24521279 - Sept 2020 (9 comments)
lordnacho 3y ago
The critical distinction is IMO that the examples he gives are all examples of where the prediction forms part of its own outcome. Whether interest rate go up depends on whether people think interest rates will go up. Same with GDP and all the other examples. Having this recursion is a massive problem, because there might not be a stable point where the prediction and its outcome are fixed. This may be why it's hard to steer cars automatically. I don't know enough about the subject, but it would seem to fit in the category of "fields where what you think affects what happens". By contrast, a lot of other phenomena are complicated but do not have this prediction-feedback effect. The weather, astronomical observations, a lot of engineering systems.
TrevorMaynard 3y ago
[dead]
TrevorMaynard 3y ago
[dead]
TrevorMaynard 3y ago
[dead]

91 points by rognjen 3y ago

wslh 3y ago

I like the article but not the final conclusion "We can’t predict much at all", I think we can predict more than we think, it is similar to asking if the glass is half empty and half full. For example, sci-fi, scientists, and some economists have predicted a lot of things before but we don't have an accurate time of happening: one thing is to predict an event for next year and another see a trend that will happen in the next 50 years. There is even a futurist science. Regarding AI, and forgetting "AI cults" it is incredible that the neural networks that we are using now are similar to the ones studied decades ago but there was a breakthrough in other aspects such as computing capacity and techniques [1]. [1] https://www.nature.com/articles/323533a0 https://www.nature.com/articles/323533a0

wslh 3y ago
Randomly, today's Fred Wilson's "What Happened In 2023" post [1] includes the following content: "...The second is the emergence of a new tech megatrend, AI, which has been developing in front of our very eyes for as long as I have been in tech, so that is over forty years now." [1] https://news.ycombinator.com/item?id=38825073 https://news.ycombinator.com/item?id=38825073

philipswood 3y ago

What?! The article ends with a Alan Kay quote attributed to Cindy Gallup: >Or as Cindy Gallup likes to say: > >“In order to predict the future, you have to invent it". So she does like to say it (see quote below), but it seemed strange to end with a "second hand" quote. From https://www.lbbonline.com/news/5-minutes-with-cindy-gallop https://www.lbbonline.com/news/5-minutes-with-cindy-gallop LBB > ‘In order to predict the future, you have to invent it’, Alan Kay, is reportedly your favourite quote. Why? CG > Because I am all about inventing the future. Too many people feel that the future is something that happens without us, that we have no control over, that simply rolls us over in its wake. I believe in deciding what you want the future to be, and then inventing it.

darkerside 3y ago

Language is important. We can predict anything and everything. There are some things we can't _reliably_ predict. This article ironically suffers from its own thesis. It assumes that because we haven't provided some things successfully in the past, we will never predict anything in the future. A simple counterexample should dispel this silly notion. We used to consider the weather completely unpredictable. Now we have elaborate systems and theories that allow us to predict the weather with at least some accuracy. A more reasonable thesis might be, we can't reliably predict human behavior, because much like the uncertainty principle, each prediction that is published, which it must be to be meaningful, affects the behavior it is trying to predict.

skippyboxedhero 3y ago

You can predict what will occur quite easily, you can't necessarily predict when it will occur. A lot of the "failed" predictions relate to markets...the reason why you can't predict this stuff is because humans are irrational and those irrational humans control outcomes in the short and medium term. For example, you can see that a recession should have occurred. What people didn't expect is fiscal stimulus worth about 50% of GDP, tens of trillions in monetary stimulus, etc. Yes, if the government just deposits hundreds of billions into people's bank accounts then it is going to impact growth. I remember back in 2007, Blackstone RE made insane leveraged bets at the very top of the market, it is very easy to point out rationally "these are absolutely terrible investments, the price is awful, these aren't economic"...today, all these bets got bailed out by the government (after a short period of bankruptcy/restructuring), the person responsible is probably going to be made head of Blackstone, that unit has hundreds of billions in AUM, etc. The assumption that people make with forecasts has to be: the long-term is today. That is it. You will often be wrong but that does not mean that your model is wrong (indeed, the reason why this stuff is so predictable is because people believe that the models have stopped working repeatedly). If you take something as apparently "unpredictable" as the market, you can predict returns to within 10bps very easily over the long-term because the fundamentals do not change (but, again, the current period has been the most unpredictable because of the level of government intervention, it is unprecedented...the government cannot hold back the waves forever though). EDIT: referencing the 2005 interest rate prediction is quite humorous too, 2% against a predicted 5%...this was basically the start of it. Back then, no-one thought the Fed would cut rates to this level for, essentially, no reason...the Fed cut, the result was a financial crash. Turns out those predictions (which were essentially the long-term neutral rate) were right and the Fed was wrong...but the only account you hear about is: those damn forecasters, they failed to predict the Fed torching the economy, so stupid. Lol.

kbrkbr 3y ago

I think this article has two shortcomings that make its sweeping conclusions shaky. First, it identifies forecasting with point forecasting. There are other ways to put forecasting questions, e.g. lower and upper level with a certain probability. Also it mentions Tetlock, but only his negative findings, not his positive ones that lead to Good Judgement Project, which suggest the contrary of the conclusion of this article [1]. Thus I think it is not up to the latest research results. See you over at gjopen.com, if you are interested and have lots of time to waste... [1] https://en.m.wikipedia.org/wiki/The_Good_Judgment_Project https://en.m.wikipedia.org/wiki/The_Good_Judgment_Project

btilly 3y ago
Tetlock paints a different picture of his positive findings than you do. Specifically, Tetlock's project opens with key issues of scope about what to even try to forecast. Based on his previous work in expert prediction, he concluded that geopolitics is sufficiently chaotic to be impossible to predict 10 years out. So while he did a lot of work on forecasting, it is generally focused on the next year or to. Which means that Tetlock agrees that we can't predict 10 years out.
- kbrkbr 3y ago
  I agree with the assessment that there are not many systems we can predict 10 years out with great confidence, specifically geopolitics. But I do not think I painted much of a picture of Tetlock's results. I read the article as concluding: let's stop predicting, it does not work. Let's start building. (After stating we cannot predict this, and we cannot predict that.) And I think Tetlock"s result contradict that, as I said. Sometimes and under certain circumstances we can predict quite well.
  - mcshicks 3y ago
    I have a graph from "Expert Political Judgement" that I've kept on a cork board for over a decade. It's from page 55 in my edition. It charts "Objective Frequency" vs "Subjective Probability" It has three curves, Experts (people in Government, paid to make political assessments), Dilettantes (people who are well read, read NYT, WSJ and the like), and College Undergrads. The Expert and Dilettante lines are more or less on top of each other. The undergrads are observably much worse and farther from the "Perfect Calibration line" that is a 45 degree line between objective frequency and subjective probability. So it's not the case that there is no difference in people's ability to predict political events, it's that so called "experts" are no better than people who follow current events closely. This was for me the main takeaway from the book, is that nobody can predict political events very well, but some groups are measurably worse than others. Tetlock has a brief section that somewhat mirrors your argument on page 186 "Misunderstanding what game is being played" where one expert tells him making predictions is all about getting your sound bite out, not about being correct. In this game, stronger, incorrect predictions might be advantageous in that they can change the narrative.

hardlianotion 3y ago

Everyone that provides a forecast that others depend on should really be on the hook to report on the outcome, and to provide the forecast error distribution, if the forecast is one they make regularly.

jakderrida 3y ago
I remember actually importing all the WSJ economic forecast surveys and testing the accuracy of each forecaster to find out which one I should bet my money on in the future. The results were... unimpressive for all of them. What they call forecasts aren't really rigorously modeled predictions of the future. Pretty much all of them come down to a very simple formula of long-term mean reverting growth rates. The forecasters that got awards for best forecast were either always optimistic and got the award during good economic periods or always pessimistic and got it during an unprecedented recession. So disappointing. I really believed in it before I studied the numbers.

sega_sai 3y ago

Any forecasts that don't involve probabilities or confidence intervals are useless. Also, if any forecasters were really serious, they would register their past forecasts and show how good they have been in the past. But I think most of forecasters are probably afraid of showing their true track record.

qznc 3y ago
I agree. The article succumbs to a False Binary Fallacy, where forecasts are either correct or wrong. The real question about forecasts is how certain they are. https://en.wikipedia.org/wiki/False_dilemma https://en.wikipedia.org/wiki/False_dilemma
- hef19898 3y ago
  First rule of forecasts in supply chain management: the forecast is always wrong. And still people ignore that cardinal, and a lotnof smaller, rules all the time.
  - kqr 3y ago
    Right or wrong is not the measure by which to evaluate forecasts, just as it's not the appropriate yardstick for models.

Zolde 3y ago

One should ask economists what a recession is, not how to predict one. Good modelers do not necessarily need (or want) to know what they are predicting and still beat "domain experts". Authority without clear track-record is a net negative to getting good results. It is better to stick to anonymity, and only let the track-record do the talking/weighting. Without a clear track-record it does not even matter if the prediction-maker has skin in the game. If you do have skin in the game, there is no reason to sell your hide cheaply, or even give it away. You instead take the profit others say does and can not exist beyond "luck": If you can't even beat a random walk, you have no business evaluating the limitations of predictive modeling. The big consultancy companies making bold predictions don't even need to be right. Customers read the predictions these consultancy companies peddle, because these customers are not bold enough to make their own predictions. And nobody ever got fired for buying the predictions from big consultancy companies and incorporating them into a business strategy.

notahacker 3y ago
> One should ask economists what a recession is, not how to predict one. Most economists would agree. It's everyone else that says "well if you know so much about how shocks and policy changes cause recessions, why can't you tell me if there will be a recession in $country in Q2 2025?". And in economics, "skin in the game" means policy responses to avoid dire forecast outcomes (or lack of them when nobody expect oil prices to change or a major bank to collapse). There's no shortage of opportunity to make money by beating everyone else at the prediction game, but the funds that have consistently profited from spotting the recessions ahead of everyone else don't exist any more than the always-right public expert forecasters.
hef19898 3y ago
Consultancies predicting something isn't forecasting, it is marketing. And there or only a rare few thing I disagree more stongly with the statement, that good modellers / data scientist / whatever only need knowledge about how to model stuff to beat domain experts. It takes domain experts to judge whether or not a model correct, to identify the known and unknown unknowns and limitations of these models. Claiming otherwise is deeply arrogant, and it ended in disaster everytime I saw it tried. Good modellers need enough domain knowledge to properly work with, and understand, domain experts. And domain experts need sufficient knowledge about modelling to do the same. Both need the willingness to do so. And every modeller needs to accept that reality beats models, always.

INGELRII 3y ago

A forecasting system in aircraft autopilot that can accurately forecast when the plane hits the mountain is always wrong. Forecasting when the forecast depends on the actions of agents that can be informed by the forecast changes the game. If the Fed model forecasts recession and the Fed takes action to prevent it from happening, it changes everything. Only a forecasting model that is not observed/believed by policy makers can predict without intervention. Layman's idea of forecasting: Predict what happens in the future. Economic forecasting: Forecast is input for actions. Predict what happens in the future, using this model, these variables, and everything else stay the same. You can check afterward if the model is an accurate forecaster by removing the changes caused by variables outside the model.

Zolde 3y ago
It is always harder to accurately forecast actual recession, than it is to forecast the predictions of the Fed model. You don't need an information edge there, just information parity. When the Fed takes action, it is usually a very rational action, with a clear-defined goal of long-term economic health. This makes their actions easier to predict than other market participants. So you went the hard route, forecasting the highly complex system directly, but then "variables outside the model" caused the "accurate" model to not perform well? You don't buy anything with that, since you live in a world with outside variables which mess up your predictions. The solution is to make your model actually accurate, by incorporating these "variables outside the model": Predict what others will predict.
paulpauper 3y ago
Yeah, if forecasts did work, ppl would change behavior accordingly, rendering the model useless eventually

jxf 3y ago

The author would also conclude: * Collision avoidance systems are terrible at forecasting collisions because they almost never result in a collision. (The point of the system is to help you avoid an upcoming collision.) * The prediction that Y2K would happen was a bad one since it didn't happen. (We spent billions of dollars to make sure it didn't.) * The 1978 prediction that the ozone layer would be depleted by 2010 was a bad one since it didn't happen. (Humans took action to reverse CFCs and the ozone layer began to regenerate.) When you make a forecast about an event wherein agents can change the course of the event, the correct evaluation of the forecast is not "did the event happen?" but "would the event have happened but for intervention?". The author seems to miss this larger point.

troupe 3y ago
> The author would also conclude: > The prediction that Y2K would happen was a bad one since it didn't happen. (We spent billions of dollars to make sure it didn't.) I would argue that we don't really know what would have happened had the world not spent all the money on upgrading systems. It appears a very large number of them would have continued to work as expected and it isn't immediately clear if the ones that were replaced would have resulted in a catastrophe.
- snowwrestler 3y ago
  The people who were working on Y2K did know what would happen in many cases. Their work avoided known huge messes in banking, infrastructure, aviation, and healthcare, among others. What they didn’t do, much, is write or blog about their work. A lot of fixes were to commercial or government systems running on commercial or government hardware. Publicly disclosing problems and fixes was not part of those cultures. So it is very hard, today, for members of the public to go back and reconstruct the problems and solutions to “prove” that there were real issues. Which has led some people to believe, incorrectly, that there were not real issues.
  - troupe 3y ago
    I was part of that remediation effort in the healthcare sector. And yes there were things that were fixed that prevented problems. However, given how many things were not fixed, it is amazing how few problems actually happened. (Someone got charged for 100 years of late library fees...) Is that because we found and patched all the systems that would have actually had a problem? Maybe. I'm guessing it is because many of the things that were fixed, wouldn't have actually caused any signifiant problem--at least not at the scale that was being predicted. But as you point out, if there were any systems that would have failed in a catastrophic way, those were evidently fixed.

cesaref 3y ago

If you want a counter example, go and investigate algo trading hedge funds - you'll find they do a pretty solid job of predicting the future. Sure, some of them predict only a few ms into the future, some a few minutes (the one I worked for was in that category) and others will do interday strategies. I'm pretty sure there are examples which have a track record of decent returns above the markets they trade in with longer term strategies. So, i'd say there are examples of forecasting working, but generally the people who are good at it don't write about it, and instead use their knowledge and ideas to quietly make money from their insights :)

chadash 3y ago
I don’t have a background in this but I was under the impression that much of algorithmic trading is that there are trillions of pennies lying around and if you have an algorithm that picks up those pennies faster than anyone else, you make a lot of money. So it’s capitalizing on tiny market inefficiencies rather than directional predictions.
- WJW 3y ago
  There's a wide variety of strategies available. The type you mention of picking up small inefficiencies certainly exists but there are plenty of other strategies that involve having some sort of informational edge. Some hedge fund managers just read a lot of earnings releases, but there are also more sophisticated approaches: a famous example would be the fund that paid for satellite imagery of the parking lots of certain shops, so that they could count how many cars there were and extrapolate that into whether the chain was growing or not. Another straightforward example would involve using proprietary weather forecasting software to try and predict the global grain/cocoa/coffee/whatever harvest, so that you can then trade accordingly if you can see a bumper crop coming up.
  - altdataseller 3y ago
    Both of those examples have been exploited to death and no longer are profitable

civilized 3y ago

There's a germ of insight here that could use some development and nuancing. > The future is uncertain. You cannot predict it. But you can create it. For millions of years, prediction has been the engine by which humans have created the future. We don't always call it that, but prediction is the engine. Let's start from the most basic facts of life. We know from experience (our own and others') what plants will sustain us and what will kill us, so we can predict what present-day choices of food will create a positive future. We create our positive future by making choices in accordance with those predictions. It seems that we need to figure out what separates the kind of prediction that is the engine of human life and progress from the kind that is just useless blather.

CharlesW 3y ago
Also to your point, it makes no sense to say, "You cannot predict it. But you can create it." The reason a person or entity creates something is that they've predicted a desired outcome for that creation. > It seems that we need to figure out what separates the kind of prediction that is the engine of human life and progress from the kind that is just useless blather. For sure. The author treats content marketing by management consultancies — blather — as serious efforts to predict outcomes. But these are stories about potential outcomes created to lure customers to the rim of their sales funnel. In other words, their actual prediction is that publishing thousands of "thought leadership" pieces will improve SEO and sales engagement, which is probably true.
cheschire 3y ago
Really? I think rather than prediction as our future creating engine, it's been critical thinking and problem solving. Most inventions are solutions to problems, not solutions in search of problems. Industrialization was in response to a need to scale up production. The internet was in response to a discoverability problem. Smartphones were in response to a need to do personal computing on the go. Monetizing those things was the inflection point for success in all those cases, but even prior to monetization most human ingenuity has been based in problem solving. Which is looking backwards. Not forwards.
- 3y ago

hamilyon2 3y ago

Isn't inability to accurately predict some economic metrics consequence of efficient market hypothesis? All available and some unavailable information is already reflected in market. So, sum of reasonable guesses of next year GDP more or less is today's market index. Anything over that is some baseless speculation with no skin in the game.

nimbius 3y ago
correct me if im wrong but the approximate cycle of boom/bust each decade or so for capitalism is a well documented feature? that it sort of has to "reinvent" itself each time in order for continued existence? couldnt one plan around this in broader strokes that dont involve the sorts of precision quantitative analysis that wallstreet seems so fond of?
bee_rider 3y ago
We need a market uncertainty principle or something, haha.
adrianN 3y ago
Afaik the efficient market hypothesis says nothing about how long the market takes to optimize after new information is available (and I believe the market needs to solve an np-hard optimization problem). So in principle you could beat the market, by using a better algorithm or more compute.

jaygray0919 3y ago

I just read Hari Seldon to find out what will happen in the future.

hk__2 3y ago

(2020)

diab0lic 3y ago

Much of the discussion here including the linked article fail to make an important distinction between domains. Prediction can be done quite effectively on thin tailed processes. A lot of the counter examples listed in the comments here are physical systems which are thin tailed. I see aircraft autopilot, collision detection, ozone depletion. These are all well understood physical phenomenon in which large deviations do not occur — your car doesn’t get teleported elsewhere in the middle of avoiding a collision. If a large deviation did occur, say a meteor striking between your car and the object it is attempting to avoid, the collision avoidance system would almost certainly fail. These events occur so infrequently that the system can just assume they won’t and boast a high success rate. Meanwhile the examples from the linked article are fat tailed processes. Recessions, GDP, interest rates, exchange rates. These are all subject to large discontinuous jumps. Anyone doing a 5 year rate prediction in July 2019 would have been required to predict the pandemic in order to accurately forecast. This is a single example but predictions in this domain are regularly blown out by being teleported to a completely different world. Unlike the thin tailed domain these events happen frequently enough that they’re the only thing that matters for the forecast. Knowing which class your generating process belongs to is critical to understanding whether forecasting will be effective or not. I’ll take collision detection and leave economic forecasts at the door any day.

kqr 3y ago
It's not that we can't forecast heavy tailed processes -- it's just that the forecasts are used wrong. The appropriate layman's forecast of a recession within the next year is something like a constant 11 %. I'm willing to bet this outperforms most "predictions" out there. But! When people see that number they go, "right, so it's vastly more likely it does not happen" and then completely ignore the possibility. The problem is not in the probability, but in the failure to adequately assign a cost function to the less likely outcomes.
- diab0lic 3y ago
  > The appropriate layman's forecast of a recession within the next year is something like a constant 11 %. I'm willing to bet this outperforms most "predictions" out there. I think we agree here but the unspoken measure is the amount of uncertainty in each forecast. This is still a very inaccurate prediction compared to the collision detection system which nearly always gets it right. You hit the nail on the head in your last paragraph. Doing something useful doesn’t require an accurate prediction. I agree entirely that assigning an appropriate cost function and responding accordingly guides you to useful actions. Edit: Just discovered your blog through your profile. The topics look super relevant to my interests. Thank you for sharing your thoughts online, I’m looking forward to reading them!

FergusArgyll 3y ago

Some questions from an economic analysis standpoint. If someone can predict the future reliably, why don't financial firms hire them? If a firm did hire them, how much money are they getting paid? why so little? Is the economic value of correct predictions lower than you'd think? Does the market believe "past performance is no guarantee of future results"?

jncfhnb 3y ago

Ugh. This annoys me. Can we predict things super accurately? Often no. But you know what’s better than anecdotes about times predictions were bad? Training and testing sets to judge how good we expect models and predictions to be from the beginning. Because a lot of these are not high confidence predictions. And no, it’s not “black swans”. Are those a thing? Sure. It’s ok that models can’t account for things that are not modeled or seen before. But if these things are common enough that they’re systemic and the mode is just not actually accommodating for the world of relevant factors, then it’s not going to have been a good model on the test set to begin with. And we would know that.

paulpauper 3y ago

does this matter if they cannot predict? Doctors cannot predict who gets heart attacks or cancer but they are still needed anyway.

dang 3y ago

Discussed at the time: The Forecasting Fallacy - https://news.ycombinator.com/item?id=24521279 https://news.ycombinator.com/item?id=24521279 - Sept 2020 (9 comments)

lordnacho 3y ago

The critical distinction is IMO that the examples he gives are all examples of where the prediction forms part of its own outcome. Whether interest rate go up depends on whether people think interest rates will go up. Same with GDP and all the other examples. Having this recursion is a massive problem, because there might not be a stable point where the prediction and its outcome are fixed. This may be why it's hard to steer cars automatically. I don't know enough about the subject, but it would seem to fit in the category of "fields where what you think affects what happens". By contrast, a lot of other phenomena are complicated but do not have this prediction-feedback effect. The weather, astronomical observations, a lot of engineering systems.

TrevorMaynard 3y ago

[dead]

TrevorMaynard 3y ago

[dead]

TrevorMaynard 3y ago

[dead]