6 ms·

A Better Crystal Ball?

17 points by howsilly 6y ago

cl42 6y ago
I’ve been doing something similar for investing in stocks and it has worked well. I’m exploring building an NLP-powered version of this (based on crawling news events). If anyone is interested in chatting or collaborating, let me know. :)
- arthurcolle 6y ago
  I'm working on this space as well. Would love to collaborate.
  - cl42 6y ago
    Excellent! I just emailed you.
    - carsonkahn1 6y ago
      Ditto - side project. Can you add me to the thread? carson@carsonkahn.com / carsonkahn.com/who
- jacques_chester 6y ago
  Anecdote time: an NLP specialist I once worked with told me about some research he did into determining the "goodness" or "badness" of press releases and SEC filings. Buy on good news, sell on bad news, hopefully fast enough to beat others to it. After a great deal of crunching and study he said they did come up with a model that could do it reasonably well. Then realised it could be replaced with a simple rule: Are there a bunch of numbers at the top? Good news. Are the numbers buried way down? Bad news. Another simple rule, by the way, is timing vs executive compensation events. If there's an announcement just before a big block of options or RSUs are vesting, then it's good likely to be good news. If it's being posted far away from vesting dates, then it's likely to be bad news.
trishankkarthik 6y ago
This is the most dangerously stupid thing I've ever heard. Tetlock is a known academic charlatan pushing his absolutely useless "superforecasting" nonsense which @nntaleb keeps debunking on Twitter. If Tetlock and friends are so good about forecasting the future, why didn't they predict and warn us about COVID-19 BEFORE @nntaleb and friends (including myself) did? P.S. I have been downvoted for saying this.
- cl42 6y ago
  I think you’re partially right. However, it also depends on what you’re comparing to and what the baseline is. There are lots of projects that try to predict the future and using a framework to do so in a systematic way that can improve over time with feedback is really valuable. That part resonates with me. I do agree that keeping this high level and abstract (scenario planning + probabilistic estimates) isn’t necessarily doing them any favours. Their book on the topic is better, and I do appreciate their argument that it’s not about being a perfect predictor of the future, but simply better than what we have today.
  - trishankkarthik 6y ago
    You cannot fundamentally predict the future, fuhgedaboutit, partly for the same reasons we cannot use Turing machines to solve the halting problem for Turing machines. What you can do, however, is control your exposure to outcomes of unpredictable events. You don't know when a pandemic will hit, but you know it is is matter of time, just like getting hacked. So prepare and design accordingly. Simple. No need for this superforecasting nonsense which doesn't even work.
    - cl42 6y ago
      They would actually argue the same thing. Certain events are impossible to predict. Other events are not. That’s part of the beauty of the problem. You need to be able to pick the right events to predict and ensure the event is well-defined enough to actually test. Your own examples of why this wouldn’t work are exactly what they discuss as the problem... Eventually a lot of forecasts are correct; what governments, companies, and leaders need are forecasts that are time bound, as that’s how they plan.
vosper 6y ago
> Well-calibrated forecasters, for instance, can estimate the likelihood that a skirmish with the Chinese navy in the South China Sea will result in at least two American deaths by December 31. But what policymakers really want to know is the extent to which China will threaten U.S. interests in the coming years and decades. This has never happened (a skirmish, let alone deaths) so how can we think that the forecasters are well-calibrated to predict the likelihood of this event? Because they predicted other, different, stuff?
- deleted 6y ago
  [deleted]
- jacques_chester 6y ago
  > Because they predicted other, different, stuff? In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness. The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions. For example: what are the components of the question? They might include: - The likelihood of deaths due to combat action. That can in turn include predictions of the relative effects of ship armour, weapons effectiveness, preparedness for combat conditions and so on. - The number of people involved. If a skirmish involves a hundred ships on each side, 2 deaths is a very low estimate. How many ships are in the area? How many could reach it by December 31st? How many would be likely to be sent? - The causes of conflict. What are the triggering events, and how likely are they to occur? I'm sure smarter and better-qualified folks than me can argue all the details, but that's the point. You don't accept the question as a coin-flip with no priors. You go prior-hunting.
  - enkid 6y ago
    The other big component is using Bayesian inference. For that you need a prior. What's the prior for this? You could look at incidents the US has had in similar freedom of navigation actions, what China has done with other countries, or other similar situations. For example, I can only think of a few similar incidents, and none that escalated to fatalities at sea. So your prior would probably be pretty small, say 1%. After that you would adjust according to new data. For example, China seems willing to do more risky stuff lately, with India being a perfect example. You might raise your estimate based on that. So maybe that raises your estimate to 2%. But it probably shouldn't completely alter your estimate.
motohagiography 6y ago
Odd to see an article about forecasting without mentioning DeMesquita and Smith's predictioneer models and stakeholder salience models. Also without Charlie Munger's comments on incentives, which have had a better track record than pretty much anyone. There was much point scoring done after the 2016 U.S. election about polling and predictions being wrong, particularly against Fivethirtyeight, but where I think critics got it wrong was, when someone says there's a %95 chance of an event going in one way, it's not valuable as a prediction, but it certainly is as an accurate assessment of the impact of the results. In 2016, if you told people that the %95 implied probability of winning translated to what felt like paying out a 19/1 loss, a lot of people would have agreed with you. You can reliably bet on the effect of the upset being inversely proportional to the predicted probabilities. When I see election predictions that are 60:40 I think the 2nd order impacts of the result will be slightly lopsided, but within the realms of expectations. In this sense, %54 chance of winning close races may be better for stability than perceived polarized %95 ones because of how people perceive the results of a loss.
- jacques_chester 6y ago
  What's described here is more of a framework than an exact forecasting method. DeMesquita's methods would fit quite well, actually. > You can reliably bet on the effect of the upset being inversely proportional to the predicted probabilities. Broadly, yes, if you think of it in an information-theoretic sense. Rare signals carry more information than common ones.
mensetmanusman 6y ago
A travesty that the U.S. pays over a trillion dollars a year while hosting soldiers in 70 countries... You would think with that much planning we would have paid attention to experts decades ago warning that we needed more charged fiber (N95) in storage in case of a pandemic at a cost of ~0.0001 trillion ...
- tim333 6y ago
  The US used to have pandemic planning... https://twitter.com/paulg/status/1244995102818304000 https://twitter.com/paulg/status/1244995102818304000

17 points by howsilly 6y ago

cl42 6y ago

I’ve been doing something similar for investing in stocks and it has worked well. I’m exploring building an NLP-powered version of this (based on crawling news events). If anyone is interested in chatting or collaborating, let me know. :)

arthurcolle 6y ago
I'm working on this space as well. Would love to collaborate.
- cl42 6y ago
  Excellent! I just emailed you.
  - carsonkahn1 6y ago
    Ditto - side project. Can you add me to the thread? carson@carsonkahn.com / carsonkahn.com/who
jacques_chester 6y ago
Anecdote time: an NLP specialist I once worked with told me about some research he did into determining the "goodness" or "badness" of press releases and SEC filings. Buy on good news, sell on bad news, hopefully fast enough to beat others to it. After a great deal of crunching and study he said they did come up with a model that could do it reasonably well. Then realised it could be replaced with a simple rule: Are there a bunch of numbers at the top? Good news. Are the numbers buried way down? Bad news. Another simple rule, by the way, is timing vs executive compensation events. If there's an announcement just before a big block of options or RSUs are vesting, then it's good likely to be good news. If it's being posted far away from vesting dates, then it's likely to be bad news.

trishankkarthik 6y ago

This is the most dangerously stupid thing I've ever heard. Tetlock is a known academic charlatan pushing his absolutely useless "superforecasting" nonsense which @nntaleb keeps debunking on Twitter. If Tetlock and friends are so good about forecasting the future, why didn't they predict and warn us about COVID-19 BEFORE @nntaleb and friends (including myself) did? P.S. I have been downvoted for saying this.

cl42 6y ago
I think you’re partially right. However, it also depends on what you’re comparing to and what the baseline is. There are lots of projects that try to predict the future and using a framework to do so in a systematic way that can improve over time with feedback is really valuable. That part resonates with me. I do agree that keeping this high level and abstract (scenario planning + probabilistic estimates) isn’t necessarily doing them any favours. Their book on the topic is better, and I do appreciate their argument that it’s not about being a perfect predictor of the future, but simply better than what we have today.
- trishankkarthik 6y ago
  You cannot fundamentally predict the future, fuhgedaboutit, partly for the same reasons we cannot use Turing machines to solve the halting problem for Turing machines. What you can do, however, is control your exposure to outcomes of unpredictable events. You don't know when a pandemic will hit, but you know it is is matter of time, just like getting hacked. So prepare and design accordingly. Simple. No need for this superforecasting nonsense which doesn't even work.
  - cl42 6y ago
    They would actually argue the same thing. Certain events are impossible to predict. Other events are not. That’s part of the beauty of the problem. You need to be able to pick the right events to predict and ensure the event is well-defined enough to actually test. Your own examples of why this wouldn’t work are exactly what they discuss as the problem... Eventually a lot of forecasts are correct; what governments, companies, and leaders need are forecasts that are time bound, as that’s how they plan.

vosper 6y ago

> Well-calibrated forecasters, for instance, can estimate the likelihood that a skirmish with the Chinese navy in the South China Sea will result in at least two American deaths by December 31. But what policymakers really want to know is the extent to which China will threaten U.S. interests in the coming years and decades. This has never happened (a skirmish, let alone deaths) so how can we think that the forecasters are well-calibrated to predict the likelihood of this event? Because they predicted other, different, stuff?

deleted 6y ago
[deleted]
jacques_chester 6y ago
> Because they predicted other, different, stuff? In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness. The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions. For example: what are the components of the question? They might include: - The likelihood of deaths due to combat action. That can in turn include predictions of the relative effects of ship armour, weapons effectiveness, preparedness for combat conditions and so on. - The number of people involved. If a skirmish involves a hundred ships on each side, 2 deaths is a very low estimate. How many ships are in the area? How many could reach it by December 31st? How many would be likely to be sent? - The causes of conflict. What are the triggering events, and how likely are they to occur? I'm sure smarter and better-qualified folks than me can argue all the details, but that's the point. You don't accept the question as a coin-flip with no priors. You go prior-hunting.
- enkid 6y ago
  The other big component is using Bayesian inference. For that you need a prior. What's the prior for this? You could look at incidents the US has had in similar freedom of navigation actions, what China has done with other countries, or other similar situations. For example, I can only think of a few similar incidents, and none that escalated to fatalities at sea. So your prior would probably be pretty small, say 1%. After that you would adjust according to new data. For example, China seems willing to do more risky stuff lately, with India being a perfect example. You might raise your estimate based on that. So maybe that raises your estimate to 2%. But it probably shouldn't completely alter your estimate.

motohagiography 6y ago

Odd to see an article about forecasting without mentioning DeMesquita and Smith's predictioneer models and stakeholder salience models. Also without Charlie Munger's comments on incentives, which have had a better track record than pretty much anyone. There was much point scoring done after the 2016 U.S. election about polling and predictions being wrong, particularly against Fivethirtyeight, but where I think critics got it wrong was, when someone says there's a %95 chance of an event going in one way, it's not valuable as a prediction, but it certainly is as an accurate assessment of the impact of the results. In 2016, if you told people that the %95 implied probability of winning translated to what felt like paying out a 19/1 loss, a lot of people would have agreed with you. You can reliably bet on the effect of the upset being inversely proportional to the predicted probabilities. When I see election predictions that are 60:40 I think the 2nd order impacts of the result will be slightly lopsided, but within the realms of expectations. In this sense, %54 chance of winning close races may be better for stability than perceived polarized %95 ones because of how people perceive the results of a loss.

jacques_chester 6y ago
What's described here is more of a framework than an exact forecasting method. DeMesquita's methods would fit quite well, actually. > You can reliably bet on the effect of the upset being inversely proportional to the predicted probabilities. Broadly, yes, if you think of it in an information-theoretic sense. Rare signals carry more information than common ones.

mensetmanusman 6y ago

A travesty that the U.S. pays over a trillion dollars a year while hosting soldiers in 70 countries... You would think with that much planning we would have paid attention to experts decades ago warning that we needed more charged fiber (N95) in storage in case of a pandemic at a cost of ~0.0001 trillion ...

tim333 6y ago
The US used to have pandemic planning... https://twitter.com/paulg/status/1244995102818304000 https://twitter.com/paulg/status/1244995102818304000