11 ms·

A ragtag band of internet friends became the best at forecasting world events

229 points by thyrsus 3y ago

RugnirViking 3y ago
This is very interesting. A discussion of the process of estimating and predicting future events, specifically going into their focus on "base rates". Apparently they are significantly better than other groups that enter most predicting competitions. (who knew there was such a thing?) I particularly liked the idea of base rates and averaging them. [1] I can see here that the idea of them is not to be on their own accurate (some of the methodologies individually are dumb), but do get an idea for the scales of numbers to talk about (the difference between a 5% event and a 10% event is very hard to notice as a human) [1] "One was the rate at which provinces claimed by China (like Hong Kong, Macau, and Tibet) have eventually been absorbed, peacefully or by force; another was how often control of Taiwan has changed over the last few hundred years (twice; once when Japan took over from the Qing Empire in 1895 and once when the Chinese Nationalists did in 1945); the third base rate used Laplace’s rule. Laplace’s rule states that the probability of something that hasn’t happened before happening is 1 divided by N+2, where N is the number of times it hasn’t happened in the past. So the odds of the People’s Republic of China invading Taiwan this year is 1 divided by 75 (the number of years since 1949 when this has not happened) plus 2, or 1/77, or 1.3 percent. Sempere averaged his three base rates to get his initial prediction: 8 percent"
- NunoSempere 3y ago
  Thanks!
  - danielmarkbruce 3y ago
    Out of interest - why don't you guys do way more of this for financial markets?
    - theptip 3y ago
      Presumably you don’t publish your strats there.
      - danielmarkbruce 3y ago
        I'm not sure it's quite the same thing though. It's not a strat, it's a judgement call. And, they can put a bet on at time t and then publish at time t + 1 min. This is done by hedge funds, they do publish/pitch their ideas sometimes.
    - NunoSempere 3y ago
      Yeah, good question. A few years ago the answer may have been that we thought we could influence more money than we could make. Fwiw, when I've tried it, it has tended to suck in a whole lot of my time. For me, other projects seem more meaningful. E.g., right now I'm setting up this: https://alert-team.org https://alert-team.org, another member is behind https://theaidigest.org/progress-and-dangers https://theaidigest.org/progress-and-dangers, a third works at https://forecastingresearch.org/ https://forecastingresearch.org/, etc. Maybe we're an idealistic bunch, but also maybe we're making a bad judgment call, and still yet it's possible we couldn't cut it.
      - 4ggr0 3y ago
        > Maybe we're an idealistic bunch I mean, it makes me quite happy that y'all apparently use your skills for wholesome, humanitarian things instead of just focusing on making $$$.
      - danielmarkbruce 3y ago
        Aren't you just itching to see if you can do it against the hottest competition on the planet? Ie, and I mean this with respect - financial markets are the major leagues of prediction - it feels a little bit like you are playing in the minor leagues. And it's not just a chest beating exercise - you guys appear genuine and to want to push forward the science/art of prediction. But if you can't beat the current best, you aren't state of the art, let alone able to push things forward. You might be pushing things backwards. fwiw, it appears you'd have a decent chance. You guys seem to have an extreme level of objectivity and rationality. There is a lot of domain knowledge required to bet on say, healthcare stocks or whatever, but maybe too much domain knowledge harms rather than hurts.
  - hackerlight 3y ago
    Has your group made predictions on nuclear fusion timelines, or mean temperature rise due to climate change?
    - surgicalcolor 3y ago
      Eh, the second part to me seems like a less useful forecasting. Generally it seems fore asting is focused on human/political events that obviously don't already have useful modelling. For climate change, there's already a lot of existing models out there calculated by much more rigorous metrics than "the Laplace rule" or just naively adding together averages.
      - hackerlight 3y ago
        Climate models just give us scenarios. Conceptually they give us the function f(emissions) = warming + error. But we don't know the probability distribution of emissions, so we don't know the probability distribution of f. Coming up with a reliable probability distribution requires considering things like politics, world events, economic growth, technology breakthoughs, etc. IPCC etc make attempts to model the distribution of future emissions, but it's less rigorous and scientific than the actual underlying climate model.
        exe34 3y ago
        > the probability distribution of emissions We know that every species consumes all available resources until an external force, whether it's predators or environmental collapse checks its growth. Humans are no different - any time a poor person uses a paper straw, a rich one will fly their private jet. Muh backyard arguments on nuclear means we will continue to burn coal and gas as fast as possible, even while renewables help drive up more demand. So I think it's safe to assume that emissions will continue growing.
        hackerlight 3y ago
        I'm hoping for a rigorous analysis. For example, X := "arrival time of fusion with a cost lower than natural gas in $ per kWh" is a random variable with a distribution. Our assumptions about the distribution of X will impact our estimated distribution of emissions, and hence impact our estimated distribution of f(emissions). There's lots of these questions that contribute, both technological and behavioral in nature.
- lz400 3y ago
  Honestly, that sounds completely arbitrary, especially the one about 1/75. It also ignores anything that would alter any thinking person's probability calculations in the real world. For example, if tomorrow the Chinese premier says that Taiwan belongs to China and starts amassing navies, the % wouldn't change. I'd personally much _much_ rather listen to a view from some scholar or expert in the geopolitical situation than these "base rate" calculations.
  - valval 3y ago
    It's supposed to be a completely uninformed prior. The only information, as you can see, is introduced in the formula with the above explanation. Forecasting in general is a silly affair. As a data scientist I get all kinds of requests for all kinds of predictions, and my only real job is to part those requesting parties from their money.
  - yorwba 3y ago
    The Chinese government has been saying that Taiwan belongs to China and amassed navies every year for almost 80 years now. (Longer if you count previous governments.) Base rates are important because what you think is a highly significant trigger event that will surely have major consequences may actually be a frequent occurrence that is typically inconsequential. Or in other words, if you want to say "this time is different" and deviate from the base rate, you actually need to look at the previous times to find out whether it really is different this time, and if so, by how much.
    - lz400 3y ago
      Ok, you can adjust what geopolitical event we think it's significant. Again, since I'm not an expert in the matter, I'm likely to get it wrong, as it seems I did here :) But my point stands, I'd trust expert opinion over these simplistic, falsely confident quantitative approaches.
  - picadores 3y ago
    The percentages of democratic citizens assimilated into china would reach quiet a interesting percentage with taiwan assmiliated into the mainland. You can censor and propagandize all you want, but people who travelled or know better, just wont blieve a word of it happily ever after. One wonders what the internal "knowing" citizen rate is by now in china. The party relies on divide and conquer of uniformed fools. But the fools are in the minority by now - and the divide and conquer is one sneaky trustfall decentralized communication platform away from never working again.
jemmyw 3y ago
This is what drives me mad about this kind of article: > The team had an answer, and it’s an answer that goes some way toward explaining why this group has managed to get so good at predicting the future. Then a whole section and a half of stuff I don't care about to get to the answer: > there was broad agreement that the base rate of war — between China and Taiwan or just between countries in general — is not very high
- LastTrain 3y ago
  I feel like that sometimes too, and I get especially annoyed when I think it is being done to to make me scroll through more ad impressions. I don't think that is the case with this one though. Writers are creative, and they want people to enjoy their writing on a level that is more than just digesting the pertinent facts. My guess is that this author thought they'd whet our appetite enough that we'd want to read the rest of the article. The rest of the article IS interesting but IMO kind of a non-sequitur to the main premise and to your point, gets in the way.
  - brunblanc 3y ago
    At least half of the article was a synopsis of Tetlock's books. So as a summary, it was okay. If it gets people to go and read the real research though, it's worthwhile.
vlovich123 3y ago
The base rate arguing seems like specious reasoning. For example, if you had a volcano that erupts roughly ever 100 years, base rate reasoning using the past 99 years of data would suggest that the probability is 0 and using 1000 years of data would suggest it's ~10% when in reality your base rate in the year following an eruption is 0 with every passing year your probability of an eruption would increase & increase past 10% for every year past 100 that goes without explosion. Same goes for something like war where pressures build up and war becomes more likely rather than less. So getting judged that you're better at predicting by giving low probabilities for rare events isn't that insightful because you'd be outperformed by someone who predicts a black swan event because the magnitude of the event matters. > The prediction got some press attention and earned rejoinders from nuclear experts like Peter Scoblic, who argued it significantly understated the risk of a nuclear exchange. It was a big moment for the group — but also an example of a prediction that’s very, very difficult to get right. The further you’re straying from the ordinary course of history (and a nuclear bomb going off in London would be straying very far), the harder this is. Yup, the group got it right but predicting a rare event doesn't happen isn't that difficult, it's just notable because everyone was overly freaked out, particularly in the media due to self-repeated sensationalism. Peter Scoblic is correct that the risk is significantly understated because it's not correctly adjusting for the impact of the black swan event happening (e.g. if a nuclear explosion were to occur, you'd expect nuclear retaliations).
- Terr_ 3y ago
  A hermit lived by the volcano, and every day he would put up a sign for all the tourists hiking up to see it:"100% accurate prediction: Won't Erupt Today." One day, 30 years later, the volcano's caldera began to spit magma and bubble with gas, so that morning the old hermit walked outside and changed his sign: "99.99% accurate"...
  - aleph_minus_one 3y ago
    That is why it is important not only to consider the accuracy, but also the information gain of a prediction.
jemmyw 3y ago
I see a particular problem with this kind of prediction which maybe someone else can explain. If you are predicting one off events like China invading Taiwan, or London being hit by a nuclear weapon, as a percentage, how can you ever know if the prediction was accurate? Whether the event happens or not does not validate or invalidate the percent chance of it happening. Is there some kind of aggregate that happens to all the predictions to validate them? If you give invasion an 8% chance and invasion happens it doesn't mean you're wrong, just that the world fell into that 8%.
- dadrian 3y ago
  You look across unrelated events that you scored similarly. In aggregate, across all things you predict as having an 8% chance to happen, 8% should actually happen.
  - jemmyw 3y ago
    That sounds reasonable. The article doesn't dig into it. Does this group actually make enough predictions to have this kind of validation?
    - draaglom 3y ago
      Yes, serious forecasters predict on hundreds of questions & the relevant websites prominently feature "binary calibration" plots. This is mine on Metaculus (mentioned in the article): https://imgur.com/a/d3s67xk https://imgur.com/a/d3s67xk There's still a bit of noise at 205 predictions but you can see a pattern emerging!
      - anonymous-panda 3y ago
        But if you can pick what you’re predicting on and pick the percentage you assign to an event, I fail to see how that gives you a calibration.
        tunesmith 3y ago
ManitobaForMe 3y ago
It is meaningless to put any number on an event like this. Unless you can find the very large number of universes where that same situation arises, you can never know if you were right. Maybe a prediction market-based approach can weigh the beliefs of people about the event, but that doesn't mean it's a real probability. Probability theory has given people the mistaken belief that everything is like a card or dice game. It isn't.
- Vecr 3y ago
  https://en.wikipedia.org/wiki/Dutch_book https://en.wikipedia.org/wiki/Dutch_book https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenster... https://en.wikipedia.org/wiki/Bayesian_inference https://en.wikipedia.org/wiki/Bayesian_inference Maybe everything's not a card game, but you really don't want to find yourself in a situation that isn't.
groby_b 3y ago
The "base rate" argument is a rather odd one to make. The base rate of war was hasn't changed much from 2022, that didn't stop Ukraine from happening. Neither did it stop people from calling it. (It certainly didn't stop Samotsvety, who made that call correctly as well) And given that each year sees ~50[1] new armed conflicts (for a total of 195 countries total) I'd posit that the base rate significantly exceeds their estimate of 22%. Yes, of course, a war between China & Taiwan is not just like any war, and there are significant characteristics that influence the likelihood that aren't present in other wars, but at that point, talking of a base rate is just misleading. And when you then "average" the "base rate" of hand picked special cases like "how often did control of Taiwan change", you're waving your hands. Wildly. Which means the point isn't the idea of base rates per se, but choosing the right base rates - or answering the question "what is relevant here", and then analyzing from there. The other success factor for forecasting orgs is choosing what you forecast. As such, I have questions about the actual forecasts made being locked behind a paywall: https://samotsvety.org/track-record/ https://samotsvety.org/track-record/ (It doesn't mean they're not good at forecasting, but it's hard to judge how good) [1] https://ourworldindata.org/conflict-measures-how-do-researchers-measure-how-common-and-deadly-armed-conflicts-are https://ourworldindata.org/conflict-measures-how-do-research...
- gs17 3y ago
  It really is silly. > One was the rate at which provinces claimed by China (like Hong Kong, Macau, and Tibet) have eventually been absorbed, peacefully or by force; another was how often control of Taiwan has changed over the last few hundred years (twice; once when Japan took over from the Qing Empire in 1895 and once when the Chinese Nationalists did in 1945); the third base rate used Laplace’s rule. Laplace’s rule states that the probability of something that hasn’t happened before happening is 1 divided by N+2, where N is the number of times it hasn’t happened in the past. So the odds of the People’s Republic of China invading Taiwan this year is 1 divided by 75 (the number of years since 1949 when this has not happened) plus 2, or 1/77, or 1.3 percent. The first rate is maybe since 1949 (or maybe 1841 or 1557?), the second is over an arbitrary number of centuries, and the last is also since 1949, but could be calculated wildly differently if rephrased as "years mainland Chinese government has control over Taiwan" and extended back to the Qing dynasty. Somehow these are compatible to do an average of. Then "He nudged up to 12 percent." Not that estimated percent chances can ever really be "wrong".
deleted 3y ago
[deleted]
ggm 3y ago
RAND has been in the consensus prediction business since .. well.. RAND was a thing. This is pretty much normal. It's just that it carries on under the surface, most people are unaware where "sources state that there is a 20% chance of .." comments from the military-strategic orbit come from. These % figures aren't very meaningful. I personally think they are mis-read more than they are understood. I would not cross the road on a 1 in 10 chance of death. I undertake activity which at least formally has a 1 in 100 chance of negative outcome frequently, and most of us do 1 in 1000 or upward without thinking about it. The Anti-Vax Phobics were rioting on a 1 in 100,000 risk issue. I tend to think China-Taiwan by 2030 is in the 5% bucket as well for a number of reasons. Principally, the join over news stating the Chinese Military is massively corrupt, and there are generals weeding out over-claimed capacity, the lack of visible improvement in their naval fleet beyond one Aircraft carrier, and the massive bright red wave of blood which will stem from an opposed landing from sea, in modern warfare. Blood is a remarkable thing, in terms of it's influence on the people at large. China is not Russia, they are nothing like as fatalist about their sons and daughters. An occupation of Taiwan will incur massive loss in the generation which is the one-child policy outcome: Many chinese families will lose their investment in the future. Not to mention the missiles coming back the other way will make this invasion a weapon with two sides, one of which faces coastal investment on the mainland. That and the highly internally facing quality of the rhetoric: Talking up Taiwan is seasonal politics as people jockey for control of the structures. It's not preparatory to invasion.
- rufus_foreman 3y ago
  >> China is not Russia, they are nothing like as fatalist about their sons and daughters What is your evidence for this claim? How many people needlessly died due to Chinese communism? Maybe 50,000,000? All sacrificed to an ideology. What has happened before can happen again.
  - wolverine876 3y ago
    Also, at least the histories I've read say the Chinese Communists used 'human wave' tactics in the Korean War.
- 3y ago
wslh 3y ago
Sorry, is there a TLDR? I found the article difficult to follow in their main argument. Where can we compete with this team? Who and how will evaluate our results?
- tunesmith 3y ago
  https://www.infer-pub.com https://www.infer-pub.com
myth_drannon 3y ago
What is the point of forecasting events? The groups who need to act on the events will ignore the forecast or use it to self delude themselves into not acting, others will prepare for events by ignoring the forecasting. Ukraine for example, prepared itself despite forecasts of low probability of an attack. Israel's military intelligence on the other hand knew the details of Hamas attack to the point and chose to ignore it despite high probability of the event.
- draaglom 3y ago
  I'd answer this from two perspectives. As a forecaster: It's fun! It's an interesting way to learn about the world -- rather than gathering inert facts, you're forced to integrate them into a mental model, and then your model is tested for its validity empirically. It's also difficult and competitive, if you like that sort of thing. As a consumer of forecasts: There's good research that prediction markets and other forecast aggregators are the best technology we have as a society for quantifying uncertainty. Not everyone will listen to them, just like not everyone eats their veggies, but c'est la vie. I don't think there's too much risk of self-defeating forecasts (where a low forecast lulls decision makers into a false sense of security) - at least not yet. They're still pretty niche.
- aleph_minus_one 3y ago
  > The groups who need to act on the events will ignore the forecast I know pretty surely that for example people in respective areas of finance are pretty interested in forecasts and will often act on them.
  - fnordpiglet 3y ago
    Forecasts are specifically important in hedging macro risk exposures as well as in black swan event long tail strategies.
- fnordpiglet 3y ago
  It’s hard to believe after the Crimea annexation anyone forecasted a low likelihood of invasion. In fact my understanding (limited) of the forecasting community consensus is that Ukraine and other neighboring Russian former Soviet nations with strategic access to ports have been considered high risk of invasion for some time. I know the US government has depended on forecasting in intelligence and defense for a long time built on the RAND legacy. I find it hard to believe most other sophisticated nations don’t also do similar forecasting and planning around forecasts given how influential these efforts in the US are.
postsantum 3y ago
That reads like a puff piece. There should be a list of correct predictions at the very beginning
IncreasePosts 3y ago
I assume the people who are actually the best at predicting world events are making huge sums of money in the stock market. Or posting on /pol on 4chan. Not entering prediction competitions.
- advael 3y ago
  This assumes that prediction of world events tracks stock performance well. Predicting the performance of stocks requires also predicting a lot of the noise in factors like the error in prediction made by other investors, what marketing moves are made by the people who have an interest in the company looking good in a given quarter, the degree to which companies will be manipulated by large-scale investors for reasons orthogonal to that companies ostensible business models... etc. Some quantitative finance algos obviously do well, but it's a different skill from predicting the coarser phenomena we call "world events". Also, if this all basically sounds like the economy is run like a mob casino... yes
- baq 3y ago
  plenty of people in the stock market make money when they're wrong and lose money when they're right. it's only a game of predicting the future if you're talking about prices.
- wolverine876 3y ago
  > I assume the people who are actually the best at predicting world events are making huge sums of money in the stock market. Or posting on /pol on 4chan. Why assume that those would be people's highest priorities? If I had some superpower, I would not spend my time using it for either of those things.
treme 3y ago
tl;dr Bayes'
- brunblanc 3y ago
  Nassim Taleb just published something interesting, about 'What do you do when you must estimate a probability and there is ZERO information?' https://fooledbyrandomness.com/blog/index.php/2021/09/07/estimating-medical-error-rate-an-intuitive-max-entropy-method/ https://fooledbyrandomness.com/blog/index.php/2021/09/07/est...
bitcurious 3y ago
If I wanted to try my hand at this hobby, what’s the best place to get started?
- lancewiggs 3y ago
  Read Superforecasters. Look at the two linked-from-article forecasting sites. Look up Adam Braff's blog - he runs an annual forecasting contest. Enter prediction markets. Read up about Brier scores. Try, fail, try and learn.
- mhb 3y ago
  Kalshi, Metaculus.
- brunblanc 3y ago
  Most superforecasters are retired academics or quants. I wouldn't recommend the investment in time that it would require.
  - ornornor 3y ago
    But what would be your odds of success? ;)
- NunoSempere 3y ago
  Good Judgment Open, Metaculus
- jellyberg 3y ago
  Forecasting tournaments: https://www.metaculus.com/ https://www.metaculus.com/ https://www.infer-pub.com/ https://www.infer-pub.com/ Forecasting your own life: https://fatebook.io/ https://fatebook.io/ Training your calibration: https://programs.clearerthinking.org/calibrate_your_judgment.html https://programs.clearerthinking.org/calibrate_your_judgment... https://www.quantifiedintuitions.org/calibration https://www.quantifiedintuitions.org/calibration Books to read: Superforecasting, Scout Mindset
aaron695 3y ago
[dead]
horrysith 3y ago
[flagged]
lawrenceyan 3y ago
For people who are religious, how do you factor in an external entity (omnipotent, benevolent in nature, cares about guiding humanity's karma, etc.) weighting the odds of certain events? This is something I've been thinking deeply about for a while now. Curious to hear other people's thoughts.
- UncleOxidant 3y ago
  I mean there are a huge number of viewpoints about this kind of thing even within a single religion like, for example Christianity. The long running divide between Calvinists and Armenians, for example. Calvinists say we don't have free will that God is pulling all the strings, Armenians say that God doesn't go against human free will. I have to say I'm more on the Armenian side - God isn't orchestrating events so much, we humans are given a large amount of latitude. ... but I guess I'm not following how this question is related to the article? They're not claiming to use some kind of supernatural means to predict the future.
  - lawrenceyan 3y ago
    If you really wanted to accurately predict the future, wouldn't you need to take into account the omnipotent external entity's opinion on things?
    - coldtea 3y ago
      Opinion, no. Interventions yes. And even the latter only if you mainly wanted to predict events where said intervention will occur and be a major factor (e.g. predicting an earthquake doesn't really need to take into account that someone will be spared from dying there by divine intervention).
  - coldtea 3y ago
    Just Armenians? Don't Catholic and Eastern Orthodox in general believe the same? (Besides, iirc, Armenians are part of the wider Eastern Orthodox faith).
djyaz1200 3y ago
Xi Jinping is on the record, saying they are taking Taiwan. https://www.nbcnews.com/news/china/xi-warned-biden-summit-beijing-will-reunify-taiwan-china-rcna130087 https://www.nbcnews.com/news/china/xi-warned-biden-summit-be... The only questions are when and how and what will the rest of the world will do about it.
dalore 3y ago
How is this different from the scam where someone sends out stock tips for different companies to different people? The ones that were successful they send out another round of random stock tips. Some of those will be right. Then they can target the people they sent the right ones to and say, look we predicted these so we are successful. Basically, if you have many people predicting something, just from randomness some will be more successful and some won't be.
- tasuki 3y ago
  You could watch Samotsvety to see if they continue getting lucky in the future.
jredwards 3y ago
While I think this is really interesting, I'm not sure how useful it is. The world doesn't change because something may have a 5% chance of happening; it changes because it happened anyway. As a general rule, the most predictable outcomes have the lowest impact.
nuc1e0n 3y ago
Vested interest skews your opinions more than if you don't care either way. Who knew?

229 points by thyrsus 3y ago

This is very interesting. A discussion of the process of estimating and predicting future events, specifically going into their focus on "base rates". Apparently they are significantly better than other groups that enter most predicting competitions. (who knew there was such a thing?) I particularly liked the idea of base rates and averaging them. [1] I can see here that the idea of them is not to be on their own accurate (some of the methodologies individually are dumb), but do get an idea for the scales of numbers to talk about (the difference between a 5% event and a 10% event is very hard to notice as a human) [1] "One was the rate at which provinces claimed by China (like Hong Kong, Macau, and Tibet) have eventually been absorbed, peacefully or by force; another was how often control of Taiwan has changed over the last few hundred years (twice; once when Japan took over from the Qing Empire in 1895 and once when the Chinese Nationalists did in 1945); the third base rate used Laplace’s rule. Laplace’s rule states that the probability of something that hasn’t happened before happening is 1 divided by N+2, where N is the number of times it hasn’t happened in the past. So the odds of the People’s Republic of China invading Taiwan this year is 1 divided by 75 (the number of years since 1949 when this has not happened) plus 2, or 1/77, or 1.3 percent. Sempere averaged his three base rates to get his initial prediction: 8 percent"

NunoSempere 3y ago
Thanks!
- danielmarkbruce 3y ago
  Out of interest - why don't you guys do way more of this for financial markets?
  - theptip 3y ago
    Presumably you don’t publish your strats there.
    - danielmarkbruce 3y ago
      I'm not sure it's quite the same thing though. It's not a strat, it's a judgement call. And, they can put a bet on at time t and then publish at time t + 1 min. This is done by hedge funds, they do publish/pitch their ideas sometimes.
  - NunoSempere 3y ago
    Yeah, good question. A few years ago the answer may have been that we thought we could influence more money than we could make. Fwiw, when I've tried it, it has tended to suck in a whole lot of my time. For me, other projects seem more meaningful. E.g., right now I'm setting up this: https://alert-team.org https://alert-team.org, another member is behind https://theaidigest.org/progress-and-dangers https://theaidigest.org/progress-and-dangers, a third works at https://forecastingresearch.org/ https://forecastingresearch.org/, etc. Maybe we're an idealistic bunch, but also maybe we're making a bad judgment call, and still yet it's possible we couldn't cut it.
    - 4ggr0 3y ago
      > Maybe we're an idealistic bunch I mean, it makes me quite happy that y'all apparently use your skills for wholesome, humanitarian things instead of just focusing on making $$$.
    - danielmarkbruce 3y ago
      Aren't you just itching to see if you can do it against the hottest competition on the planet? Ie, and I mean this with respect - financial markets are the major leagues of prediction - it feels a little bit like you are playing in the minor leagues. And it's not just a chest beating exercise - you guys appear genuine and to want to push forward the science/art of prediction. But if you can't beat the current best, you aren't state of the art, let alone able to push things forward. You might be pushing things backwards. fwiw, it appears you'd have a decent chance. You guys seem to have an extreme level of objectivity and rationality. There is a lot of domain knowledge required to bet on say, healthcare stocks or whatever, but maybe too much domain knowledge harms rather than hurts.
- hackerlight 3y ago
  Has your group made predictions on nuclear fusion timelines, or mean temperature rise due to climate change?
  - surgicalcolor 3y ago
    Eh, the second part to me seems like a less useful forecasting. Generally it seems fore asting is focused on human/political events that obviously don't already have useful modelling. For climate change, there's already a lot of existing models out there calculated by much more rigorous metrics than "the Laplace rule" or just naively adding together averages.
    - hackerlight 3y ago
      Climate models just give us scenarios. Conceptually they give us the function f(emissions) = warming + error. But we don't know the probability distribution of emissions, so we don't know the probability distribution of f. Coming up with a reliable probability distribution requires considering things like politics, world events, economic growth, technology breakthoughs, etc. IPCC etc make attempts to model the distribution of future emissions, but it's less rigorous and scientific than the actual underlying climate model.
      - exe34 3y ago
        > the probability distribution of emissions We know that every species consumes all available resources until an external force, whether it's predators or environmental collapse checks its growth. Humans are no different - any time a poor person uses a paper straw, a rich one will fly their private jet. Muh backyard arguments on nuclear means we will continue to burn coal and gas as fast as possible, even while renewables help drive up more demand. So I think it's safe to assume that emissions will continue growing.
        hackerlight 3y ago
        I'm hoping for a rigorous analysis. For example, X := "arrival time of fusion with a cost lower than natural gas in $ per kWh" is a random variable with a distribution. Our assumptions about the distribution of X will impact our estimated distribution of emissions, and hence impact our estimated distribution of f(emissions). There's lots of these questions that contribute, both technological and behavioral in nature.
lz400 3y ago
Honestly, that sounds completely arbitrary, especially the one about 1/75. It also ignores anything that would alter any thinking person's probability calculations in the real world. For example, if tomorrow the Chinese premier says that Taiwan belongs to China and starts amassing navies, the % wouldn't change. I'd personally much _much_ rather listen to a view from some scholar or expert in the geopolitical situation than these "base rate" calculations.
- valval 3y ago
  It's supposed to be a completely uninformed prior. The only information, as you can see, is introduced in the formula with the above explanation. Forecasting in general is a silly affair. As a data scientist I get all kinds of requests for all kinds of predictions, and my only real job is to part those requesting parties from their money.
- yorwba 3y ago
  The Chinese government has been saying that Taiwan belongs to China and amassed navies every year for almost 80 years now. (Longer if you count previous governments.) Base rates are important because what you think is a highly significant trigger event that will surely have major consequences may actually be a frequent occurrence that is typically inconsequential. Or in other words, if you want to say "this time is different" and deviate from the base rate, you actually need to look at the previous times to find out whether it really is different this time, and if so, by how much.
  - lz400 3y ago
    Ok, you can adjust what geopolitical event we think it's significant. Again, since I'm not an expert in the matter, I'm likely to get it wrong, as it seems I did here :) But my point stands, I'd trust expert opinion over these simplistic, falsely confident quantitative approaches.
- picadores 3y ago
  The percentages of democratic citizens assimilated into china would reach quiet a interesting percentage with taiwan assmiliated into the mainland. You can censor and propagandize all you want, but people who travelled or know better, just wont blieve a word of it happily ever after. One wonders what the internal "knowing" citizen rate is by now in china. The party relies on divide and conquer of uniformed fools. But the fools are in the minority by now - and the divide and conquer is one sneaky trustfall decentralized communication platform away from never working again.

jemmyw 3y ago

This is what drives me mad about this kind of article: > The team had an answer, and it’s an answer that goes some way toward explaining why this group has managed to get so good at predicting the future. Then a whole section and a half of stuff I don't care about to get to the answer: > there was broad agreement that the base rate of war — between China and Taiwan or just between countries in general — is not very high

LastTrain 3y ago
I feel like that sometimes too, and I get especially annoyed when I think it is being done to to make me scroll through more ad impressions. I don't think that is the case with this one though. Writers are creative, and they want people to enjoy their writing on a level that is more than just digesting the pertinent facts. My guess is that this author thought they'd whet our appetite enough that we'd want to read the rest of the article. The rest of the article IS interesting but IMO kind of a non-sequitur to the main premise and to your point, gets in the way.
- brunblanc 3y ago
  At least half of the article was a synopsis of Tetlock's books. So as a summary, it was okay. If it gets people to go and read the real research though, it's worthwhile.

vlovich123 3y ago

The base rate arguing seems like specious reasoning. For example, if you had a volcano that erupts roughly ever 100 years, base rate reasoning using the past 99 years of data would suggest that the probability is 0 and using 1000 years of data would suggest it's ~10% when in reality your base rate in the year following an eruption is 0 with every passing year your probability of an eruption would increase & increase past 10% for every year past 100 that goes without explosion. Same goes for something like war where pressures build up and war becomes more likely rather than less. So getting judged that you're better at predicting by giving low probabilities for rare events isn't that insightful because you'd be outperformed by someone who predicts a black swan event because the magnitude of the event matters. > The prediction got some press attention and earned rejoinders from nuclear experts like Peter Scoblic, who argued it significantly understated the risk of a nuclear exchange. It was a big moment for the group — but also an example of a prediction that’s very, very difficult to get right. The further you’re straying from the ordinary course of history (and a nuclear bomb going off in London would be straying very far), the harder this is. Yup, the group got it right but predicting a rare event doesn't happen isn't that difficult, it's just notable because everyone was overly freaked out, particularly in the media due to self-repeated sensationalism. Peter Scoblic is correct that the risk is significantly understated because it's not correctly adjusting for the impact of the black swan event happening (e.g. if a nuclear explosion were to occur, you'd expect nuclear retaliations).

Terr_ 3y ago
A hermit lived by the volcano, and every day he would put up a sign for all the tourists hiking up to see it:"100% accurate prediction: Won't Erupt Today." One day, 30 years later, the volcano's caldera began to spit magma and bubble with gas, so that morning the old hermit walked outside and changed his sign: "99.99% accurate"...
- aleph_minus_one 3y ago
  That is why it is important not only to consider the accuracy, but also the information gain of a prediction.

jemmyw 3y ago

I see a particular problem with this kind of prediction which maybe someone else can explain. If you are predicting one off events like China invading Taiwan, or London being hit by a nuclear weapon, as a percentage, how can you ever know if the prediction was accurate? Whether the event happens or not does not validate or invalidate the percent chance of it happening. Is there some kind of aggregate that happens to all the predictions to validate them? If you give invasion an 8% chance and invasion happens it doesn't mean you're wrong, just that the world fell into that 8%.

dadrian 3y ago
You look across unrelated events that you scored similarly. In aggregate, across all things you predict as having an 8% chance to happen, 8% should actually happen.
- jemmyw 3y ago
  That sounds reasonable. The article doesn't dig into it. Does this group actually make enough predictions to have this kind of validation?
  - draaglom 3y ago
    Yes, serious forecasters predict on hundreds of questions & the relevant websites prominently feature "binary calibration" plots. This is mine on Metaculus (mentioned in the article): https://imgur.com/a/d3s67xk https://imgur.com/a/d3s67xk There's still a bit of noise at 205 predictions but you can see a pattern emerging!
    - anonymous-panda 3y ago
      But if you can pick what you’re predicting on and pick the percentage you assign to an event, I fail to see how that gives you a calibration.
      - tunesmith 3y ago

ManitobaForMe 3y ago

It is meaningless to put any number on an event like this. Unless you can find the very large number of universes where that same situation arises, you can never know if you were right. Maybe a prediction market-based approach can weigh the beliefs of people about the event, but that doesn't mean it's a real probability. Probability theory has given people the mistaken belief that everything is like a card or dice game. It isn't.

Vecr 3y ago
https://en.wikipedia.org/wiki/Dutch_book https://en.wikipedia.org/wiki/Dutch_book https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenster... https://en.wikipedia.org/wiki/Bayesian_inference https://en.wikipedia.org/wiki/Bayesian_inference Maybe everything's not a card game, but you really don't want to find yourself in a situation that isn't.

groby_b 3y ago

The "base rate" argument is a rather odd one to make. The base rate of war was hasn't changed much from 2022, that didn't stop Ukraine from happening. Neither did it stop people from calling it. (It certainly didn't stop Samotsvety, who made that call correctly as well) And given that each year sees ~50[1] new armed conflicts (for a total of 195 countries total) I'd posit that the base rate significantly exceeds their estimate of 22%. Yes, of course, a war between China & Taiwan is not just like any war, and there are significant characteristics that influence the likelihood that aren't present in other wars, but at that point, talking of a base rate is just misleading. And when you then "average" the "base rate" of hand picked special cases like "how often did control of Taiwan change", you're waving your hands. Wildly. Which means the point isn't the idea of base rates per se, but choosing the right base rates - or answering the question "what is relevant here", and then analyzing from there. The other success factor for forecasting orgs is choosing what you forecast. As such, I have questions about the actual forecasts made being locked behind a paywall: https://samotsvety.org/track-record/ https://samotsvety.org/track-record/ (It doesn't mean they're not good at forecasting, but it's hard to judge how good) [1] https://ourworldindata.org/conflict-measures-how-do-researchers-measure-how-common-and-deadly-armed-conflicts-are https://ourworldindata.org/conflict-measures-how-do-research...

gs17 3y ago
It really is silly. > One was the rate at which provinces claimed by China (like Hong Kong, Macau, and Tibet) have eventually been absorbed, peacefully or by force; another was how often control of Taiwan has changed over the last few hundred years (twice; once when Japan took over from the Qing Empire in 1895 and once when the Chinese Nationalists did in 1945); the third base rate used Laplace’s rule. Laplace’s rule states that the probability of something that hasn’t happened before happening is 1 divided by N+2, where N is the number of times it hasn’t happened in the past. So the odds of the People’s Republic of China invading Taiwan this year is 1 divided by 75 (the number of years since 1949 when this has not happened) plus 2, or 1/77, or 1.3 percent. The first rate is maybe since 1949 (or maybe 1841 or 1557?), the second is over an arbitrary number of centuries, and the last is also since 1949, but could be calculated wildly differently if rephrased as "years mainland Chinese government has control over Taiwan" and extended back to the Qing dynasty. Somehow these are compatible to do an average of. Then "He nudged up to 12 percent." Not that estimated percent chances can ever really be "wrong".

deleted 3y ago

[deleted]

ggm 3y ago

RAND has been in the consensus prediction business since .. well.. RAND was a thing. This is pretty much normal. It's just that it carries on under the surface, most people are unaware where "sources state that there is a 20% chance of .." comments from the military-strategic orbit come from. These % figures aren't very meaningful. I personally think they are mis-read more than they are understood. I would not cross the road on a 1 in 10 chance of death. I undertake activity which at least formally has a 1 in 100 chance of negative outcome frequently, and most of us do 1 in 1000 or upward without thinking about it. The Anti-Vax Phobics were rioting on a 1 in 100,000 risk issue. I tend to think China-Taiwan by 2030 is in the 5% bucket as well for a number of reasons. Principally, the join over news stating the Chinese Military is massively corrupt, and there are generals weeding out over-claimed capacity, the lack of visible improvement in their naval fleet beyond one Aircraft carrier, and the massive bright red wave of blood which will stem from an opposed landing from sea, in modern warfare. Blood is a remarkable thing, in terms of it's influence on the people at large. China is not Russia, they are nothing like as fatalist about their sons and daughters. An occupation of Taiwan will incur massive loss in the generation which is the one-child policy outcome: Many chinese families will lose their investment in the future. Not to mention the missiles coming back the other way will make this invasion a weapon with two sides, one of which faces coastal investment on the mainland. That and the highly internally facing quality of the rhetoric: Talking up Taiwan is seasonal politics as people jockey for control of the structures. It's not preparatory to invasion.

rufus_foreman 3y ago
>> China is not Russia, they are nothing like as fatalist about their sons and daughters What is your evidence for this claim? How many people needlessly died due to Chinese communism? Maybe 50,000,000? All sacrificed to an ideology. What has happened before can happen again.
- wolverine876 3y ago
  Also, at least the histories I've read say the Chinese Communists used 'human wave' tactics in the Korean War.
3y ago

wslh 3y ago

Sorry, is there a TLDR? I found the article difficult to follow in their main argument. Where can we compete with this team? Who and how will evaluate our results?

tunesmith 3y ago
https://www.infer-pub.com https://www.infer-pub.com

myth_drannon 3y ago

What is the point of forecasting events? The groups who need to act on the events will ignore the forecast or use it to self delude themselves into not acting, others will prepare for events by ignoring the forecasting. Ukraine for example, prepared itself despite forecasts of low probability of an attack. Israel's military intelligence on the other hand knew the details of Hamas attack to the point and chose to ignore it despite high probability of the event.

draaglom 3y ago
I'd answer this from two perspectives. As a forecaster: It's fun! It's an interesting way to learn about the world -- rather than gathering inert facts, you're forced to integrate them into a mental model, and then your model is tested for its validity empirically. It's also difficult and competitive, if you like that sort of thing. As a consumer of forecasts: There's good research that prediction markets and other forecast aggregators are the best technology we have as a society for quantifying uncertainty. Not everyone will listen to them, just like not everyone eats their veggies, but c'est la vie. I don't think there's too much risk of self-defeating forecasts (where a low forecast lulls decision makers into a false sense of security) - at least not yet. They're still pretty niche.
aleph_minus_one 3y ago
> The groups who need to act on the events will ignore the forecast I know pretty surely that for example people in respective areas of finance are pretty interested in forecasts and will often act on them.
- fnordpiglet 3y ago
  Forecasts are specifically important in hedging macro risk exposures as well as in black swan event long tail strategies.
fnordpiglet 3y ago
It’s hard to believe after the Crimea annexation anyone forecasted a low likelihood of invasion. In fact my understanding (limited) of the forecasting community consensus is that Ukraine and other neighboring Russian former Soviet nations with strategic access to ports have been considered high risk of invasion for some time. I know the US government has depended on forecasting in intelligence and defense for a long time built on the RAND legacy. I find it hard to believe most other sophisticated nations don’t also do similar forecasting and planning around forecasts given how influential these efforts in the US are.

postsantum 3y ago

That reads like a puff piece. There should be a list of correct predictions at the very beginning

IncreasePosts 3y ago

I assume the people who are actually the best at predicting world events are making huge sums of money in the stock market. Or posting on /pol on 4chan. Not entering prediction competitions.

advael 3y ago
This assumes that prediction of world events tracks stock performance well. Predicting the performance of stocks requires also predicting a lot of the noise in factors like the error in prediction made by other investors, what marketing moves are made by the people who have an interest in the company looking good in a given quarter, the degree to which companies will be manipulated by large-scale investors for reasons orthogonal to that companies ostensible business models... etc. Some quantitative finance algos obviously do well, but it's a different skill from predicting the coarser phenomena we call "world events". Also, if this all basically sounds like the economy is run like a mob casino... yes
baq 3y ago
plenty of people in the stock market make money when they're wrong and lose money when they're right. it's only a game of predicting the future if you're talking about prices.
wolverine876 3y ago
> I assume the people who are actually the best at predicting world events are making huge sums of money in the stock market. Or posting on /pol on 4chan. Why assume that those would be people's highest priorities? If I had some superpower, I would not spend my time using it for either of those things.

treme 3y ago

tl;dr Bayes'

brunblanc 3y ago
Nassim Taleb just published something interesting, about 'What do you do when you must estimate a probability and there is ZERO information?' https://fooledbyrandomness.com/blog/index.php/2021/09/07/estimating-medical-error-rate-an-intuitive-max-entropy-method/ https://fooledbyrandomness.com/blog/index.php/2021/09/07/est...

bitcurious 3y ago

If I wanted to try my hand at this hobby, what’s the best place to get started?

lancewiggs 3y ago
Read Superforecasters. Look at the two linked-from-article forecasting sites. Look up Adam Braff's blog - he runs an annual forecasting contest. Enter prediction markets. Read up about Brier scores. Try, fail, try and learn.
mhb 3y ago
Kalshi, Metaculus.
brunblanc 3y ago
Most superforecasters are retired academics or quants. I wouldn't recommend the investment in time that it would require.
- ornornor 3y ago
  But what would be your odds of success? ;)
NunoSempere 3y ago
Good Judgment Open, Metaculus
jellyberg 3y ago
Forecasting tournaments: https://www.metaculus.com/ https://www.metaculus.com/ https://www.infer-pub.com/ https://www.infer-pub.com/ Forecasting your own life: https://fatebook.io/ https://fatebook.io/ Training your calibration: https://programs.clearerthinking.org/calibrate_your_judgment.html https://programs.clearerthinking.org/calibrate_your_judgment... https://www.quantifiedintuitions.org/calibration https://www.quantifiedintuitions.org/calibration Books to read: Superforecasting, Scout Mindset

aaron695 3y ago

[dead]

horrysith 3y ago

[flagged]

lawrenceyan 3y ago

For people who are religious, how do you factor in an external entity (omnipotent, benevolent in nature, cares about guiding humanity's karma, etc.) weighting the odds of certain events? This is something I've been thinking deeply about for a while now. Curious to hear other people's thoughts.

UncleOxidant 3y ago
I mean there are a huge number of viewpoints about this kind of thing even within a single religion like, for example Christianity. The long running divide between Calvinists and Armenians, for example. Calvinists say we don't have free will that God is pulling all the strings, Armenians say that God doesn't go against human free will. I have to say I'm more on the Armenian side - God isn't orchestrating events so much, we humans are given a large amount of latitude. ... but I guess I'm not following how this question is related to the article? They're not claiming to use some kind of supernatural means to predict the future.
- lawrenceyan 3y ago
  If you really wanted to accurately predict the future, wouldn't you need to take into account the omnipotent external entity's opinion on things?
  - coldtea 3y ago
    Opinion, no. Interventions yes. And even the latter only if you mainly wanted to predict events where said intervention will occur and be a major factor (e.g. predicting an earthquake doesn't really need to take into account that someone will be spared from dying there by divine intervention).
- coldtea 3y ago
  Just Armenians? Don't Catholic and Eastern Orthodox in general believe the same? (Besides, iirc, Armenians are part of the wider Eastern Orthodox faith).

djyaz1200 3y ago

Xi Jinping is on the record, saying they are taking Taiwan. https://www.nbcnews.com/news/china/xi-warned-biden-summit-beijing-will-reunify-taiwan-china-rcna130087 https://www.nbcnews.com/news/china/xi-warned-biden-summit-be... The only questions are when and how and what will the rest of the world will do about it.

dalore 3y ago

How is this different from the scam where someone sends out stock tips for different companies to different people? The ones that were successful they send out another round of random stock tips. Some of those will be right. Then they can target the people they sent the right ones to and say, look we predicted these so we are successful. Basically, if you have many people predicting something, just from randomness some will be more successful and some won't be.

tasuki 3y ago
You could watch Samotsvety to see if they continue getting lucky in the future.

jredwards 3y ago

While I think this is really interesting, I'm not sure how useful it is. The world doesn't change because something may have a 5% chance of happening; it changes because it happened anyway. As a general rule, the most predictable outcomes have the lowest impact.

nuc1e0n 3y ago

Vested interest skews your opinions more than if you don't care either way. Who knew?