5 ms·

I see a particular problem with this kind of prediction which maybe someone else can explain. If you are predicting one off events like China invading Taiwan, o

• dadrian 3y ago
  You look across unrelated events that you scored similarly. In aggregate, across all things you predict as having an 8% chance to happen, 8% should actually happen.
  • jemmyw 3y ago
    That sounds reasonable. The article doesn't dig into it. Does this group actually make enough predictions to have this kind of validation?
    • draaglom 3y ago
      Yes, serious forecasters predict on hundreds of questions & the relevant websites prominently feature "binary calibration" plots. This is mine on Metaculus (mentioned in the article): https://imgur.com/a/d3s67xk https://imgur.com/a/d3s67xk There's still a bit of noise at 205 predictions but you can see a pattern emerging!
      • anonymous-panda 3y ago
        But if you can pick what you’re predicting on and pick the percentage you assign to an event, I fail to see how that gives you a calibration.
        • tunesmith 3y ago
          Because you make lots of predictions, and smooth out the numbers. If the events you give 8% probability actually happen 35% of the time, you aren't as calibrated as you are if they happen 10% or 5% of the time. The calibration numbers measure how much you are off by.
          • anonymous-panda 3y ago
            This makes no sense. If I have 1000 predictions and vary my estimates from 0-10%, that’s only 100 samples assuming you round to the nearest whole percent. And there’s no correlation between any of those samples. For example, I could say the probability of a lightning storm tomorrow is 1% and the probability of a war with China in the next year is 1% - calibrating amount those 1% events is clearly non sensical. You could try to calibrate among similar events but then you have no way of estimating that a 10% war between US and China vs a 1% of war between Russia and China. Basically this is an exercise of garbage in and garbage out.
            • omeze 3y ago
              I think youre kind of right, its not really meaningful to know “im right about things that happen 5% of the time!”. But if a lot of things you bet on structurally happen to be low probability and of the same type (eg you bet on one type of catastrophic event like pandemic, crop disease, or war etc) then its useful
              • anonymous-panda 3y ago
                Or you’ve just learned how to optimize your score on the game you’re playing but the game isn’t actually about predicting the future.
                • tunesmith 3y ago
                  Proper scoring rules like Brier mean that any attempt to optimize your score by definition mean that you are improving your calibration.
    • hackerlight 3y ago
      The article does dig into it. They have Brier scores.
    • jellyberg 3y ago
      See https://samotsvety.org/track-record/ https://samotsvety.org/track-record/
  • dotancohen 3y ago
    That sounds a bit like the gambler's fallacy to me.
  • avereveard 3y ago
    Doesn't seem right, if a state capital city base chance for a nuclear strike is 1% and you have 195 capitals or so 2 of them would be hit by now.
    • TOMDM 3y ago
      You're correct it mostly only works for a spread of predictions that don't have such a large influence on each other. If one capital gets nuked the odds of another one suffering the same fate skyrocket.
      • dmurray 3y ago
        Yeah, it's just about possible to imagine Kyiv or Tehran gets nuked in a "limited engagement" that never escalates further. But if, say, Lisbon gets nuked then the whole world is at war.
    • cudder 3y ago
      Maybe your base rate is way off then?
    • danmaz74 3y ago
      Nuclear strikes on N different capital cities aren't statistically independent events, and that kind of computation only makes sense for statistically independent events.
  • michaelt 3y ago
    That might give you some confidence. But if you're looking at your performance on unrelated predictions, it won't necessarily detect problems with a given prediction. I predict there's an 8% chance of one drive in my four-hard-drive array failing in the next year. I predict there's an 8% chance the next UK election ends without any party holding a clear majority, leading to a labour-lib dem coalition government. For the first prediction to be accurate, I just need to read the backblaze hard drive stats and multiply. For the second prediction to be accurate, though? That depends on a lot of factors that are a lot harder to know. For example, would the lib dems be likely to enter a coalition, given how the last one went for them in 2015?
    • yorwba 3y ago
      Sure, you cannot expect to reliably detect problems with a given prediction, since it could always be right or wrong due to chance. But also, what would you do if you had the ability to detect problems with a prediction? If you just want to use a different prediction method that fixes the problem and makes better predictions, looking at aggregate performance over many unrelated predictions does help. Compare how well different methods do, go with the best.
    • harperlee 3y ago
      But you could come up with a probability prediction on whether a random prediction from the predictor is accurate.
• contravariant 3y ago
  Probabilities turn out to be surprisingly meaningless sometimes. Especially for rare unrepeatable events. A good way to make them a bit more concrete to put some money on it, but even then it's hard to be sure. You can however use this to measure how good someone is in predicting. Take sports for example, by betting according to someone's prediction you can see how much money you earn (if their probabilities are 'true' you are almost guaranteed to earn money, eventually). But really in that case you're hoping that someone's ability to predict one sports match is indicative of their ability to predict the next, which is a good bet, but obviously doesn't hold for these kinds of miscellaneous rare events. In this case you can imagine a betting system, and you can imagine some people might turn out to be better at it than others (though the relevant measure is not the one used in this article), so I suppose you can talk about someone's ability to predict these events. Though in the end the bookie always wins, so make of that what you will. Of course this being probability theory all of these methods only work with high probability, so they may not work at all.
• yieldcrv 3y ago
  In a good prediction market the bets function as binary options, with a payout at 100 for being correct within the time frame. so its more about what price between 0 and 100 would you bet on that belief at 20 you have a 500% gain if it resolves at 100 and this willingness to bet is telegraphed to the crowd, it doesn't really mean conviction but maps to the crowds aggregate understanding of the probability occurring or not
• danielmarkbruce 3y ago
  You don't know for that event. If you predict 100s of events, then use a method like Brier scores, you can get a good idea of how good you are. Even then it's probabilistic, but with enough samples it becomes incredibly unlikely you are just the worlds luckiest guesser.
  • dotancohen 3y ago
    > it becomes incredibly unlikely you are just the worlds luckiest guesser And yet that is still more likely than the idea that you can predict the future. In any case, we are not talking about _you_ becoming the world's best guesser, we are talking about _somebody_ becoming the world's best guesser. That is going to be someone, so no need to be surprised when they emerge.
    • omeze 3y ago
      you're right, but people predict the future all the time, if there's a pre-established model for that prediction. I just threw a ball in the air and predicted how long it'd stay there </akshually> so I think this comes down to how the predictions were arrived. One way to do this is to ask individuals who both have similarly high predictive scores and bet on the same types of events to explain some of their past predictions, and if their methods are similar then you've learned a new predictive tool.
      • valval 3y ago
        How is the difficulty of a certain prediction scored and weighted?
        • danielmarkbruce 3y ago
          In these competitions it generally isn't. So, you can't compare, for example, my brier scores for predicting events a, b, c and yours for events x, y, z.
    • danielmarkbruce 3y ago
• legulere 3y ago
  > Is there some kind of aggregate that happens to all the predictions to validate them? Most likely an application of the law of large numbers https://en.wikipedia.org/wiki/Law_of_large_numbers https://en.wikipedia.org/wiki/Law_of_large_numbers
• coldtea 3y ago
  Taleb's works and often posts cover exactly this.
• im3w1l 3y ago
  The standard solution in ML for this is cross entropy loss. You can think about it like this: from the probability estimate you derive two codewords, one saying it will happen and one say it won't happen. Then when the event happens (or doesn't) you write down the corresponding code in your log file. Then after a large number of predictions you can compare the size of people's log files. Someone having a shorter logfile means they used a more efficient encoding which means their probability estimates were better. But yeah it's not possible to do this after just one event, you need some track record to be able to say that someone is statistically significantly better.
• valval 3y ago
  Previous responses have given you some great food for thought, but my two cents is that what's being discussed in the article is no different from horoscopes or the old lady in the tent with the crystal ball.
  • jemmyw 3y ago
    I'm inclined to agree with you. Events like China invading Taiwan and a nuclear strike on London cannot be aggregated with other predictions because they are unique. There's no correlation between predictions to allow for it. Showing that you have some accuracy predicting any number of other things does nothing to prove you have accuracy predicting a % chance of a unique event like those ones. Further, if those events happen or not there's no way to validate the prediction. There aren't 100 Chinas ready to invade 100 Taiwans. Even expanding to other invasions of a very broadly similar nature doesn't give enough data points. Looking at historical data is bum too, there's too much geopolitical and technological change since. And that makes the base rates meaningless. All my opinion of course.
    • Asraelite 3y ago
      > Showing that you have some accuracy predicting any number of other things does nothing to prove you have accuracy predicting a % chance of a unique event like those ones. When someone has a track record of consistently predicting these "unique" events better than average, then clearly there must be some pattern there that they're picking up on and the events aren't as unique as one would think. At the end of the day, someone who financially invests based on these predictions will eventually end up richer than someone who doesn't, whether you believe that should be possible or not.
      • jemmyw 3y ago
        They are mixing predictions that do allow for accuracy on base rates with ones that don't. There is clearly a bias that this particular group has leveraged rather than them being better at prediction.
      • valval 3y ago
• amadeuspagel 3y ago
  This is the great advantage of financial markets, where a prediction is not expressed in a percentage, but betting a certain amount of money on an outcome.
• saghm 3y ago
  I think this is essentially the divide that separates frequentists from bayesians. I'm sure someone can chime in and correct my summary if it's wrong, but broadly speaking, my understanding is that frequentists consider probabilities to only make sense in the context of a repeatable event where you can measure how often something occurred, whereas bayesians consider probabilities to represent the "likelihood" of something happening even in a one-off event, which often is conveyed in terms of whether you'd be willing to take a given bet with certain odds; if a bayesian thinks something has an 80% chance of happening, they'd consider it a fair bet to offer someone $4 if it does happen but get $1 from them if it doesn't, since 80% is four times more likely than 20%. The downside of bayesian probability that you point out is that isn't not straightforward to evaluate whether odds were correct or not. An example of this would be the 2016 US presidential elections; even the highest estimations of the odds of Trump beating Clinton were 30%, which after the fact was often cited as the predictions being "wrong". It's not easy to falsify this though, because a 30% chance doesn't mean something is _guaranteed_ not to happen; maybe it really was 70% likely not to happen, but we happened to end up in the 30%! Frequentism doesn't suffer from this issue, but it also makes it impossible to talk about certain types of events (like presidential elections). In practice, bayesian probability gets used a lot when people want to talk about those sorts of events because there's not really any obvious alternative.
  • anonymous-panda 3y ago
    No that’s not the divide. Bayesian requires you to have an estimate of the prior and a reasonable way to update that estimate by validating against ground truth. Here there’s no ground truth and no principled way that the prior is being picked. It’s numerology masking itself as statistical rigor. I have no problem with 538 saying that politician A has a 30% chance of winning because it’s a blended weighted average of several independent measurements of the ground truth (polling). There’s no independent measurements of the ground truth happening here. Indeed, things regarding war plans would be classified documents random people wouldn’t have to come up with a better estimate.