3 ms·

"Lesson 1: Use probabilities instead of words to avoid misinterpretation." Probabilities are meaningless unless it’s a repeatable experiment otherwise its a lu

"Lesson 1: Use probabilities instead of words to avoid misinterpretation." Probabilities are meaningless unless it’s a repeatable experiment otherwise its a ludic fallacy eg "There's a 70% chance of Hillary winning". This is an un-provable statement. Either she wins and prediction was right, or she loses and it counts as part of the 30%. This is Nate Silver's get-out-of-jail-free card so even when he's wrong he comes off as being right. i.e. this statement makes sense in a casino and nowhere else. "Lesson 2: Use structured approaches to set probabilities." Probabilities are meaningless by themselves. Path dependency matters a great deal to actual humans but not to business professors. A strategy that works well for the ensemble wont necessarily work well for the individual. e.g. if you save for retirement for the average life expectancy then 50% of the people would be screwed and 50% of the people would have saved too much. i.e. The cost of being right/wrong is what matters and not the probability.

fjsolwmv 8y ago
Nate silver makes hundreds of predictions every year. You can compute his average and error in his estimates.
archgoon 8y ago
> Probabilities are meaningless unless it’s a repeatable experiment otherwise its a ludic fallacy Um... this goes against the entirety of the Bayesian approach to statistics. I think you'd find a lot of very intelligent people who disagree strongly with this statement. The Bayesian approach takes probabilities as subjective confidences. You can describe confidences as "well calibrated" if, when you look at their historical guesses, if their 70% assessments are correct 70% of the time.
- safgasCVS 8y ago
  ". I think you'd find a lot of very intelligent people who disagree strongly with this statement. " I never claimed to be intelligent. " You can describe confidences as "well calibrated" if, when you look at their historical guesses, if their 70% assessments are correct 70% of the time." - Again, if you're trying to figure out if a coin is split 50/50 then yes, but without being able to repeat the same experiment you're fooling yourself and the whole aspect of bayesian thinking I think goes out the window. eg me being right about unrelated topics doesn't mean I'm right/wrong about specific topics.
caf 8y ago
It is true that you cannot evaluate a single probabilistic prediction as being right or wrong (so you cannot say that Nate Silver was either wrong or right in some example - he gave a probability, and that probability was either accurate or inaccurate, but in isolation that cannot be evaluated). However, if a consistent process is used to generate a sequence of probabilistic predictions, the accuracy of the predictions generated by that process certainly can be evaluated.
- safgasCVS 8y ago
  Absolutely - and my issue is conflating experiments that ARE repeatable (eg a probabalistic game of chance) with experiments that arent (eg election results of two particular candidates). So its one of those heads I win tails you lose bets when it comes to Nate
  - caf 8y ago
    It works even if the process is "we ask an oracle for a prediction". The more that Nate Silver produces new prior probability estimates for events, the better an idea we can get of the accuracy of his process. This measure is calibration, as the original article mentions. If our oracle tells us that an event has a 70% chance of happening, then regardless of whether the event happens or not we can't say much about the oracle's calibration. But if, out of 100 independent events the oracle has predicted to each have a 70% chance of happening, 69 of those events actually happened, then that tells us quite a bit.

fjsolwmv 8y ago

Nate silver makes hundreds of predictions every year. You can compute his average and error in his estimates.

archgoon 8y ago

> Probabilities are meaningless unless it’s a repeatable experiment otherwise its a ludic fallacy Um... this goes against the entirety of the Bayesian approach to statistics. I think you'd find a lot of very intelligent people who disagree strongly with this statement. The Bayesian approach takes probabilities as subjective confidences. You can describe confidences as "well calibrated" if, when you look at their historical guesses, if their 70% assessments are correct 70% of the time.

safgasCVS 8y ago
". I think you'd find a lot of very intelligent people who disagree strongly with this statement. " I never claimed to be intelligent. " You can describe confidences as "well calibrated" if, when you look at their historical guesses, if their 70% assessments are correct 70% of the time." - Again, if you're trying to figure out if a coin is split 50/50 then yes, but without being able to repeat the same experiment you're fooling yourself and the whole aspect of bayesian thinking I think goes out the window. eg me being right about unrelated topics doesn't mean I'm right/wrong about specific topics.

caf 8y ago

It is true that you cannot evaluate a single probabilistic prediction as being right or wrong (so you cannot say that Nate Silver was either wrong or right in some example - he gave a probability, and that probability was either accurate or inaccurate, but in isolation that cannot be evaluated). However, if a consistent process is used to generate a sequence of probabilistic predictions, the accuracy of the predictions generated by that process certainly can be evaluated.

safgasCVS 8y ago
Absolutely - and my issue is conflating experiments that ARE repeatable (eg a probabalistic game of chance) with experiments that arent (eg election results of two particular candidates). So its one of those heads I win tails you lose bets when it comes to Nate
- caf 8y ago
  It works even if the process is "we ask an oracle for a prediction". The more that Nate Silver produces new prior probability estimates for events, the better an idea we can get of the accuracy of his process. This measure is calibration, as the original article mentions. If our oracle tells us that an event has a 70% chance of happening, then regardless of whether the event happens or not we can't say much about the oracle's calibration. But if, out of 100 independent events the oracle has predicted to each have a 70% chance of happening, 69 of those events actually happened, then that tells us quite a bit.