9 ms·

This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and m

This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and modeling issues here. I’m surprised the author published it in this form - you don’t want to jump into a controversy like this with such an unclear discussion. As just one example, the whole digression on a “decision boundary” is conceptually mistaken. Once you report a posterior probability, it’s up to the user to establish a simple real number threshold for placing a bet (if we’re talking about means to put skin in the game). The posterior you just learned holds all information, your only action is to threshold it. That’s the Neyman-Pearson lemma. If you allow generalizations to interval-valued probability, which might be sensible under these conditions, the situation gets more complicated. But the writer of the post did not mention this. Another place this came out is his initial discussion of aleatory vs. epistemic uncertainty — this can be done clearly, but here we read: > Aleatory uncertainty is concerned with the fundamental system (probability of rolling a six on a standard die). Epistemic uncertainty is concerned with the uncertainty of the system (how many sides does a die have? And what is the probability of rolling a six?). This is just not helpful. He has said “probability of rolling a six” twice. Source: do UQ in day job.

hn_throwaway_99 8y ago
Agreed. The more I read the more I got embarrassed for the author, mainly because his points were so ridiculously flimsy. I thought this was one of the worst: > Because FiveThirtyEight only predicts probabilities, they do not ever take an absolute stand on an outcome: No ‘skin in the game’ as Taleb would say. This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting). Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability. In fact, Silver is constantly trying to get across how likely < 50% probabilities are with his football analogies (e.g. "a team down by 3 at the half actually winning") to emphasize how likely something like "20%" actually is. If anything, I love Silver's reality-based approach exactly because he does acknowledge that all he can do is offer a best estimate based on available evidence, instead of saying things with false-certainty and then basking in self-proclaimed genius when luck happened to run in his favor.
- VintageCool 8y ago
  This "Twitter fight" began with Nate Silver talking to the media, trying to convey both his prediction and its uncertainty: https://thehill.com/homenews/sunday-talk-shows/414759-nate-silver-says-dems-could-retake-house-or-not-both-extremely https://thehill.com/homenews/sunday-talk-shows/414759-nate-s... > "So in the House we have Democrats with about a 4 in 5 chance of winning," Silver told ABC's "This Week." > "But no one should be surprised if they only win 19 seats and no one should be surprised if they win 51 seats," Silver added. "Those are both extremely possible, based on how accurate polls are in the real world." Dinesh D'Souza jumps on him for hedging his bets: > And just like that, an 80 percent chance for Democratic takeover of the House goes to 50-50. Nassim Taleb also jumps on him in the same way: > @DineshDSouza is right: you don't change a forecast from 80% to 50% under uncertainty. Second time klueless Nate Silver makes a mistake. > When someone says are event and its opposite are extremely possible I infer either 1) 50/50 or 2) the predictor is shamelessly hedging, in other words, BS. > 4- Things are actually worse: that @NateSilver538 doesn't get that if BOTH X & Non-X are "extremely possible = Probability converging to 50-50 is EXACTLY my problem w/his misunderstanding of probability in forecasting, & point of my paper. > 5- FOR THE RECORD I am not translating @DineshDSouza's point that it was 50-50: I ALSO understood ~50-50 DIRECTLY from Silver's "extremely" in the linked "The Hill". The Twitter fight happened on Nov 4, if you want to go look. In this article, the author notes that 538 only presents its model to readers in terms of probabilities, but the public tends to round their predictions up and down. > This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting) This isn't the public's fault, says the author, it's 538's fault for not clearly presenting the uncertainty in the prediction. Additionally, the author writes, the 538 polling-based model swung too wildly in the 2016 election. In doing so, it failed to present a single prediction on which it could be judged. It also failed to acknowledge that unforeseen events could change the polls. Personally, I'd argue that the swinginess of the conveyed the uncertainty of the polls. I spent the last month of 2016 citing Nate Silver to argue on Facebook and Reddit that people were overestimating the certainty of the election forecasts. Fifteen percent of the electorate was up for grabs with just two weeks left in the election! (about 7.5% undecided; and 7.5% saying they'd vote 3rd party) > This is Taleb’s primary argument; FiveThirtyEight’s predictions do not behave like probabilities that incorporate all uncertainty and should not be passed off as them. Nate Silver is being blamed both for trying to convey to the public that his predictions involve uncertainty and for the public not perceiving the uncertainties in his predictions.
acidburnNSA 8y ago
Peripherally, I like using nuclear reactor design analogies for Aleatory vs. Epistemic. Aleatory is the inherent randomness of a system. You can characterize it but it's often hard to reduce. For example, how much boron impurities are mixed into your steel at the time of manufacture. Epistemic is things that are knowable but you don't know with much certainty (because it's hard to measure), like the probability that an incident neutron at 1 MeV will inelasatically scatter off a Uranium-238 nucleus and emerge at 200 keV in some direction.
- robbiep 8y ago
  That might be really useful for you, but for those of us who don’t design nuclear reactors it is utterly meaningless - what are the consequences of each of the results and how much should I care about them? I would be blown away if more than 1 in 10,000 could understand what the consequences are of your statement. Care to share?
  - duckerude 8y ago
    If I'm understanding it correctly, aleatory uncertainty is uncertainty you understand, and epistemic uncertainty is uncertainty due to lack of knowledge. Uncertainty about the outcome of an unbiased die roll is aleatory. You don't know what's going to happen, but that's because of inherent uncertainty, not because you don't know enough about the die. So while you can't be sure what the outcome will be, you can know that each outcome has a 1/6 probability, and you shouldn't expect to adjust that probability as you gain more knowledge. But uncertainty about the outcome of an election is epistemic, because it depends on a lot of present unknowns. If you run more polls you could reach a different, more accurate probability. Part or all of the uncertainty is because of things you don't know.
    - CompelTechnic 8y ago
      To continue the dice analogy- I can imagine a world where gambling sharks arrive at the roulette wheel in casinos, wearing hidden google-glass-ish devices, and place bets while the ball is in motion. Unbeknownst to everyone else, their device performed real time computation based on the position and velocity of the ball and wheel, and gave an output of a tight probability distribution (10% chance 25 red, 35% chance 29 black, 40% chance 12 red, 10% chance 8 black, 5% other). The sharks place bets based on these distributions, and make a bunch of money over the course of a few games. This raises the questions: 1. To the public (not the sharks), all of the possible outcomes of the game have equal probability. Would it be correct to say that all of their uncertainty regarding the outcome of the roulette wheel is aleatory? 2. To the card sharks, they DO have a model with knowledge of game outcomes. So should one say that the distribution output by the device is aleatory uncertainty, and the remaining uncertainty (the fact that there is a distribution instead of a single predicted outcome) in the system is epistimic? 3. #1 and #2 differ in presuming that such a device is present. In absense of such a device, or in general, in absense of a tested model, is it correct to say that all uncertainty is epistimic?
nkurz 8y ago
I think you may not be appreciating the subtlety of the repetition of the "probability of rolling a six". His claim is that aleatory probability starts with the assumption that you have a standard six-sided die with all sides weighted equally, but that epistemic uncertainty requires accounting for the uncertainty that you have a fair die, or even that it has six sides. So in both cases you are indeed trying to calculate the "probability of rolling a six", but the answers, and the process for creating them, are not the same. So while it might not have been the best phrasing for clarity, he's making a meaningful distinction.
- fhars 8y ago
  Yes, the description is technically correct (the best kind of correct...), but as an explanation it is garbage because it hides the point to be explained behind subtlety.
- mturmon 8y ago
  Yes, agreed. I noticed that potential interpretation, mostly because I already knew the distinction. Now suppose you didn’t already know.
  - andrelaszlo 8y ago
    I didn't know and I had to read that part twice.
i_am_proteus 8y ago
It seems less intellectually confused once one recognizes that it's mostly just a plug for the author's company: he links to his product at both beginning and end of the 'article.'
coldtea 8y ago
>This is just not helpful. He has said “probability of rolling a six” twice. For different purposes. One is "probability of rolling a six on a standard die" -- e.g. a systemic property, where we know it's 1 in 6 but we have inherent randomness in how we roll the dice (alea in aleatory comes from the latin for dice btw, as in the famous J.Ceasar quote "alea iacta est" -- well, famous from Asterix at least). The other is the probability of rolling a six based on what we don't know but in theory could (do we have a standard 6-sided dice? Are we asked to predict an event featuring some bizarro D&D dice we haven't seen? Is it really cubic? Have the edges been treated with a file? How about its balance?)
YeGoblynQueenne 8y ago
>> Source: do UQ in day job. UQ: Uncertainty Quantification. (Not, say, Universal Quantifiers, etc).
safgasCVS 8y ago
Unlikely. Taleb seems to like how his argument has been summarised. The crux of the issue taleb has with silver is showing probabilities without making a decision/declaration is cowardly. When asked about making a prediction he says one thing then follows it up with a “ but don’t be surprised if it could be completely random.” And that’s the point - the fact he covers his ass is the problem. Election forecasting is hard so don’t pretend you know something you don’t if you won’t stake something on your predictions. Oh so it could be anything and you don’t want to be held accountable? Then don’t say a damn thing. Basically Silver never wants to be held accountable for his models but always have uncertainty covering his ass as a cop out. so I’m With taleb on this one. Either silver makes a claim and sticks with it or don’t show probabilities and pretend he knows things
- OscarCunningham 8y ago
  I disagree. Probabilities can be useful even if they're not 0% or 100%. For example weather forecasts often give the probability of rain. My plans will be different if the probability is 20% vs 80%. My plans don't tend to vary based on who wins the election, but I imagine that that information is very useful to some people. And if your data is telling you to be 80% sure, I don't think you should fudge it and claim 100%. Report the uncertainty that you actually have.
  - safgasCVS 8y ago
    But look at what you did - you looked at the prob and then made a decision. And that’s kinda Talebs whole spiel - quoting probabilities is moot without an associated action else its just masturbating for attention. Nate shows the probabilities and puts his hands up and says well it could be anything etc. It’s reminds one of those pundits who are bullish on a stock but won’t put their money where their mouths are. If you the viewer buy it because of their recommendations and they are right then they are happy to get the praise but if it goes wrong and you the viewer pay the price they get to say “well the market is highly uncertain etc”. If the guy won’t eat his own dog food then why should others etc
  - lftl 8y ago
    With the weather man, I could keep track of when it rains and doesn't. If it rains significantly more or less than 80% of the time they say there's an 80% chance of rain, I can say that their model is bad. I can't think of a similar way to evaluate Silver's election forecasting model. They very clearly aren't independent probabilities, and his model changes significantly from cycle to cycle. Was his model good in 2012 when every state went to where he predicted the likely probability was? Was it bad when his model said Hillary had a 71.4% chance of winning?
Rainymood 8y ago
I kind of feel the same. I literally shouted out "what the fuck is this shit" mid-way reading the post. Then I looked up the author's credentials: Ph.D. Candidate at Stanford | Chief Data Scientist at MatrixDS ... I'm not sure what to make of this.
briantakita 8y ago
The author was expressing apparently advanced philosophy that is missing from normal discourse, so I can understand why it goes over most peoples' heads. "Instead, epistemically uncertain events are ignored a priori and then FiveThirtyEight assumes wild fluctuations in a prediction from unforeseen events are a normal part of forecasting. Which should lead us to ask ‘If the model is ignoring some of the most consequential uncertainties, are we really getting a reliable probability?’" We don't know what we don't know. Any probabilistic model is constructed with a snapshot of the perceived variables. If the model-maker is not able to conceive of variables, then they will not be in the model, which could have asymmetric effects on accuracy of the model.

hn_throwaway_99 8y ago

Agreed. The more I read the more I got embarrassed for the author, mainly because his points were so ridiculously flimsy. I thought this was one of the worst: > Because FiveThirtyEight only predicts probabilities, they do not ever take an absolute stand on an outcome: No ‘skin in the game’ as Taleb would say. This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting). Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability. In fact, Silver is constantly trying to get across how likely < 50% probabilities are with his football analogies (e.g. "a team down by 3 at the half actually winning") to emphasize how likely something like "20%" actually is. If anything, I love Silver's reality-based approach exactly because he does acknowledge that all he can do is offer a best estimate based on available evidence, instead of saying things with false-certainty and then basking in self-proclaimed genius when luck happened to run in his favor.

VintageCool 8y ago
This "Twitter fight" began with Nate Silver talking to the media, trying to convey both his prediction and its uncertainty: https://thehill.com/homenews/sunday-talk-shows/414759-nate-silver-says-dems-could-retake-house-or-not-both-extremely https://thehill.com/homenews/sunday-talk-shows/414759-nate-s... > "So in the House we have Democrats with about a 4 in 5 chance of winning," Silver told ABC's "This Week." > "But no one should be surprised if they only win 19 seats and no one should be surprised if they win 51 seats," Silver added. "Those are both extremely possible, based on how accurate polls are in the real world." Dinesh D'Souza jumps on him for hedging his bets: > And just like that, an 80 percent chance for Democratic takeover of the House goes to 50-50. Nassim Taleb also jumps on him in the same way: > @DineshDSouza is right: you don't change a forecast from 80% to 50% under uncertainty. Second time klueless Nate Silver makes a mistake. > When someone says are event and its opposite are extremely possible I infer either 1) 50/50 or 2) the predictor is shamelessly hedging, in other words, BS. > 4- Things are actually worse: that @NateSilver538 doesn't get that if BOTH X & Non-X are "extremely possible = Probability converging to 50-50 is EXACTLY my problem w/his misunderstanding of probability in forecasting, & point of my paper. > 5- FOR THE RECORD I am not translating @DineshDSouza's point that it was 50-50: I ALSO understood ~50-50 DIRECTLY from Silver's "extremely" in the linked "The Hill". The Twitter fight happened on Nov 4, if you want to go look. In this article, the author notes that 538 only presents its model to readers in terms of probabilities, but the public tends to round their predictions up and down. > This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting) This isn't the public's fault, says the author, it's 538's fault for not clearly presenting the uncertainty in the prediction. Additionally, the author writes, the 538 polling-based model swung too wildly in the 2016 election. In doing so, it failed to present a single prediction on which it could be judged. It also failed to acknowledge that unforeseen events could change the polls. Personally, I'd argue that the swinginess of the conveyed the uncertainty of the polls. I spent the last month of 2016 citing Nate Silver to argue on Facebook and Reddit that people were overestimating the certainty of the election forecasts. Fifteen percent of the electorate was up for grabs with just two weeks left in the election! (about 7.5% undecided; and 7.5% saying they'd vote 3rd party) > This is Taleb’s primary argument; FiveThirtyEight’s predictions do not behave like probabilities that incorporate all uncertainty and should not be passed off as them. Nate Silver is being blamed both for trying to convey to the public that his predictions involve uncertainty and for the public not perceiving the uncertainties in his predictions.

acidburnNSA 8y ago

Peripherally, I like using nuclear reactor design analogies for Aleatory vs. Epistemic. Aleatory is the inherent randomness of a system. You can characterize it but it's often hard to reduce. For example, how much boron impurities are mixed into your steel at the time of manufacture. Epistemic is things that are knowable but you don't know with much certainty (because it's hard to measure), like the probability that an incident neutron at 1 MeV will inelasatically scatter off a Uranium-238 nucleus and emerge at 200 keV in some direction.

robbiep 8y ago
That might be really useful for you, but for those of us who don’t design nuclear reactors it is utterly meaningless - what are the consequences of each of the results and how much should I care about them? I would be blown away if more than 1 in 10,000 could understand what the consequences are of your statement. Care to share?
- duckerude 8y ago
  If I'm understanding it correctly, aleatory uncertainty is uncertainty you understand, and epistemic uncertainty is uncertainty due to lack of knowledge. Uncertainty about the outcome of an unbiased die roll is aleatory. You don't know what's going to happen, but that's because of inherent uncertainty, not because you don't know enough about the die. So while you can't be sure what the outcome will be, you can know that each outcome has a 1/6 probability, and you shouldn't expect to adjust that probability as you gain more knowledge. But uncertainty about the outcome of an election is epistemic, because it depends on a lot of present unknowns. If you run more polls you could reach a different, more accurate probability. Part or all of the uncertainty is because of things you don't know.
  - CompelTechnic 8y ago
    To continue the dice analogy- I can imagine a world where gambling sharks arrive at the roulette wheel in casinos, wearing hidden google-glass-ish devices, and place bets while the ball is in motion. Unbeknownst to everyone else, their device performed real time computation based on the position and velocity of the ball and wheel, and gave an output of a tight probability distribution (10% chance 25 red, 35% chance 29 black, 40% chance 12 red, 10% chance 8 black, 5% other). The sharks place bets based on these distributions, and make a bunch of money over the course of a few games. This raises the questions: 1. To the public (not the sharks), all of the possible outcomes of the game have equal probability. Would it be correct to say that all of their uncertainty regarding the outcome of the roulette wheel is aleatory? 2. To the card sharks, they DO have a model with knowledge of game outcomes. So should one say that the distribution output by the device is aleatory uncertainty, and the remaining uncertainty (the fact that there is a distribution instead of a single predicted outcome) in the system is epistimic? 3. #1 and #2 differ in presuming that such a device is present. In absense of such a device, or in general, in absense of a tested model, is it correct to say that all uncertainty is epistimic?

nkurz 8y ago

I think you may not be appreciating the subtlety of the repetition of the "probability of rolling a six". His claim is that aleatory probability starts with the assumption that you have a standard six-sided die with all sides weighted equally, but that epistemic uncertainty requires accounting for the uncertainty that you have a fair die, or even that it has six sides. So in both cases you are indeed trying to calculate the "probability of rolling a six", but the answers, and the process for creating them, are not the same. So while it might not have been the best phrasing for clarity, he's making a meaningful distinction.

fhars 8y ago
Yes, the description is technically correct (the best kind of correct...), but as an explanation it is garbage because it hides the point to be explained behind subtlety.
mturmon 8y ago
Yes, agreed. I noticed that potential interpretation, mostly because I already knew the distinction. Now suppose you didn’t already know.
- andrelaszlo 8y ago
  I didn't know and I had to read that part twice.

i_am_proteus 8y ago

It seems less intellectually confused once one recognizes that it's mostly just a plug for the author's company: he links to his product at both beginning and end of the 'article.'

coldtea 8y ago

>This is just not helpful. He has said “probability of rolling a six” twice. For different purposes. One is "probability of rolling a six on a standard die" -- e.g. a systemic property, where we know it's 1 in 6 but we have inherent randomness in how we roll the dice (alea in aleatory comes from the latin for dice btw, as in the famous J.Ceasar quote "alea iacta est" -- well, famous from Asterix at least). The other is the probability of rolling a six based on what we don't know but in theory could (do we have a standard 6-sided dice? Are we asked to predict an event featuring some bizarro D&D dice we haven't seen? Is it really cubic? Have the edges been treated with a file? How about its balance?)

YeGoblynQueenne 8y ago

>> Source: do UQ in day job. UQ: Uncertainty Quantification. (Not, say, Universal Quantifiers, etc).

safgasCVS 8y ago

Unlikely. Taleb seems to like how his argument has been summarised. The crux of the issue taleb has with silver is showing probabilities without making a decision/declaration is cowardly. When asked about making a prediction he says one thing then follows it up with a “ but don’t be surprised if it could be completely random.” And that’s the point - the fact he covers his ass is the problem. Election forecasting is hard so don’t pretend you know something you don’t if you won’t stake something on your predictions. Oh so it could be anything and you don’t want to be held accountable? Then don’t say a damn thing. Basically Silver never wants to be held accountable for his models but always have uncertainty covering his ass as a cop out. so I’m With taleb on this one. Either silver makes a claim and sticks with it or don’t show probabilities and pretend he knows things

OscarCunningham 8y ago
I disagree. Probabilities can be useful even if they're not 0% or 100%. For example weather forecasts often give the probability of rain. My plans will be different if the probability is 20% vs 80%. My plans don't tend to vary based on who wins the election, but I imagine that that information is very useful to some people. And if your data is telling you to be 80% sure, I don't think you should fudge it and claim 100%. Report the uncertainty that you actually have.
- safgasCVS 8y ago
  But look at what you did - you looked at the prob and then made a decision. And that’s kinda Talebs whole spiel - quoting probabilities is moot without an associated action else its just masturbating for attention. Nate shows the probabilities and puts his hands up and says well it could be anything etc. It’s reminds one of those pundits who are bullish on a stock but won’t put their money where their mouths are. If you the viewer buy it because of their recommendations and they are right then they are happy to get the praise but if it goes wrong and you the viewer pay the price they get to say “well the market is highly uncertain etc”. If the guy won’t eat his own dog food then why should others etc
- lftl 8y ago
  With the weather man, I could keep track of when it rains and doesn't. If it rains significantly more or less than 80% of the time they say there's an 80% chance of rain, I can say that their model is bad. I can't think of a similar way to evaluate Silver's election forecasting model. They very clearly aren't independent probabilities, and his model changes significantly from cycle to cycle. Was his model good in 2012 when every state went to where he predicted the likely probability was? Was it bad when his model said Hillary had a 71.4% chance of winning?

Rainymood 8y ago

I kind of feel the same. I literally shouted out "what the fuck is this shit" mid-way reading the post. Then I looked up the author's credentials: Ph.D. Candidate at Stanford | Chief Data Scientist at MatrixDS ... I'm not sure what to make of this.

briantakita 8y ago

The author was expressing apparently advanced philosophy that is missing from normal discourse, so I can understand why it goes over most peoples' heads. "Instead, epistemically uncertain events are ignored a priori and then FiveThirtyEight assumes wild fluctuations in a prediction from unforeseen events are a normal part of forecasting. Which should lead us to ask ‘If the model is ignoring some of the most consequential uncertainties, are we really getting a reliable probability?’" We don't know what we don't know. Any probabilistic model is constructed with a snapshot of the perceived variables. If the model-maker is not able to conceive of variables, then they will not be in the model, which could have asymmetric effects on accuracy of the model.