4 ms·

Not a Nate Silver fan, here, but I don't think the "right once" sentiment about him captures his record accurately. What really happened is that millions of peo

by pohl 4y ago

Not a Nate Silver fan, here, but I don't think the "right once" sentiment about him captures his record accurately. What really happened is that millions of people who have an underdeveloped understanding of what it means for a lower-probability event to happen in a single trial are telling on themselves whenever they refer to him as having been "wrong". I mean, if you play only one round of russian roulette — but you lose — it's not really a surprise, is it. There are only six chambers.

bena 4y ago
Yeah, Silver simply tells us the odds. Someone did the analysis on it at one point and 538 was "right" at about the rate of their percentages. i.e. For all the times they called an event 65%, it happened about 65% of the time. So they weren't "wrong" the other 35% of the time, it's just those are the times the 35% chance bore out. People have a habit of assuming "is likely" is the same as "guaranteed". Or that that's what the caller believe will happen. And then using the event once passed to discredit the initial analysis.
- mabbo 4y ago
  This is exactly why the 2016 election was interesting. Silver called it about 33% chance Trump would win. And then he did, and people said "AHA! You were WRONG!" But he gave Trump a higher probability of winning than just about anyone, other than Trump himself. One of Silver's detractors, Sam Wang, even said it was so impossible Trump would win that he'd eat a bug if he got more than 240 electoral votes. (he chose a gourmet cricket https://www.politico.com/story/2016/11/sam-wang-poll-expert-eats-bug-231278 https://www.politico.com/story/2016/11/sam-wang-poll-expert-...)
  - greesil 4y ago
    Nate and team do modeling well. Fat tails, correlated populations, and they deal with the crap datasets they get from pollsters. I always tune into his podcast when an election rolls around.
  - yamtaddle 4y ago
    > This is exactly why the 2016 election was interesting. Silver called it about 33% chance Trump would win. And then he did, and people said "AHA! You were WRONG!" What was funny about that one was he also got a bunch of "LOL no way you lunatic, he can't possibly have that large a chance and no-one else is saying so, so you're dumb, or maybe you're secretly a pro-Trump shill". Caught it going both ways.
- oneoff786 4y ago
  That “someone” includes 538 themselves. They’re very transparent about their track record and methodology.
- refurb 4y ago
  That logic means you can never prove them wrong (or judge accuracy) unless they give a 100% probability to an outcome which they never would. Sounds kinda useless? I mean, if they said the probabilities were 99 to 1 or 1 to 99, regardless the outcome they can say “well, we never said it couldnt happen”
  - green_on_black 4y ago
    No, we know how to deal with statistics. Sounds like you need a course in it.
    - refurb 4y ago
      Who is "we"? In that case, please "educate" me on how to understand the quality of a prediction (was it accurate or not), of a binary event (say a Presidential candidate winning election), when both outcomes are both "probable" to a non-zero degree. I'm eagerly looking forward to my "education".
      - wasabi991011 4y ago
        For 538 specifically, can start by reading their own evaluations, as written in their helpfully titled "How Good Are FiveThirtyEight Forecasts?"[1]. You can then look up keywords describing their methodology to understand strengths/limitations of those methods. On a broader scale, what you are asking about is called a "Scoring Rule". Wikipedia, as usual, provides an overview[2]. You can take the mean of a forecaster's score, which allows you to compare forecaster methodologies. I'm not sure why your comment needed to be so aggressive. [1] https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/ [2] https://en.m.wikipedia.org/wiki/Scoring_rule https://en.m.wikipedia.org/wiki/Scoring_rule
      - mananaysiempre 4y ago
        A single prediction? You can't. Multiple predictions? Compare the predicted and the actual frequencies, that is, check whether, for example, (roughly) a third of the events that were predicted to happen with 33% certainty to happen actually happened. It is as bad to be not confident enough as it is to be too confident. The standard term for this is calibration[1,2]. Note that this is the easy part: a sports model that predicted a 50% win for the first team, a weather model that predicted historical averages, or a language model that predicted letter frequencies, would have near-perfect calibration, but would at the same time be pretty useless. The other part is discrimination: how educated your guesses are. That is not so simple to quantify, although the 538 articles above mention some of the possible measures. That's it for what an uncertain binary prediction means; but why do we want one? Well, if you're betting (literally or figuratively) on an outcome, it probably makes a difference to you whether the "losing" possibility will come up 1% of the time or 40% of the time; but that does not seem that easy to formalize and may feel unsatisfactory. In that case, here's a formal result. An always-certain prediction service is obviously equivalent to a deterministic decision rule, which churns some data about the situation and says yes or no based on that. (They are the same thing.) An uncertain prediction service is (less obviously) equivalent to a randomized decision rule, which churns some data about the situation, tosses some (known) coins, and says yes or no based on both. (Take the service's result, output yes or no with the probabilities it gave.) Of course there's always a probabilistic decision rule that performs at least as well as any given deterministic one (run the deterministic rule, choose not to toss any coins, output its result). It turns out (see e.g. the introduction[3] to Chentsov's monograph[4]) there can be randomized decision rules that are strictly better on average than any possible deterministic rule. [1] https://fivethirtyeight.com/features/when-we-say-70-percent-it-really-means-70-percent/ https://fivethirtyeight.com/features/when-we-say-70-percent-... [2] https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/ [3] https://books.google.lv/books?id=iqMluWtSFdoC&pg=PA8 https://books.google.lv/books?id=iqMluWtSFdoC&pg=PA8 [4] https://openlibrary.org/books/OL26831120M/ https://openlibrary.org/books/OL26831120M/, https://bookstore.ams.org/mmono-53 https://bookstore.ams.org/mmono-53
- deleted 4y ago
  [deleted]

by pohl 4y ago

bena 4y ago

Yeah, Silver simply tells us the odds. Someone did the analysis on it at one point and 538 was "right" at about the rate of their percentages. i.e. For all the times they called an event 65%, it happened about 65% of the time. So they weren't "wrong" the other 35% of the time, it's just those are the times the 35% chance bore out. People have a habit of assuming "is likely" is the same as "guaranteed". Or that that's what the caller believe will happen. And then using the event once passed to discredit the initial analysis.

mabbo 4y ago
This is exactly why the 2016 election was interesting. Silver called it about 33% chance Trump would win. And then he did, and people said "AHA! You were WRONG!" But he gave Trump a higher probability of winning than just about anyone, other than Trump himself. One of Silver's detractors, Sam Wang, even said it was so impossible Trump would win that he'd eat a bug if he got more than 240 electoral votes. (he chose a gourmet cricket https://www.politico.com/story/2016/11/sam-wang-poll-expert-eats-bug-231278 https://www.politico.com/story/2016/11/sam-wang-poll-expert-...)
- greesil 4y ago
  Nate and team do modeling well. Fat tails, correlated populations, and they deal with the crap datasets they get from pollsters. I always tune into his podcast when an election rolls around.
- yamtaddle 4y ago
  > This is exactly why the 2016 election was interesting. Silver called it about 33% chance Trump would win. And then he did, and people said "AHA! You were WRONG!" What was funny about that one was he also got a bunch of "LOL no way you lunatic, he can't possibly have that large a chance and no-one else is saying so, so you're dumb, or maybe you're secretly a pro-Trump shill". Caught it going both ways.
oneoff786 4y ago
That “someone” includes 538 themselves. They’re very transparent about their track record and methodology.
refurb 4y ago
That logic means you can never prove them wrong (or judge accuracy) unless they give a 100% probability to an outcome which they never would. Sounds kinda useless? I mean, if they said the probabilities were 99 to 1 or 1 to 99, regardless the outcome they can say “well, we never said it couldnt happen”
- green_on_black 4y ago
  No, we know how to deal with statistics. Sounds like you need a course in it.
  - refurb 4y ago
    Who is "we"? In that case, please "educate" me on how to understand the quality of a prediction (was it accurate or not), of a binary event (say a Presidential candidate winning election), when both outcomes are both "probable" to a non-zero degree. I'm eagerly looking forward to my "education".
    - wasabi991011 4y ago
      For 538 specifically, can start by reading their own evaluations, as written in their helpfully titled "How Good Are FiveThirtyEight Forecasts?"[1]. You can then look up keywords describing their methodology to understand strengths/limitations of those methods. On a broader scale, what you are asking about is called a "Scoring Rule". Wikipedia, as usual, provides an overview[2]. You can take the mean of a forecaster's score, which allows you to compare forecaster methodologies. I'm not sure why your comment needed to be so aggressive. [1] https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/ [2] https://en.m.wikipedia.org/wiki/Scoring_rule https://en.m.wikipedia.org/wiki/Scoring_rule
    - mananaysiempre 4y ago
      A single prediction? You can't. Multiple predictions? Compare the predicted and the actual frequencies, that is, check whether, for example, (roughly) a third of the events that were predicted to happen with 33% certainty to happen actually happened. It is as bad to be not confident enough as it is to be too confident. The standard term for this is calibration[1,2]. Note that this is the easy part: a sports model that predicted a 50% win for the first team, a weather model that predicted historical averages, or a language model that predicted letter frequencies, would have near-perfect calibration, but would at the same time be pretty useless. The other part is discrimination: how educated your guesses are. That is not so simple to quantify, although the 538 articles above mention some of the possible measures. That's it for what an uncertain binary prediction means; but why do we want one? Well, if you're betting (literally or figuratively) on an outcome, it probably makes a difference to you whether the "losing" possibility will come up 1% of the time or 40% of the time; but that does not seem that easy to formalize and may feel unsatisfactory. In that case, here's a formal result. An always-certain prediction service is obviously equivalent to a deterministic decision rule, which churns some data about the situation and says yes or no based on that. (They are the same thing.) An uncertain prediction service is (less obviously) equivalent to a randomized decision rule, which churns some data about the situation, tosses some (known) coins, and says yes or no based on both. (Take the service's result, output yes or no with the probabilities it gave.) Of course there's always a probabilistic decision rule that performs at least as well as any given deterministic one (run the deterministic rule, choose not to toss any coins, output its result). It turns out (see e.g. the introduction[3] to Chentsov's monograph[4]) there can be randomized decision rules that are strictly better on average than any possible deterministic rule. [1] https://fivethirtyeight.com/features/when-we-say-70-percent-it-really-means-70-percent/ https://fivethirtyeight.com/features/when-we-say-70-percent-... [2] https://projects.fivethirtyeight.com/checking-our-work/ https://projects.fivethirtyeight.com/checking-our-work/ [3] https://books.google.lv/books?id=iqMluWtSFdoC&pg=PA8 https://books.google.lv/books?id=iqMluWtSFdoC&pg=PA8 [4] https://openlibrary.org/books/OL26831120M/ https://openlibrary.org/books/OL26831120M/, https://bookstore.ams.org/mmono-53 https://bookstore.ams.org/mmono-53
deleted 4y ago
[deleted]