6 ms·

His tone comes off as very defensive about his own work rather than constructive criticism of the Economist model. >In that sense, the best tip-off that the fo

His tone comes off as very defensive about his own work rather than constructive criticism of the Economist model. >In that sense, the best tip-off that the forecasts are different is they have Biden at 97% to win the popular vote, and have sometimes been as high as 98% (and 99% before they revised their model) whereas we are at 82%. You're getting up to 97-98%, you're getting VERY confident.

TMWNN 6y ago
>You're getting up to 97-98%, you're getting VERY confident. Silver made the same point about probability before the 2016 election, and got into a similar Twitter argument (https://www.politico.com/story/2016/11/nate-silver-huffington-post-polls-twitter-230815 https://www.politico.com/story/2016/11/nate-silver-huffingto...). One guess on who proved to be correct.
- WoahNoun 6y ago
  Winning the election != winning the popular vote. There are plenty of qualitative reasons to believe the popular vote distribution matches the Economist model. He isn't arguing about how they came up with distribution. He is arguing that the particular kind of distribution is unreasonable in all instances. Which I disagree with. There are plenty of places in statistical modeling where a narrow head makes absolute sense.
- matthewmacleod 6y ago
  Nobody was “proved to be correct”, because the winner of a poll does not by itself show anything about the probability of that particular candidate having won.
  - martingoodson 6y ago
    The probability of winning is based on the popular support for a candidate. The poll directly measures that support. How can it possibly not 'show anything about the probability of that particular candidate having won'?
    - xupybd 6y ago
      I think you are right it can. But the last election is proof that the polls can fail to predict the outcome of an election.
    - WoahNoun 6y ago
      >The probability of winning is based on the popular support for a candidate This isn't actually true. The probability of winning is based on aggregation of regional probabilities. The same national vote probability distribution can lead to very, very, different election probability distributions due to regional variation. The electoral college essentially guarantees that the national probability distribution is worthless for actually predicting who will be President.
    - matthewmacleod 6y ago
      Let's say we have two candidates A and B for some kind of election. We analyse our polling and other data, and estimate there is a 90% chance of candidate A winning and a 10% chance of candidate B winning. If candidate B then wins, it does not mean that our analysis was "proved to be [in]correct". By itself, it doesn't actually say anything about the quality of our analysis. After all, we explicitly pointed out that this was a possibility, and it would be strange to argue "your analysis said this might happen, and then it did, so your analysis was incorrect". There's just not enough information to draw any conclusions.
      - martingoodson 6y ago
        If one candidate won 1000 consecutive polls would it also tell you nothing about your estimate? This is obviously absurd: of course it would. How about 100 times? 10 times? 2 times? At what point does evidence cease to 'say anything about the quality of our analysis'? The answer is never. Every datapoint can be used to update your priors according to bayesian statistics.
        WoahNoun 6y ago
        As I said in my other comment that you chose to ignore, the probability of winning national popular vote does not indicate the probability of who will be President.
  - aaaaardvark 6y ago
    I think Nate's _interpretation_ was proved correct, and in the vein of "all models are wrong, some were useful," his proved useful (or would have if people were taking it more seriously) in both forecasting a more uncertain result as well as ancillary things like showing which states were"tipping point states" e.g. where should you put your money if you want to win.
    - aaaaardvark 6y ago
      Or rather, his interpretation was supported by the results, not proved
  - dap 6y ago
    What does it mean to predict the probability of an event that can only possibly be tested once? Isn’t the prediction unfalsifiable?
    - panarky 6y ago
      Sure, if the forecaster and the model were only used one time. But in 538's case they use the similar methods to forecast many individual contests like presidential primaries, presidential elections, senate elections, house elections, governor elections, club soccer, college football, March Madness men and women, MLB-NBA-NFL games and playoffs, etc. Multiple-year track records over many events so you can compare their forecasts to actual results over time. Get the data and compute your own error rate here: https://github.com/fivethirtyeight/checking-our-work-data https://github.com/fivethirtyeight/checking-our-work-data
      - dap 6y ago
        All of those things are very different though. It’s possible you could be excellent at predicting Senate races and terrible at the electoral college, right? And you only get one data point every 4 years.
    - dllthomas 6y ago
      There's a philosophical side to your question that I am not going to engage with, but for the practical you might be interested in this: https://en.m.wikipedia.org/wiki/Scoring_rule https://en.m.wikipedia.org/wiki/Scoring_rule
bioipbiop 6y ago
Reading through both Nate’s twitter thread and the article, it reads like Nate never read the article. It seems to address every point he brought up in a reasonable manner. Odd.
- OJFord 6y ago
  Could you link the article? I'm no statistician, but I do find his '50-54% delta is different than 95-99%' compelling.
  - bioipbiop 6y ago
    https://statmodeling.stat.columbia.edu/2020/08/14/new-fivethirtyeight-com-election-forecast/ https://statmodeling.stat.columbia.edu/2020/08/14/new-fiveth...
dragonwriter 6y ago
> His tone comes off as very defensive about his own work rather than constructive criticism of the Economist model. Most of the linked thread isn't about his own work or the Economist model, but about the false description of their forecasts as being pretty close, made as part of the (true) description that fairly small changes in intermediate results in the Economist model would lead to the same bottom line forecast as Silver's model.
- iainmerrick 6y ago
  I don’t agree with the “fairly small changes” part -- it looked like they were talking about 1% change in mean estimate and something like 25% change in variance. Those sound small, but they’re really not! Or I guess another way to put it is, given the highly polarised two party system in the US, elections tend to be fairly balanced and small differences have an inflated impact on the results. But whether you see the difference as small (1% difference in popular vote) or large (3x difference in chance of victory), the models are different, and one of them must be better, although it’s probably impossible to tell which it is! Comparing models across multiple elections and calculating the Bayesian regret is one way to do it. The models get tweaked each election so this isn’t exact, but it could give a sense of the skill of each forecaster. Does anyone have links for previous election forecasts from Morris and/or Gelman?
  - dragonwriter 6y ago
    > Comparing models across multiple elections and calculating the Bayesian regret is one way to do it. The models get tweaked each election so this isn’t exact, As long as the same inputs aren’t fundamentally unavailable (and even then if the model has systematic handling of missing data, though the validity of that comparison is less clear) you can run the tweaked model on past elections (the main problem there is since those are probably the data used to generate them, it rewards overfitting; you might do better by running them against past elections with differing sets of random dropout of input data points [individual instances of particular polls, etc.] to mitigate that.)
deleted 6y ago
[deleted]

TMWNN 6y ago

>You're getting up to 97-98%, you're getting VERY confident. Silver made the same point about probability before the 2016 election, and got into a similar Twitter argument (https://www.politico.com/story/2016/11/nate-silver-huffington-post-polls-twitter-230815 https://www.politico.com/story/2016/11/nate-silver-huffingto...). One guess on who proved to be correct.

WoahNoun 6y ago
Winning the election != winning the popular vote. There are plenty of qualitative reasons to believe the popular vote distribution matches the Economist model. He isn't arguing about how they came up with distribution. He is arguing that the particular kind of distribution is unreasonable in all instances. Which I disagree with. There are plenty of places in statistical modeling where a narrow head makes absolute sense.
matthewmacleod 6y ago
Nobody was “proved to be correct”, because the winner of a poll does not by itself show anything about the probability of that particular candidate having won.
- martingoodson 6y ago
  The probability of winning is based on the popular support for a candidate. The poll directly measures that support. How can it possibly not 'show anything about the probability of that particular candidate having won'?
  - xupybd 6y ago
    I think you are right it can. But the last election is proof that the polls can fail to predict the outcome of an election.
  - WoahNoun 6y ago
    >The probability of winning is based on the popular support for a candidate This isn't actually true. The probability of winning is based on aggregation of regional probabilities. The same national vote probability distribution can lead to very, very, different election probability distributions due to regional variation. The electoral college essentially guarantees that the national probability distribution is worthless for actually predicting who will be President.
  - matthewmacleod 6y ago
    Let's say we have two candidates A and B for some kind of election. We analyse our polling and other data, and estimate there is a 90% chance of candidate A winning and a 10% chance of candidate B winning. If candidate B then wins, it does not mean that our analysis was "proved to be [in]correct". By itself, it doesn't actually say anything about the quality of our analysis. After all, we explicitly pointed out that this was a possibility, and it would be strange to argue "your analysis said this might happen, and then it did, so your analysis was incorrect". There's just not enough information to draw any conclusions.
    - martingoodson 6y ago
      If one candidate won 1000 consecutive polls would it also tell you nothing about your estimate? This is obviously absurd: of course it would. How about 100 times? 10 times? 2 times? At what point does evidence cease to 'say anything about the quality of our analysis'? The answer is never. Every datapoint can be used to update your priors according to bayesian statistics.
      - WoahNoun 6y ago
        As I said in my other comment that you chose to ignore, the probability of winning national popular vote does not indicate the probability of who will be President.
- aaaaardvark 6y ago
  I think Nate's _interpretation_ was proved correct, and in the vein of "all models are wrong, some were useful," his proved useful (or would have if people were taking it more seriously) in both forecasting a more uncertain result as well as ancillary things like showing which states were"tipping point states" e.g. where should you put your money if you want to win.
  - aaaaardvark 6y ago
    Or rather, his interpretation was supported by the results, not proved
- dap 6y ago
  What does it mean to predict the probability of an event that can only possibly be tested once? Isn’t the prediction unfalsifiable?
  - panarky 6y ago
    Sure, if the forecaster and the model were only used one time. But in 538's case they use the similar methods to forecast many individual contests like presidential primaries, presidential elections, senate elections, house elections, governor elections, club soccer, college football, March Madness men and women, MLB-NBA-NFL games and playoffs, etc. Multiple-year track records over many events so you can compare their forecasts to actual results over time. Get the data and compute your own error rate here: https://github.com/fivethirtyeight/checking-our-work-data https://github.com/fivethirtyeight/checking-our-work-data
    - dap 6y ago
      All of those things are very different though. It’s possible you could be excellent at predicting Senate races and terrible at the electoral college, right? And you only get one data point every 4 years.
  - dllthomas 6y ago
    There's a philosophical side to your question that I am not going to engage with, but for the practical you might be interested in this: https://en.m.wikipedia.org/wiki/Scoring_rule https://en.m.wikipedia.org/wiki/Scoring_rule

bioipbiop 6y ago

Reading through both Nate’s twitter thread and the article, it reads like Nate never read the article. It seems to address every point he brought up in a reasonable manner. Odd.

OJFord 6y ago
Could you link the article? I'm no statistician, but I do find his '50-54% delta is different than 95-99%' compelling.
- bioipbiop 6y ago
  https://statmodeling.stat.columbia.edu/2020/08/14/new-fivethirtyeight-com-election-forecast/ https://statmodeling.stat.columbia.edu/2020/08/14/new-fiveth...

dragonwriter 6y ago

> His tone comes off as very defensive about his own work rather than constructive criticism of the Economist model. Most of the linked thread isn't about his own work or the Economist model, but about the false description of their forecasts as being pretty close, made as part of the (true) description that fairly small changes in intermediate results in the Economist model would lead to the same bottom line forecast as Silver's model.

iainmerrick 6y ago
I don’t agree with the “fairly small changes” part -- it looked like they were talking about 1% change in mean estimate and something like 25% change in variance. Those sound small, but they’re really not! Or I guess another way to put it is, given the highly polarised two party system in the US, elections tend to be fairly balanced and small differences have an inflated impact on the results. But whether you see the difference as small (1% difference in popular vote) or large (3x difference in chance of victory), the models are different, and one of them must be better, although it’s probably impossible to tell which it is! Comparing models across multiple elections and calculating the Bayesian regret is one way to do it. The models get tweaked each election so this isn’t exact, but it could give a sense of the skill of each forecaster. Does anyone have links for previous election forecasts from Morris and/or Gelman?
- dragonwriter 6y ago
  > Comparing models across multiple elections and calculating the Bayesian regret is one way to do it. The models get tweaked each election so this isn’t exact, As long as the same inputs aren’t fundamentally unavailable (and even then if the model has systematic handling of missing data, though the validity of that comparison is less clear) you can run the tweaked model on past elections (the main problem there is since those are probably the data used to generate them, it rewards overfitting; you might do better by running them against past elections with differing sets of random dropout of input data points [individual instances of particular polls, etc.] to mitigate that.)

deleted 6y ago

[deleted]