6 ms·
This is a shoddy analysis. 1. There's a statistical gadget specifically for doing this—a "scoring rule" [1] which is a principled way to compare different prob
by benkuhn 6y ago
This is a shoddy analysis.
1. There's a statistical gadget specifically for doing this—a "scoring rule" [1] which is a principled way to compare different probabilistic predictions. A bunch of scatterplots of random quantities against each other are... not that.
By comparing only binary win/loss predictions instead of probabilities, like in the first chart, you throw away almost all information contained in the probabilistic estimates—if Democrats win a state, there's no bonus for predicting (say) 95% Dem instead of 55% dem.
It's plausible that 538 would actually win under a proper scoring rule, because betting markets were underconfident (relative to 538) in deep dem/rep states (predicting e.g. <95% Dem win in VT, vs 538's >99%). [2]
2. The calibration analysis assumes that different state win/loss rates are independent, but that's really untrue: 538's predictions were specifically not independent because they assumed polling errors were correlated between states.
3. Many of the other scatterplots look outlier-driven and don't include r^2 or p-values. With so few datapoints, it's unclear if they are meaningful at all.
[1]: https://en.wikipedia.org/wiki/Scoring_rule https://en.wikipedia.org/wiki/Scoring_rule
[2]: Maybe we should cut prediction markets some slack here because liquidity constraints make them inaccurate for small probabilities. If that's the article's position, though, they should address this instead of just... not using a scoring rule.
- birken 6y agoNot to mention that 538 gave Biden a ~90% chance to win right before the election, whereas betting markets gave Biden a ~60-65% chance to win. So if you go by the most important metric of what was trying to be predicted, 538 was far more accurate for the one giant data point.
- nstj 6y agoI’m interested in the liquidity constraints theory - Betfair has ~$1bn matched for the most recent election. At present there is ~$700k bid and offered at the market for both candidates (the market is still open). Could you expand on your thinking?
- benkuhn 6y agoSorry, that was imprecise. My impression is that, at least on some prediction markets, transaction fees (and maybe also inflation?) make it low-return to buy high-probability contracts. I don't bet on prediction markets myself so I may be wrong about this though!
- amznthrwaway 6y agoYou're correct; there are often fees to deposit or withdrawal, as well as fees on profits. Additionally, consistent winners may face other fees (e.g. Betfair's "premium charge", which will hit long-term consistent winners with a 20% fee on overall profits) Combine those fees with the need to tie up the money until a decision is made, and it makes it hard to make profit from high probability bets. (or from any bet where the line is reasonably near the true odds).
- b0b10101 6y agoyep, predictit takes a 5% fee on profit + 10% on withdrawals
- bhitov 6y agoOther way around. 10% profit 5% withdrawal.
- nstj 6y agoHow would the fees make it any less profitable for high probability contracts vs low probability contracts? And could you expand on the inflation theory? Appreciate the insight!
- Retric 6y agoOpportunity costs on long term high probability events eats up significant portions of the value of winning. Add withdrawal fees and you could easily lose money while making an accurate prediction.
- Closi 6y agoAlso betting markets are highly influenced by both polls and published election models, so it’s hard to say that they ‘beat’ them while also being influenced by them.
- JoshuaDavid 6y agoYou might be interested in this blog post, which _does_ use a scoring rule: http://zackmdavis.net/blog/2020/11/scoring-2020-us-presidential-election-predictions/ http://zackmdavis.net/blog/2020/11/scoring-2020-us-president....
- benkuhn 6y agoOh, great! And the prediction market did not beat polls (it tied with them on swing states and beat them on safe states, as I speculated). I guess between this and the other commenter who made money betting 538's model on predictit, I consider this pretty thoroughly debunked.
- nwienert 6y agoYou can’t do the analysis on PredictIt, they limit market size heavily.
- thebean11 6y agoDo you know of any similar sites that don't limit bet size? I saw some sports betting sites were offering odds, but not really a true market like predictit I think
- anonymoushn 6y agoBetfair seems to offer most of the same contracts as PredictIt as two-sided markets with lower fees and no cap on bet size or number of participants.
- Morizero 6y agoThey are geolocked for some features, however
- anonymoushn 6y ago
- noxvilleza 6y agoIn addition to just using Brier scores (or alternatives) to evaluate their performance, I'm also interested in seeing how 538's underlying Monte Carlo models shift based on x% of error/uncertainty? (for example instead of evaluating a polling advantage as a single value K within a single simulation rather consider it a normally distributed variable).
- tempestn 6y agoThey also only looked at a single election where Republicans outperformed polls across the board. If you look at the 2016 election, 538 was more bullish on Republicans than betting markets, and came out ahead.