6 ms·
It absolutely has been! In general prediction markets can’t be “correct” or “incorrect” - for instance if a prediction market says there’s a 60% chance of an e
by enjeyw 9mo ago
It absolutely has been!
In general prediction markets can’t be “correct” or “incorrect” - for instance if a prediction market says there’s a 60% chance of an event occurring, and it doesn’t occur, was the market right or wrong? Well it’s hard to say - certainly the market said the event was more likely to occur than not, but only just, and who knows? Maybe the event _only just_ occurred, and very nearly didn’t!
So generally we say a prediction market is “correct” if it is “well calibrated”, which is to say that if we took all the events that the market said had a 60% chance of occurring, then approximately 60% percent of these events occurred (with the same holding true for all other percentages).
On this note, an interesting phenomenon that used to occur was “favorite-longshot bias”, where markets would consistently overestimate the likelihood of longshot events occurring - so events that the market predicted would occur 10% of the time would only occur 5% of the time. What’s fascinating is that once people realized that this bias exited, they began to exploit it by making bets against longshots, which had the effect of moving the market and removing the biases, making the markets well calibrated. It’s a pretty neat example of the efficient market hypothesis in action!
- kurtis_reed 9mo agoNo, markets are evaluated on accuracy, not calibration
- enjeyw 9mo agoWell markets are evaluated on a number of different metrics depending on what you’re trying to determine. If you want to go be pedantic about it and select one metric, markets are evaluated on their Brier Score or some other Proper Scoring Rule, not accuracy. However, I prefer calibration as a high level way to explain prediction market performance to people, as it’s more intuitive.
- kurtis_reed 9mo agoProper scoring rules measure accuracy
- A4ET8a8uTh0_v2 9mo agoI suspect that you are arguing semantics, where parent and grandparent focus on the nuance of what is ACTUALLY being measured. I am saying it like this, because while I never used prediction markets, I briefly looked into them to see if I could use them well. The question of accuracy came up, which is why I happen to align with posters above. With that in mind, what do mean exactly.
- bitshiftfaced 9mo agoYeah it's a good way to introduce the idea. But I don't think someone would really grasp it until they understand why both calibration and "discrimination" are necessary in determining if a prediction market is accurate.
- stingraycharles 9mo agoNoob question from me: what’s the difference between accuracy and calibration? A well calibrated market would be more accurate and vice versa, not? Edit: just found the answer myself: “accuracy measures the percentage of correct predictions out of total predictions, while calibration assesses whether a prediction market's assigned probabilities align with the actual observed frequency of those outcomes”
- kqr 9mo agoA forecaster can be calibrated but almost only assign probabilities in the 40--60 % range. This is not as ueful as one assiging calibrated probabilities in the full range. We try to measure the increased usefulness of the latter with proper scoring rules.
- kurtis_reed 9mo agoSuppose there are 1000 events and 500 will have outcome A and 500 will have outcome B. If you predict a 50% chance of A for every event you'll be perfectly calibrated. On the other hand, if you predict a 90% chance of a certain outcome and you're right for 800 events, you're not perfectly calibrated but you have a lower Brier score (lower is better).
- jjmarr 9mo agoCharlie Kirk has a 3% chance of winning a Nobel Peace Prize right now according to Polymarket. He's climbed from 1% since Maduro was arrested. It seems unlikely since Nobels aren't awarded posthumously.
- rich_sasha 9mo agoHow much volume on this bet? Let's ignore black swan events and say it's a guaranteed 3% return. On how much? $1? $10? $1m? I'd weigh the accuracy by how much money is at stake... Even then, a "perfect" prediction market need not be accurate, if people use it for hedging. If some low probability event is really bad for me, I may pay over odds (pushing the implied probability up) to get paid if it happens. The equilibrium probability may be efficient, reasonable and biased.
- hobofan 9mo agoNormally they aren't, but maybe the US will take over Sweden and the Nobel Foundation and make it happen.
- Boltgolt 9mo ago...only to find out they invaded the wrong country! (Nobel peace prizes are awarded in Oslo)
- vintermann 9mo agoWell, Nobel peace prizes aren't usually awarded to people calling for invasions of their home country either, or cheering for the extrajudicial double-tap killing of smugglers/random fishermen. Who's to say a dead person can't have done the most to "promote peace conferences" as mentioned in Nobel's will? These days, I'd say dead people make a larger net contribution to peace than most politicians.
- BrenBarn 9mo agoSome of the longshot biases still exists and can't be removed due to technical constraints on the platforms. A lot of times there is a minimum contract price, which effectively means the probability of unlikely events cannot be modeled as lower than 1% or 0.1% or whatever. But there are contracts for events much less likely than that.
- pseudo0 9mo agoThere are also issues with the time value of money for long-shot events. Someone has to be willing to buy a share of "No", and if that works out to a return lower than the risk-free rate (eg. buying t-bills) there will be no incentive to take the "No" position. That makes anything roughly under 3-4% per year pretty unreliable.
- bitshiftfaced 9mo agoPolymarket and Kalshi both pay interest on long term bets around the same as the risk free rate.
- thaumasiotes 9mo ago> for instance if a prediction market says there’s a 60% chance of an event occurring, and it doesn’t occur, was the market right or wrong? Well it’s hard to say - certainly the market said the event was more likely to occur than not, but only just, and who knows? Maybe the event _only just_ occurred, and very nearly didn’t! For most events like this, you'd want to see the market spike to 0% or 100% as the deadline approached. And in particular for an event that happens, you want to see the spike to 100% before it happens. Remaining at 60% until after the fact is wrong because the occurrence of the event becomes more certain as it gets closer. Being "well-calibrated" as you describe is a very bad quality metric in the sense that two sets of predictions can achieve the same calibration profile while differing markedly in quality. The farther the predictions are from 50%, the better they are, but your calibration metric doesn't take this into account.