3 ms·

No, markets are evaluated on accuracy, not calibration

enjeyw 10mo ago
Well markets are evaluated on a number of different metrics depending on what you’re trying to determine. If you want to go be pedantic about it and select one metric, markets are evaluated on their Brier Score or some other Proper Scoring Rule, not accuracy. However, I prefer calibration as a high level way to explain prediction market performance to people, as it’s more intuitive.
- kurtis_reed 10mo ago
  Proper scoring rules measure accuracy
  - A4ET8a8uTh0_v2 10mo ago
    I suspect that you are arguing semantics, where parent and grandparent focus on the nuance of what is ACTUALLY being measured. I am saying it like this, because while I never used prediction markets, I briefly looked into them to see if I could use them well. The question of accuracy came up, which is why I happen to align with posters above. With that in mind, what do mean exactly.
- bitshiftfaced 10mo ago
  Yeah it's a good way to introduce the idea. But I don't think someone would really grasp it until they understand why both calibration and "discrimination" are necessary in determining if a prediction market is accurate.
stingraycharles 10mo ago
Noob question from me: what’s the difference between accuracy and calibration? A well calibrated market would be more accurate and vice versa, not? Edit: just found the answer myself: “accuracy measures the percentage of correct predictions out of total predictions, while calibration assesses whether a prediction market's assigned probabilities align with the actual observed frequency of those outcomes”
- kqr 10mo ago
  A forecaster can be calibrated but almost only assign probabilities in the 40--60 % range. This is not as ueful as one assiging calibrated probabilities in the full range. We try to measure the increased usefulness of the latter with proper scoring rules.
- kurtis_reed 10mo ago
  Suppose there are 1000 events and 500 will have outcome A and 500 will have outcome B. If you predict a 50% chance of A for every event you'll be perfectly calibrated. On the other hand, if you predict a 90% chance of a certain outcome and you're right for 800 events, you're not perfectly calibrated but you have a lower Brier score (lower is better).

enjeyw 10mo ago

Well markets are evaluated on a number of different metrics depending on what you’re trying to determine. If you want to go be pedantic about it and select one metric, markets are evaluated on their Brier Score or some other Proper Scoring Rule, not accuracy. However, I prefer calibration as a high level way to explain prediction market performance to people, as it’s more intuitive.

kurtis_reed 10mo ago
Proper scoring rules measure accuracy
- A4ET8a8uTh0_v2 10mo ago
  I suspect that you are arguing semantics, where parent and grandparent focus on the nuance of what is ACTUALLY being measured. I am saying it like this, because while I never used prediction markets, I briefly looked into them to see if I could use them well. The question of accuracy came up, which is why I happen to align with posters above. With that in mind, what do mean exactly.
bitshiftfaced 10mo ago
Yeah it's a good way to introduce the idea. But I don't think someone would really grasp it until they understand why both calibration and "discrimination" are necessary in determining if a prediction market is accurate.

stingraycharles 10mo ago

Noob question from me: what’s the difference between accuracy and calibration? A well calibrated market would be more accurate and vice versa, not? Edit: just found the answer myself: “accuracy measures the percentage of correct predictions out of total predictions, while calibration assesses whether a prediction market's assigned probabilities align with the actual observed frequency of those outcomes”

kqr 10mo ago
A forecaster can be calibrated but almost only assign probabilities in the 40--60 % range. This is not as ueful as one assiging calibrated probabilities in the full range. We try to measure the increased usefulness of the latter with proper scoring rules.
kurtis_reed 10mo ago
Suppose there are 1000 events and 500 will have outcome A and 500 will have outcome B. If you predict a 50% chance of A for every event you'll be perfectly calibrated. On the other hand, if you predict a 90% chance of a certain outcome and you're right for 800 events, you're not perfectly calibrated but you have a lower Brier score (lower is better).