3 ms·

Not sure which sentence you're referring to. And I'm not saying there's no way to tell which probability assignments are better. I'm just saying that the mere

by hsitz 10y ago

Not sure which sentence you're referring to. And I'm not saying there's no way to tell which probability assignments are better. I'm just saying that the mere fact that an event assigned a lower probability happens (e.g., if Romney would have beaten Obama in Florida) does not mean the assignments were wrong. This is consistent with Silver's own explanations of what he's doing, and (I'm pretty sure ) he is careful to say he doesn't make "predictions". (Actually, not sure, it may be merely that he often clarifies that a different result than predicted does not make the probability assignment that prompted the prediction "wrong". The prediction and the probability analysis it's based on are different things.) I'm not saying that the Brier Score can't differentiate between better and worse probability assignments, but I am saying it's confusing the issue if it calls probability assignments "right" or "wrong". They may be "better" or "worse" than each other, but not "right" or "wrong". This may be using language in more technical manner, but it seems to me (and to Silver) to be an important distinction.

basch 10y ago
"Also note that Wang & Ferguson got a better Brier score than Silver despite getting Florida (barely) wrong while Silver got Florida right. The Atlantic Wire gave Wang & Ferguson only a “Silver Star” for this reason, but our more detailed analysis shows that Wang & Ferguson probably should have gotten a “Gold Star.”" I think being concerned about this authors usage of the words "right and wrong" misses the point. The media will say "you were wrong" if you were on the wrong side of a 51/49% prediction, even if you shouldn't be penalized for it. Wang & Ferguson got dinged by The Atlantic Wire for being wrong, when they shouldnt have. The last quote in my post, supports your point. The person who is least wrong is the person with the lowest aggregate Brier score. The author was using right and wrong, because the articles they were referencing used those terms. The quote was written with the colloquial language of what it addressed.
- hsitz 10y ago
  We're not too far off. Main thng in original quote that bothered me was "So if Romney had won Florida, Silver would have been wrong, but ONLY A LITTLE WRONG." Predictions of victory or loss are either right or wrong. There are not degrees of rightness or wrongness with the prediction; it's either right or wrong. The probability analysis that the prediction is based on does admit of degrees; one probability assignment may be better or worse than another. But saying a prediction is "only a little wrong" implicitly refers to the probability assignment it was based on; it confuses the concepts of "prediction" and "probability". The probability assignment was not necessarily "wrong" to assign a higher probability to the event that didn't occur. It may have been a bad probability assignment, but discerning that would involve considering factors other than the outcome of the event.

by hsitz 10y ago

basch 10y ago

"Also note that Wang & Ferguson got a better Brier score than Silver despite getting Florida (barely) wrong while Silver got Florida right. The Atlantic Wire gave Wang & Ferguson only a “Silver Star” for this reason, but our more detailed analysis shows that Wang & Ferguson probably should have gotten a “Gold Star.”" I think being concerned about this authors usage of the words "right and wrong" misses the point. The media will say "you were wrong" if you were on the wrong side of a 51/49% prediction, even if you shouldn't be penalized for it. Wang & Ferguson got dinged by The Atlantic Wire for being wrong, when they shouldnt have. The last quote in my post, supports your point. The person who is least wrong is the person with the lowest aggregate Brier score. The author was using right and wrong, because the articles they were referencing used those terms. The quote was written with the colloquial language of what it addressed.

hsitz 10y ago
We're not too far off. Main thng in original quote that bothered me was "So if Romney had won Florida, Silver would have been wrong, but ONLY A LITTLE WRONG." Predictions of victory or loss are either right or wrong. There are not degrees of rightness or wrongness with the prediction; it's either right or wrong. The probability analysis that the prediction is based on does admit of degrees; one probability assignment may be better or worse than another. But saying a prediction is "only a little wrong" implicitly refers to the probability assignment it was based on; it confuses the concepts of "prediction" and "probability". The probability assignment was not necessarily "wrong" to assign a higher probability to the event that didn't occur. It may have been a bad probability assignment, but discerning that would involve considering factors other than the outcome of the event.