3 ms·

My pet peeve: predictions not associated with probabilities or odds. It's impossible to verify or score accuracy without that. Nobody predicts anything at 100

by kqr 4y ago

My pet peeve: predictions not associated with probabilities or odds. It's impossible to verify or score accuracy without that. Nobody predicts anything at 100 % certainty (otherwise it wouldn't be a prediction, it would just be stating inevitable facts), so there's always an implicit level of confidence. It's just that when people don't say what it is, their predictions become meaningless to anyone but themselves. A prediction without a probability is also not open to critique, because the way we settle differences in beliefs about uncertainties is through betting against each other. But a bet requires odds to price it with.

stonewareslord 4y ago
What do you mean? Adding odds defeats the purpose of the prediction since you can't confirm it to be true or not. If I predict with 99.99% odds something will happen and it doesn't, my prediction could still have been correct as that fits in the remaining .01%. Whereas if I predict something will or won't happen after some time, you can confirm if it did or didn't happen after that time.
- kqr 4y ago
  All predictions come with uncertainty. That's what makes them predictions. Even if someone says "there will be an ice cream shortage in France in the summer of 2025" they don't mean they are 100 % certain of it. They mean they find it somewhat likely -- but exactly how likely? That's what they have to tell us, because we don't know. For an individual prediction, you can never tell whether it's true or not, because individual predictions always come with some level of uncertainty. What you can do, if you know at what confidence level it was made, is - Score individual predictions. One made at 99.9 % certainty that doesn't come true gets a heavy negative score, while if it would have been made at 60 % it gets penalised much less. - Verify the accuracy of a collection of predictions. If I predict five things at confidence levels of 0.8 0.7 0.9 0.5 0.5 then I should get roughly 3--4 of them right. If I don't, my predictions aren't very accurate.
melony 4y ago
We can go all bayesian about predictions but at the end of the day dropbox didn't build itself. At some point we need to act on it.
- kqr 4y ago
  Right and it's specifically for action that we need to know the odds. To pick one set of actions in advance of an uncertain event we need to know three things: - The cost in case the event turns out bad, - The upside in case the event turns out good, and - The probability that the event turns out good. We can summarise the two first ones as "the payoff function of the event" and simplify the situation into requiring knowledge about the payoff and the probability. But without the probability we cannot take informed action. (Unless the payoff is structured such that the probability doesn't matter -- but then it's also not much of a difficult decision.)
c3534l 4y ago
Predictions and probabilites seem unrelated, except in mathematical models which attempt to make "predictions" through probabalistic models. Perhaps you've spent so much time reading about machine learning that you've gotten the concepts confused. I would go so far as to say that a prediction that comes with a probability is not a true prediction, but an estimate masquarading as one.
- kqr 4y ago
  Quite the opposite! A prediction is an estimation of the most likely outcome in a binary event. I mean, I get that just saying the above seems to confirm your "too much machine learning" hypothesis, but I guarantee you that's not the case. At my own peril, I'm too old fashioned to have read much about machine learning. What I can say is that when making strategic and tactical decisions that affect a lot of people, I have found it incredible to know things like "I am 85 % sure we will not make the next release in time" (an actionable prediction) compared to "there's no way we'll make the next release in time" (when in fact there's still a 15 % chance we will, if we take proper action now.)
  - c3534l 4y ago
    It most definitely is not. Confidence and predictions are orthogonal concepts outside of stats and ML. A gambler makes predictions, but may have no confidence in winning. A scientific model may produce predictions with no associatel claims of probability or even validity. It is perfectly grammatically and semantically correct to say that someone made a prediction that they themselves do not believe to be the most likely outcome.
    - kqr 4y ago
      Any gambler that makes a prediction without internally associating it with a probability is one that's most likely losing money. You simply cannot consistently make positive-EV bets without considering the probabilities of your predictions. (Heck, EV is the linear combination of probability and payoff.) A scientific model almost always includes a very specific kind of probability: the p value. The best scientific models are also validated externally and have the results of that experiment reported -- as probabilities: false positive and false negative rates are common concepts to discuss that. I completely agree that one can predict something that one does not believe to be the most likely outcome. But in order to even say that, you have to be implying that there is at least a partial order from "least likely" to "most likely" and that order is commonly referred to as probability. (Edit: oh, I see where you got the "most likely" from. By convention, in a binary prediction, the prediction itself is stated in terms of the most likely outcome, with a confidence between 50 % and 100 %. It is of course possible to state the other side of the event with a confidence level < 50 % -- it's just not the way these things are usually presented.)

by kqr 4y ago

stonewareslord 4y ago

What do you mean? Adding odds defeats the purpose of the prediction since you can't confirm it to be true or not. If I predict with 99.99% odds something will happen and it doesn't, my prediction could still have been correct as that fits in the remaining .01%. Whereas if I predict something will or won't happen after some time, you can confirm if it did or didn't happen after that time.

kqr 4y ago
All predictions come with uncertainty. That's what makes them predictions. Even if someone says "there will be an ice cream shortage in France in the summer of 2025" they don't mean they are 100 % certain of it. They mean they find it somewhat likely -- but exactly how likely? That's what they have to tell us, because we don't know. For an individual prediction, you can never tell whether it's true or not, because individual predictions always come with some level of uncertainty. What you can do, if you know at what confidence level it was made, is - Score individual predictions. One made at 99.9 % certainty that doesn't come true gets a heavy negative score, while if it would have been made at 60 % it gets penalised much less. - Verify the accuracy of a collection of predictions. If I predict five things at confidence levels of 0.8 0.7 0.9 0.5 0.5 then I should get roughly 3--4 of them right. If I don't, my predictions aren't very accurate.

melony 4y ago

We can go all bayesian about predictions but at the end of the day dropbox didn't build itself. At some point we need to act on it.

kqr 4y ago
Right and it's specifically for action that we need to know the odds. To pick one set of actions in advance of an uncertain event we need to know three things: - The cost in case the event turns out bad, - The upside in case the event turns out good, and - The probability that the event turns out good. We can summarise the two first ones as "the payoff function of the event" and simplify the situation into requiring knowledge about the payoff and the probability. But without the probability we cannot take informed action. (Unless the payoff is structured such that the probability doesn't matter -- but then it's also not much of a difficult decision.)

c3534l 4y ago

Predictions and probabilites seem unrelated, except in mathematical models which attempt to make "predictions" through probabalistic models. Perhaps you've spent so much time reading about machine learning that you've gotten the concepts confused. I would go so far as to say that a prediction that comes with a probability is not a true prediction, but an estimate masquarading as one.

kqr 4y ago
Quite the opposite! A prediction is an estimation of the most likely outcome in a binary event. I mean, I get that just saying the above seems to confirm your "too much machine learning" hypothesis, but I guarantee you that's not the case. At my own peril, I'm too old fashioned to have read much about machine learning. What I can say is that when making strategic and tactical decisions that affect a lot of people, I have found it incredible to know things like "I am 85 % sure we will not make the next release in time" (an actionable prediction) compared to "there's no way we'll make the next release in time" (when in fact there's still a 15 % chance we will, if we take proper action now.)
- c3534l 4y ago
  It most definitely is not. Confidence and predictions are orthogonal concepts outside of stats and ML. A gambler makes predictions, but may have no confidence in winning. A scientific model may produce predictions with no associatel claims of probability or even validity. It is perfectly grammatically and semantically correct to say that someone made a prediction that they themselves do not believe to be the most likely outcome.
  - kqr 4y ago
    Any gambler that makes a prediction without internally associating it with a probability is one that's most likely losing money. You simply cannot consistently make positive-EV bets without considering the probabilities of your predictions. (Heck, EV is the linear combination of probability and payoff.) A scientific model almost always includes a very specific kind of probability: the p value. The best scientific models are also validated externally and have the results of that experiment reported -- as probabilities: false positive and false negative rates are common concepts to discuss that. I completely agree that one can predict something that one does not believe to be the most likely outcome. But in order to even say that, you have to be implying that there is at least a partial order from "least likely" to "most likely" and that order is commonly referred to as probability. (Edit: oh, I see where you got the "most likely" from. By convention, in a binary prediction, the prediction itself is stated in terms of the most likely outcome, with a confidence between 50 % and 100 %. It is of course possible to state the other side of the event with a confidence level < 50 % -- it's just not the way these things are usually presented.)