3 ms·

Accuracy is almost never the right statistic. Here, profitability is. As soon as you realise the target statistic applies a human-goal-value-weighting to the pr

Accuracy is almost never the right statistic. Here, profitability is. As soon as you realise the target statistic applies a human-goal-value-weighting to the prediction outcomes, it should be easier to see why ML systems are often unsuited to the task they're set. Here, those 5% of cases are likely business-ending or business-making: precisely because they arent naively routine.

kqr 5y ago
I think you can be even bolder: accuracy never matters. It's always about the consequences, not probabilities. Sometimes the two are the same, but most of the time they are not. Also worth noting that the arithmetic expectation is only a good way to measure the profitability if we are talking about small amounts compared to your total wealth. For any other case, you should use the geometric expectation of total wealth to evaluate options. (This is equivalent to maximising log wealth, the Kelly criterion.)
- yobbo 5y ago
  Outcome * probability = expectancy, which is the quantity usually maximised or minimized in AI. The problem in this case was defined as reproducing the human scores. Mainly, it's a demonstration of the information content in the data and the scores. To me, it demonstrates how simplistic the human scores are. With information about historic outcomes, we could have compared the effectiveness of the "credit score method" with some other AI algorithm that was optimizing total value.
  - kqr 5y ago
    One little expansion: what you described first is the arithmetic expectation, which is a good approximation when the numbers involved are small compared to total wealth. When you start taking larger bets, you want the geometric expectation of total wealth, i.e. (current wealth + outcome)^probability. (This is the Kelly criterion for judging significant opportunities.)

3 ms·

Accuracy is almost never the right statistic. Here, profitability is. As soon as you realise the target statistic applies a human-goal-value-weighting to the pr

by mjburgess 5y ago

kqr 5y ago
I think you can be even bolder: accuracy never matters. It's always about the consequences, not probabilities. Sometimes the two are the same, but most of the time they are not. Also worth noting that the arithmetic expectation is only a good way to measure the profitability if we are talking about small amounts compared to your total wealth. For any other case, you should use the geometric expectation of total wealth to evaluate options. (This is equivalent to maximising log wealth, the Kelly criterion.)
- yobbo 5y ago
  Outcome * probability = expectancy, which is the quantity usually maximised or minimized in AI. The problem in this case was defined as reproducing the human scores. Mainly, it's a demonstration of the information content in the data and the scores. To me, it demonstrates how simplistic the human scores are. With information about historic outcomes, we could have compared the effectiveness of the "credit score method" with some other AI algorithm that was optimizing total value.
  - kqr 5y ago
    One little expansion: what you described first is the arithmetic expectation, which is a good approximation when the numbers involved are small compared to total wealth. When you start taking larger bets, you want the geometric expectation of total wealth, i.e. (current wealth + outcome)^probability. (This is the Kelly criterion for judging significant opportunities.)