3 ms·

In my opinion there is a difference. You can build a classifier that says "does the person have a 30% chance of defaulting this year." And you can in aggregat

by jmount 13y ago

In my opinion there is a difference. You can build a classifier that says "does the person have a 30% chance of defaulting this year." And you can in aggregate estimate the accuracy of such a classifier. However, for an individual example you don't know if such a classifier is right or wrong which can confuse partners. You say somebody has a 30% chance of default and they don't default (were you right?), or they do default (again, were you right on this single instance?). Obviously you can work something out- but you have moved to building a classifier where you no longer have an easily observable ground truth on individuals (only on aggregates). You then want to move to forecasting and rating of scores or quality of sorting (which is what we encourage in the writeup).

TrainedMonkey 13y ago
"And we build a good forecast or scoring procedure that for 2% of the population returns a score of 0.3 and for the remaining 98% of the population returns a score near 0.01. Further suppose our scoring algorithm is well calibrated and excellent: the 2% of the population that it returns a score of 0.3 and above on actually tends to default at a rate of 30%." In this case your classifier has two outputs. Let's label them safe and default prone respectively. Now we can transform question for each account into: "Is this account default prone?". In the example provided answers are obviously yes or no. If your classifier has two options, there must exist a question that would yield a yes/no answer. Other alternative is many option classifiers (neural-networks, decision trees, etc...). Part of confusion might stem from the fact that your example is simplified. From problem statement and description it sounds you guys built a scoring algorithm, that for each account outputs probability that account will default this year. In that case 0.01 and 0.3 would directly translate into 1% and 30% chance of default respectively. Obviously algorithm would not just output two options, it would output a range from 0 to 1. This is not yet classifier, however it is trivially easy to build a classifier with desired number of buckets on top of this. For example with two buckets: ask a question "does this account has at least 30% of chance to default this year?", with scoring algorithm solving that is as simple as filtering accounts with score in excess of 0.3.

by jmount 13y ago

TrainedMonkey 13y ago

"And we build a good forecast or scoring procedure that for 2% of the population returns a score of 0.3 and for the remaining 98% of the population returns a score near 0.01. Further suppose our scoring algorithm is well calibrated and excellent: the 2% of the population that it returns a score of 0.3 and above on actually tends to default at a rate of 30%." In this case your classifier has two outputs. Let's label them safe and default prone respectively. Now we can transform question for each account into: "Is this account default prone?". In the example provided answers are obviously yes or no. If your classifier has two options, there must exist a question that would yield a yes/no answer. Other alternative is many option classifiers (neural-networks, decision trees, etc...). Part of confusion might stem from the fact that your example is simplified. From problem statement and description it sounds you guys built a scoring algorithm, that for each account outputs probability that account will default this year. In that case 0.01 and 0.3 would directly translate into 1% and 30% chance of default respectively. Obviously algorithm would not just output two options, it would output a range from 0 to 1. This is not yet classifier, however it is trivially easy to build a classifier with desired number of buckets on top of this. For example with two buckets: ask a question "does this account has at least 30% of chance to default this year?", with scoring algorithm solving that is as simple as filtering accounts with score in excess of 0.3.