"And we build a good forecast or scoring procedure that for 2% of the population returns a score of 0.3 and for the remaining 98% of the population returns a sc

"And we build a good forecast or scoring procedure that for 2% of the population returns a score of 0.3 and for the remaining 98% of the population returns a score near 0.01. Further suppose our scoring algorithm is well calibrated and excellent: the 2% of the population that it returns a score of 0.3 and above on actually tends to default at a rate of 30%." In this case your classifier has two outputs. Let's label them safe and default prone respectively. Now we can transform question for each account into: "Is this account default prone?". In the example provided answers are obviously yes or no. If your classifier has two options, there must exist a question that would yield a yes/no answer. Other alternative is many option classifiers (neural-networks, decision trees, etc...). Part of confusion might stem from the fact that your example is simplified. From problem statement and description it sounds you guys built a scoring algorithm, that for each account outputs probability that account will default this year. In that case 0.01 and 0.3 would directly translate into 1% and 30% chance of default respectively. Obviously algorithm would not just output two options, it would output a range from 0 to 1. This is not yet classifier, however it is trivially easy to build a classifier with desired number of buckets on top of this. For example with two buckets: ask a question "does this account has at least 30% of chance to default this year?", with scoring algorithm solving that is as simple as filtering accounts with score in excess of 0.3.