I use AUC often, and always look at the confusion matrix. Metrics based off of it (recall, precision) are both useful to use as well. In the end the probabilit

I use AUC often, and always look at the confusion matrix. Metrics based off of it (recall, precision) are both useful to use as well. In the end the probability cutoffs you choose are more often than not based on business context and costs associated with various actions. For example - if we're trying to predict whether a customer is going to leave or not, I'd choose a different threshold based on whether we're mass e-mailing people, or if we're having customer service reps call people, since the underlying costs for each action are so different. In the first case it's fine to cast a wide net, and e-mail lots of people who we're less certain about, in the second we'd optimize to target people we're most sure are going to leave, since it costs so much more to reach out to them.