5 ms·

This blog post presents good techniques, but misses the point. The most important thing to do when designing a machine learning algorithm is to choose your los

by bravura 14y ago

This blog post presents good techniques, but misses the point. The most important thing to do when designing a machine learning algorithm is to choose your loss function. The loss function quantifies how much your model sucks. Ideally, your loss function should be the true loss, in real life. In a business, your true loss function is: -profit over the lifetime of the business. A lot of time it's difficult to measure the true loss, so your loss function should be a "surrogate" (approximation) of the true loss. The author is poking around with techniques, without considering what he actually wants to optimize. Brandon Ballinger said it well: http://www.quora.com/What-are-the-keys-to-operationalizing-a-machine-learning-ranking-system-from-an-organization-engineering-management-point-of-view/answer/Brandon-Ballinger http://www.quora.com/What-are-the-keys-to-operationalizing-a... "Make your success metric user happiness. Traditional accuracy measures like precision, square error, ROC, etc. don't capture what you really care about--how users react to your model. For example, if you're running an ad system, your metrics should be revenue per pageview and click through rate. It's completely possible to have a model with a reduced error rate which lowers revenue, due to Simpson's paradox and a host of other reasons." The author of this post hasn't really considered these questions, and gets all hand-wavey about precision and recall vs. confusion matrices. How do you know that any of these are what you really care about? So: What's the value of a false positive? What's the value of a false negative? Here's where it gets interesting. For airport screening, one person might say that a false negative (terrorist slips through and kills 100 people) is 100M times worse than a false positive (I strip-search an innocent person). So this person has made a value judgment that the death of a random innocent person is 1 million times worse than humiliating an innocent person. The TSA, on the other hand, decides that a false negative is 100B times worse than a false positive, so they assign 1000 times as much value to catching terrorists vs. humiliating people as the other person. Who's right? It's a value judgment, and machine learning can't answer that. What I'm getting at here is that the choice of a loss function is ultimately an aesthetic decision. Once we can optimize loss perfectly, what loss will we optimize?

numlocked 14y ago
It's a great point, and well articulated. In the case of the example in the post (fraud), the credit card industry has actually termed the false positive rate the "insult rate" (because the customer feels insulted in the store that their charge has been declined), which certainly gives you an idea of how they feel about false positives.
rm999 14y ago
I used to build fraud models where we had insane amounts of negative data and precious positive data. Our performance metric was well-reasoned; as you mention, we cared about the happiness of our customer, and baked all sorts of useful information into our evaluation metrics. The problem is there is not always a cheap way to encode a complex evaluation metric into a loss function for many machine learning algorithms. In fact most of the really popular algorithms are popular because they work well mathematically - for example, generalized linear models are convex on a simple quadratic loss function, and neural networks have gradients when their loss functions are differentiable. Also, I found optimizing for simple but sensible loss functions often gave pretty good performance. From a practical perspective, most of the gains of understanding the performance metrics we cared about came in the data preparation, evaluation and calibration stages of modeling, not model selection or training.
tel 14y ago
I love and reiterate this point consistently. ML is ultimately quite often very dumb. It's simply a good tool that must be wielded with understanding, intuition, and ingenuity if you're going to make it valuable.

by bravura 14y ago

numlocked 14y ago

It's a great point, and well articulated. In the case of the example in the post (fraud), the credit card industry has actually termed the false positive rate the "insult rate" (because the customer feels insulted in the store that their charge has been declined), which certainly gives you an idea of how they feel about false positives.

rm999 14y ago

I used to build fraud models where we had insane amounts of negative data and precious positive data. Our performance metric was well-reasoned; as you mention, we cared about the happiness of our customer, and baked all sorts of useful information into our evaluation metrics. The problem is there is not always a cheap way to encode a complex evaluation metric into a loss function for many machine learning algorithms. In fact most of the really popular algorithms are popular because they work well mathematically - for example, generalized linear models are convex on a simple quadratic loss function, and neural networks have gradients when their loss functions are differentiable. Also, I found optimizing for simple but sensible loss functions often gave pretty good performance. From a practical perspective, most of the gains of understanding the performance metrics we cared about came in the data preparation, evaluation and calibration stages of modeling, not model selection or training.

tel 14y ago

I love and reiterate this point consistently. ML is ultimately quite often very dumb. It's simply a good tool that must be wielded with understanding, intuition, and ingenuity if you're going to make it valuable.