I use this as an example of why we need to be careful to think about what we're optimising for in ML. They wanted to reduce the number of rats, but incentivised

I use this as an example of why we need to be careful to think about what we're optimising for in ML. They wanted to reduce the number of rats, but incentivised increasing the number of rat tails. Do your performance measures correspond directly to things you care about? Are there ways of optimising the number while hurting something that matters to you? A good example to me is Google marking a picture of a black couple as "gorillas" - to a training algorithm that can easily be counted as a single wrong answer equivalent to any other when it is actually very significant. Edit - this is the same for optimising any KPI, but ML is a case where your optimiser has no wider context and so there's more surprising cases. Worth thinking about overall though, what are you rewarding?