The thing that bothers me here is less that the model cheated and more that it found a way to improve the score that the people running the test didn't intend.

The thing that bothers me here is less that the model cheated and more that it found a way to improve the score that the people running the test didn't intend. That's a pretty nasty failure once you start giving these things more control