I'm also reminded of https://blog.openai.com/faulty-reward-functions/ https://blog.openai.com/faulty-reward-functions/ (which I saw on HN a while ago), where an

I'm also reminded of https://blog.openai.com/faulty-reward-functions/ https://blog.openai.com/faulty-reward-functions/ (which I saw on HN a while ago), where an AI unexpected learned to > turn in a large circle and repeatedly knock over three targets, timing its movement so as to always knock over the targets just as they repopulate. Despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track, our agent manages to achieve a higher score using this strategy than is possible by completing the course in the normal way.