There have been quite a few papers on this probem. If you don't get your simulator AND your cost function right ... fun things happen. Things you probably didn'

There have been quite a few papers on this probem. If you don't get your simulator AND your cost function right ... fun things happen. Things you probably didn't intend. Last year someone made a compilation paper about a series of fun techniques algorithms have used to satisfy the demands of their creators ... while not solving the problem. It was featured in popular mechanics. https://www.popularmechanics.com/technology/robots/a19445627/the-hilarious-and-terrifying-ways-algorithms-have-outsmarted-their-creators/ https://www.popularmechanics.com/technology/robots/a19445627... The way to solve it is something engineers hate for some reason. You explicitly design (and simulate) VERY bad hardware. What's bad hardware ? A camera that has a noise floor of 30% it's measurements. Yes, even in low light conditions (also: noise floor must vary a lot between runs of the algorithm). An actuator that goes the right way 90% of the time, and the result of a particular voltage on the motor varies by 20-30%. And in 10% of cases, it's just entirely stuck, without giving any feedback abou tthat. https://arxiv.org/pdf/1804.10332.pdf https://arxiv.org/pdf/1804.10332.pdf Or for the more visually inclined: https://www.youtube.com/watch?v=lUZUr7jxoqM https://www.youtube.com/watch?v=lUZUr7jxoqM The lesson is that what engineers always do, open-loop designs (I send voltage, motor moves), can be incredibly outperformed in control, accuracy, resiliency, and more by much worse hardware closed-loop designs (I send voltage, motor moves, I check how it moves, I change voltage). And yet somehow people seem to have incredible issues trusting such systems. For instance, autopilots are mostly open-loop designs. That's like a pilot flying a plane with his eyes glued shut, and no sense of balance (ie. he HAS to trust one instrument, and have no way to verify that, say, the plane actually goes up when they pull the stick. So if it's keeling over backwards or something, they'd just keep pulling the stick right up to the point they hit the decor). They implicitly trust the plane does what the autopilot orders it to do. If for some reason it doesn't ... it's not going to end well. The issue is that closed-loop designs are much harder to write. The solution to that of course is to not write them, learn them. An autopilot that flies a plane, and when the plane breaks, rapidly learns how to fly a broken plane rather than just trusting it's (now inaccurate) model and killing everyone doing that.