This is also known as the Lucas critique, dating from 1976 https://en.wikipedia.org/wiki/Lucas_critique https://en.wikipedia.org/wiki/Lucas_critique and also

This is also known as the Lucas critique, dating from 1976 https://en.wikipedia.org/wiki/Lucas_critique https://en.wikipedia.org/wiki/Lucas_critique and also https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law Edit: I think this is not the first time the good people from lesswrong dug up some well known idea and gave it a new name. Good thing, too, giving this important concept more attention. Too often we forget how many people have dealt with the problem of modeling complex systems in the past. And while we can not read everything, it's often a good idea to have at least a glance at where they failed! Too often I read/review some new "revolutionary" paper based on the idea that hey, we can model this process (involving people) like XYZ from physics, where this stuff works great! Surely, this is better than the plebian approaches in the literature! And then, to the shock of all involved, it doesn't work great.... Also: https://xkcd.com/793/ https://xkcd.com/793/