A nice formulation of the first part of yout post is Goodhart's Law: If you use a correlated metric to measure something qualitative, people will optimize towar

A nice formulation of the first part of yout post is Goodhart's Law: If you use a correlated metric to measure something qualitative, people will optimize toward the metric, and it'll stop correlating. Some reading: http://en.wikipedia.org/wiki/Goodhart's_law http://en.wikipedia.org/wiki/Goodhart's_law http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/ http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/