This is a thoughtful response, put in words I've been missing for a while, to some reasons why the EA crowd seem to be hyperoptimizing on the wrong cost functio

This is a thoughtful response, put in words I've been missing for a while, to some reasons why the EA crowd seem to be hyperoptimizing on the wrong cost functions (not to mention the ethical dangers that come about when insisting all questions of ethics can be quantified).