Your comment made me think of the Vietnam War. Many of the military leaders in the war came of age during World War II and Korea, where the most obvious metric

Your comment made me think of the Vietnam War. Many of the military leaders in the war came of age during World War II and Korea, where the most obvious metric was land: if Nazis or communists were there, then bad. If allies, then good. But that completely broke down in Vietnam, when the placid villagers in South Vietnam by day became the ruthless Viet Cong by night. American generals had no idea how to approximate their success like they had in previous wars, so they settled on kill scores, and as long as more Vietnamese died than Americans, the DoD could go on television and claim that America was “winning”, even though the incentives for kill scores resulted in a lot of bystander villages being torched and civilians being killed, which only further put public opinion against the Americans. I wonder if a sufficiently advanced AI system could correct its use of (or abandon) a bad metric even when it is incentivized to use it. The problem I cited above was effectively so hard for the people at the time that it was effectively intractable. I’m not arguing that an AI would have had to solve the problem (setting aside the likelihood that even had an AI come up with a better solution other inertial forces like politics and sunk cost would have prevented the solution from being implemented), but any system that would claim to be a “strong” AI, it would at least have to be up to the task of trying.