A "sufficiently advanced intelligence" would presumably understand the inaccuracy of reducing billion's of peoples' worth of preferences down to a single Englis

A "sufficiently advanced intelligence" would presumably understand the inaccuracy of reducing billion's of peoples' worth of preferences down to a single English sentence, the effects of incomplete information, the effects of uncertainty on decision making, the differences in time preferences and in general the effects of adding "time" into dynamic systems rather than static models, human's inabilities to explain the real reasons why they do things to themselves or anybody else, and so on and so on. To bring it to a more human scale, an alcoholic can both sincerely desire to stop drinking alcohol and also drink themselves into a stupor every night, and a "sufficiently advanced intelligence" won't sit there spinning trying to figure out exactly which statement "The human loves alcohol" and "The human hates alcohol" is true to the total exclusion of the other. (And I mean this as merely one example dimension, of which the "global warming" problem contains hundreds or thousands of such issues at a minimum.)