The hole in that argument is that we can annotate token sequences with a "good/bad" score, train a model on annotated token sequences, then add "high good score

The hole in that argument is that we can annotate token sequences with a "good/bad" score, train a model on annotated token sequences, then add "high good score" as a condition during inference.