RLHF does nothing to "filter" or guarantee that the output will be good. It changes the weights of the NN to push the probabilities of naughty tokens to be low.

RLHF does nothing to "filter" or guarantee that the output will be good. It changes the weights of the NN to push the probabilities of naughty tokens to be low. Actual "filtering" would be filtering assisted decoding where you remove tokens from the vocabulary you don't want at a particular time step, which is described in this paper: https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be...