I suspect a lot of low-quality content still makes it into the training process and the RLHF layer is specifically designed to improve the quality of the output

I suspect a lot of low-quality content still makes it into the training process and the RLHF layer is specifically designed to improve the quality of the output despite noisy input. i.e. without RLHF, these models would for sure be vulgar, explicit, etc.