It's far too easy to destroy any type of RLHF done to try to prevent bad behavior from an LLM, and just "fixing the dataset" doesn't really help. For example,

It's far too easy to destroy any type of RLHF done to try to prevent bad behavior from an LLM, and just "fixing the dataset" doesn't really help. For example, if you want a LLM to generate things that look like social security numbers, you may try to prompt it asking for social security numbers. It will of course give you "I'm sorry hal I can't do that..." Then start using a technique like token filtering/filter assisted decoding, to make it where the LLM can only generate hyphens and numbers, and suddenly it does what you ask despite RLHF I explored this a tiny bit in the later sections of my paper studying what happens when you restrict an LLMs vocabulary: https://aclanthology.org/2022.cai-1.pdf#page=17 https://aclanthology.org/2022.cai-1.pdf#page=17 You can even play with this with open source models using CTGS: https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio https://github.com/Hellisotherpeople/Constrained-Text-Genera... Now we have even more sophisticated stuff like Guidance from microsoft, LMQL, and other template languages which also filter vocabularies to force behavior we want. The reality is that LLMs are basically impossible to remove the risk of bad behavior in.