Good thing I am not your grad student... filtering out the worst humanity has to offer is a terrible job. But anyways, even filtering out bad content is not go

Good thing I am not your grad student... filtering out the worst humanity has to offer is a terrible job. But anyways, even filtering out bad content is not going to guarantee the LLM won't say terrible things. LLMs can do negation, and can easily turn sources that are about preventing harm into doing harm. And there is also fictional work, we are fine with terrible things in fiction because we understand it is fiction and furthermore, it is the bad guy doing it. If a LLM acts like a fictional bad guy, it will say terrible things because it is what bad guys do. They do use people to filter the output though, it is called RLHF, all of the major publicly available LLMs do it.