They do both. Broadly, Large Language Models (LLMs) are initially trained on a massive amount of unfiltered text. Removing unpleasant content from the initial

They do both. Broadly, Large Language Models (LLMs) are initially trained on a massive amount of unfiltered text. Removing unpleasant content from the initial training corpus is intractible due to its sheer size. These models can produce pretty unpleasant output, because of the unpleasant messages present in the training data. Accordingly, LLM models are then trained further using Reinforcement Learning from Human Feedback (RLHF). This training phase uses a much smaller corpus of hand picked examples demonstrating desired output. This higher quality corpus is too small to train a high quality LLM from scratch, so it can only be used for fine tuning. Effectively, it "bakes in" to the model the desired form of the output, but it's not perfect, because most of the training occured before this phase. Therefore instruction inserted at the beggining of every prompt or session are used to further increases the chance of the model producing desireable output.