4 ms·

I run through a lot of these concepts, specifically RLHF, in my latest coding stream where I finetune LLama 2 if anyone's interested in getting a LLM deep dive

• bilsbie 3y ago
  When you’re doing RLHF are you actually modifying the weights of llama itself? Or is something on top?
  • jawerty 3y ago
    So I'm not doing RLHF that's how LLama is pre-trained. It's in the loss/optimization phase in their training I believe. For the finetuning i'm using LoRA to freeze most of the layers for parameter optimization. Using PEFT from huggingface
    • hallqv 3y ago
      RLHF is not part of LLaMa pretraining, or pretraning of any other models for that matter. RLHF comes after pretraining. https://twitter.com/Jeande_d/status/1661833563069620247/photo/1 https://twitter.com/Jeande_d/status/1661833563069620247/phot...
      • jsmith45 3y ago
        Seems like a classic case of a term of art overlapping with normal English terminology. Knowing that you will be doing further training on a provided model (even "just" extensive fine-tuning), one would want to distinguish the training done before you get your hands on it, from the training you do. An obvious word for that previous training is pre-training, which unfortunately conflicts with a term of art.
      • jawerty 3y ago
        I see, that’s my misunderstanding I was grouping all training as pretraining
        • wilhelm____ 3y ago
          pre-training is developing the language model's base understanding of conditional word probabilities. SFT and RLHF is attempting to further guide the model in terms of steerability + alignment of output. In fact, the InstructGPT authors were worried about losing the pre-trained model's underlying probability distribution, so they try a version where it penalizes the model deviating too significantly from the original distribution (using KL). I don't remember them seeing a significant difference in performance.
  • RC_ITR 3y ago
    RLHF does change the parameters. The way to think about it is that backpropagation changes the parameters of a model so they get closer to some sort of desired output. In pre-training and SFT, the parameters are changed so the model does a better job of replicating the next word in the training data, given the words it has already seen. In RLHF, the parameters are changed so the model does a better job of outputting the response that aligns to the human's preference (see: the feedback screen in the linked article).
    • bilsbie 3y ago
      Thanks. That helps. So how can you update weights without doing back-propagation? Or is it still back propagation but with a different metric?
      • bradfox2 3y ago
        It's still loss being backproped, but the loss is calculated over a different criteria
        • bilsbie 3y ago
          Ok that makes a lot of sense. Why do they call it reinforcement learning then? Is it not traditional RE such as Q learning?
          • dgant 3y ago
            The distinction making it RL is that the model is training on data produced by the model itself. The benefit of RL in general is that you're training on states the agent is likely to find itself in, and the cost is needing an agent which explores salient states. Which is why we keep seeing RL as a finishing step after imitation (eg AlphaStar first learning StarCraft from replays)
          • bradfox2 3y ago
            LLM output is scored by another model that produces a reward for the entire sequence emitted by the LLM. The reward model is trained on human preferences or some other metric usually. It's RL because we train on the reward and not some language modeling objective. The LLM is trained to increase this reward score (or minimize the inverse), which is what makes it RL.
• deleted 3y ago
  [deleted]