3 ms·

RLHF is a very important part of the training process. Without it, the LLMs would definitely behave much more similar to a glorified autocomplete than something

RLHF is a very important part of the training process. Without it, the LLMs would definitely behave much more similar to a glorified autocomplete than something resembling "intelligence". So the answer is that if you just feed it a bunch of data, it might be good at spitting out some similarly looking data but ultimately has little use. If you manage to get some experts sitting with it and give it instructions and explain the data, i.e. relating data to concepts, then you may get a model that can generate interesting things based on natural language, because it was taught how to interpret the data with natural language. That is the next crop of AI I believe. For example, in biotech, people are feeding it protein sequences and teach it what each protein are and how to relate certain concepts with the sequences. The result is an AI that can generate new proteins with certain characteristics and functions.

iroddis 3y ago
To save others from searching, RLHF [1] is Reinforcement Learning from Human Feedback [1] https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback?wprov=sfti1 https://en.wikipedia.org/wiki/Reinforcement_learning_from_hu...
- belter 3y ago
  And to put it in context...RLHF is the special sauce behind ChatGPT
gota 3y ago
Isn't RLHF for building _chatbots_ over LLMs (and not part of LLMs themselves)?
- nailer 3y ago
  RLHF is where the model reaches out to people when it's not confident of the output. It's good because it makes the model 'push into' classes the model is weak on.
  - T-A 3y ago
    That doesn't look like any definition of RLHF I'm familiar with, which amounts to first training a reward model on human feedback and then using that reward model to train the target model [1]. No reaching out to humans involved. [1] https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback https://en.wikipedia.org/wiki/Reinforcement_learning_from_hu...
    - nailer 3y ago
      You're right, I was confusing RLHF with https://en.wikipedia.org/wiki/Human-in-the-loop https://en.wikipedia.org/wiki/Human-in-the-loop

iroddis 3y ago

To save others from searching, RLHF [1] is Reinforcement Learning from Human Feedback [1] https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback?wprov=sfti1 https://en.wikipedia.org/wiki/Reinforcement_learning_from_hu...

belter 3y ago
And to put it in context...RLHF is the special sauce behind ChatGPT

gota 3y ago

Isn't RLHF for building _chatbots_ over LLMs (and not part of LLMs themselves)?

nailer 3y ago
RLHF is where the model reaches out to people when it's not confident of the output. It's good because it makes the model 'push into' classes the model is weak on.
- T-A 3y ago
  That doesn't look like any definition of RLHF I'm familiar with, which amounts to first training a reward model on human feedback and then using that reward model to train the target model [1]. No reaching out to humans involved. [1] https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback https://en.wikipedia.org/wiki/Reinforcement_learning_from_hu...
  - nailer 3y ago
    You're right, I was confusing RLHF with https://en.wikipedia.org/wiki/Human-in-the-loop https://en.wikipedia.org/wiki/Human-in-the-loop