3 ms·

How about alignment/ability to answer prompt queries and chain of thought reasoning capabilities? Without the fine tuning RLHF phase to make it like instructgp

How about alignment/ability to answer prompt queries and chain of thought reasoning capabilities? Without the fine tuning RLHF phase to make it like instructgpt I'm assuming it won't be as good as ChatGPT, is that right? How hard would it be to fine tune the 65B model on commodity hardware? Found answer here: > Out-of-scope use cases LLaMA is a base, or foundational, model. As such, it should not be used on downstream applications without further risk evaluation and mitigation. In particular, our model has not been trained with human feedback, and can thus generate toxic or offensive content, incorrect information or generally unhelpful answers. https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md https://github.com/facebookresearch/llama/blob/main/MODEL_CA...

zozbot234 4y ago
Yes but fine tuning for RL is not expected to be hard. You're essentially limited by how much human feedback is available, so it's very different from training the foundational model on random bulk data.
- simonw 4y ago
  At this point I fully expect that someone will release a usable RLHF-fine-tuned language model that can run on consumer hardware, based on the methodology used for LLaMA (and other similar papers e.g. https://github.com/FMInference/FlexGen https://github.com/FMInference/FlexGen ), at some point in the next 6-24 months.
  - monkeydust 4y ago
    What are the functional steps that need to happen to get to that point ?
  - ntonozzi 4y ago
    How are they different from models that already exist like FLAN-T5?
- stevenhuang 4y ago
  Of course, but even still the level of scale, clean data, and human supervision needed may be significant. It's reported OpenAI used an army of humans to generate question answer prompts and rate the model output. They kept the details closely guarded and only hinted at how they did RLHF and transitioned the architecture to self supervised learning.
  - pps 4y ago
    I'm stupid in this area and have only a very broad understanding of "AI". But isn't this already happening in this project? https://github.com/LAION-AI/Open-Assistant https://github.com/LAION-AI/Open-Assistant

zozbot234 4y ago

Yes but fine tuning for RL is not expected to be hard. You're essentially limited by how much human feedback is available, so it's very different from training the foundational model on random bulk data.

simonw 4y ago
At this point I fully expect that someone will release a usable RLHF-fine-tuned language model that can run on consumer hardware, based on the methodology used for LLaMA (and other similar papers e.g. https://github.com/FMInference/FlexGen https://github.com/FMInference/FlexGen ), at some point in the next 6-24 months.
- monkeydust 4y ago
  What are the functional steps that need to happen to get to that point ?
- ntonozzi 4y ago
  How are they different from models that already exist like FLAN-T5?
stevenhuang 4y ago
Of course, but even still the level of scale, clean data, and human supervision needed may be significant. It's reported OpenAI used an army of humans to generate question answer prompts and rate the model output. They kept the details closely guarded and only hinted at how they did RLHF and transitioned the architecture to self supervised learning.
- pps 4y ago
  I'm stupid in this area and have only a very broad understanding of "AI". But isn't this already happening in this project? https://github.com/LAION-AI/Open-Assistant https://github.com/LAION-AI/Open-Assistant