3 ms·

How do you know it’s pretty much just scale? “Open”AI has been pretty tight-lipped about the details of its training and merely “claims” it was scale. It hired

by EGreg 4y ago

How do you know it’s pretty much just scale? “Open”AI has been pretty tight-lipped about the details of its training and merely “claims” it was scale. It hired a ton of humans to train the model in little ways. If that’s the “scale” you’re talking about then it’s humans all the way down: https://www.forbes.com/sites/kenrickcai/2023/04/11/how-alexandr-wang-turned-an-army-of-clickworkers-into-a-73-billion-ai-unicorn/ https://www.forbes.com/sites/kenrickcai/2023/04/11/how-alexa...

famouswaffles 4y ago
It's just been scale up to before GPT-4. Open AI hasn't said anything specific about 4 so feel free to think there's something major going on if you want but there's basically nothing to support that.
- EGreg 4y ago
  How do we know it’s just been scale up to before GPT-4? As I said, OpenAI hasn’t told us why ChatGPT is so much better than other models (Bloom, LaMDA, LLaMa) and yet we know that they employ thousands of people to do RLHF, including for the coding models: https://www.semafor.com/article/01/27/2023/openai-has-hired-an-army-of-contractors-to-make-basic-coding-obsolete https://www.semafor.com/article/01/27/2023/openai-has-hired-... Doesn’t quite sound like it’s “just scale”. I asked ChatGPT about its training and corpus and it explicitly disavows having that information.
  - famouswaffles 4y ago
    >OpenAI hasn’t told us why ChatGPT is so much better than other models Here's the thing.... It's not. Unless you restrict the pool to models you know about. https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios And Again, chatGPT was not the general SOTA LLM. https://arxiv.org/abs/2210.11416 https://arxiv.org/abs/2210.11416 So in the research world with detailed papers explaining what and what not, we had models that were better.
famouswaffles 4y ago
Also human feedback is not the only way to get a competitive model. Claude's models (Anthropic) are trained from AI feedback and they work just fine. https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios Again there's nothing suggesting any big architectural changes. It's just scale.

by EGreg 4y ago

famouswaffles 4y ago

It's just been scale up to before GPT-4. Open AI hasn't said anything specific about 4 so feel free to think there's something major going on if you want but there's basically nothing to support that.

EGreg 4y ago
How do we know it’s just been scale up to before GPT-4? As I said, OpenAI hasn’t told us why ChatGPT is so much better than other models (Bloom, LaMDA, LLaMa) and yet we know that they employ thousands of people to do RLHF, including for the coding models: https://www.semafor.com/article/01/27/2023/openai-has-hired-an-army-of-contractors-to-make-basic-coding-obsolete https://www.semafor.com/article/01/27/2023/openai-has-hired-... Doesn’t quite sound like it’s “just scale”. I asked ChatGPT about its training and corpus and it explicitly disavows having that information.
- famouswaffles 4y ago
  >OpenAI hasn’t told us why ChatGPT is so much better than other models Here's the thing.... It's not. Unless you restrict the pool to models you know about. https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios And Again, chatGPT was not the general SOTA LLM. https://arxiv.org/abs/2210.11416 https://arxiv.org/abs/2210.11416 So in the research world with detailed papers explaining what and what not, we had models that were better.

famouswaffles 4y ago

Also human feedback is not the only way to get a competitive model. Claude's models (Anthropic) are trained from AI feedback and they work just fine. https://crfm.stanford.edu/helm/latest/?group=core_scenarios https://crfm.stanford.edu/helm/latest/?group=core_scenarios Again there's nothing suggesting any big architectural changes. It's just scale.