4 ms·

Let's not ignore better architectures, training techniques and computing power.

by GaggiX 2y ago

kaycebasques 2y ago
Suppose you are competing to create the "best" frontier model and have finite R&D budget to allocate into the 4 buckets (plus a catchall in case we're missing something): * Data * Architecture * Training techniques * Compute * Other What allocation gives you the best chance for success? And how are you defining "best"?
- NitpickLawyer 2y ago
  Right now I'd prioritise compute over anything, because it allows for more experiments, and some of those experiments might turn out to be the key to better models (either specific applications or overall generalist models). Meta did this with L3. They used L2 to pre-filter the training data, filtering out a lot of junk. They also used it to classify datasets. Then after pre-training (involving lots of compute) they also used almost exclusively synthetic data for fine-tuning (forgoing RLHF when it was surpassed). So yet more compute. The results are pretty good, L3, 3.1 and 3.2 are pretty high up there in terms of open access SotA. oAI did this with their o1 models. They used lots of compute to have the models go over the space of generating tokens, analysing, correcting, and so on. They RLd the "reasoning traces" in a way. Lots of compute. The results seem to be pretty good, with impressive showings on "reasoning" tasks, math, code, and so on. The thing is, they weren't the first ones to propose these techniques! What differentiates them is the available compute. WizardML tried and were really successful with their RLAIF implementation (tho never released code afaik) about a year ago. And while they were connected to MS research, they probably didn't have as much compute available as Meta. But the WizardML fine-tunes on open models like Mistral and Mixtral were pretty much SotA when released, scoring way higher than the creator's own fine-tunes. In the same vein, but at lower scales is the team behind DeepSeek. They used RL on math problems, in their DeepSeekMath-7bRL model, and that model was SotA at the time of release as well. It took a team of multiple really talented folks to fine-tune a better model (in the AIMO kaggle competition) and everyone except the 1st place used the RL model. The 1st place used the base model, with different fine-tuning. So again, the methods were tried, just at much lower scales. Yeah, I think compute would be my bet in the short run.
JCharante 2y ago
using human feedback for reinforcement learning is a training technique
ben_w 2y ago
It's both. I recently saw a comparison of various models on two IQ tests, one of which was public and the other of which was carefully curated to be not directly learnable from the likely training sets. On public tests, LLMs vary between "just below average human" and "genius". On the hopefully-private test (it's difficult to be sure*), the best was o1, which was "merely" just below an average human, Claude-3 Opus which was stupid, and all the rest were "would need a full time caretaker". In both cases, the improvements to the models came with higher scores; but there's still a lot you can do by learning for the test — and one thing that LLMs are definitely superhuman at is that. https://www.maximumtruth.org/p/massive-breakthrough-in-ai-intelligence https://www.maximumtruth.org/p/massive-breakthrough-in-ai-in... * I could have said the same last year about last year's models, so I'm emphatically not saying o1 really is as smart as this test claims; I'm only saying this demonstrates these IQ tests are a learnable skill up to at least this magnitude of difference.

Let's not ignore better architectures, training techniques and computing power.

by GaggiX 2y ago

Let's not ignore better architectures, training techniques and computing power.

Suppose you are competing to create the "best" frontier model and have finite R&D budget to allocate into the 4 buckets (plus a catchall in case we're missing something): * Data * Architecture * Training techniques * Compute * Other What allocation gives you the best chance for success? And how are you defining "best"?

NitpickLawyer 2y ago
Right now I'd prioritise compute over anything, because it allows for more experiments, and some of those experiments might turn out to be the key to better models (either specific applications or overall generalist models). Meta did this with L3. They used L2 to pre-filter the training data, filtering out a lot of junk. They also used it to classify datasets. Then after pre-training (involving lots of compute) they also used almost exclusively synthetic data for fine-tuning (forgoing RLHF when it was surpassed). So yet more compute. The results are pretty good, L3, 3.1 and 3.2 are pretty high up there in terms of open access SotA. oAI did this with their o1 models. They used lots of compute to have the models go over the space of generating tokens, analysing, correcting, and so on. They RLd the "reasoning traces" in a way. Lots of compute. The results seem to be pretty good, with impressive showings on "reasoning" tasks, math, code, and so on. The thing is, they weren't the first ones to propose these techniques! What differentiates them is the available compute. WizardML tried and were really successful with their RLAIF implementation (tho never released code afaik) about a year ago. And while they were connected to MS research, they probably didn't have as much compute available as Meta. But the WizardML fine-tunes on open models like Mistral and Mixtral were pretty much SotA when released, scoring way higher than the creator's own fine-tunes. In the same vein, but at lower scales is the team behind DeepSeek. They used RL on math problems, in their DeepSeekMath-7bRL model, and that model was SotA at the time of release as well. It took a team of multiple really talented folks to fine-tune a better model (in the AIMO kaggle competition) and everyone except the 1st place used the RL model. The 1st place used the base model, with different fine-tuning. So again, the methods were tried, just at much lower scales. Yeah, I think compute would be my bet in the short run.

JCharante 2y ago

using human feedback for reinforcement learning is a training technique

ben_w 2y ago

It's both. I recently saw a comparison of various models on two IQ tests, one of which was public and the other of which was carefully curated to be not directly learnable from the likely training sets. On public tests, LLMs vary between "just below average human" and "genius". On the hopefully-private test (it's difficult to be sure*), the best was o1, which was "merely" just below an average human, Claude-3 Opus which was stupid, and all the rest were "would need a full time caretaker". In both cases, the improvements to the models came with higher scores; but there's still a lot you can do by learning for the test — and one thing that LLMs are definitely superhuman at is that. https://www.maximumtruth.org/p/massive-breakthrough-in-ai-intelligence https://www.maximumtruth.org/p/massive-breakthrough-in-ai-in... * I could have said the same last year about last year's models, so I'm emphatically not saying o1 really is as smart as this test claims; I'm only saying this demonstrates these IQ tests are a learnable skill up to at least this magnitude of difference.