3 ms·

Suppose you are competing to create the "best" frontier model and have finite R&D budget to allocate into the 4 buckets (plus a catchall in case we're missing s

Suppose you are competing to create the "best" frontier model and have finite R&D budget to allocate into the 4 buckets (plus a catchall in case we're missing something): * Data * Architecture * Training techniques * Compute * Other What allocation gives you the best chance for success? And how are you defining "best"?

NitpickLawyer 2y ago
Right now I'd prioritise compute over anything, because it allows for more experiments, and some of those experiments might turn out to be the key to better models (either specific applications or overall generalist models). Meta did this with L3. They used L2 to pre-filter the training data, filtering out a lot of junk. They also used it to classify datasets. Then after pre-training (involving lots of compute) they also used almost exclusively synthetic data for fine-tuning (forgoing RLHF when it was surpassed). So yet more compute. The results are pretty good, L3, 3.1 and 3.2 are pretty high up there in terms of open access SotA. oAI did this with their o1 models. They used lots of compute to have the models go over the space of generating tokens, analysing, correcting, and so on. They RLd the "reasoning traces" in a way. Lots of compute. The results seem to be pretty good, with impressive showings on "reasoning" tasks, math, code, and so on. The thing is, they weren't the first ones to propose these techniques! What differentiates them is the available compute. WizardML tried and were really successful with their RLAIF implementation (tho never released code afaik) about a year ago. And while they were connected to MS research, they probably didn't have as much compute available as Meta. But the WizardML fine-tunes on open models like Mistral and Mixtral were pretty much SotA when released, scoring way higher than the creator's own fine-tunes. In the same vein, but at lower scales is the team behind DeepSeek. They used RL on math problems, in their DeepSeekMath-7bRL model, and that model was SotA at the time of release as well. It took a team of multiple really talented folks to fine-tune a better model (in the AIMO kaggle competition) and everyone except the 1st place used the RL model. The 1st place used the base model, with different fine-tuning. So again, the methods were tried, just at much lower scales. Yeah, I think compute would be my bet in the short run.

NitpickLawyer 2y ago

Right now I'd prioritise compute over anything, because it allows for more experiments, and some of those experiments might turn out to be the key to better models (either specific applications or overall generalist models). Meta did this with L3. They used L2 to pre-filter the training data, filtering out a lot of junk. They also used it to classify datasets. Then after pre-training (involving lots of compute) they also used almost exclusively synthetic data for fine-tuning (forgoing RLHF when it was surpassed). So yet more compute. The results are pretty good, L3, 3.1 and 3.2 are pretty high up there in terms of open access SotA. oAI did this with their o1 models. They used lots of compute to have the models go over the space of generating tokens, analysing, correcting, and so on. They RLd the "reasoning traces" in a way. Lots of compute. The results seem to be pretty good, with impressive showings on "reasoning" tasks, math, code, and so on. The thing is, they weren't the first ones to propose these techniques! What differentiates them is the available compute. WizardML tried and were really successful with their RLAIF implementation (tho never released code afaik) about a year ago. And while they were connected to MS research, they probably didn't have as much compute available as Meta. But the WizardML fine-tunes on open models like Mistral and Mixtral were pretty much SotA when released, scoring way higher than the creator's own fine-tunes. In the same vein, but at lower scales is the team behind DeepSeek. They used RL on math problems, in their DeepSeekMath-7bRL model, and that model was SotA at the time of release as well. It took a team of multiple really talented folks to fine-tune a better model (in the AIMO kaggle competition) and everyone except the 1st place used the RL model. The 1st place used the base model, with different fine-tuning. So again, the methods were tried, just at much lower scales. Yeah, I think compute would be my bet in the short run.