3 ms·

> It's a pretty well accepted fact now that bigger LLM = moar better without exceptions. That's not true, the amount of training is a MAJOR factor. See the Ch

> It's a pretty well accepted fact now that bigger LLM = moar better without exceptions. That's not true, the amount of training is a MAJOR factor. See the Chinchilla paper - https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 tl;dr - a "fully" trained small model can outperform a "undertrained" larger model. If you have a fixed amount of compute (budget), then you need to optimize for the largest model that you can fully train, and not simply up the parameter count. EDIT: Also you can't necessarily compare the parameter count across model architectures* This thing seems to outperform the finetuned 30B llama models I've seen.

moffkalast 3y ago
Well if you're set on training something for a specific budget then it does of course make sense to pick the most optimal model size, true. But the problem is that these models don't exist in a vacuum, and have to go against slightly larger ones that are also compute optimal and use more data, which will definitely perform better. Then again maybe there is a sweet spot for a model that's small enough to run effortlessly on regular machines while only serving as a control node in an autoGPT style setting, where it fetches the context it can't possibly have from a curated online database to make up for its shortcomings.
- thewataccount 3y ago
  > But the problem is that these models don't exist in a vacuum, and have to go against slightly larger ones that are also compute optimal and use more data, which will definitely perform better. They don't have to go against those though. Most of these models are research models, either from academia or from companies experimenting to see what works. From my understanding, most of these are a - "We have X amount of USD for the next month or so, we'll try a few things, then whatever our best bet is we'll stick the time out on that". Very few companies have the resources to train big models with as much compute as Google/OpenAI/Microsoft/Facebook. These are also not being monetized as they're open source. Going from their 2.7B model to 10B would be ~10X the compute (FLOPS) required for an optimal model. And this is likely their first open model and not their last, since Replit likely doesn't have the budget that openai does it makes sense they didn't want to blow their entire year's budget on their first open model. 2.7B would also be a really nice if anyone can get it working because it's more likely to be able to run in the IDE at that point instead of needed a massively scaled cloud (which might be valuable for replit).

moffkalast 3y ago

Well if you're set on training something for a specific budget then it does of course make sense to pick the most optimal model size, true. But the problem is that these models don't exist in a vacuum, and have to go against slightly larger ones that are also compute optimal and use more data, which will definitely perform better. Then again maybe there is a sweet spot for a model that's small enough to run effortlessly on regular machines while only serving as a control node in an autoGPT style setting, where it fetches the context it can't possibly have from a curated online database to make up for its shortcomings.

thewataccount 3y ago
> But the problem is that these models don't exist in a vacuum, and have to go against slightly larger ones that are also compute optimal and use more data, which will definitely perform better. They don't have to go against those though. Most of these models are research models, either from academia or from companies experimenting to see what works. From my understanding, most of these are a - "We have X amount of USD for the next month or so, we'll try a few things, then whatever our best bet is we'll stick the time out on that". Very few companies have the resources to train big models with as much compute as Google/OpenAI/Microsoft/Facebook. These are also not being monetized as they're open source. Going from their 2.7B model to 10B would be ~10X the compute (FLOPS) required for an optimal model. And this is likely their first open model and not their last, since Replit likely doesn't have the budget that openai does it makes sense they didn't want to blow their entire year's budget on their first open model. 2.7B would also be a really nice if anyone can get it working because it's more likely to be able to run in the IDE at that point instead of needed a massively scaled cloud (which might be valuable for replit).