4 ms·
Training smaller models can be really compute intensive (a 7B models should get trained on 1.4T tokens to follow the "LLaMA laws"). So that would be C = 6 * 1.4
by FanaHOVA 3y ago
Training smaller models can be really compute intensive (a 7B models should get trained on 1.4T tokens to follow the "LLaMA laws"). So that would be C = 6 * 1.4T * 7B = 58.8T FLOP-seconds. That's 1/5th the compute of GPT3 for example, but it's still a lot. We asked Quentin to do a similar post but for fine tuning math; that's still a very underexplored space.
(Not to self plug too much, but this is exactly what last episode's with Tianqi Chen was about if you're interested :) https://www.latent.space/p/llms-everywhere#details https://www.latent.space/p/llms-everywhere#details
- swyx 3y agothat said there's more data efficiency to be gained in the smol models space - Phi-1 achieved 51% on HumanEval with only a 5.3x token-param ratio: https://www.latent.space/p/cogrev-tinystories#details https://www.latent.space/p/cogrev-tinystories#details
- arugulum 3y agoI want to jump in and correct your usage of "LLaMA Laws" (even you are using it informally, but I just want to clarify). There is no "LLaMA scaling law". There are a set of LLaMA training configurations. Scaling laws describe the relationship between training compute, data, and expected loss (performance). Kaplan et al., estimated one set of laws, and the Chinchilla folks refined that estimate (mainly improving it by adjusting the learning rate schedule). The LLaMA papers do not posit any new law nor contradict any prior one. They chose a specific training configuration that still abide by the scaling laws but with a different goal in mind. (Put another way: a scaling law doesn't tell you what configuration to train on. It tells you what to expect given a configuration, but you're free to decide on whatever configuration you want.)
- FanaHOVA 3y agoYep, +1. That's why I used the quotes. :) Thanks for expanding!
- arugulum 3y agoYep I understood that you were using it informally, just trying to keep things informative for other folks reading too.
- swyx 3y agothere frankly needs to be a paper calling this out tho, because at this point there are a bunch of industry models following “llama laws” and nobody’s really done the research, its all monkey see monkey do
- arugulum 3y agoBut what would they be calling out? If industry groups want to run a training run based on the configurations of a well-performing model, I don't see anything wrong with that. Now, if they were to claim that what they are doing is somehow "optimal", then there would be something to criticize.
- npsomaratna 3y agoIsn't the Chinchilla estimate considered to be wrong now? https://espadrine.github.io/blog/posts/chinchilla-s-death.html https://espadrine.github.io/blog/posts/chinchilla-s-death.ht...
- quickthrower2 3y ago> C = 6 * 1.4T * 7B = 58.8T FLOP-seconds. Sounds off by a few orders of magnitude since a RTX3090 can smash through about 78TFLOP/s. I trained nanoGPT on a rented RTX3090 which is a much tinier model and takes about 12 hours on 100k steps (I need to check the params again but maybe 10M tokens tops) And the reason is it is not a single flop per parameter per token. You have a deep architecture, compute intensive attention heads and back propagations to do as well.