3 ms·

Chinchilla scaling was only good for academics - it prioritises model training cost at the expense of serving cost. If you plan to, you know, actually use the m

Chinchilla scaling was only good for academics - it prioritises model training cost at the expense of serving cost. If you plan to, you know, actually use the model, not just publish a paper on it, you need the LLaMA approach of training smaller models past the Chinchilla point. But the smaller the model, the more compute is needed to cover the gap. To reach the loss of a 67B model - A 33B model needs 2.3x compute - A 13B model needs 25x compute - A 7B model needs 7837x compute - A 3B model can't match the 67B. Ever.

haldujai 3y ago
> Chinchilla scaling was only good for academics I don't know if it's only good of academics, the point as the paper (as it says) is a scaling law for optimal loss given a fixed compute budget. By design it doesn't address inference costs and isn't a recipe for "how you should train a LLM for your use case". If you're serving LLMs in a low throughput high cost scenario optimizing loss at the expense of inference cost may very well be your goal, or if you cant pay up front for 25x compute.