4 ms·
True. There was a good blog post published about this a few weeks ago: https://finbarr.ca/llms-not-trained-enough/ https://finbarr.ca/llms-not-trained-enough/
by gamegoblin 4y ago
True. There was a good blog post published about this a few weeks ago: https://finbarr.ca/llms-not-trained-enough/ https://finbarr.ca/llms-not-trained-enough/
Money quote for those who don't want to read the whole thing:
'''
When people talk about training a Chinchilla-optimal model, this is what they mean: training a model that matches their estimates for optimality. They estimated the optimal model size for a given compute budget, and the optimal number of training tokens for a given compute budget.
However, when we talk about “optimal” here, what is meant is “what is the cheapest way to obtain a given loss level, in FLOPS.” In practice though, we don’t care about the answer! This is exactly the answer you care about if you’re a researcher at DeepMind/FAIR/AWS who is training a model with the goal of reaching the new SOTA so you can publish a paper and get promoted. If you’re training a model with the goal of actually deploying it, the training cost is going to be dominated by the inference cost. This has two implications:
1) there is a strong incentive to train smaller models which fit on single GPUs
2) we’re fine trading off training time efficiency for inference time efficiency (probably to a ridiculous extent).
Chinchilla implicitly assumes that the majority of the total cost of ownership (TCO) for a LLM is the training cost. In practice, this is only the case if you’re a researcher at a research lab who doesn’t support products (e.g. FAIR/Google Brain/DeepMind/MSR). For almost everyone else, the amount of resources spent on inference will dwarf the amount of resources spent during training.
'''
- haldujai 4y agoWhile true I think this also misses that “for almost everyone else” you’re probably not (or at least should not) be trying to optimize zero-shot performance if you have an intended high inference use case so I don’t think Chinchilla would be all that relevant.
- vintermann 4y agoI have a suspicion that good zero-shot performance is a good starting point for fine-tuning. If you have more than one intended high inference use case, or can imagine a couple of new ones on the horizon, it might still be best to not target the first use case directly.
- haldujai 4y agoWell yeah that’s kind of intuitive, my point is that if you just optimize for zero-shot you end up with something like GPT4 when an enterprise could probably be using finetuned LLaMA-7B with similar performance.
- sebzim4500 4y ago>Chinchilla implicitly assumes that the majority of the total cost of ownership (TCO) for a LLM is the training cost. In practice, this is only the case if you’re a researcher at a research lab who doesn’t support products (e.g. FAIR/Google Brain/DeepMind/MSR). For almost everyone else, the amount of resources spent on inference will dwarf the amount of resources spent during training. I'm not so convinced, especially if people are doing multiple training runs for hyperparameter tuning, cleaning data, fixing bugs, etc. I would be very interested in knowing what portion of OpenAI's compute budget is training. I would not be surprised if it was a significant minority.
- aiappreciator 4y ago"the training cost is going to be dominated by the inference cost." That's only true for general-mass-consumer models. Companies may want to fine-tune/train their own models, which don't have that many users for their narrow use cases (possibly only internal staff), will find that training cost is a substantial chunk of the TCO