2 ms·
I didn't follow entirely on a fast read, but this confused me especially: The parameter count of an LLM defines a certain bit budget. This bit budget must be
by andy99 1y ago
I didn't follow entirely on a fast read, but this confused me especially:
The parameter count of an LLM defines a certain bit budget. This bit budget must be spread across many, many tasks
I'm pretty sure that LLMs, like all big neural networks, are massively under-specified, as in there are way more parameters than data to fit (understanding the training data set is bigger than the size of the model, but the point is the same loss can be achieved with many different combinations of parameters).
And I think of this underspecification as the reason neural networks extrapolate cleanly and this generalize.
- vonneumannstan 1y agoThis doesn't seem right and most people recognize that 'neurons' encode for multiple activations. https://transformer-circuits.pub/2022/toy_model/index.html https://transformer-circuits.pub/2022/toy_model/index.html
- Der_Einzige 1y agoThey’re 1000% right on the idea that most models are hilariously undertrained
- vonneumannstan 1y agoPretty sure that since the Chinchilla paper this probably isn't the case. https://arxiv.org/pdf/2203.15556 https://arxiv.org/pdf/2203.15556
- kazinator 1y ago> the same loss can be achieved with many different combinations of parameters Perhaps it can be, but it isn't. The loss in a given model was achieved with that particular combination of parameters that the model has, and there exists no other combination of parameters which which that model can appeal to for more information. To have the other combinations, we would need to train more models and then somehow combine them so that the combinations are available; but that's just conceptually the same as making one larger model with more parameters, and lower loss.