4 ms·
How does training on just 800k pieces of data need 7b parameters?
by m3kw9 4y ago
How does training on just 800k pieces of data need 7b parameters?
- sp332 4y agoLlama 7B was trained on a trillion tokens. The Lora is a small fraction of extra neurons that get integrated into the structure, and those are what get trained on the new data. It's like fine-tuning but takes less RAM and compute than retraining the whole model.
- comex 4y agoBecause it’s fine-tuning an existing 7B-parameter language model, not training from scratch.
- KRAKRISMOTT 4y agoThe chinchilla formula demands a 20:1 ratio
- airstrike 4y ago"demands" is great
- KRAKRISMOTT 4y agoHey I didn't invent Chinchilla, blame google for that.
- adt 4y agoDeepMind (apparently there's some friction between those two!)