3 ms·
You don't need to store it in weights. It would be too expensive to constantly update model with new data. It wouldn't be able to contain everything anyways. LL
by terafo 4y ago
You don't need to store it in weights. It would be too expensive to constantly update model with new data. It wouldn't be able to contain everything anyways. LLaMa 30B with access to 100TB of storage(you can store whole libgen plus arxiv and much much more in there) and trained to use it should be ahead of LLaMa 65B with no such access in terms of world knowledge. And it would cost less.
- Imnimo 4y agoYou can retrieve small amounts of specific information this way. But suppose the model was asked to write code in a new programming language that it doesn't have existing knowledge of. Could it do that given unguided access to the docs?
- terafo 4y agoIt might, but in this specific instance some kind of fine-tuning would be better.
- Imnimo 4y agoIt feels pretty obvious to me that it could not, but then again I've been wrong about LLM abilities before!
- webmaven 4y agoWe've already seen zero-shot learning of novel syntaxes (that is, chatting with an LLM to interactively devise a new language that is then used, without any actual retraining of the model).