4 ms·
For the numerically challenged like me: 100TB is 100 trillion bytes, giving you 1 byte per parameter at 100T params. LLaMA can apparently run quantized to 4 bi
by andai 3y ago
For the numerically challenged like me: 100TB is 100 trillion bytes, giving you 1 byte per parameter at 100T params.
LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though), which would allow you to run a 200TB model on one of these cards if I'm understanding right.
- valine 3y agoThat would work for inference, but for efficient training you’d also want you training set to fit in memory.
- airgapstopgap 3y agoI think people talking about a 100T GPT didn't mean a dense transformer but some sort of extreme Mixture-of-Experts which is much more amenable to low-resource setups and complicates this discussion. In any case, it's almost certainly not bigger than 1T, even if it's not a dense transformer (PaLM-2 is and makes do with 340B, but it isn't exactly on par).
- renonce 3y agoYou can’t quantize it for training due to numerical instability. For inference you don’t usually use such a big cluster.
- Tepix 3y ago> LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though) From the GPTQ paper https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323: "... with negligible accuracy degradation relative to the uncompressed baseline"