3 ms·
From the paper[1]: > We additionally scale our inference code to support million-length sequences by implementing RingAttention for decoding. Inference for suc
by woadwarrior01 3y ago
From the paper[1]:
> We additionally scale our inference code to support million-length sequences by implementing RingAttention for decoding. Inference for such long sequences requires a minimum of v4-128 with a TPU mesh sharding of 32 tensor parallelism, and 4 sequence parallelism (ring dimension).
Each TPU-v4 has 32 GiB of HBM memory, so about 4 TiB (128 x 32GiB) of memory in fp16, without quantization.
[1]: https://arxiv.org/abs/2402.08268 https://arxiv.org/abs/2402.08268
- Mathnerd314 3y agoThe model itself is only 13.6GB though: https://huggingface.co/LargeWorldModel/LWM-Chat-1M-Jax/tree/main https://huggingface.co/LargeWorldModel/LWM-Chat-1M-Jax/tree/...
- woadwarrior01 3y agoIndeed. Almost all of the inference memory overhead comes from attention matrices and the KV cache.
- moffkalast 3y agoSo uh, they thought "we have this datacenter full of GPUs laying around, you know what would be best, if we made a model that needs the entire thing just for inference". Brute forcing a quadratic complexity problem seems like wastefulness at its worst.
- cyanydeez 3y agoaren't most LLMs just a brute force probability engine?
- 3abiton 3y agoSounds like they're loading the internet into those vrams.