3 ms·
6 GPUs because they want fast storage and it uses PCIe lanes. Besides the goal was to run a 70b FP16 model (requiring roughly 140GB VRAM). 6*24GB = 144GB
by Tepix 3y ago
6 GPUs because they want fast storage and it uses PCIe lanes.
Besides the goal was to run a 70b FP16 model (requiring roughly 140GB VRAM).
6*24GB = 144GB
- andersa 3y agoThat calculation is incorrect. You need to fit both the model (140GB) and the KV cache (5GB at 32k tokens FP8 with flash attention 2) * batch size into VRAM. If the goal is to run a FP16 70B model as fast as possible, you would want 8 GPUs with P2P, for a total of 192GB VRAM. The model is then split across all 8 GPUs with 8-way tensor parallelism, letting you make use of the full 8TB/s memory bandwidth on every iteration. Then you have 50GB spread out remaining for KV cache pages, so you can raise the batch size up to 8 (or maybe more).
- renewiltord 3y agoI’ve got a few 4090s that I’m planning on doing this with. Would appreciate even the smallest directional tip you can provide on splitting the model that you believe is likely to work.
- andersa 3y agoThe split is done automatically by the inference engine if you enable tensor parallelism. TensorRT-LLM, vLLM and aphrodite-engine can all do this out of the box. The main thing is just that you need either 4 or 8 GPUs for it to work on current models.
- renewiltord 3y agoThank you! Can I run with 2 GPUs or with heterogeneous GPUs that have same RAM? I will try. Just curious if you already have tried.
- andersa 3y ago2 GPUs works fine too, as long as your model fits. Using different GPUs with same VRAM however, is highly highly sketchy. Sometimes it works, sometimes it doesn't. In any case, it would be limited by the performance of the slower GPU.
- renewiltord 3y agoAll right, thank you. I can run it on 2x 4090 and just put the 3090s in different machine.
- deleted 3y ago[deleted]
- Tepix 3y agoI know there's some overhead, it's not my calculation. https://www.tweaktown.com/news/97110/tinycorps-new-tinybox-ai-system-amd-gpu-starts-at-15k-nvidia-25k/index.html https://www.tweaktown.com/news/97110/tinycorps-new-tinybox-a... Quote: "Runs 70B FP16 LLaMA-2 out of the box using tinygrad" Related: https://github.com/tinygrad/tinygrad/issues/3791 https://github.com/tinygrad/tinygrad/issues/3791