4 ms·

The best value GPU right now for SoTA LLMs are probably the Nvidia RTX 6000. You can connect them together with NVLink, they have 48GB, and you can fit 4 of the

The best value GPU right now for SoTA LLMs are probably the Nvidia RTX 6000. You can connect them together with NVLink, they have 48GB, and you can fit 4 of them in a PCI 4.0 high end consumer motherboard. Enough to fine tune even LLama-2, albeit with a batch size of 1 :)

CamperBob2 3y ago
Unfortunately, you have to specify which generation of "RTX 6000" you mean, since nVidia likes to reuse product names for no good reason besides confusing customers. The one you're talking about is around 8 grand USD. The one that comes up first when you Google "RTX 6000" is $2400 and change, and not particularly useful.
- jamesblonde 3y ago
  Thanks for the clarification. I bought them for $5k (without VAT) in Sweden last month.
brucethemoose2 3y ago
> Enough to fine tune even LLama-2, albeit with a batch size of 1 :) At FP16? I think you need much less (~2 48GB cards) to finetune 70B with increasing levels of optimization. I think you can even do it on a single card with QLORA.
- buildbot 3y ago
  I have not been able to get it working with 2 4090s yet, but I think that's because the library I choose to use (axolotl) does not support model parallel at all.
  - brucethemoose2 3y ago
    I wish Facebook would release the raw 34B model... Or better yet, that Mistral would release one.
buildbot 3y ago
Only The RTX A6000 (ampere) (Not RTX 6000! those are 24GB Turings!) has NVlink, and as far as Nevada states, they only work in 2x NVLink. Memory pooling is not really a real thing as much as they say it is. They run about 4K on eBay, and have 303TFlops of tensor core perf in sparse Bfloat 16. So 150 Dense. They do have 48GB of memory which is great, but at 768GB/s. Source: https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/proviz-print-nvidia-rtx-a6000-datasheet-us-nvidia-1454980-r9-web%20(1).pdf https://www.nvidia.com/content/dam/en-zz/Solutions/design-vi... The 4090s run 1599 for a founders edition or a Gigabyte Windforce V2 (my choice). They have at least 165 Tflops of Bfloat16, 330 TFlops of FP16 with FP16 accumulate, and 660 TFlops if you use sparsity. They also support FP8 at 660 TFlops Dense, 1321 TFlops Sparse(!!). Unless you need the 2x 25% slower memory, the 4090s are much better choices. You get the same scaling over 2 cards anyway. There is also the RTX A6000 Ada, which is 8K, and based on the same Ada chip as the 4090, except with 48GB of memory. Lower power and clock speeds result in slightly lower peak TFlops numbers. You really really pay for memory.

CamperBob2 3y ago

Unfortunately, you have to specify which generation of "RTX 6000" you mean, since nVidia likes to reuse product names for no good reason besides confusing customers. The one you're talking about is around 8 grand USD. The one that comes up first when you Google "RTX 6000" is $2400 and change, and not particularly useful.

jamesblonde 3y ago
Thanks for the clarification. I bought them for $5k (without VAT) in Sweden last month.

brucethemoose2 3y ago

> Enough to fine tune even LLama-2, albeit with a batch size of 1 :) At FP16? I think you need much less (~2 48GB cards) to finetune 70B with increasing levels of optimization. I think you can even do it on a single card with QLORA.

buildbot 3y ago
I have not been able to get it working with 2 4090s yet, but I think that's because the library I choose to use (axolotl) does not support model parallel at all.
- brucethemoose2 3y ago
  I wish Facebook would release the raw 34B model... Or better yet, that Mistral would release one.

buildbot 3y ago

Only The RTX A6000 (ampere) (Not RTX 6000! those are 24GB Turings!) has NVlink, and as far as Nevada states, they only work in 2x NVLink. Memory pooling is not really a real thing as much as they say it is. They run about 4K on eBay, and have 303TFlops of tensor core perf in sparse Bfloat 16. So 150 Dense. They do have 48GB of memory which is great, but at 768GB/s. Source: https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/proviz-print-nvidia-rtx-a6000-datasheet-us-nvidia-1454980-r9-web%20(1).pdf https://www.nvidia.com/content/dam/en-zz/Solutions/design-vi... The 4090s run 1599 for a founders edition or a Gigabyte Windforce V2 (my choice). They have at least 165 Tflops of Bfloat16, 330 TFlops of FP16 with FP16 accumulate, and 660 TFlops if you use sparsity. They also support FP8 at 660 TFlops Dense, 1321 TFlops Sparse(!!). Unless you need the 2x 25% slower memory, the 4090s are much better choices. You get the same scaling over 2 cards anyway. There is also the RTX A6000 Ada, which is 8K, and based on the same Ada chip as the 4090, except with 48GB of memory. Lower power and clock speeds result in slightly lower peak TFlops numbers. You really really pay for memory.