4 ms·
What would be the absolute top notch Nvidia card I should buy now, that would last long and would run the most (if not all) of bigger-end SotA models like CodeL
by ewuhic 3y ago
What would be the absolute top notch Nvidia card I should buy now, that would last long and would run the most (if not all) of bigger-end SotA models like CodeLlama or Stable Diffusion XL blazingly fast?
Also, should I buy multiple and bundle them, in case a single top card won't satisfy the above requirements?
- jokethrowaway 3y agoit depends how much you're willing to spend. You can get a H100 (80GB) for around 28-30k$, an A100 (40GB) for around 6-7k$, a 4090 (24GB) for 1.5-2k$. You can build a normal pc with a 4090, otherwise you're in datacenter territory. You can bundle up more GPUs. You can decrease the precision of your models to get them to run on cards with less RAM.
- ewuhic 3y agoCould I really buy a single A100 as a consumer? Any links for that? And could I fit models bigger than 40GB into A100?
- quickthrower2 3y agoAs a consumer isn’t the A100 costing you unnecessary “cloud tax”
- wincy 3y agoYou can snag them on Ebay[0], the A100 with 40GB is ~$5000, the 80GB is like $13,000, good luck, hope it doesn’t break, since you’re not getting a warranty! [0] https://www.ebay.com/sch/i.html?_nkw=a100+40gb https://www.ebay.com/sch/i.html?_nkw=a100+40gb
- timthelion 3y agoI'm seeing a bunch of results from China. I was under the impression it was illegal to sell A100 to China, so where are the chinese sellers getting these cards?
- M4v3R 3y agoA 4060 Ti with 16GB memory costs around $700-800 over here, can I connect two of these to get 32 GB memory to run the bigger models (13B stock or 30B quantized)?
- wincy 3y agoA 3090 with 24GB of vram is $650 used on Facebook where I live. I think that would be a better card as you can use nvlink. Alternatively, the 4000 series cards rely on the PCI express lanes for linking, so if you want to get the benefits of linking the 4000 series cards as a consumer you’ll want a Threadripper or even maybe an EPYC CPU/motherboard combo.
- coder543 3y agoOne additional caveat worth considering... a lot of LLM computation is often memory bandwidth bound. The 4060 Ti is infamous for how deeply Nvidia cut the memory system. The 3060 Ti had a 256-bit bus that was capable of 448GB/s of memory bandwidth. The 4060 Ti has a paltry 128-bit bus that can only do 288GB/s, about 64% of the memory bandwidth of the previous generation. For comparison, an RTX 3090 has 935GB/s of memory bandwidth. As the other person mentioned, that would likely be a much better card if you can find a used one for a reasonable price... but that's just my opinion.
- M4v3R 3y agoThanks for the info!
- quickthrower2 3y agoHave a play on vast.ai first. I don’t know about these models but for training smaller models the humble 3090 packs a punch.