3 ms·
This. If you insist on being as cheap as possible, shoot for a 12gb card but be aware that you'll be missing out on the throughput of higher-end models. The 306
by smoldesu 3y ago
This. If you insist on being as cheap as possible, shoot for a 12gb card but be aware that you'll be missing out on the throughput of higher-end models. The 3060 is popular for this I think, but you'll probably want a better card with more CUDA cores to max out performance.
Cards like the A770 are awesome, but barely even support raster drivers on DirectX. Your best bang-for-buck options are going to be Nvidia-only for now, with a few competing AMD cards that have fast-tracked Pytorch support.
- matthewaveryusa 3y agoI purchased a 3060 specifically for the 12gb of memory last November and I've been able to run llama, alpaca, stable diffusion out of the box for everything without ever having any memory issues. Training is usually overnight, and a stable diffusion will render in ~5 seconds, llama will do 20 tokens/second. I would say start with the 3060 for 250 bucks, and if you're still loving it after a couple months, drop 10x more on a quadro. My only word of advice is get docker setup and install the nvidia docker toolkit to passthrough your gpu to docker images -- the package management for all these python ai tools is a hell-scape, especially if you want to try a bunch of different things.