8 ms·

It's surprisingly simple to run all of this at home. I just took a bunch of GPUs out of storage and racked up a machine yesterday. From 0 to Linux/Docker insta

It's surprisingly simple to run all of this at home. I just took a bunch of GPUs out of storage and racked up a machine yesterday. From 0 to Linux/Docker install to Stable Diffusion / llama in about 2 hours.

psshank 3y ago
Consumer GPUs are great for inference at scale. We're a distributed GPU cloud with 10k+ consumer GPUs on our network. We can crank out 4500+ stable diffusion images per dollar (highest in the market), all on consumer 3060s. The high-end GPUs definitely have a role but for a bulk of production/inference serving can be done on consumer GPUs at 80-90% of the cost.
buildbot 3y ago
Most people don’t have multiple GPUs in storage, a server chassis, or a place to rack up said chassis.
- nickstinemates 3y ago
  A lot of people have PC's with modern video cards in them, especially this crowd. It's really no different.
  - 6gvONxR4sf7o 3y ago
    An H100 or A100 have very different capabilities from the cards you're talking about.
    - nickstinemates 3y ago
      Thanks, I wasn't aware.
neilv 3y ago
Many old consumer gaming GPUs will run an implementation of Stable Diffusion. But this page seems to be about getting use of H100 and A100, such as one might want for running or training decent-sized LLMs.
- kkielhofner 3y ago
  Training anything resembling a current LLM from scratch is so far beyond the ability/capital of anyone using a site like this. Facebook/Meta used 8,000 A100s to train LLaMA (for example). If you’re doing anything with an LLM it’s fine-tuning and there’s a new approach nearly weekly to do it better, faster, cheaper, and easier on 24GB cards which can be had for less than $1000. Same for inference.
  - nickstinemates 3y ago
    We just don't understand.
    - kkielhofner 3y ago
      It's only four words and I'm not understanding what you're saying here as all of your other comments (including the one I replied to) seem to be very much in agreement with my position.
  - ukuina 3y ago
    Perhaps we need a new term... "Armchair LLMer"?
  - neilv 3y ago
    Is the earlier point that people should see what they can do with "common household ingredients", before they assume they need to pay cloud providers for bigger/more iron? I agree, and I have a 3090 for that purpose, and once wrote a tutorial for others wanting to do ML stuff on a GPU at home rather than rent from a cloud provider. But a consumer GPU (or eBay older Tesla card) can't do everything that a rental pool of H100 and A100 can do, and I and other readers here will sometimes want to do those other things. I didn't want to confuse people that all they'd need was to buy up a bunch of random retired Ethereum miner GPUs, no matter what they wanted to do with ML.
    - kkielhofner 3y ago
      Don't get me wrong - websites like this still have a lot of value. My overall point is the assumption that you need A100/H100 to make productive use of LLMs isn't accurate. You can go a very long way with a single 3090 in a workstation for $1000 (as frequently noted on HN, including this thread, r/localLLamA, etc). Or you can rent nearly anything you want on various platforms (most of the consumer RTX stuff is usually only available on platforms like Vast.ai). Whatever works for you in your situation but the somewhat common belief (especially in the more mainstream press) that you need at least 40-80GB of VRAM to do anything LLM related is flat out wrong. The other benefit I'd add with buying your own GPUs is availability. They are yours and always yours, in a commercial application with deadlines, etc it's a real risk to depend on being able to get the necessary on-demand GPU compute on various cloud platforms at any point in time. There is nothing worse than logging into a cloud provider console and seeing "no availability" when you really need to get something done. For me personally this is what pushed me to buying vs cloud because I ended up in scenarios where Vast.ai was the only option left and I haven't had the best experiences with Vast.ai in terms of reliability and performance (I'm pretty sure many of the benchmarks are gamed, although I'm not sure how). Speaking of performance, I've also seen very real issues with virtualized CPUs, what I assume is network attached storage, etc feeding data to high end GPUs fast enough (again, noted elsewhere in this thread). In benchmarking that I've done with various cloud providers unless you go for the much more expensive options on GCP and elsewhere with directly attached NVMe storage a single NVMe drive and decent CPU in a workstation will run circles around many of these cloud providers.
Ologn 3y ago
I have an RTX 3060 on my desktop, and my friend has an RTX 4090 on his desktop turned server. We are both running Stable Diffusion with Dreambooth etc., Llama (well, he is running Llama, I am running 4-bit Llama), Deep Floyd and so on. Both machines are running Ubuntu, don't know what distros are good for servers. Incidentally, NVDA closed at 429.97 today, an all time high (it was 108.13 last year, after they were banned from selling A100s/H100s to China - it has almost quadrupled in price in a year).
d4rkp4ttern 3y ago
Awesome , looking forward to a blog post on this :)
- nickstinemates 3y ago
  I was thinking of writing one - will do so.
  - nickstinemates 3y ago
    I wrote something very bare bones. https://keeb.dev/2023/06/15/NVIDIA-on-Docker/ https://keeb.dev/2023/06/15/NVIDIA-on-Docker/

psshank 3y ago

Consumer GPUs are great for inference at scale. We're a distributed GPU cloud with 10k+ consumer GPUs on our network. We can crank out 4500+ stable diffusion images per dollar (highest in the market), all on consumer 3060s. The high-end GPUs definitely have a role but for a bulk of production/inference serving can be done on consumer GPUs at 80-90% of the cost.

buildbot 3y ago

Most people don’t have multiple GPUs in storage, a server chassis, or a place to rack up said chassis.

nickstinemates 3y ago
A lot of people have PC's with modern video cards in them, especially this crowd. It's really no different.
- 6gvONxR4sf7o 3y ago
  An H100 or A100 have very different capabilities from the cards you're talking about.
  - nickstinemates 3y ago
    Thanks, I wasn't aware.

neilv 3y ago

Many old consumer gaming GPUs will run an implementation of Stable Diffusion. But this page seems to be about getting use of H100 and A100, such as one might want for running or training decent-sized LLMs.

kkielhofner 3y ago
Training anything resembling a current LLM from scratch is so far beyond the ability/capital of anyone using a site like this. Facebook/Meta used 8,000 A100s to train LLaMA (for example). If you’re doing anything with an LLM it’s fine-tuning and there’s a new approach nearly weekly to do it better, faster, cheaper, and easier on 24GB cards which can be had for less than $1000. Same for inference.
- nickstinemates 3y ago
  We just don't understand.
  - kkielhofner 3y ago
    It's only four words and I'm not understanding what you're saying here as all of your other comments (including the one I replied to) seem to be very much in agreement with my position.
- ukuina 3y ago
  Perhaps we need a new term... "Armchair LLMer"?
- neilv 3y ago
  Is the earlier point that people should see what they can do with "common household ingredients", before they assume they need to pay cloud providers for bigger/more iron? I agree, and I have a 3090 for that purpose, and once wrote a tutorial for others wanting to do ML stuff on a GPU at home rather than rent from a cloud provider. But a consumer GPU (or eBay older Tesla card) can't do everything that a rental pool of H100 and A100 can do, and I and other readers here will sometimes want to do those other things. I didn't want to confuse people that all they'd need was to buy up a bunch of random retired Ethereum miner GPUs, no matter what they wanted to do with ML.
  - kkielhofner 3y ago
    Don't get me wrong - websites like this still have a lot of value. My overall point is the assumption that you need A100/H100 to make productive use of LLMs isn't accurate. You can go a very long way with a single 3090 in a workstation for $1000 (as frequently noted on HN, including this thread, r/localLLamA, etc). Or you can rent nearly anything you want on various platforms (most of the consumer RTX stuff is usually only available on platforms like Vast.ai). Whatever works for you in your situation but the somewhat common belief (especially in the more mainstream press) that you need at least 40-80GB of VRAM to do anything LLM related is flat out wrong. The other benefit I'd add with buying your own GPUs is availability. They are yours and always yours, in a commercial application with deadlines, etc it's a real risk to depend on being able to get the necessary on-demand GPU compute on various cloud platforms at any point in time. There is nothing worse than logging into a cloud provider console and seeing "no availability" when you really need to get something done. For me personally this is what pushed me to buying vs cloud because I ended up in scenarios where Vast.ai was the only option left and I haven't had the best experiences with Vast.ai in terms of reliability and performance (I'm pretty sure many of the benchmarks are gamed, although I'm not sure how). Speaking of performance, I've also seen very real issues with virtualized CPUs, what I assume is network attached storage, etc feeding data to high end GPUs fast enough (again, noted elsewhere in this thread). In benchmarking that I've done with various cloud providers unless you go for the much more expensive options on GCP and elsewhere with directly attached NVMe storage a single NVMe drive and decent CPU in a workstation will run circles around many of these cloud providers.

Ologn 3y ago

I have an RTX 3060 on my desktop, and my friend has an RTX 4090 on his desktop turned server. We are both running Stable Diffusion with Dreambooth etc., Llama (well, he is running Llama, I am running 4-bit Llama), Deep Floyd and so on. Both machines are running Ubuntu, don't know what distros are good for servers. Incidentally, NVDA closed at 429.97 today, an all time high (it was 108.13 last year, after they were banned from selling A100s/H100s to China - it has almost quadrupled in price in a year).

d4rkp4ttern 3y ago

Awesome , looking forward to a blog post on this :)

nickstinemates 3y ago
I was thinking of writing one - will do so.
- nickstinemates 3y ago
  I wrote something very bare bones. https://keeb.dev/2023/06/15/NVIDIA-on-Docker/ https://keeb.dev/2023/06/15/NVIDIA-on-Docker/