7 ms·

Someone needs to write a buyer's guide for GPUs and LLMs. For example, what's the best course of action if don't need to train anything but do want to eventuall

by causi 4y ago

Someone needs to write a buyer's guide for GPUs and LLMs. For example, what's the best course of action if don't need to train anything but do want to eventually run whatever model becomes the first local-capable equivalent to ChatGPT? Do you go with Nvidia for the CUDA cores or with AMD for more VRAM? Do you do neither and wait another generation?

avereveard 4y ago
Depends on whether you just want to play around or have actual business needs, but to play around I'm waiting for the next generation of unified memory architectures
- ilteris 4y ago
  Can you point any source re: next generation of unified memory architectures? How far are they away?
  - avereveard 4y ago
    Well currently there's just apple and they are super secretive about their roadmap, the M1/2 top line chip already perform quite well, and while the top version look expensive, a pair of a6000 will definitely eat more money both to purchase and to run. M3 is around the corner tho, and there's some announcement to come from intel or arm following their partnership. There's also the new card coming from intel that is supposed to be aimed squarely at machine learning workloads, and they don't have to segment their market by memory sizing like Nvidia do, but they aren't well supported as device targets, but a pair of these will likely be very cost effective if and only if they will get credible compatibility with the libraries and models
redox99 4y ago
It's pretty simple actually. Get a 3090 or 4090. Forget about AMD.
- mewpmewp2 4y ago
  Should I get a gaming PC with 4090 if I want to get into personal side projects in ML and also games? Do I need dualboot? Or is Windows good?
  - thewataccount 4y ago
    Windows generally works but there may be a somewhat small performance hit. IMO linux is much easier to get to work judging by all the github issue threads I see able SD/LLaMa stuff on windows - but I don't use windows so I dont have personal experience. 4090 24GB is 1800USD, The Ada A6000 48GB is like 8000USD and idk where you buy it? So if you want to run games and models locally the 4090 is honestly the best option. EDIT: I forgot - there is a rumored 4090ti with 48gb of vram, no idea if thats worth waiting for.
    - macrolime 4y ago
      The A6000 is actually the old generation, Ampere. The new Ada generation one is called 6000. Seems many places still sell A6000 (Ampere) for the same price as RTX 6000 (Ada) though, even though the new one is twice as fast. Seems you can get used RTX A6000s for around $3000 on ebay.
      - thewataccount 4y ago
        That.... That explains why I can't find it and makes a ton of sense..... I think that's such a silly name for it, but oh well Thanks for the correction!
        macrolime 4y ago
        Just to add to the confusion, there's another older RTX 6000 with 24GB ram. This is from an even older generation, same as the GeForce 20 series.
        4y ago
dannyw 4y ago
Definitely NVIDIA. Good luck running any real ML models on AMD GPUs.
thewataccount 4y ago
Nvidia and the highest amount of vram you can get. Currently the 4090, the rumor is the 4090ti will have 48gb of vram, idk if its worth waiting or not. The more VRAM the higher paremeter count you can run all in memory (fastest by far). AMD is almost a joke in ML. The lack of CUDA support (which is nvidia proprietary) is straight lethal, and also even though ROCM does have much better support these days, from what I've seen it's still a fraction of the performance of what it should be. I'm also not sure if you need projects to support it or not, I know pytorch has backend support for it but I'm not sure how easy it is to drop in.
- causi 4y ago
  That's good to know. A lot of people might look at the 7900XTX with its 24GB of ram for a grand and think "ah, a good deal for fitting an LLM in".
- brandall10 4y ago
  I think the most recent rumors were amended to it having 24, unfortunately.
  - thewataccount 4y ago
    Darn. I mean in all honestly there's no reason a gaming card would need 48gb at the moment when so few games even use 24gb. 48GB really only makes sense for workstation cards.
seydor 4y ago
At least for llama derivatives, reddit has a list https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/ https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...
l33tman 4y ago
You don't need a GPU, alpaca.cpp runs purely on the CPU if you have 32+ GB of RAM for the 30B Llama model. It's a bit slow, but you can toy with it..
mk_stjames 4y ago
Ask: how many parameters is the model? At how many bits? and has it been quantized to a smaller number of bits or if not can it be without losing quality? multiply number of bits by the number of parameters, convert to GB, and then add so amount of overhead for however much else needs to load and/or how horrible/amazing of an implementation the inference code is. Done. Example: A 30B parameter model trained at 16bit FP gets quantized down to 4 bit ints. 4 bits = 0.5 byte. 30 billion * 0.5 byte = 15GB of VRAM (plus a GB or few of other overhead) For more real world discussion see https://github.com/ggerganov/llama.cpp/issues/13 https://github.com/ggerganov/llama.cpp/issues/13
lhl 4y ago
For a general guide, I recommend: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni... There's a subreddit r/LocalLLaMA that seems like the most active community focused on self-hosting LLMs. Here's a recent discussion on hardware: https://www.reddit.com/r/LocalLLaMA/comments/12lynw8/is_anyone_else_super_eager_to_upgrade_their/ https://www.reddit.com/r/LocalLLaMA/comments/12lynw8/is_anyo... If you're looking just for local inference, you're best bet is probably to buy a consumer GPU w/ 24GB of RAM (3090 is fine, 4090 more performance potential), which can fit a 30B parameter 4-bit quantized model that can probably be fine-tuned to ChatGPT (3.5) level quality. If not, then you can probably add a second card later on. Alternatively, if you have an Apple Silicon Mac, llama.cpp performs surprisingly well, it's easy to try for free: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Current AMD consumer cards have terrible software support and IMO isn't really an option. On Windows you might be able to use SHARK or DirectML ports, but nothing will run out of the box. ROCm still has no RDNA3 support (supposedly coming w/ 5.5 but no release date announced) and it's unclear how well it'll work - basically, unless you would rather be fighting w/ hardware than playing around w/ ML, it's probably best to avoid (the older RDNA cards also don't have tensor cores, so perf would be hobbled even if you could get things running. Lots of software has been written w/ CUDA-only in mind).
- BeefWellington 4y ago
  FWIW I had no real issues getting StableDiffusion to run on a 6800 I have in one of my systems. I haven't tried with LLaMA at all.
  - Auracle 4y ago
    Running is one thing. Running with any sort of speed is another.
  - lhl 4y ago
    6800 is RDNA2, not RDNA3. The latter is still waiting for ROCm support 4 months post-launch: https://github.com/RadeonOpenCompute/ROCm/issues/1813 https://github.com/RadeonOpenCompute/ROCm/issues/1813
Tepix 4y ago
Read Tim Dettmer's guide. Basically, you want nVidia, and you want lots of VRAM. Buy used for much more bang for the buck. Depending on your budget, get: - an RTX 3060 with 12GB or - 1 used RTX 3090 with 24GB (approx twice as expensive as the 3060 but twice the VRAM and much faster) or - 2 used RTX 3090 cards if you need more than 24GB. Everything beyond that gets quite a bit more expensive because then you need a platform with more PCIe lanes, you may need more than one PSU and you will have problems fitting and cooling everything. With two cards and 2x24GB you can run the largest version of the LLaMA model (the 65B variant) and all its descendants with 4-bit quantization inside your GPU's VRAM, i.e. with good performance. Can can also try some low resource fine-tuning variants (LoRa etc). Oh and while you're at it also get a decent amount of RAM like 64GB or 128GB (it's very cheap right now) and a NVMe SSD. These models are quite large.
boppo1 4y ago
3090 or 4090. That's it as far as LLMs go. You can run 13b on a 4080 but 13B sucks compared to 30B. And ROCm is a poorly supported hair-pulling nightmare.

by causi 4y ago

avereveard 4y ago

Depends on whether you just want to play around or have actual business needs, but to play around I'm waiting for the next generation of unified memory architectures

ilteris 4y ago
Can you point any source re: next generation of unified memory architectures? How far are they away?
- avereveard 4y ago
  Well currently there's just apple and they are super secretive about their roadmap, the M1/2 top line chip already perform quite well, and while the top version look expensive, a pair of a6000 will definitely eat more money both to purchase and to run. M3 is around the corner tho, and there's some announcement to come from intel or arm following their partnership. There's also the new card coming from intel that is supposed to be aimed squarely at machine learning workloads, and they don't have to segment their market by memory sizing like Nvidia do, but they aren't well supported as device targets, but a pair of these will likely be very cost effective if and only if they will get credible compatibility with the libraries and models

redox99 4y ago

It's pretty simple actually. Get a 3090 or 4090. Forget about AMD.

mewpmewp2 4y ago
Should I get a gaming PC with 4090 if I want to get into personal side projects in ML and also games? Do I need dualboot? Or is Windows good?
- thewataccount 4y ago
  Windows generally works but there may be a somewhat small performance hit. IMO linux is much easier to get to work judging by all the github issue threads I see able SD/LLaMa stuff on windows - but I don't use windows so I dont have personal experience. 4090 24GB is 1800USD, The Ada A6000 48GB is like 8000USD and idk where you buy it? So if you want to run games and models locally the 4090 is honestly the best option. EDIT: I forgot - there is a rumored 4090ti with 48gb of vram, no idea if thats worth waiting for.
  - macrolime 4y ago
    The A6000 is actually the old generation, Ampere. The new Ada generation one is called 6000. Seems many places still sell A6000 (Ampere) for the same price as RTX 6000 (Ada) though, even though the new one is twice as fast. Seems you can get used RTX A6000s for around $3000 on ebay.
    - thewataccount 4y ago
      That.... That explains why I can't find it and makes a ton of sense..... I think that's such a silly name for it, but oh well Thanks for the correction!
      - macrolime 4y ago
        Just to add to the confusion, there's another older RTX 6000 with 24GB ram. This is from an even older generation, same as the GeForce 20 series.
        4y ago

dannyw 4y ago

Definitely NVIDIA. Good luck running any real ML models on AMD GPUs.

thewataccount 4y ago

Nvidia and the highest amount of vram you can get. Currently the 4090, the rumor is the 4090ti will have 48gb of vram, idk if its worth waiting or not. The more VRAM the higher paremeter count you can run all in memory (fastest by far). AMD is almost a joke in ML. The lack of CUDA support (which is nvidia proprietary) is straight lethal, and also even though ROCM does have much better support these days, from what I've seen it's still a fraction of the performance of what it should be. I'm also not sure if you need projects to support it or not, I know pytorch has backend support for it but I'm not sure how easy it is to drop in.

causi 4y ago
That's good to know. A lot of people might look at the 7900XTX with its 24GB of ram for a grand and think "ah, a good deal for fitting an LLM in".
brandall10 4y ago
I think the most recent rumors were amended to it having 24, unfortunately.
- thewataccount 4y ago
  Darn. I mean in all honestly there's no reason a gaming card would need 48gb at the moment when so few games even use 24gb. 48GB really only makes sense for workstation cards.

seydor 4y ago

At least for llama derivatives, reddit has a list https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/ https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_...

l33tman 4y ago

You don't need a GPU, alpaca.cpp runs purely on the CPU if you have 32+ GB of RAM for the 30B Llama model. It's a bit slow, but you can toy with it..

mk_stjames 4y ago

Ask: how many parameters is the model? At how many bits? and has it been quantized to a smaller number of bits or if not can it be without losing quality? multiply number of bits by the number of parameters, convert to GB, and then add so amount of overhead for however much else needs to load and/or how horrible/amazing of an implementation the inference code is. Done. Example: A 30B parameter model trained at 16bit FP gets quantized down to 4 bit ints. 4 bits = 0.5 byte. 30 billion * 0.5 byte = 15GB of VRAM (plus a GB or few of other overhead) For more real world discussion see https://github.com/ggerganov/llama.cpp/issues/13 https://github.com/ggerganov/llama.cpp/issues/13

lhl 4y ago

For a general guide, I recommend: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni... There's a subreddit r/LocalLLaMA that seems like the most active community focused on self-hosting LLMs. Here's a recent discussion on hardware: https://www.reddit.com/r/LocalLLaMA/comments/12lynw8/is_anyone_else_super_eager_to_upgrade_their/ https://www.reddit.com/r/LocalLLaMA/comments/12lynw8/is_anyo... If you're looking just for local inference, you're best bet is probably to buy a consumer GPU w/ 24GB of RAM (3090 is fine, 4090 more performance potential), which can fit a 30B parameter 4-bit quantized model that can probably be fine-tuned to ChatGPT (3.5) level quality. If not, then you can probably add a second card later on. Alternatively, if you have an Apple Silicon Mac, llama.cpp performs surprisingly well, it's easy to try for free: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Current AMD consumer cards have terrible software support and IMO isn't really an option. On Windows you might be able to use SHARK or DirectML ports, but nothing will run out of the box. ROCm still has no RDNA3 support (supposedly coming w/ 5.5 but no release date announced) and it's unclear how well it'll work - basically, unless you would rather be fighting w/ hardware than playing around w/ ML, it's probably best to avoid (the older RDNA cards also don't have tensor cores, so perf would be hobbled even if you could get things running. Lots of software has been written w/ CUDA-only in mind).

BeefWellington 4y ago
FWIW I had no real issues getting StableDiffusion to run on a 6800 I have in one of my systems. I haven't tried with LLaMA at all.
- Auracle 4y ago
  Running is one thing. Running with any sort of speed is another.
- lhl 4y ago
  6800 is RDNA2, not RDNA3. The latter is still waiting for ROCm support 4 months post-launch: https://github.com/RadeonOpenCompute/ROCm/issues/1813 https://github.com/RadeonOpenCompute/ROCm/issues/1813

Tepix 4y ago

Read Tim Dettmer's guide. Basically, you want nVidia, and you want lots of VRAM. Buy used for much more bang for the buck. Depending on your budget, get: - an RTX 3060 with 12GB or - 1 used RTX 3090 with 24GB (approx twice as expensive as the 3060 but twice the VRAM and much faster) or - 2 used RTX 3090 cards if you need more than 24GB. Everything beyond that gets quite a bit more expensive because then you need a platform with more PCIe lanes, you may need more than one PSU and you will have problems fitting and cooling everything. With two cards and 2x24GB you can run the largest version of the LLaMA model (the 65B variant) and all its descendants with 4-bit quantization inside your GPU's VRAM, i.e. with good performance. Can can also try some low resource fine-tuning variants (LoRa etc). Oh and while you're at it also get a decent amount of RAM like 64GB or 128GB (it's very cheap right now) and a NVMe SSD. These models are quite large.

boppo1 4y ago

3090 or 4090. That's it as far as LLMs go. You can run 13b on a 4080 but 13B sucks compared to 30B. And ROCm is a poorly supported hair-pulling nightmare.