5 ms·

I think your take is a bit optimistic. I like quantization as much as the next person, but even the 2-bit model won’t fit entirely on a 4090: https://huggingfac

I think your take is a bit optimistic. I like quantization as much as the next person, but even the 2-bit model won’t fit entirely on a 4090: https://huggingface.co/TheBloke/Llama-2-70B-GGUF https://huggingface.co/TheBloke/Llama-2-70B-GGUF I would be uncomfortable recommending less than 4-bit quantization on a non-MoE model, which is ~40GB on a 70B model.

nox101 3y ago
fortunately it will run on my UMA mac. it's made me curious what the trade offs are. Would I be better off with a 4090 or a Mac with 128+gig of uma memory
- coder543 3y ago
  Even the M3 Max seems to be slower than my 3090 for LLMs that fit onto the 3090, but it’s hard to find comprehensive numbers. The primary advantage is that you can spec out more memory with the M3 Max to fit larger models, but with the exception of CodeLlama-70B today, it really seems like the trend is for models to be getting smaller and better, not bigger. Mixtral runs circles around Llama2-70B and arguably ChatGPT-3.5. Mistral-7B often seems fairly close to Llama2-70B. Microsoft accidentally leaked that ChatGPT-3.5-Turbo is apparently only 20B parameters. 24GB of VRAM is enough to run ~33B parameter models, and enough to run Mixtral (which is a MoE, which makes direct comparisons to “traditional” LLMs a little more confusing.) I don’t think there’s a clear answer of what hardware someone should get. It depends. Should you give up performance on the models most people run locally in hopes of running very large models, or give up the ability to run very large models in favor of prioritizing performance on the models that are popular and proven today?
  - int_19h 3y ago
    M3 Max is actually less than ideal because it peaks at 400 Gb/s for memory. What you really want is M1 or M2 Ultra, which offers up to 800 Gb/s (for comparison, RTX 3090 runs at 936 GB/s). A Mac Studio suitable for running 70B models with speeds fast enough for realtime chat can be had for ~$3K The downside of Apple's hardware at the moment is that the training ecosystem is very much focused on CUDA; llama.cpp has an open issue about Metal-accelerated training: https://github.com/ggerganov/llama.cpp/issues/3799 https://github.com/ggerganov/llama.cpp/issues/3799 - but no work on it so far. This is likely because training at any significant sizes requires enough juice that it's pretty much always better to do it in the cloud currently, where, again, CUDA is the well-established ecosystem, and it's cheaper and easier for datacenter operators to scale. But, in principle, much faster training on Apple hardware should be possible, and eventually someone will get it done.
    - coder543 3y ago
      Yep, I seriously considered a Mac Studio a few months ago when I was putting together an “AI server” for home usage, but I had my old 3090 just sitting around, and I was ready to upgrade the CPU on my gaming desktop… so then I had that desktop’s previous CPU. I just had too many parts already, and it deeply annoys me that Apple won’t put standard, user-upgradable NVMe SSDs on their desktops. Otherwise, the Mac Studio is a very appealing option for sure.
- zten 3y ago
  Well, the workstation-class equivalent of a 4090 -- RTX 6000 Ada -- has enough RAM to work with a quantized model, but it'll blow away anyone's budget at anywhere between $7,000 and $10,000.
Eisenstein 3y ago
The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. It will be slower, but waiting a couple minutes for a prompt response isn't the worst thing if you are the type that would get use out of a local 70b parameter model. Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.
- coder543 3y ago
  > The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. No… that’s not such a great thing. Helpful in a pinch, but if you’re not running at least 70% of your layers on the GPU, then you barely get any benefit from the GPU in my experience. The vast gulf in performance between the CPU and GPU means that the GPU is just spinning its wheels waiting on the CPU. Running half of a model on the GPU is not useful. > Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit. I agree with this, if someone has a desktop that can fit two GPUs.
  - deleted 3y ago
    [deleted]
  - sp332 3y ago
    The main benefit of a GPU in that case is much faster prompt reading. Could be useful for Code Llama cases where you want the model to read a lot of code and then write a line or part of a line.
  - zten 3y ago
    Multi-GPU in desktop chassis gets crazy pretty quickly. If you don't care about aesthetics and can figure out both the power delivery and PCI-E lane situation, https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni... has an example that will make a Bitcoin mining rig look clean. Water cooling can get you down to 2x slot height, with all of the trouble involved in water cooling. NVIDIA really segmented the market quite well. Gamers hate blower cards, but they are the right physical dimensions to make multi-GPU work well, and they are exclusively on the workstation cards.

nox101 3y ago

fortunately it will run on my UMA mac. it's made me curious what the trade offs are. Would I be better off with a 4090 or a Mac with 128+gig of uma memory

coder543 3y ago
Even the M3 Max seems to be slower than my 3090 for LLMs that fit onto the 3090, but it’s hard to find comprehensive numbers. The primary advantage is that you can spec out more memory with the M3 Max to fit larger models, but with the exception of CodeLlama-70B today, it really seems like the trend is for models to be getting smaller and better, not bigger. Mixtral runs circles around Llama2-70B and arguably ChatGPT-3.5. Mistral-7B often seems fairly close to Llama2-70B. Microsoft accidentally leaked that ChatGPT-3.5-Turbo is apparently only 20B parameters. 24GB of VRAM is enough to run ~33B parameter models, and enough to run Mixtral (which is a MoE, which makes direct comparisons to “traditional” LLMs a little more confusing.) I don’t think there’s a clear answer of what hardware someone should get. It depends. Should you give up performance on the models most people run locally in hopes of running very large models, or give up the ability to run very large models in favor of prioritizing performance on the models that are popular and proven today?
- int_19h 3y ago
  M3 Max is actually less than ideal because it peaks at 400 Gb/s for memory. What you really want is M1 or M2 Ultra, which offers up to 800 Gb/s (for comparison, RTX 3090 runs at 936 GB/s). A Mac Studio suitable for running 70B models with speeds fast enough for realtime chat can be had for ~$3K The downside of Apple's hardware at the moment is that the training ecosystem is very much focused on CUDA; llama.cpp has an open issue about Metal-accelerated training: https://github.com/ggerganov/llama.cpp/issues/3799 https://github.com/ggerganov/llama.cpp/issues/3799 - but no work on it so far. This is likely because training at any significant sizes requires enough juice that it's pretty much always better to do it in the cloud currently, where, again, CUDA is the well-established ecosystem, and it's cheaper and easier for datacenter operators to scale. But, in principle, much faster training on Apple hardware should be possible, and eventually someone will get it done.
  - coder543 3y ago
    Yep, I seriously considered a Mac Studio a few months ago when I was putting together an “AI server” for home usage, but I had my old 3090 just sitting around, and I was ready to upgrade the CPU on my gaming desktop… so then I had that desktop’s previous CPU. I just had too many parts already, and it deeply annoys me that Apple won’t put standard, user-upgradable NVMe SSDs on their desktops. Otherwise, the Mac Studio is a very appealing option for sure.
zten 3y ago
Well, the workstation-class equivalent of a 4090 -- RTX 6000 Ada -- has enough RAM to work with a quantized model, but it'll blow away anyone's budget at anywhere between $7,000 and $10,000.

Eisenstein 3y ago

The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. It will be slower, but waiting a couple minutes for a prompt response isn't the worst thing if you are the type that would get use out of a local 70b parameter model. Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.

coder543 3y ago
> The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. No… that’s not such a great thing. Helpful in a pinch, but if you’re not running at least 70% of your layers on the GPU, then you barely get any benefit from the GPU in my experience. The vast gulf in performance between the CPU and GPU means that the GPU is just spinning its wheels waiting on the CPU. Running half of a model on the GPU is not useful. > Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit. I agree with this, if someone has a desktop that can fit two GPUs.
- deleted 3y ago
  [deleted]
- sp332 3y ago
  The main benefit of a GPU in that case is much faster prompt reading. Could be useful for Code Llama cases where you want the model to read a lot of code and then write a line or part of a line.
- zten 3y ago
  Multi-GPU in desktop chassis gets crazy pretty quickly. If you don't care about aesthetics and can figure out both the power delivery and PCI-E lane situation, https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni... has an example that will make a Bitcoin mining rig look clean. Water cooling can get you down to 2x slot height, with all of the trouble involved in water cooling. NVIDIA really segmented the market quite well. Gamers hate blower cards, but they are the right physical dimensions to make multi-GPU work well, and they are exclusively on the workstation cards.