5 ms·

32B is a good choice of size, as it allows running on a 24GB consumer card at ~4 bpw (RTX 3090/4090) while using most of the VRAM. Unlike llama 3.1, which had 8

32B is a good choice of size, as it allows running on a 24GB consumer card at ~4 bpw (RTX 3090/4090) while using most of the VRAM. Unlike llama 3.1, which had 8b, 70B (much too big to fit), and 405B.

Svoka 2y ago
what do you mean? I can easily run 70b on my macbook. Fits easily.
- diggan 2y ago
  > what do you mean? I can easily run 70b on my macbook. Fits easily Does your MacBook really have a 24GB VRAM consumer (GPU) card?
  - rcxdude 2y ago
    Newer mac books have a unified memory architecture, so the GPU can use all the available memory if it wants.
    - blincoln 2y ago
      There's an important caveat, though. The software you're working with needs to support Metal/MPS, or you'll end up using the CPU instead of the GPU. For example, as of a few weeks ago, PyTorch still didn't support nested tensors on Metal. Probably not a deal-breaker for a lot of folks, but important to figure out before one spends thousands on hardware.
- alwayslikethis 2y ago
  Macs that can run it are quite a bit more expensive than a 3090. GPUs can also do finetuning and run other models with larger batch sizes which Macs would struggle with. Also, for the models that fit both, an nvidia card can run it much faster.
  - seanmcdirmid 2y ago
    It’s really not that expensive to get an M4 pro with 64GB. But you really want a Max or ultra, and you aren’t going to be able to do much with images. You are limited on the GPU part. 3090s aren’t sold anymore, so you have to scrounge for them in the used market, 4090s are a couple K? The local LLM scene is still pricey and confusing.
    - ponector 2y ago
      >>not that expensive to get an M4 pro with 64GB $3900 as the starting price does not sound 'not that expensive' for me.
      - seanmcdirmid 2y ago
        M4 Pro with 64GB of ram starts at $3900? It doesn't take that much searching to know that is BS: https://www.apple.com/shop/buy-mac/mac-mini/apple-m4-pro-chip-with-12-core-cpu-16-core-gpu-24gb-memory-512gb https://www.apple.com/shop/buy-mac/mac-mini/apple-m4-pro-chi... An upgraded M4 Pro mac mini to 64GB is $1999. You'll probably want the extra GPUs, though, bringing you up to $2199. But you probably want the max (not offered for the mini) or ultra to do serious LLM work. An M4 Max in a 14 inch MBP with 64GB of ram will cost $3900, but that's a whole step up from the M4 pro. I'm waiting to see what they do with the Mac Studio refresh (I don't need a max in a portable, and I'm betting I can get an M4 Ultra with 64GB ram for $4000 or so).
        alwayslikethis 2y ago
        You're also going to want some more SSD space, because working with these model files on a 512 GB root is going to quickly become a problem. If you get the 2TB option, that's another $600.
        seanmcdirmid 2y ago
        Thunderbolt 5 mostly takes care of the storage problem.
- do_not_redeem 2y ago
  Yes and I can fit 70b in system RAM on my linux box. But parent was talking about discrete GPUs, and shared memory architectures like Intel's integrated GPUs or Apple's unified memory will never perform as well as dedicated GPU memory.
  - zozbot234 2y ago
    > shared memory architectures like Intel's integrated GPUs or Apple's unified memory can't compare to dedicated GPU memory. It depends. For the initial prompt+context processing, iGPUs (and NPUs) are still a win because you're compute bound at that stage. For token generation, CPU is typically faster than iGPU (or even plausibly NPU) for local quantized models because it can make better use of all available memory bandwidth, with no wasteful padding reducing your usable throughput.
  - alwayslikethis 2y ago
    It is slower, but I would say fairly usable for just LLM inference (still much faster than you can read). I'd say it's more of a matter of control. You can be reasonably sure that your linux box is free from ordinary forms of telemetry, which is hard to do with a Mac.
qeternity 2y ago
IIRC Llama 3 had a 32b variant but it didn't pass red teaming and Meta have said nothing about it since.

32B is a good choice of size, as it allows running on a 24GB consumer card at ~4 bpw (RTX 3090/4090) while using most of the VRAM. Unlike llama 3.1, which had 8b, 70B (much too big to fit), and 405B.

Svoka 2y ago

what do you mean? I can easily run 70b on my macbook. Fits easily.

diggan 2y ago
> what do you mean? I can easily run 70b on my macbook. Fits easily Does your MacBook really have a 24GB VRAM consumer (GPU) card?
- rcxdude 2y ago
  Newer mac books have a unified memory architecture, so the GPU can use all the available memory if it wants.
  - blincoln 2y ago
    There's an important caveat, though. The software you're working with needs to support Metal/MPS, or you'll end up using the CPU instead of the GPU. For example, as of a few weeks ago, PyTorch still didn't support nested tensors on Metal. Probably not a deal-breaker for a lot of folks, but important to figure out before one spends thousands on hardware.
alwayslikethis 2y ago
Macs that can run it are quite a bit more expensive than a 3090. GPUs can also do finetuning and run other models with larger batch sizes which Macs would struggle with. Also, for the models that fit both, an nvidia card can run it much faster.
- seanmcdirmid 2y ago
  It’s really not that expensive to get an M4 pro with 64GB. But you really want a Max or ultra, and you aren’t going to be able to do much with images. You are limited on the GPU part. 3090s aren’t sold anymore, so you have to scrounge for them in the used market, 4090s are a couple K? The local LLM scene is still pricey and confusing.
  - ponector 2y ago
    >>not that expensive to get an M4 pro with 64GB $3900 as the starting price does not sound 'not that expensive' for me.
    - seanmcdirmid 2y ago
      M4 Pro with 64GB of ram starts at $3900? It doesn't take that much searching to know that is BS: https://www.apple.com/shop/buy-mac/mac-mini/apple-m4-pro-chip-with-12-core-cpu-16-core-gpu-24gb-memory-512gb https://www.apple.com/shop/buy-mac/mac-mini/apple-m4-pro-chi... An upgraded M4 Pro mac mini to 64GB is $1999. You'll probably want the extra GPUs, though, bringing you up to $2199. But you probably want the max (not offered for the mini) or ultra to do serious LLM work. An M4 Max in a 14 inch MBP with 64GB of ram will cost $3900, but that's a whole step up from the M4 pro. I'm waiting to see what they do with the Mac Studio refresh (I don't need a max in a portable, and I'm betting I can get an M4 Ultra with 64GB ram for $4000 or so).
      - alwayslikethis 2y ago
        You're also going to want some more SSD space, because working with these model files on a 512 GB root is going to quickly become a problem. If you get the 2TB option, that's another $600.
        seanmcdirmid 2y ago
        Thunderbolt 5 mostly takes care of the storage problem.
do_not_redeem 2y ago
Yes and I can fit 70b in system RAM on my linux box. But parent was talking about discrete GPUs, and shared memory architectures like Intel's integrated GPUs or Apple's unified memory will never perform as well as dedicated GPU memory.
- zozbot234 2y ago
  > shared memory architectures like Intel's integrated GPUs or Apple's unified memory can't compare to dedicated GPU memory. It depends. For the initial prompt+context processing, iGPUs (and NPUs) are still a win because you're compute bound at that stage. For token generation, CPU is typically faster than iGPU (or even plausibly NPU) for local quantized models because it can make better use of all available memory bandwidth, with no wasteful padding reducing your usable throughput.
- alwayslikethis 2y ago
  It is slower, but I would say fairly usable for just LLM inference (still much faster than you can read). I'd say it's more of a matter of control. You can be reasonably sure that your linux box is free from ordinary forms of telemetry, which is hard to do with a Mac.

qeternity 2y ago

IIRC Llama 3 had a 32b variant but it didn't pass red teaming and Meta have said nothing about it since.