3 ms·

Simon, what is your local GPU setup? (No doubt you've covered this, but I'm not sure where to dig up).

by tomrod 2y ago

simonw 2y ago
MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.
- Elucalidavah 2y ago
  > MacBook Pro M2 with 64GB of RAM Are there non-mac options with similar capabilities?
  - simonw 2y ago
    Yes, but I don't really know anything about those. https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ is full of people running models on PCs with NVIDIA cards. The unique benefit of an Apple Silicon Mac at the moment is that the 64GB of RAM is available to both the GPU and the CPU at once. With other hardware you usually need dedicated separate VRAM for the GPU.
  - _neil 2y ago
    It’s not out yet, but the upcoming Framework desktop [0] is supposed to have a similar unified memory setup. [0] https://frame.work/desktop https://frame.work/desktop
  - dwood_dev 2y ago
    Anything with the Radeon 8060S/Ryzen AI Max+ 395. One of the popular MiniPC Chinese brands has them for preorder[0] with shipping starting May 7th. Framework also has them, but shipping Q3. 0: https://www.gmktec.com/products/prepaid-deposit-amd-ryzen™-ai-max-395-evo-x2-ai-mini-pc https://www.gmktec.com/products/prepaid-deposit-amd-ryzen™-a...
    - chpatrick 2y ago
      I've never been able to get ROCm working reliably personally.
  - danans 2y ago
    Nvidia Orin AGX if a desktop form factor works for you.
  - chpatrick 2y ago
    I remember seeing a post about someone running the full size DeepSeek model in a dual-Xeon server with a ton of RAM.
- jychang 2y ago
  MLX is slower than GGUFs on Macs. On my M1 Max macbook pro, the GGUF version bartowski/google_gemma-3-27b-it-qat-GGUF is 15.6gb and runs at 17tok/sec, whereas mlx-community/gemma-3-27b-it-qat-4bit is 16.8gb and runs at 15tok/sec. Note that both of these are the new QAT 4bit quants.
  - phaedrix 2y ago
    No, in general mlx versions are always faster, ice tested most of them.
    - 85392_school 2y ago
      What TPS difference are you getting?

Simon, what is your local GPU setup? (No doubt you've covered this, but I'm not sure where to dig up).

by tomrod 2y ago

Simon, what is your local GPU setup? (No doubt you've covered this, but I'm not sure where to dig up).

MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.

Elucalidavah 2y ago
> MacBook Pro M2 with 64GB of RAM Are there non-mac options with similar capabilities?
- simonw 2y ago
  Yes, but I don't really know anything about those. https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ is full of people running models on PCs with NVIDIA cards. The unique benefit of an Apple Silicon Mac at the moment is that the 64GB of RAM is available to both the GPU and the CPU at once. With other hardware you usually need dedicated separate VRAM for the GPU.
- _neil 2y ago
  It’s not out yet, but the upcoming Framework desktop [0] is supposed to have a similar unified memory setup. [0] https://frame.work/desktop https://frame.work/desktop
- dwood_dev 2y ago
  Anything with the Radeon 8060S/Ryzen AI Max+ 395. One of the popular MiniPC Chinese brands has them for preorder[0] with shipping starting May 7th. Framework also has them, but shipping Q3. 0: https://www.gmktec.com/products/prepaid-deposit-amd-ryzen™-ai-max-395-evo-x2-ai-mini-pc https://www.gmktec.com/products/prepaid-deposit-amd-ryzen™-a...
  - chpatrick 2y ago
    I've never been able to get ROCm working reliably personally.
- danans 2y ago
  Nvidia Orin AGX if a desktop form factor works for you.
- chpatrick 2y ago
  I remember seeing a post about someone running the full size DeepSeek model in a dual-Xeon server with a ton of RAM.
jychang 2y ago
MLX is slower than GGUFs on Macs. On my M1 Max macbook pro, the GGUF version bartowski/google_gemma-3-27b-it-qat-GGUF is 15.6gb and runs at 17tok/sec, whereas mlx-community/gemma-3-27b-it-qat-4bit is 16.8gb and runs at 15tok/sec. Note that both of these are the new QAT 4bit quants.
- phaedrix 2y ago
  No, in general mlx versions are always faster, ice tested most of them.
  - 85392_school 2y ago
    What TPS difference are you getting?