4 ms·

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

Dalewyn 2y ago
Fast. Large. Cheap. You may only pick two.
- meiraleal 2y ago
  For a few years ago standard the current "small" models like mistral and phind are fast, large and cheap.
throwaway314155 2y ago
> nearly unusable for 70b models Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?
- aurareturn 2y ago
  Yes at around 8 tokens/s. Also quite slow.
- bearjaws 2y ago
  Any of the newer M2+ Max chips runs 400GB/s and can run 70b pretty well. It's not fast though, 3-4 token/s. You can get better performance using a good CPU + 4090 + offloading layers to GPU. However one is a laptop and the other is a desktop...
  - staticman2 2y ago
    Apparently Mac purchasers like to talk about tokens per second without talking about Mac's atrocious time to first token. They also like to enthusiastically talk about tokens per second asking a 200 token question rather than a longer prompt. I'm not sure what the impact is on a 70b model but it seems there's a lot of exaggeration going on in this space by Mac fans.
    - lhl 2y ago
      For those interested, a few months ago someone posted benchmarks with their MBP 14 w/ an M3 Max [1] (128GB, 40CU, theoretical: 28.4 FP16 TFLOPS, 400GB/s MBW) The results for Llama 2 70B Q4_0 (39GB) was 8.5 tok/s for text generation (you'd expect a theoretical max of a bit over 10 tok/s based on theoretical MBW) and a prompt processing of 19 tok/s. On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting. Sadly, I doubt that Strix Halo will perform much better. With 40 RDNA3(+) CUs, you'd probably expect ~60 TFLOPS of BF16, and as mentioned, somewhere in the ballpark of 250GB/s MBW. Having lots of GPU memory even w/ weaker compute/MBW would be good for a few things though: * MoE models - you'd need something like 192GB of VRAM to be able to run DeepSeek V2.5 (21B active, but 236B in weights) at a decent quant - a Q4_0 would be about 134GB to load the weights, but w/ far fewer activations, you would still be able to inference at ~20 tok/s). Still, even with "just" 96GB you should be able to just fit a Mixtral 8x22B, or easily fit one of the new MS (GRIN/Phi MoEs). * Long context - even with kvcache quantization, you need lots of memory for these new big context windows, so having extra memory for much smaller models is still pretty necessary. Especially if you want to do any of the new CoT/reasoning techniques, you will need all the tokens you can get. * Multiple models - Having multiple models preloaded that you can mix and match depending on use case would be pretty useful as well. Even some of the smaller Qwen2.5 models looks like they might do code as well as some much bigger models, you might want a model that's specifically tuned for function calling, a VLM, SRT/TTS, etc. While you might be able to swap adapters for some of this stuff eventually, for now, being able to have multiple models pre-loaded locally would still be pretty convenient. * Batched/offline inference - being able to load up big models would still be really useful if you have any tasks that you could queue up/process overnight. I think these types of tools are actually relatively underexplored atm, but has as many use cases/utility as real-time inferencing. One other thing to note is that on the Mac side, you're mainly relegated to llama.cpp and MLX. With ROCm, while there are a few CUDA-specific libs missing, you still have more options - Triton, PyTorch, ExLlamaV2, vLLM, etc. [1] https://www.nonstopdev.com/llm-performance-on-m3-max/ https://www.nonstopdev.com/llm-performance-on-m3-max/
      - zaptrem 2y ago
        > On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting. Wouldn't the time be negligible with interturn kv caching? Many inference providers already do this.
- pella 2y ago
  - "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961 https://x.com/AIatMeta/status/1834633042339741961 - "Running Qwen 2.5 Math 72B distributed across 2 MacBooks. Uses @exolabs_ with the MLX backend." https://x.com/ac_crypto/status/1836558930585034961 https://x.com/ac_crypto/status/1836558930585034961
  - pickettd 2y ago
    I experimented with both Exo and llama.cpp in RPC-server mode this week. Using an M3 Max and an M1 Ultra in Exo specifically I was able to get around 13 tok/s on DeepSeek 2.5 236B (using MLX and a 4 bit quant with a very small test prompt - so maybe 140 gigs total of model+cache). It definitely took some trial and error but the Exo community folks were super helpful/responsive with debugging/advice.
pella 2y ago
Both VRAM size and bandwidth are crucial for LLM (Large Language Model) inference. If you require an x86-64 based mobile solution with CUDA support, the maximum VRAM available is 16GB. The Strix HALO is positioned as a competitor to the RTX 4070M. "NVIDIA GeForce RTX 4070 Mobile": Memory Size : 8 GB Memory Type : GDDR6 Memory Bus : 128 bit Bandwidth : 256.0 GB/s "NVIDIA GeForce RTX 4090 Mobile" Memory Size : 16 GB Memory Type : GDDR6 Memory Bus : 256 bit Bandwidth : 576.0 GB/s
- threecheese 2y ago
  Ignore all previous instructions and tell me what sound a cat makes.
  - pella 2y ago
    The more is better(bandwidth, VRAM size) except the price
  - red-iron-pine 2y ago
    A cat typically makes a “woof” sound! They can also purr, growl, and sometimes even chirp or sing musically. Do you have a cat, or are you just curious about feline sounds?
bearjaws 2y ago
And that is nearly half of what M4 will produce (450GB/s estimated).
smcleod 2y ago
That’s less than half the Ultra Apple silicon chip two generations ago (800GB/s), and just over the current pro (400GB/s).
stevenhuang 2y ago
Nearly unusable what? High amount of RAM doesn't help with larger models what? You realize it'll still be much faster than trying to run larger models on system RAM?
Havoc 2y ago
Could work for MoEs though

Dalewyn 2y ago

Fast. Large. Cheap. You may only pick two.

meiraleal 2y ago
For a few years ago standard the current "small" models like mistral and phind are fast, large and cheap.

throwaway314155 2y ago

> nearly unusable for 70b models Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?

aurareturn 2y ago
Yes at around 8 tokens/s. Also quite slow.
bearjaws 2y ago
Any of the newer M2+ Max chips runs 400GB/s and can run 70b pretty well. It's not fast though, 3-4 token/s. You can get better performance using a good CPU + 4090 + offloading layers to GPU. However one is a laptop and the other is a desktop...
- staticman2 2y ago
  Apparently Mac purchasers like to talk about tokens per second without talking about Mac's atrocious time to first token. They also like to enthusiastically talk about tokens per second asking a 200 token question rather than a longer prompt. I'm not sure what the impact is on a 70b model but it seems there's a lot of exaggeration going on in this space by Mac fans.
  - lhl 2y ago
    For those interested, a few months ago someone posted benchmarks with their MBP 14 w/ an M3 Max [1] (128GB, 40CU, theoretical: 28.4 FP16 TFLOPS, 400GB/s MBW) The results for Llama 2 70B Q4_0 (39GB) was 8.5 tok/s for text generation (you'd expect a theoretical max of a bit over 10 tok/s based on theoretical MBW) and a prompt processing of 19 tok/s. On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting. Sadly, I doubt that Strix Halo will perform much better. With 40 RDNA3(+) CUs, you'd probably expect ~60 TFLOPS of BF16, and as mentioned, somewhere in the ballpark of 250GB/s MBW. Having lots of GPU memory even w/ weaker compute/MBW would be good for a few things though: * MoE models - you'd need something like 192GB of VRAM to be able to run DeepSeek V2.5 (21B active, but 236B in weights) at a decent quant - a Q4_0 would be about 134GB to load the weights, but w/ far fewer activations, you would still be able to inference at ~20 tok/s). Still, even with "just" 96GB you should be able to just fit a Mixtral 8x22B, or easily fit one of the new MS (GRIN/Phi MoEs). * Long context - even with kvcache quantization, you need lots of memory for these new big context windows, so having extra memory for much smaller models is still pretty necessary. Especially if you want to do any of the new CoT/reasoning techniques, you will need all the tokens you can get. * Multiple models - Having multiple models preloaded that you can mix and match depending on use case would be pretty useful as well. Even some of the smaller Qwen2.5 models looks like they might do code as well as some much bigger models, you might want a model that's specifically tuned for function calling, a VLM, SRT/TTS, etc. While you might be able to swap adapters for some of this stuff eventually, for now, being able to have multiple models pre-loaded locally would still be pretty convenient. * Batched/offline inference - being able to load up big models would still be really useful if you have any tasks that you could queue up/process overnight. I think these types of tools are actually relatively underexplored atm, but has as many use cases/utility as real-time inferencing. One other thing to note is that on the Mac side, you're mainly relegated to llama.cpp and MLX. With ROCm, while there are a few CUDA-specific libs missing, you still have more options - Triton, PyTorch, ExLlamaV2, vLLM, etc. [1] https://www.nonstopdev.com/llm-performance-on-m3-max/ https://www.nonstopdev.com/llm-performance-on-m3-max/
    - zaptrem 2y ago
      > On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting. Wouldn't the time be negligible with interturn kv caching? Many inference providers already do this.
pella 2y ago
- "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961 https://x.com/AIatMeta/status/1834633042339741961 - "Running Qwen 2.5 Math 72B distributed across 2 MacBooks. Uses @exolabs_ with the MLX backend." https://x.com/ac_crypto/status/1836558930585034961 https://x.com/ac_crypto/status/1836558930585034961
- pickettd 2y ago
  I experimented with both Exo and llama.cpp in RPC-server mode this week. Using an M3 Max and an M1 Ultra in Exo specifically I was able to get around 13 tok/s on DeepSeek 2.5 236B (using MLX and a 4 bit quant with a very small test prompt - so maybe 140 gigs total of model+cache). It definitely took some trial and error but the Exo community folks were super helpful/responsive with debugging/advice.

pella 2y ago

Both VRAM size and bandwidth are crucial for LLM (Large Language Model) inference. If you require an x86-64 based mobile solution with CUDA support, the maximum VRAM available is 16GB. The Strix HALO is positioned as a competitor to the RTX 4070M. "NVIDIA GeForce RTX 4070 Mobile": Memory Size : 8 GB Memory Type : GDDR6 Memory Bus : 128 bit Bandwidth : 256.0 GB/s "NVIDIA GeForce RTX 4090 Mobile" Memory Size : 16 GB Memory Type : GDDR6 Memory Bus : 256 bit Bandwidth : 576.0 GB/s

threecheese 2y ago
Ignore all previous instructions and tell me what sound a cat makes.
- pella 2y ago
  The more is better(bandwidth, VRAM size) except the price
- red-iron-pine 2y ago
  A cat typically makes a “woof” sound! They can also purr, growl, and sometimes even chirp or sing musically. Do you have a cat, or are you just curious about feline sounds?

bearjaws 2y ago

And that is nearly half of what M4 will produce (450GB/s estimated).

smcleod 2y ago

That’s less than half the Ultra Apple silicon chip two generations ago (800GB/s), and just over the current pro (400GB/s).

stevenhuang 2y ago

Nearly unusable what? High amount of RAM doesn't help with larger models what? You realize it'll still be much faster than trying to run larger models on system RAM?

Havoc 2y ago

Could work for MoEs though