3 ms·

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you

"A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model." Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able to run models locally though.

jbellis 4mo ago
That's a reasonable option, just be aware that you get about 1/3 as much memory bandwidth with the M5 Pro, or 2/3 with the M5 Max [now you're at $4100 for the lowest-end]. So both your prefill (flops-bound, M5 has a lot less) and decode (bw-bound) will be slower.
LeBit 4mo ago
I’m an idiot who is unable to project itself in situations I’ve never experienced before. So, I always thought local LLMs were toys not worth pursuing. Only once have I tried something decent like Gemma 4 31B and Qwen 3.6 27B did I realize how incredibly useful they are. You stop fearing you are sharing sensitive information. You stop fearing you will run out of tokens. You stop fearing about the availability of the remote AI. Local LLMs are extremely valuable.
- bityard 4mo ago
  *for many tasks
Aurornis 4mo ago
I have an M5 MacBook Pro and I also have a separate GPU setup for running models. The difference in speed is significant. It's not just token generation speed, but time to first token (prompt processing). The M5 hardware is amazing for what it is, but GPUs are still so much faster. Running the models on the GPU box also means I can use the laptop on my lap instead of turning it into a hot plate.
- amelius 4mo ago
  What is your GPU setup?
  - brcmthrowaway 4mo ago
    tinygpu kernel driver
boredatoms 4mo ago
The standalone mini/studio is better if you dont want to have a constantly hot laptop Get a regular laptop and use the network to access the LLM
amelius 4mo ago
You can also buy a Jetson Orin with 64GB of unified memory.
WithinReason 4mo ago
I'm running Qwen3.6-27B on a single 24GB GPU at 80 tok/s, you don't even need 2 of them
- npodbielski 4mo ago
  Yeah but 4 bits very often loops needlessly. Which is not that bad because you do not pay for tokens. But you paid for hardware and you want use it for something useful. Q6 is better but then you have like 40t/s prefill. Which is really tiring. But at least it says sorry when you ask it what is wrong! I heard there is some extension for PI preventing that. I need to look into it. Otherwise I am quite happy.
  - Der_Einzige 4mo ago
    You can fix looping with proper repetition penalties. Turn on the one called “DRY” that PeW invented and got merged into llama cpp
    - npodbielski 4mo ago
      I added repetition penalty of two. I do not know. Maybe it is not applied correctly somehow by llama swap, that I am using, but I do not consider it that much of a nuisance to so I did not tried to fix it yet. Do you have this DRY docs?
  - Zambyte 4mo ago
    "Very often" sounds like a lot more than I would say. I've been using Qwen 3.6 27b Q4 in Pi (with out any anti-looping extension) daily for weeks now, and I've had it get stuck in an infinite loop maybe 3 or 4 times.
mips_avatar 4mo ago
The cool thing about the 3090s is the RAM bandwidth. Token generation is mostly bottlenecked on memory bandwidth. Dual 3090s have 1.87 TB/s memory bandwidth (0.936 TB/s each), vs the M5 Macbook pro with only 0.3 TB/s (max chip has up to 0.63 TB/s but it's a $10k machine at that config). This translates to qwen 27b actually working fast enough for useful work on dual 3090s and being painfully slow on Macbook Pros. Also if you're running a big model on a macbook pro the UI gets laggy and the keyboard gets hot. Much better to run dual 3090s in your basement and connect to them from your Macbook.
- CobaltFire 4mo ago
  $4.8k for 48GB Max (what the parent said). Half of your quote. Even a 128GB is $6.8k today. Still only 2/3 your quote. Bandwidth is relevant (I have both a 5090 and an M4 Max 128GB Studio, so have direct comparison right here), but quote the cost appropriately!
  - mips_avatar 4mo ago
    You need the 128gb ram config to get the 614 GB/s bandwidth (which is $6999), you could skip out on upgrading the storage to save money but at that point I think most people upgrade the storage too at which point it's $8-10k + tax.
    - CobaltFire 4mo ago
      No? Any M5 Max with the upgraded GPU has the full bandwidth, which includes the 48GB model the original poster mentioned. Same as the M4 Max, where only the trimmed part had a lower bandwidth. Why are you throwing in extra cost for something thats not necessary? I know multiple people with 128GB Macs and none of us upgraded the storage. Especially not on a Studio (which isn't currently available). I will say that their $3k number is off. I somehow missed that, and its too low.
      - mips_avatar 4mo ago
        I made a mistake, there is a $5k config with high memory bandwidth. The Max chip has two tiers (I incorrectly thought the tiers were based on memory capacity), you need the higher tier Max GPU upgrade (+$300) to get the 614 GB/s memory bandwidth but you don't need to upgrade the RAM to 128 GB to get the full memory bandwidth. So to get the 614 GB/s you need to upgrade to the max chip + upgraded GPU, but you can spec it at only 48gb if you want. So the total for an m5 max with 614 GB/s memory bandwidth is $4999-$9999 depending on config. Still 3x lower memory Bandwidth than a dual 3090 setup which you can build for $3k with parts from facebook marketplace and run in your basement.
satvikpendem 4mo ago
To summarize a video I saw recently [0] rebutting your arguments, MacBooks can get crazy slow when running local models or even just Claude Code and Codex due to their poor implementation, to the point that the laptop itself becomes unusable. There are other arguments for running an ssh-able box in a closet somewhere too as with KVMs you can give an agent remote control over the machine itself such that it has vastly more capabilities than if it were controlling its own machine it's running on, as well as not needing to keep the MacBook open all the time just to have the agent finish running. [0] https://youtu.be/9tGrhrVKCrE https://youtu.be/9tGrhrVKCrE

jbellis 4mo ago

That's a reasonable option, just be aware that you get about 1/3 as much memory bandwidth with the M5 Pro, or 2/3 with the M5 Max [now you're at $4100 for the lowest-end]. So both your prefill (flops-bound, M5 has a lot less) and decode (bw-bound) will be slower.

LeBit 4mo ago

I’m an idiot who is unable to project itself in situations I’ve never experienced before. So, I always thought local LLMs were toys not worth pursuing. Only once have I tried something decent like Gemma 4 31B and Qwen 3.6 27B did I realize how incredibly useful they are. You stop fearing you are sharing sensitive information. You stop fearing you will run out of tokens. You stop fearing about the availability of the remote AI. Local LLMs are extremely valuable.

bityard 4mo ago
*for many tasks

Aurornis 4mo ago

I have an M5 MacBook Pro and I also have a separate GPU setup for running models. The difference in speed is significant. It's not just token generation speed, but time to first token (prompt processing). The M5 hardware is amazing for what it is, but GPUs are still so much faster. Running the models on the GPU box also means I can use the laptop on my lap instead of turning it into a hot plate.

amelius 4mo ago
What is your GPU setup?
- brcmthrowaway 4mo ago
  tinygpu kernel driver

boredatoms 4mo ago

The standalone mini/studio is better if you dont want to have a constantly hot laptop Get a regular laptop and use the network to access the LLM

amelius 4mo ago

You can also buy a Jetson Orin with 64GB of unified memory.

WithinReason 4mo ago

I'm running Qwen3.6-27B on a single 24GB GPU at 80 tok/s, you don't even need 2 of them

npodbielski 4mo ago
Yeah but 4 bits very often loops needlessly. Which is not that bad because you do not pay for tokens. But you paid for hardware and you want use it for something useful. Q6 is better but then you have like 40t/s prefill. Which is really tiring. But at least it says sorry when you ask it what is wrong! I heard there is some extension for PI preventing that. I need to look into it. Otherwise I am quite happy.
- Der_Einzige 4mo ago
  You can fix looping with proper repetition penalties. Turn on the one called “DRY” that PeW invented and got merged into llama cpp
  - npodbielski 4mo ago
    I added repetition penalty of two. I do not know. Maybe it is not applied correctly somehow by llama swap, that I am using, but I do not consider it that much of a nuisance to so I did not tried to fix it yet. Do you have this DRY docs?
- Zambyte 4mo ago
  "Very often" sounds like a lot more than I would say. I've been using Qwen 3.6 27b Q4 in Pi (with out any anti-looping extension) daily for weeks now, and I've had it get stuck in an infinite loop maybe 3 or 4 times.

mips_avatar 4mo ago

The cool thing about the 3090s is the RAM bandwidth. Token generation is mostly bottlenecked on memory bandwidth. Dual 3090s have 1.87 TB/s memory bandwidth (0.936 TB/s each), vs the M5 Macbook pro with only 0.3 TB/s (max chip has up to 0.63 TB/s but it's a $10k machine at that config). This translates to qwen 27b actually working fast enough for useful work on dual 3090s and being painfully slow on Macbook Pros. Also if you're running a big model on a macbook pro the UI gets laggy and the keyboard gets hot. Much better to run dual 3090s in your basement and connect to them from your Macbook.

CobaltFire 4mo ago
$4.8k for 48GB Max (what the parent said). Half of your quote. Even a 128GB is $6.8k today. Still only 2/3 your quote. Bandwidth is relevant (I have both a 5090 and an M4 Max 128GB Studio, so have direct comparison right here), but quote the cost appropriately!
- mips_avatar 4mo ago
  You need the 128gb ram config to get the 614 GB/s bandwidth (which is $6999), you could skip out on upgrading the storage to save money but at that point I think most people upgrade the storage too at which point it's $8-10k + tax.
  - CobaltFire 4mo ago
    No? Any M5 Max with the upgraded GPU has the full bandwidth, which includes the 48GB model the original poster mentioned. Same as the M4 Max, where only the trimmed part had a lower bandwidth. Why are you throwing in extra cost for something thats not necessary? I know multiple people with 128GB Macs and none of us upgraded the storage. Especially not on a Studio (which isn't currently available). I will say that their $3k number is off. I somehow missed that, and its too low.
    - mips_avatar 4mo ago
      I made a mistake, there is a $5k config with high memory bandwidth. The Max chip has two tiers (I incorrectly thought the tiers were based on memory capacity), you need the higher tier Max GPU upgrade (+$300) to get the 614 GB/s memory bandwidth but you don't need to upgrade the RAM to 128 GB to get the full memory bandwidth. So to get the 614 GB/s you need to upgrade to the max chip + upgraded GPU, but you can spec it at only 48gb if you want. So the total for an m5 max with 614 GB/s memory bandwidth is $4999-$9999 depending on config. Still 3x lower memory Bandwidth than a dual 3090 setup which you can build for $3k with parts from facebook marketplace and run in your basement.

satvikpendem 4mo ago

To summarize a video I saw recently [0] rebutting your arguments, MacBooks can get crazy slow when running local models or even just Claude Code and Codex due to their poor implementation, to the point that the laptop itself becomes unusable. There are other arguments for running an ssh-able box in a closet somewhere too as with KVMs you can give an agent remote control over the machine itself such that it has vastly more capabilities than if it were controlling its own machine it's running on, as well as not needing to keep the MacBook open all the time just to have the agent finish running. [0] https://youtu.be/9tGrhrVKCrE https://youtu.be/9tGrhrVKCrE