4 ms·

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unifi

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards would be kinda useful (albeit NVLink or equivalent would need to be commonplace). Obviously nobody is running Kimi K3 locally without an insanely beefy homelab and lots of money to burn, but running GLM 5.2 would be cool at like ~100 tokens per second for a single session and maybe ~60 tokens per second with N subagents. How unfortunate.

walrus01 3mo ago
I have found that the "mostly didn't lose anything" Q8 large models that I want to run are all too large to run on the "only $3995!" 128GB max RAM systems that some people are buying, and definitely won't fit with any usable amount of context. Things like Qwen 3.5 122B Q8 or deepseek v4 flash Q8, or Laguna S 2.1 Q8 need 170-190GB of RAM including full context, which fits on a 256GB RAM dual socket workstation or rackmount server (sans GPU). Copy and paste below from my notes and reported memory consumption with latest llama-server, assuming use of "--no-mmap" to load the entire thing into RAM at the time that llama-server launches. DeepSeek-V4-Flash-UD-Q4_K_XL via unsloth 145GB on disk GGUF 0.03.323.204 I common_params_fit_impl: projected to use 178175 MiB of host memory DeepSeek-V4-Flash-UD-Q8_K_XL via unsloth 151GB on disk GGUF 0.02.215.885 I common_params_fit_impl: projected to use 184636 MiB of host memory Laguna-S-2.1-UD-Q8_K_X via unsloth 120GB on disk 0.01.616.119 I common_params_fit_impl: projected to use 172860 MiB of host memory Qwen3.5-122B-A10B-UD-Q8_K_XL via unsloth 160GB on disk GGUF 165GB RAM use on launch, fresh context 0.04.976.905 I common_params_fit_impl: projected to use 170038 MiB of host memory
- Iolaum 3mo ago
  There's an emerging practice of using Q4 quants and Q8 KV cache for local inference. At that point you can run both Qwen3.5-122B-A10B (my personal choice on Framework Desktop 128gb) and Laguna-S-2.1. Now whether that's good enough for one's use-case remains to be determined. You can also get more out of those (local models and quantizations) if you further tweak the harness you use them with, but tbh this is where it gets too much work (at least for me and the time I have available).
  - zargon 3mo ago
    > emerging practice of using Q4 quants and Q8 KV cache for local inference That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.
    - walrus01 3mo ago
      I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.
      - trollbridge 3mo ago
        Gemma's QAT is surprisingly good (although Gemma isn't that great to begin with).
        KerrAvon 3mo ago
        IME: Gemma is not great for programming, but it is fantastic at following directions compared to anything else in its size class.
  - apitman 3mo ago
    Artificial Analysis ranks qwen3.6-27b higher than qwen3.5-122b-a10b on both intelligence and coding. Does that run counter to your experience?
    - nekitamo 3mo ago
      This tracks, in my experience the 27B is better at coding and instruction following. I'm shocked at how much of a difference the dense models vs MoE makes. But it's a moot point, because for local inference on consumer hardware, the MoE is so much faster.
- johndough 3mo ago
  DeepSeek-V4 should use only 5GB for context due to CSA and HCA, see figure here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro But not every framework implements it properly yet.
  - walrus01 3mo ago
    Yeah, or close enough to 5GB for estimation purposes, for example a just spawned qwen 3.5 122B llama-server instance reports as: 0.07.015.888 I common_memory_breakdown_print: | - Host | 170038 = 162913 + 6740 + 384 | The 6740 is the cache size.
embedding-shape 3mo ago
Yeah, I've been experimenting with DiffusionGemma which sadly isn't as smart as Gemma itself, but holy hell is it FAST, and has image input as well, so doing things like "take a screenshot once per second + ask the model to categorize/model it WITH reasoning before" becomes realistic and doable. I ended up implementing DiffusionGemma myself with Candle in Rust + CUDA, and it's quite literally the fastest model I've managed to run on my hardware.
bjackman 3mo ago
I agree but worth noting that it's never gonna be very practical to run LLMs like this at home. Unless we have some sort of design breakthrough, the only "sensible" way to run them is at high batch levels on shared HW. Like, yeah if I could spend a few grand on such a GPU I probably would coz I'm a rich nerd, but I'd acknowledge it as an extremely inefficient luxury, kinda like a sports car. So I think you could say the real misfortune is that we don't really have the technology (be it computer tech or political/social tech) to do that shared-HW thing in way we can truly trust.
- taneq 3mo ago
  ‘Never’ is a big word in the computing world. 10 years from now a model this size will probably run on a high-end phone. Of course, by then we’ll want to run something commensurately larger.
- ttoinou 3mo ago
  We could make LLM inference 100x cheaper to run at home efficiently, but that solution might need to be updated every 1-2 years, whereas current GPU are useful for various others tasks and last longer
  - nektro 3mo ago
    > We could make LLM inference 100x cheaper to run at home efficiently do you genuinely think that's going to happen?
- Tepix 3mo ago
  "Never" is a long time. Just think about how much ram we had 10 or 20 years ago. 1.5TB isn't a lot really.
  - usef- 3mo ago
    The typical ram has surprisingly not increased very much in 10 years. > April 2016, 8 GB was standard across the 13-inch MacBook Air range ... Now it's 16. Rich nerds will have quite a bit more. But I suspect the standard of model rich nerds want to use will have gone up somewhat too.
acchow 3mo ago
> like 180W-250W TDP > running GLM 5.2 would be cool at like ~100 tokens per second for a single session Your power consumption estimates are off for this generation of GPUs. A 27B dense model gets 50-80 tps on an RTX 6000 using 600 watts.
- ThunderSizzle 3mo ago
  An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.
  - syntaxing 3mo ago
    Which quants? I get these speed (only 20-30 TPS) at Q4_K_M with MTP for 27B on my framework desktop. I draw sub 130W for the whole machine.
  - acchow 3mo ago
    That's about right. Half the TPS with half the power. GLM-5.2 will be much more demanding tho
  - mhjkl 3mo ago
    If you’re on AMD, use llamacpp with the vulkan backend. For the 35B I get around 115 tps without mtp and 150-170tps with it. Also the pcie slot you use affects it
    - ThunderSizzle 3mo ago
      I have issues with the vulkan backend. It seems to cause over all GUI sluggishness, but maybe that was actually Firefox. I'll give it another go.
adityas02 3mo ago
The individual-shaped-hardware problem gets even sharper at the phone end. Shipping a 3B model on-device, the usable RAM budget after the OS and everything else is more like 2-4GBtotal, not per-model so it's not 'can I afford more VRAM', it's 'can I fit a language model and an STT model and embeddings without the OS killing my process'. Feels like phones are the most hardware-constrained 'individual' tier and get the least airtime in these kind of discussion. Is that because the models that fit are still too limited to be interesting, or something else?
- walrus01 3mo ago
  Recently spent a few hours messing with bonsai 27B and ternary bonsai 27B at total weights + context fitting in slightly under 6GB RAM, and it's just dumb as hell. It writes what seems like grammatically correct content but it's extremely limited. It will also happily hallucinate new names and content to fill in gaps in its knowledge, and present the hallucations in what looks like a correctly formatted sentence, so it could fool a person who doesn't know the subject matter. Like, I asked it for a description of Seattle and it hallucinated a name and description of a nonexistant tallest building in the city and suggested the view from its observation deck . https://prismml.com/news/bonsai-27b https://prismml.com/news/bonsai-27b https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf In no way was I surprised, it's asking a lot of under 6GB RAM usage. But I think for 99% of people they will get better results doing something over the network where the weights and inference engine are not on the device.
  - emulio 3mo ago
    [dead]
- deleted 3mo ago
  [deleted]
mft_ 3mo ago
You're right, and it's interesting to consider why. It's probably a combination of a few factors: 1) Local LLMs are a relatively new phenomenon and hardware takes years. Apple probably lucked into their unified memory architecture being suitable (in terms of memory size and bandwidth) for local LLMs, but it's only with the newest generations we're hearing about LLMs even being a consideration in their design process. 2) NVidia seem to be deliberately blocking consumers from taking this path - as evidenced by the removal of NVLink from the 30x0 series onwards - probably to protect their data center cards from internal competition? 3) Perhaps there's just not the market for it? It's feasible that the number of nerds interested local LLMs is very small in numbers, sales, and profit potential compared to gamers on the one side, and data centers on the other. (This would explain why AMD and Intel aren't trying to out-innovate NVidia in this area, despite it being an obvious opportunity.)
- trollbridge 3mo ago
  There will be a huge market for local inference once it's cheap and widely available. Try to imagine output token speeds of 15,000 tok/s and a time-to-first-token of 200ms. (This has already been done for Llama 8B.) Now imagine gargantuan context windows (2M, 4M, or even bigger); keep in mind the 1M context windows were science fiction a few years ago... now imagine having this on a local model on something like a phone or portable device that can be gathering data about things you're doing and constantly run inference for things useful to you. An obvious example of this would be a chatbot you can talk to that responds like a normal human conversation and doesn't have delays, but that's just scratching the surface.
  - cmrdporcupine 3mo ago
    > There will be a huge market for local inference once it's cheap and widely available. I've seen public pronouncements that the RAM shortage could persist for a decade. And then if consider that the constraint on local LLMs isn't just memory size but bandwidth ... If you take something like a DGX Spark and increase its memory to 512GB that doesn't even solve the problem. Because the bandwidth of DDR5 just can't manage reasonable speeds for decode. If you take a dense model or an MoE model uses up most of that 128GB in active decode you will only get like 15 tok/sec. "Real" datacentre inference boxes use high bandwidth memory that is 10x the speed. I think we're unfortunately a long way off, unless people learn to accept working with much less intelligent models locally. The innovation is going to have to come on the research & software side -- we need to find ways to pack more intelligence into a smaller number of parameters.
ttoinou 3mo ago
The next mac Ultra will allow to run a big model locally with acceptable speed. But we need people to optimize it for that computer, and we’ll be more limited in models we can choose from 128GB is enough to run a large model, quantized, REAPed, with MoE and fast SSD for model weights
- Davidzheng 3mo ago
  Not Kimi K3 large though
  - KerrAvon 3mo ago
    128 isn't enough, but if this report is accurate, the next Mac Studio could run it https://www.macrumors.com/2026/06/25/m5-ultra-mac-studio-2026/ https://www.macrumors.com/2026/06/25/m5-ultra-mac-studio-202...
  - jayd16 3mo ago
    They'll just have you buy two studios and thunderbolt them together.
chrysoprace 3mo ago
I've been feeling for a while that as we keep increasing model size, we're going through the opposite of the PC revolution. The "democratisation" talk from the frontier labs is especially egregious when they only release closed models (gpt-oss hardly counts) and are trying everything they can to make it harder to run open models.
schmorptron 3mo ago
LLM inference unfortunately also seems to be a task that's poorly formed for moderate consumer hardware,as a single user. For a single user use case, the load is bursty but requires the weights to be in memory already. So a multi user server that keeps the model weights in parts of its memory and then spends some more per user kv cache is wildly more efficient and the wildly expensive gpu cores aren't just sitting idle most of the time. Don't get me wrong, most desktop workloads are bursty, but the power needed to to them has gotten cheap enough that we can have way overkill for idle scenarios hardware just sitting on our desks.
- zozbot234 3mo ago
  Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K3 itself.
  - schmorptron 3mo ago
    it has to be so bursty for realtime usecases like chat, which is what most people are using it for today. of course, once (if) stuff like software dark factories start working out for the average person, then you'll be able to make full use of your hardware for workload where asynchonous execution is feasible and have it run several parallel tasks overnight, with an orchestrator managing the gpu(s) allocations.
    - zozbot234 3mo ago
      Chat doesn't have to literally be realtime though, that's just the model most users have settled on. You could fire off your request, let it work unattended and check back on it later (perhaps after getting some notification from the chat frontend via RSS, Web Notifications API or similar that the full response is ready).
  - 3mo ago
deleted 3mo ago
[deleted]
Curzel 3mo ago
RTX Spark does go up to 128gb No idea how it compares tho
- cmrdporcupine 3mo ago
  I have one. The limitation (beyond total size of the memory) with the Spark is DDR5. "Real" inference hardware is HBM (high bandwidth memory) which is like 10x the performance. So for prefill -- which is more about compute than bandwidth -- the Spark performs quite admirably. But on decode it's highly bandwidth constrained. Some smaller MoE models (e.g. Gemma4) can do 60-70 tok/second but anything dense, or anything that is actually filling up most of that 128GB is going to choke out around 15 tok/sec. Even at NVFP4. My experiments with this are at https://github.com/rdaum/eider/ https://github.com/rdaum/eider/ For my current work I get to log into trays on a real GB300. It's somewhat comical that NVIDIA is marketing the little baby on my shelf here as even in the same universe as that. Which is basically like having access to a super computer.
OkayPhysicist 3mo ago
Post-crypto, the GPU manufacturers took the proactive move to use VRAM to segment the market for the purpose of price discrimination. Sure, data centers will pay vastly more for GPUs, but Nvidia knows that the PC market is steady and reliable. They could get the best of both worlds by kneecapping their consumer cards to tiny amounts of RAM, to dissuade the cloud providers from scooping up all the consumer cards, and then charging the two segments wildly different amounts for what amounts to the same hardware (back when the cost of RAM was negligible)
stevejb 3mo ago
You can run GLM 5.2 at a reasonable quant on a cluster of DGX Sparks. On the link below, the guy is getting decent performance and links to a repo. An 8x Spark cluster can run it even better. A Spark cluster is about as good as it gets for (a) runnable at home, (b) "affordable" hardware, (c) not going to kill your power bill. https://www.youtube.com/watch?v=nbHOBvLlypY https://www.youtube.com/watch?v=nbHOBvLlypY https://www.youtube.com/watch?v=PV89U-PNUNA https://www.youtube.com/watch?v=PV89U-PNUNA

walrus01 3mo ago

I have found that the "mostly didn't lose anything" Q8 large models that I want to run are all too large to run on the "only $3995!" 128GB max RAM systems that some people are buying, and definitely won't fit with any usable amount of context. Things like Qwen 3.5 122B Q8 or deepseek v4 flash Q8, or Laguna S 2.1 Q8 need 170-190GB of RAM including full context, which fits on a 256GB RAM dual socket workstation or rackmount server (sans GPU). Copy and paste below from my notes and reported memory consumption with latest llama-server, assuming use of "--no-mmap" to load the entire thing into RAM at the time that llama-server launches. DeepSeek-V4-Flash-UD-Q4_K_XL via unsloth 145GB on disk GGUF 0.03.323.204 I common_params_fit_impl: projected to use 178175 MiB of host memory DeepSeek-V4-Flash-UD-Q8_K_XL via unsloth 151GB on disk GGUF 0.02.215.885 I common_params_fit_impl: projected to use 184636 MiB of host memory Laguna-S-2.1-UD-Q8_K_X via unsloth 120GB on disk 0.01.616.119 I common_params_fit_impl: projected to use 172860 MiB of host memory Qwen3.5-122B-A10B-UD-Q8_K_XL via unsloth 160GB on disk GGUF 165GB RAM use on launch, fresh context 0.04.976.905 I common_params_fit_impl: projected to use 170038 MiB of host memory

Iolaum 3mo ago
There's an emerging practice of using Q4 quants and Q8 KV cache for local inference. At that point you can run both Qwen3.5-122B-A10B (my personal choice on Framework Desktop 128gb) and Laguna-S-2.1. Now whether that's good enough for one's use-case remains to be determined. You can also get more out of those (local models and quantizations) if you further tweak the harness you use them with, but tbh this is where it gets too much work (at least for me and the time I have available).
- zargon 3mo ago
  > emerging practice of using Q4 quants and Q8 KV cache for local inference That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.
  - walrus01 3mo ago
    I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.
    - trollbridge 3mo ago
      Gemma's QAT is surprisingly good (although Gemma isn't that great to begin with).
      - KerrAvon 3mo ago
        IME: Gemma is not great for programming, but it is fantastic at following directions compared to anything else in its size class.
- apitman 3mo ago
  Artificial Analysis ranks qwen3.6-27b higher than qwen3.5-122b-a10b on both intelligence and coding. Does that run counter to your experience?
  - nekitamo 3mo ago
    This tracks, in my experience the 27B is better at coding and instruction following. I'm shocked at how much of a difference the dense models vs MoE makes. But it's a moot point, because for local inference on consumer hardware, the MoE is so much faster.
johndough 3mo ago
DeepSeek-V4 should use only 5GB for context due to CSA and HCA, see figure here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro But not every framework implements it properly yet.
- walrus01 3mo ago
  Yeah, or close enough to 5GB for estimation purposes, for example a just spawned qwen 3.5 122B llama-server instance reports as: 0.07.015.888 I common_memory_breakdown_print: | - Host | 170038 = 162913 + 6740 + 384 | The 6740 is the cache size.

embedding-shape 3mo ago

Yeah, I've been experimenting with DiffusionGemma which sadly isn't as smart as Gemma itself, but holy hell is it FAST, and has image input as well, so doing things like "take a screenshot once per second + ask the model to categorize/model it WITH reasoning before" becomes realistic and doable. I ended up implementing DiffusionGemma myself with Candle in Rust + CUDA, and it's quite literally the fastest model I've managed to run on my hardware.

bjackman 3mo ago

I agree but worth noting that it's never gonna be very practical to run LLMs like this at home. Unless we have some sort of design breakthrough, the only "sensible" way to run them is at high batch levels on shared HW. Like, yeah if I could spend a few grand on such a GPU I probably would coz I'm a rich nerd, but I'd acknowledge it as an extremely inefficient luxury, kinda like a sports car. So I think you could say the real misfortune is that we don't really have the technology (be it computer tech or political/social tech) to do that shared-HW thing in way we can truly trust.

taneq 3mo ago
‘Never’ is a big word in the computing world. 10 years from now a model this size will probably run on a high-end phone. Of course, by then we’ll want to run something commensurately larger.
ttoinou 3mo ago
We could make LLM inference 100x cheaper to run at home efficiently, but that solution might need to be updated every 1-2 years, whereas current GPU are useful for various others tasks and last longer
- nektro 3mo ago
  > We could make LLM inference 100x cheaper to run at home efficiently do you genuinely think that's going to happen?
Tepix 3mo ago
"Never" is a long time. Just think about how much ram we had 10 or 20 years ago. 1.5TB isn't a lot really.
- usef- 3mo ago
  The typical ram has surprisingly not increased very much in 10 years. > April 2016, 8 GB was standard across the 13-inch MacBook Air range ... Now it's 16. Rich nerds will have quite a bit more. But I suspect the standard of model rich nerds want to use will have gone up somewhat too.

acchow 3mo ago

> like 180W-250W TDP > running GLM 5.2 would be cool at like ~100 tokens per second for a single session Your power consumption estimates are off for this generation of GPUs. A 27B dense model gets 50-80 tps on an RTX 6000 using 600 watts.

ThunderSizzle 3mo ago
An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.
- syntaxing 3mo ago
  Which quants? I get these speed (only 20-30 TPS) at Q4_K_M with MTP for 27B on my framework desktop. I draw sub 130W for the whole machine.
- acchow 3mo ago
  That's about right. Half the TPS with half the power. GLM-5.2 will be much more demanding tho
- mhjkl 3mo ago
  If you’re on AMD, use llamacpp with the vulkan backend. For the 35B I get around 115 tps without mtp and 150-170tps with it. Also the pcie slot you use affects it
  - ThunderSizzle 3mo ago
    I have issues with the vulkan backend. It seems to cause over all GUI sluggishness, but maybe that was actually Firefox. I'll give it another go.

adityas02 3mo ago

The individual-shaped-hardware problem gets even sharper at the phone end. Shipping a 3B model on-device, the usable RAM budget after the OS and everything else is more like 2-4GBtotal, not per-model so it's not 'can I afford more VRAM', it's 'can I fit a language model and an STT model and embeddings without the OS killing my process'. Feels like phones are the most hardware-constrained 'individual' tier and get the least airtime in these kind of discussion. Is that because the models that fit are still too limited to be interesting, or something else?

walrus01 3mo ago
Recently spent a few hours messing with bonsai 27B and ternary bonsai 27B at total weights + context fitting in slightly under 6GB RAM, and it's just dumb as hell. It writes what seems like grammatically correct content but it's extremely limited. It will also happily hallucinate new names and content to fill in gaps in its knowledge, and present the hallucations in what looks like a correctly formatted sentence, so it could fool a person who doesn't know the subject matter. Like, I asked it for a description of Seattle and it hallucinated a name and description of a nonexistant tallest building in the city and suggested the view from its observation deck . https://prismml.com/news/bonsai-27b https://prismml.com/news/bonsai-27b https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf In no way was I surprised, it's asking a lot of under 6GB RAM usage. But I think for 99% of people they will get better results doing something over the network where the weights and inference engine are not on the device.
- emulio 3mo ago
  [dead]
deleted 3mo ago
[deleted]

mft_ 3mo ago

You're right, and it's interesting to consider why. It's probably a combination of a few factors: 1) Local LLMs are a relatively new phenomenon and hardware takes years. Apple probably lucked into their unified memory architecture being suitable (in terms of memory size and bandwidth) for local LLMs, but it's only with the newest generations we're hearing about LLMs even being a consideration in their design process. 2) NVidia seem to be deliberately blocking consumers from taking this path - as evidenced by the removal of NVLink from the 30x0 series onwards - probably to protect their data center cards from internal competition? 3) Perhaps there's just not the market for it? It's feasible that the number of nerds interested local LLMs is very small in numbers, sales, and profit potential compared to gamers on the one side, and data centers on the other. (This would explain why AMD and Intel aren't trying to out-innovate NVidia in this area, despite it being an obvious opportunity.)

trollbridge 3mo ago
There will be a huge market for local inference once it's cheap and widely available. Try to imagine output token speeds of 15,000 tok/s and a time-to-first-token of 200ms. (This has already been done for Llama 8B.) Now imagine gargantuan context windows (2M, 4M, or even bigger); keep in mind the 1M context windows were science fiction a few years ago... now imagine having this on a local model on something like a phone or portable device that can be gathering data about things you're doing and constantly run inference for things useful to you. An obvious example of this would be a chatbot you can talk to that responds like a normal human conversation and doesn't have delays, but that's just scratching the surface.
- cmrdporcupine 3mo ago
  > There will be a huge market for local inference once it's cheap and widely available. I've seen public pronouncements that the RAM shortage could persist for a decade. And then if consider that the constraint on local LLMs isn't just memory size but bandwidth ... If you take something like a DGX Spark and increase its memory to 512GB that doesn't even solve the problem. Because the bandwidth of DDR5 just can't manage reasonable speeds for decode. If you take a dense model or an MoE model uses up most of that 128GB in active decode you will only get like 15 tok/sec. "Real" datacentre inference boxes use high bandwidth memory that is 10x the speed. I think we're unfortunately a long way off, unless people learn to accept working with much less intelligent models locally. The innovation is going to have to come on the research & software side -- we need to find ways to pack more intelligence into a smaller number of parameters.

ttoinou 3mo ago

The next mac Ultra will allow to run a big model locally with acceptable speed. But we need people to optimize it for that computer, and we’ll be more limited in models we can choose from 128GB is enough to run a large model, quantized, REAPed, with MoE and fast SSD for model weights

Davidzheng 3mo ago
Not Kimi K3 large though
- KerrAvon 3mo ago
  128 isn't enough, but if this report is accurate, the next Mac Studio could run it https://www.macrumors.com/2026/06/25/m5-ultra-mac-studio-2026/ https://www.macrumors.com/2026/06/25/m5-ultra-mac-studio-202...
- jayd16 3mo ago
  They'll just have you buy two studios and thunderbolt them together.

chrysoprace 3mo ago

I've been feeling for a while that as we keep increasing model size, we're going through the opposite of the PC revolution. The "democratisation" talk from the frontier labs is especially egregious when they only release closed models (gpt-oss hardly counts) and are trying everything they can to make it harder to run open models.

schmorptron 3mo ago

LLM inference unfortunately also seems to be a task that's poorly formed for moderate consumer hardware,as a single user. For a single user use case, the load is bursty but requires the weights to be in memory already. So a multi user server that keeps the model weights in parts of its memory and then spends some more per user kv cache is wildly more efficient and the wildly expensive gpu cores aren't just sitting idle most of the time. Don't get me wrong, most desktop workloads are bursty, but the power needed to to them has gotten cheap enough that we can have way overkill for idle scenarios hardware just sitting on our desks.

zozbot234 3mo ago
Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K3 itself.
- schmorptron 3mo ago
  it has to be so bursty for realtime usecases like chat, which is what most people are using it for today. of course, once (if) stuff like software dark factories start working out for the average person, then you'll be able to make full use of your hardware for workload where asynchonous execution is feasible and have it run several parallel tasks overnight, with an orchestrator managing the gpu(s) allocations.
  - zozbot234 3mo ago
    Chat doesn't have to literally be realtime though, that's just the model most users have settled on. You could fire off your request, let it work unattended and check back on it later (perhaps after getting some notification from the chat frontend via RSS, Web Notifications API or similar that the full response is ready).
- 3mo ago

deleted 3mo ago

[deleted]

Curzel 3mo ago

RTX Spark does go up to 128gb No idea how it compares tho

cmrdporcupine 3mo ago
I have one. The limitation (beyond total size of the memory) with the Spark is DDR5. "Real" inference hardware is HBM (high bandwidth memory) which is like 10x the performance. So for prefill -- which is more about compute than bandwidth -- the Spark performs quite admirably. But on decode it's highly bandwidth constrained. Some smaller MoE models (e.g. Gemma4) can do 60-70 tok/second but anything dense, or anything that is actually filling up most of that 128GB is going to choke out around 15 tok/sec. Even at NVFP4. My experiments with this are at https://github.com/rdaum/eider/ https://github.com/rdaum/eider/ For my current work I get to log into trays on a real GB300. It's somewhat comical that NVIDIA is marketing the little baby on my shelf here as even in the same universe as that. Which is basically like having access to a super computer.

OkayPhysicist 3mo ago

Post-crypto, the GPU manufacturers took the proactive move to use VRAM to segment the market for the purpose of price discrimination. Sure, data centers will pay vastly more for GPUs, but Nvidia knows that the PC market is steady and reliable. They could get the best of both worlds by kneecapping their consumer cards to tiny amounts of RAM, to dissuade the cloud providers from scooping up all the consumer cards, and then charging the two segments wildly different amounts for what amounts to the same hardware (back when the cost of RAM was negligible)

stevejb 3mo ago

You can run GLM 5.2 at a reasonable quant on a cluster of DGX Sparks. On the link below, the guy is getting decent performance and links to a repo. An 8x Spark cluster can run it even better. A Spark cluster is about as good as it gets for (a) runnable at home, (b) "affordable" hardware, (c) not going to kill your power bill. https://www.youtube.com/watch?v=nbHOBvLlypY https://www.youtube.com/watch?v=nbHOBvLlypY https://www.youtube.com/watch?v=PV89U-PNUNA https://www.youtube.com/watch?v=PV89U-PNUNA