37 ms·

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xe

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge price difference in what you can do with buying a used 4U rackmount server and putting 3TB of RAM in it (64GB DIMMs x quantity 32 in a quad socket xeon, you can see some benchmark prices on eBay for sets of 16 or 32 matched 64GB ECC DIMMs) for <$30,000, vs the cost of trying to run it on real GPU hardware. Now obviously, as of the time I write this, the full precision hasn't been released nor has anyone like unsloth run it through quantization yet to produce a "Q8" or "Q8-XL" variant of it. But I think it's going to need more than 1536GB of RAM, with a usable and large amount of context, more like 2TB and preferably 2.5 to 3TB. I also predict that people who try to run it in Q4 and Q6 will get the worst of both worlds, less precision/lost knowledge but also not reliable output that comes out too slow. In my personal opinion if I'm going to deal with something that is smart but slow and running on limited budget hardware, I need it to be Q8.

sandworm101 3mo ago
Those old LTT videos of high core-count threadrippers running GPU benchmarks become more relevant each day.
- walrus01 3mo ago
  The performance bottleneck is not really so much the number of cores or processing power in each core, but the memory bus bandwidth to/from the CPU. I have an older dual socket xeon server here which is a CPU-only LLM test machine with 256GB of RAM and the actual CPU stress is not much, I can even quantify this by how little it spins up the CPU fans to meet thermal load (the CPUs are operating at nowhere near their 180W per socket max capacity, compared to like, crunching prime numbers or running cpuburn). But the memory bus speed is fully committed when generating tokens or thinking.
  - sandworm101 3mo ago
    Thats where the threadrippers really excelled. They had the lanes for memmory access. We might soon see the return of dinner plate-sized CPUs with thousands of pins.
    - walrus01 3mo ago
      The epyc Venice SP7 socket is apparently 9324 pins https://x.com/tomshardware/status/2066846693778510331 https://x.com/tomshardware/status/2066846693778510331
      - sandworm101 3mo ago
        We are going to need a bigger boat. https://www.cerebras.ai/ https://www.cerebras.ai/
fooker 3mo ago
> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really really narrow. I wouldn't be surprised if this is an empty set.
- walrus01 3mo ago
  There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socket Dell R940 for a month.
  - fooker 3mo ago
    Great, so the other member of the set matters for you more than cost. Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?
    - walrus01 3mo ago
      Do I really need to? No, not really. The 27B full density, 35B MoE, 70B and 122B models I have in use get me 95% of the way there on a lot of things. Particularly when dealing with languages and systems where I have at least an intermediate level of knowledge on, to know whether something is going down a dead end, using a wrong method, metaphorically chasing its tail, or is producing valid output. On the other hand, would it be cool to also have a really big thing as an ancillary tool that I could throw a request into opencode before going to bed, let it crank away and take a look at what it's done 7 hours later? Yeah, particularly if I (very much an unknown quantity at this time) could be confident that it builds high quality, syntax valid, appropriately commented and not absurd code.
embedding-shape 3mo ago
I dunno, K3 thinks a lot before it actually replies, and you might be in the ~1 tok/speed region or even "seconds / tokens", and with K3, you'd wait days if not weeks for a reply in that case. Don't get me wrong, slow is sometimes better than "not at all", but depending on the performance, it might end up way too slow to even work for batched/async jobs like that.
- walrus01 3mo ago
  I agree it's very likely to be painfully slow, I very much want to see some real world results from people who try it. Early testers will inform others on whether it's even worth trying. Results very much TBD right now. I don't have a system sitting around here with 2TB of greater of RAM that isn't already committed for other uses, regretfully.
  - embedding-shape 3mo ago
    Lets say an easy response takes 32k tokens in total, and to be generous, let's say it does 1 tok/s. This is already ~9 hours, and 32k reasoning tokens isn't even that much and as mentioned, K3 probably does the longest/most reasoning/thinking out of the available open weights models today, much like GLM. Just lowering that performance to 0.5 tok/s, would lead to ~18 hours for a simple prompt to receive an answer. And then that's just for single prompts, what about agent harnesses, where before every tool call the model could reason a bunch? I agree with you that real world results would be interesting, but I wouldn't hold my breath nor expect it to realistically be able to be useful. Still, people should try it, for science if nothing else :)
  - justincormack 3mo ago
    You can rent one in the cloud to try it
magicalhippo 3mo ago
> running it on a no GPU, but tons of RAM server Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable. [1]: https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri
- walrus01 3mo ago
  It's a great concept but I think it would cross the line from 'very slow' to 'so slow it's unusable' at this size. Even if we say you have an NVME SSD that does 7GB/s reads, that's dramatically slower than being able to hold the whole thing in DRAM. Like the difference between 1.3 tok/s in RAM vs 0.1 tok/s with a colibri-like method. edit: the results I have seen from people trying colibri with fast consumer grade PCI-E 4.0 NVME SSD are 0.1 tok/s on models that are <700B in size, things that are well under 800GB on disk. With something that's 3T in size it'll probably be a lot slower than hat.
  - magicalhippo 3mo ago
    It claims to support using multiple devices RAID-0 style, which should boost performance, but yea probably not very useful for most. But still fun you can run it at home.
  - zozbot234 3mo ago
    For single stream inference of a MoE model, the size of active sparse parameters will matter a lot more than total parameters. This is generally around half of the reported active parameter count - the other half being a dense subset that can be easily cached in VRAM even on fairly modest consumer setups. So the achievable performance may be quite a bit better than a naïve assessment might suggest.
  - LtdJorge 3mo ago
    On a server machine you can have more than 100GB/s of NVMe if you parallelize (RAID 0 and the like). But it's still gonna be noticeably slower.
    - 3mo ago
deleted 3mo ago
[deleted]
johndough 3mo ago
> But I think it's going to need more than 1536GB of RAM, with a usable and large amount of context, more like 2TB and preferably 2.5 to 3TB. The model is known to be MXFP4 according to Kimi's release blog post, so the model weights will be less than 1536GB: https://www.kimi.com/blog/kimi-k3 https://www.kimi.com/blog/kimi-k3 Also, their previous models were native INT4, so it would be weird if they went larger now.
- johndough 3mo ago
  Update: Looks like the model is larger after all (1561.44 GB). Only the MoE weights are MXFP4, while the other weights are BF16 (and a few FP32). * Sparse Experts: 1481.4 GB * Dense Experts: 1.9 GB * Self-Attention: 72.4 GB * LLM Head: 2.4 GB * Embeddings: 2.4 GB * Vision Encoder: 0.35 GB (surprisingly small) plus some miscellaneous parameters. Most importantly, we now know that the model has 104B active parameters, which is quite a lot and will make it difficult to self-host efficiently.
Havoc 3mo ago
> Even if the output is like 5-6 tok/s On a 3T model I’d imagine you’d be closer to 0.05 tks
- amluto 3mo ago
  Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.
  - zozbot234 3mo ago
    Speculative decoding is not really worthwhile for sparsely-loaded models. You end up paying in both memory bandwith and compute (loading experts based on wrongly-predicted tokens) which leaves you worse off overall. It becomes viable (even for sparse MoE) once you're batching so widely that you end up having to load most of your total weights anyway.
    - amluto 3mo ago
      > Speculative decoding is not really worthwhile for sparsely-loaded models. If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens, without too much quality loss. Another fun idea might be to try to build a model where the router chooses the expert 1-3 tokens in advance.
      - johndough 3mo ago
        > If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens You can! > AFM 3 Core Advanced makes routing decisions per prompt. A lightweight, dense block selects a fixed set of experts during initial processing, periodically reselecting them during generation. https://machinelearning.apple.com/research/introducing-third-generation-of-apple-foundation-models https://machinelearning.apple.com/research/introducing-third...
barbacoa 3mo ago
They are saying that AMD's new Epyc Venice CPU has 16 memory channels allowing up to 1.6Tb/s of bandwidth. Which is higher bandwidth than most non-HBM GPUs. So full CPU local AI inference may become viable option in coming years.
- lallysingh 3mo ago
  This is essentially guaranteed. There are lots of useful smaller models that we should be able to run locally. Over time they'll be more and more capable and require less API usage.
  - calgoo 3mo ago
    Im wondering if we are finally seeing the end of the "hard disk" era, and are entering a new era of vast instant on systems.
- adgjlsfhk1 3mo ago
  the GPU competition is using 16 gpus, so the actual comparison is that the CPU has <1/10th the bandwidth
flockonus 3mo ago
As you already went through the thought exercise of laying all this RAM over various slots, then match against the right CPU (which also you'll need multiple) - it becomes clear quite fast that it's trying to mimic the architecture of a GPU except in extremely low fidelity and bandwidth @ a higher energy cost.
holoduke 3mo ago
It will be not 5 tok/sec. More like 0.5 tok per sec with a fast cpu setup.

sandworm101 3mo ago

Those old LTT videos of high core-count threadrippers running GPU benchmarks become more relevant each day.

walrus01 3mo ago
The performance bottleneck is not really so much the number of cores or processing power in each core, but the memory bus bandwidth to/from the CPU. I have an older dual socket xeon server here which is a CPU-only LLM test machine with 256GB of RAM and the actual CPU stress is not much, I can even quantify this by how little it spins up the CPU fans to meet thermal load (the CPUs are operating at nowhere near their 180W per socket max capacity, compared to like, crunching prime numbers or running cpuburn). But the memory bus speed is fully committed when generating tokens or thinking.
- sandworm101 3mo ago
  Thats where the threadrippers really excelled. They had the lanes for memmory access. We might soon see the return of dinner plate-sized CPUs with thousands of pins.
  - walrus01 3mo ago
    The epyc Venice SP7 socket is apparently 9324 pins https://x.com/tomshardware/status/2066846693778510331 https://x.com/tomshardware/status/2066846693778510331
    - sandworm101 3mo ago
      We are going to need a bigger boat. https://www.cerebras.ai/ https://www.cerebras.ai/

fooker 3mo ago

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really really narrow. I wouldn't be surprised if this is an empty set.

walrus01 3mo ago
There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socket Dell R940 for a month.
- fooker 3mo ago
  Great, so the other member of the set matters for you more than cost. Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?
  - walrus01 3mo ago
    Do I really need to? No, not really. The 27B full density, 35B MoE, 70B and 122B models I have in use get me 95% of the way there on a lot of things. Particularly when dealing with languages and systems where I have at least an intermediate level of knowledge on, to know whether something is going down a dead end, using a wrong method, metaphorically chasing its tail, or is producing valid output. On the other hand, would it be cool to also have a really big thing as an ancillary tool that I could throw a request into opencode before going to bed, let it crank away and take a look at what it's done 7 hours later? Yeah, particularly if I (very much an unknown quantity at this time) could be confident that it builds high quality, syntax valid, appropriately commented and not absurd code.

embedding-shape 3mo ago

I dunno, K3 thinks a lot before it actually replies, and you might be in the ~1 tok/speed region or even "seconds / tokens", and with K3, you'd wait days if not weeks for a reply in that case. Don't get me wrong, slow is sometimes better than "not at all", but depending on the performance, it might end up way too slow to even work for batched/async jobs like that.

walrus01 3mo ago
I agree it's very likely to be painfully slow, I very much want to see some real world results from people who try it. Early testers will inform others on whether it's even worth trying. Results very much TBD right now. I don't have a system sitting around here with 2TB of greater of RAM that isn't already committed for other uses, regretfully.
- embedding-shape 3mo ago
  Lets say an easy response takes 32k tokens in total, and to be generous, let's say it does 1 tok/s. This is already ~9 hours, and 32k reasoning tokens isn't even that much and as mentioned, K3 probably does the longest/most reasoning/thinking out of the available open weights models today, much like GLM. Just lowering that performance to 0.5 tok/s, would lead to ~18 hours for a simple prompt to receive an answer. And then that's just for single prompts, what about agent harnesses, where before every tool call the model could reason a bunch? I agree with you that real world results would be interesting, but I wouldn't hold my breath nor expect it to realistically be able to be useful. Still, people should try it, for science if nothing else :)
- justincormack 3mo ago
  You can rent one in the cloud to try it

magicalhippo 3mo ago

> running it on a no GPU, but tons of RAM server Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable. [1]: https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri

walrus01 3mo ago
It's a great concept but I think it would cross the line from 'very slow' to 'so slow it's unusable' at this size. Even if we say you have an NVME SSD that does 7GB/s reads, that's dramatically slower than being able to hold the whole thing in DRAM. Like the difference between 1.3 tok/s in RAM vs 0.1 tok/s with a colibri-like method. edit: the results I have seen from people trying colibri with fast consumer grade PCI-E 4.0 NVME SSD are 0.1 tok/s on models that are <700B in size, things that are well under 800GB on disk. With something that's 3T in size it'll probably be a lot slower than hat.
- magicalhippo 3mo ago
  It claims to support using multiple devices RAID-0 style, which should boost performance, but yea probably not very useful for most. But still fun you can run it at home.
- zozbot234 3mo ago
  For single stream inference of a MoE model, the size of active sparse parameters will matter a lot more than total parameters. This is generally around half of the reported active parameter count - the other half being a dense subset that can be easily cached in VRAM even on fairly modest consumer setups. So the achievable performance may be quite a bit better than a naïve assessment might suggest.
- LtdJorge 3mo ago
  On a server machine you can have more than 100GB/s of NVMe if you parallelize (RAID 0 and the like). But it's still gonna be noticeably slower.
  - 3mo ago

deleted 3mo ago

[deleted]

johndough 3mo ago

> But I think it's going to need more than 1536GB of RAM, with a usable and large amount of context, more like 2TB and preferably 2.5 to 3TB. The model is known to be MXFP4 according to Kimi's release blog post, so the model weights will be less than 1536GB: https://www.kimi.com/blog/kimi-k3 https://www.kimi.com/blog/kimi-k3 Also, their previous models were native INT4, so it would be weird if they went larger now.

johndough 3mo ago
Update: Looks like the model is larger after all (1561.44 GB). Only the MoE weights are MXFP4, while the other weights are BF16 (and a few FP32). * Sparse Experts: 1481.4 GB * Dense Experts: 1.9 GB * Self-Attention: 72.4 GB * LLM Head: 2.4 GB * Embeddings: 2.4 GB * Vision Encoder: 0.35 GB (surprisingly small) plus some miscellaneous parameters. Most importantly, we now know that the model has 104B active parameters, which is quite a lot and will make it difficult to self-host efficiently.

Havoc 3mo ago

> Even if the output is like 5-6 tok/s On a 3T model I’d imagine you’d be closer to 0.05 tks

amluto 3mo ago
Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.
- zozbot234 3mo ago
  Speculative decoding is not really worthwhile for sparsely-loaded models. You end up paying in both memory bandwith and compute (loading experts based on wrongly-predicted tokens) which leaves you worse off overall. It becomes viable (even for sparse MoE) once you're batching so widely that you end up having to load most of your total weights anyway.
  - amluto 3mo ago
    > Speculative decoding is not really worthwhile for sparsely-loaded models. If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens, without too much quality loss. Another fun idea might be to try to build a model where the router chooses the expert 1-3 tokens in advance.
    - johndough 3mo ago
      > If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens You can! > AFM 3 Core Advanced makes routing decisions per prompt. A lightweight, dense block selects a fixed set of experts during initial processing, periodically reselecting them during generation. https://machinelearning.apple.com/research/introducing-third-generation-of-apple-foundation-models https://machinelearning.apple.com/research/introducing-third...

barbacoa 3mo ago

They are saying that AMD's new Epyc Venice CPU has 16 memory channels allowing up to 1.6Tb/s of bandwidth. Which is higher bandwidth than most non-HBM GPUs. So full CPU local AI inference may become viable option in coming years.

lallysingh 3mo ago
This is essentially guaranteed. There are lots of useful smaller models that we should be able to run locally. Over time they'll be more and more capable and require less API usage.
- calgoo 3mo ago
  Im wondering if we are finally seeing the end of the "hard disk" era, and are entering a new era of vast instant on systems.
adgjlsfhk1 3mo ago
the GPU competition is using 16 gpus, so the actual comparison is that the CPU has <1/10th the bandwidth

flockonus 3mo ago

As you already went through the thought exercise of laying all this RAM over various slots, then match against the right CPU (which also you'll need multiple) - it becomes clear quite fast that it's trying to mimic the architecture of a GPU except in extremely low fidelity and bandwidth @ a higher energy cost.

holoduke 3mo ago

It will be not 5 tok/sec. More like 0.5 tok per sec with a fast cpu setup.