zozbot234

7 ms·

61.
by zozbot234 3mo ago
Expert choice actually happens per layer, not just per token. It's not a limitation when doing inference at scale since all experts are then loaded in fast VRAM anyway. It's mostly just a way to enforce some kind of model sparsity
62.
by zozbot234 3mo ago
> The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.
63.
by zozbot234 3mo ago
antirez's own video (the one I referenced in my comment) shows K3 inference running on M5 Max, not M1-series silicon (which is OP). M1 series has far lower SSD read throughput and memory bandwidth, and can barely fit the model weights
64.
by zozbot234 3mo ago
That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer
65.
by zozbot234 3mo ago
> With MoE it's low probability. For even the sparsest MoE open models, having more than a handful of inferences in the batch is enough to make it more likely than not that you'll get some MoE weight reuse within any given la
66.
by zozbot234 3mo ago
If you're targeting widespread local/on prem deployment which is what many open weight models are doing, that inherently limits your scale in terms of total model weights/inference-time compute compared to running in a few ce
67.
by zozbot234 3mo ago
To be clear, the restriction on "Model as a Service" past $20M yearly revenue is new to K3. K2.x had the attribution requirement for any commercial use with more than $20M monthly revenue.
68.
by zozbot234 3mo ago
Open source AI is actually a lot less "powerful" than genuine frontier models, i.e. it has a much tighter inherent capability ceiling. This is "decelerationist" from a purely AGI-pilled point of view but it's actua
69.
by zozbot234 3mo ago
If it's 4-bit native for the sparse parameters (which is the bulk of them) why would you expect it to add up to 2+TB?
70.
by zozbot234 3mo ago
The full K3 model will probably be way more than 594GB, that's more of a plausible range for Kimi 2.x. You'll probably be able to test run this model at full or near-full precision using SSD offload, but only at very slow speed
71.
by zozbot234 3mo ago
Chat doesn't have to literally be realtime though, that's just the model most users have settled on. You could fire off your request, let it work unattended and check back on it later (perhaps after getting some notification fr
72.
by zozbot234 3mo ago
Speculative decoding is not really worthwhile for sparsely-loaded models. You end up paying in both memory bandwith and compute (loading experts based on wrongly-predicted tokens) which leaves you worse off overall. It becomes viable (eve
73.
by zozbot234 3mo ago
In principle, a slower inference ought to be easier to steer and course-correct. You'd always be able to look at partial results, especially with a local model that doesn't hide its thinking.
74.
by zozbot234 3mo ago
Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent mod
75.
by zozbot234 3mo ago
For single stream inference of a MoE model, the size of active sparse parameters will matter a lot more than total parameters. This is generally around half of the reported active parameter count - the other half being a dense subset that
76.
by zozbot234 3mo ago
> In typical MoE model they get switched "randomly" on every token, so all experts have to be readily available. It's worse than that: a typical MoE model routes a separate set of experts at every layer , not just every t
77.
by zozbot234 3mo ago
Abliteration is not magic. It cannot give the model knowledge that it wasn't specifically trained for. The people who talk about abliterated models being dangerous should discuss actual red-teaming scenarios where they managed to ask
78.
by zozbot234 3mo ago
> There’s a Twitter thread making rounds by Dean Ball about deceleration in AI development caused by open models and I can’t understand how people don’t see that it’s true: open models dismantle the frontier lab capex spend potential by
79.
by zozbot234 3mo ago
You can run larger models by offloading to SSD (for weights), it's just slow so people don't do it all that much. But you can get back at least some of that performance by using either MTP (at least for dense models; not effective
80.
by zozbot234 3mo ago
> You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow
81.
by zozbot234 3mo ago
I don't want to sound overly dismissive, but for quite a few practical cases pipeline parallelism with micro-batches will be a likely win over tensor parallelism. Of course this inherently comes with a requirement to take batched inf
82.
by zozbot234 3mo ago
> speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference This is somewhat of a nitpicking point, but AIUI speculative decoding is worthwhi
83.
by zozbot234 3mo ago
It's a very nice summary of KV cache storage requirements for the leading open weights models, but ultimately it still shows the DeepSeek V4 series winning by a huge margin on that front. And DeepSeek actually does quite well on needle
84.
by zozbot234 3mo ago
0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the co
85.
by zozbot234 3mo ago
Kimi uses INT4 as its native format, there's no such thing as "better than 4-bit precision" for that model. This is in contrast with GLM for which 16-bit precision is native and 8-bit is in common use.
86.
by zozbot234 3mo ago
> The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those doing the math, this
87.
by zozbot234 4mo ago
10k context is not a whole lot, this model theoretically supports up to 1M. But the KV cache storage takes up a whole lot more memory capacity at full context than DeepSeek V4 Pro, let alone Flash. (About ~96GB according to readily availabl
88.
by zozbot234 4mo ago
AIUI the llama.cpp implementation for this model is still quite half-baked due to missing the support for DSA sparse attention mechanism. This leads to running the model with a different mechanism that it has not been trained for, which ha
89.
by zozbot234 4mo ago
Lousy benchmark, they explicitly focus on the easiest tasks to automate for AI (i.e. heavily cherry picked outcomes) and it seems that they don't bother to test anything except just-released proprietary models.
90.
by zozbot234 4mo ago
What's "exponential" about AI development? Model parameter counts? Anthropic doesn't publish those for their own models, last I checked. Datacenter buildouts? Water consumption per request? There just isn't enough e

More ›