5 ms·
Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles
by giancarlostoro 3mo ago
Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory.
Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about. I wouldn't be surprised if frontier AI could be drastically cheaper if they just finetune and optimize their models to not consume all available RAM to only access less than 10% of the models knowledge.
- deleted 3mo ago[deleted]
- vorticalbox 3mo agoPutting the whole model in memory is far faster then swapping to disk.
- giancarlostoro 3mo agoFor local inference the cost of "speed" is not that bad I would think? I wouldn't mind a bit of a delay if it means I can run much larger models on my Mac.
- pertymcpert 3mo agoIt's pretty painful to have speeds < 30 tok/sec though. Especially if you're used to API providers at higher speeds. It makes any interactive work almost impossible to do efficiently because you have no choice but to context switch after every request.
- giancarlostoro 3mo agoI assume it will get better over time, and does it improve in speed if you use a larger buffer? Say instead of 2GB you go with 6GB? I imagine it would, and you might need to stream drastically less no?
- gitpusher42 3mo agoYeah, correct! You can set up this engine to get more expert cache slots (e.g 32 instead of 16) to get a better hit rate and better tok/s. it will be 3.5gb instead of 2gb.
- bigyabai 3mo agoYou're kinda describing the MoE architecture; you can offload expert layers and stream them as-needed if the experts are small enough and the SSD is fast enough. Dense LLMs typically perform better, but slow down much more than MoE models when you try offloading layers.
- avadodin 3mo agoThe issue is that the MoE knowledge is LLM "knowledge" and it still has a cost so, overall, it has a lower quality/cost ratio. What he's envisioning is a dense 1B model that looks at the Python specification and your prompt and goes: > Ah, I get it now! This is like Darmok and Jalad at Tanagra! Or at least: > Android UI development in Python? It's UNIX, I know this! We do work like this sometimes but in general we rely on internalized knowledge so I don't know to what extent it is a viable strategy.
- literalAardvark 3mo agoYou're right, but the "just" in "just finetune" is doing _a lot_ of work here. It's still early days and we "just" don't really know how to do it well.
- giancarlostoro 3mo agoI mean, that's fair, I guess what I mean is, it feels like we're re-using well known solutions even if it takes a bit of effort to re-apply them into how we run inference (and maybe training as well). It will be interesting to see a lot of these approaches compound into anyone with a reasonable GPU or even a Mac running a model much larger than their machine can handle.
- medicis123 3mo agoWe did something similar - Streaming experts. Maintaining an expert cache, optimizing it to simulate running a multi-model agentic workflow on a 2-DGC Spark Cluster. The models we ran were: DeepSeek V4 Flash, Gemma 4 26B A4B, and Nemotron 3 Nano Omni 30B NVFP4. The results were very encouraging in terms of performance and model switching. Check it out here - https://woolyai.com/ai-compute-software/dgx-spark-inference-stack/ https://woolyai.com/ai-compute-software/dgx-spark-inference-...
- BonerWiener 3mo agoThe benchmarks are hidden behind a sign up form. Why not just keep it open? This looks as if you are just advertising.
- oblio 3mo agoIt looks? It is advertising :-)
- gopalv 3mo ago> It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. The A in 26B-A4B is the active weights. The problem is that this is a per-token load/unload at best, not for the whole prompt. The division happened until one of these can fit in a single GPU and they stopped scaling it down any more, because you can wire up 8 of them to do their share of the work.
- swiftcoder 3mo ago> I don't care who King Charles is every single time Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...
- kanbankaren 3mo ago> Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool... I think this can be achieved already. Take a base model and train only on source code. In fact, the very early Granite models from IBM were like that though it didn't support reasoning which limited its performance. You can do it too. I don't know how much it will cost to train on just source code repos. $10K in total? Not sure.
- kccqzy 3mo agoI’m a bit skeptical. Without instructional materials from textbooks, programming language reference manuals and guides, as well as general knowledge about logic and discrete mathematics, I doubt a model could work very well.
- kanbankaren 3mo agoYes. Model would be limited in its performance, but it would perform well in the limited domain because LLM interpolate from training data. They don't think like humans. We might think that knowledge from logc and discrete math would spill over to coding. Unfortunately, it doesn't seem to work like that. Even 1T parameter LLM fail on tasks if there are no variants of it in the training data.
- tyromaniac 3mo agoI'm skeptical that the "logical ability" is much more then the elided data. Obviously some things get fully memorized and other things don't, but I don't think there's anything like functional circuits.
- Keyframe 3mo agoI don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.
- IdontKnowRust 3mo agoThis makes me think (by full ignorance) how quantum computing can be useful in this field of things reach a stable-ish point
- ComputerPerson 3mo agoSeems to me like there's a conflict of incentives in the current era of market capture. Not to say the cost-cutting wouldn't be valuable; today's race is predominantly about the model's reasoning capacity or "how hard of a math problem can the model solve". It'll be a nice day when research-oriented human capital gets redirected to things that benefit us layfolks's pockets more directly
- spot5010 3mo agoThe human analogy would be that I don’t remember everything in the books I have read, but I do recall reading a particular book and can always look it up. So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.
- bastawhiz 3mo ago> I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to the new positions of each of the weights. For billions of parameters, that's a lot of calculations. And it requires us to know what each parameter actually represents, which nobody does.
- subarctic 3mo agoI know very little about this but it seems like the kind of thing that can and eventually will be solved computationally, not by people figuring out what a parameter or group of parameters represent
- bastawhiz 3mo agoIt's not a compute problem. It's a knowledge problem. Even if you can process each parameter individually and re-link the model layers, you need enough information to know what each parameter is for which is necessarily more memory than the weights themselves. You can use the weights to know whether each parameter is useful for a given prompt, but that operation is a strict superset of just generating the answer. By the time you know which parameters are useful, you've already done all the work of generating your output tokens and the effort is pointless.
- jz391 3mo agoWould an extension to `madvise` to say: "page this region in/out as a whole" help here? Engine could define memory ranges representing each expert and leave paging to the OS (tho' "paging" at this point becomes more similar to swapping in granularity...).
- bastawhiz 3mo agoExperts are chosen per token
- walrus01 3mo ago> I don't care who King Charles is every single time There is just a stupendous amount of everything packed into a 35B size or larger model. For instance Qwen 3.6 35B A3B (Q8) can do a fairly decent job translating English to Arabic, but it can also generate python code with a reasonable layout and commenting. I try to remember that as a mental model, an epub copy of a truly gargantuan sized 1000+ page novel such as the unabridged/2nd revision of Steven King's The Stand is about 800KB, and we're talking about a GGUF file that's 37GB in size or something like that.
- oblio 3mo ago> Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about. The idea that people in 10+ frontier labs (OpenAI, Anthropic, Google, Alibaba, Z.ai, DeepSeek, xAI, Amazon, etc) in a trillion dollars industry are all dumb is frankly, hilarious.
- joefourier 3mo agoAnthropic's API has two nines availability and Claude Code is a TUI made with React that can regularly consume more than 1GB of RAM, and the codebase is utter slop. They couldn't fix the flickering bug for over a year! And yet, Fable and Opus are among the best coding models out there (matched only by GPT5.6 Sol). It's not about the people there being smart or not, it's about their and the company's priorities, resources and what they choose to focus on.
- oblio 3mo agoOne is the UX, which they don't care about because people use their models anyway. The second would be hardware savings on the order of tens of billions of dollars if they were stupid not do try all possible optimizations. Spot the difference.
- joefourier 3mo agoYou cannot just "try all possible optimisations". It takes time, effort, and money that could otherwise be spent elsewhere (especially for training, where each training run is especially costly, and optimisations might be promising early on, but cause the final performance of the model to be worse). You need smart people interested in unglamorous work, and if you're swimming in VC money, it's far more straightforward to just throw more GPUs at the problem.
- kcexn 3mo agoI don't think the problem is that memory footprint can't be reduced. The problem is that the token generation rate is simply far too low. OP suggests that the token rate of of their solution is ~5 per second. That's at least an order of magnitude slower than commercially available models.
- oezi 3mo agoOne key thing with MoE I am still not understanding is why we don't keep the same expert in memory for a larger number of tokens than 1. Why do we route to some other expert every token? Wouldn't it be more memory efficient to generate at least 2,3,4,8,12 or 32 tokens and then swap the experts?
- zozbot234 3mo agoExpert choice actually happens per layer, not just per token. It's not a limitation when doing inference at scale since all experts are then loaded in fast VRAM anyway. It's mostly just a way to enforce some kind of model sparsity and save on compute. The new version of Apple Foundation Model (AFM) Core Advanced is an exception, it actually routes experts per prompt (with sporadic rerouting perhaps?) which is more in line with what you're proposing. But this will theoretically make the model less smart than a similar one where experts are picked per layer.
- glitchc 3mo agoIndeed, it begs the question why we have "everything" models where instead we could have very efficient "something" models. Typical LLMs out there can generate code and translate between 60 different languages. Sometimes I only need the first part, sometimes the second. Two distinct models would be a lot smaller and run much faster (token-wise).
- panarky 3mo agoBut they'd be stupider. The results for English and Python are much better because the model is also trained on Mandarin and Greek and Lisp even if you never make a request or receive a response in Mandarin, Greek or Lisp.
- glitchc 3mo agoThat's news to me since it's unlikely most of those weights are activated when responding to a coding prompt. Can you point to a source/paper that validates this claim?
- panarky 3mo agoIt's well known and well documented in AI research. Instead of taking syntactic shortcuts, the richer abstractions learned from multiple languages, and code, and math, and images and audio, ultimately make English comprehension and reasoning far stronger. If you want citations, ask your favorite LLM how linguistic diversity prevents "surface memorization", overfitting on surface-level English patterns instead of representing the deeper concepts in latent vector space.
- BobbyTables2 3mo agoThe problem is that knowledge in a LLM is separated in the mathematical sense (vector direction) but not necessarily meaningfully grouped in the matrix (would be easier to split between disk/memory) if such. I think a rough analog is that it would be difficult to organize the rows of a hash table of everybody in a country by their geographic location. At this point, people are just thrilled LLMs can even function as they do…
- akmittal 3mo agoGames used to render entire map in map in memory. Later they figured only the surroundings can we rendered. We can hopefully get same in AI
- code51 3mo agoYou don't know whether King Charles or 42 is the answer to "the Question" my 2c on King Charles vs 42.
- gitpusher42 3mo agoApple does something similar with their latest foundation model. They process input prompt and based on results they preload required experts. Quite neat solution for the edge devices