15 ms·
New LLM optimization technique slashes memory costs
- Euphorbium 2y agoStop words in extra steps.
- aussieguy1234 2y agoMost people don't remember absolutely everything, just the important stuff.
- hoc 2y agoAnd we finally can sell the unoptimized models as Hires (since I can still read the differences!).
- cs702 2y agoVery clever, very meta, and it seems to work really well. The two big take-aways for me are: * It's possible to train a model to learn to summarize context from the attention matrix, based only on dot-product scores (k @ q.T * mask), regardless of how tokens are embedded. * Once the model is trained, it will work with any attention matrix, even if it's the attention matrix of another model. I've added this to my ever-growing list of things to try.
- xpl 2y agoIs there any intuition why does it even work? It seems very unexpected.
- cs702 2y agoThe intuition is that the relative frequency at which past tokens get attention from future tokens is a good proxy for their relative importance. The model the authors use, in fact, maps attention scores to features in the frequency domain.
- skellington 2y agoThis only reduces the working memory, not the base model itself?
- pizzreatr 2y ago[flagged]
- ComputerGuru 2y agoThis only decreases memory cost of input context window, not the memory cost to load and run the models.
- freehorse 2y agoContext window requires ram too.
- ynniv 2y ago"Only"
- keyle 2y agoI agree with you though, the title is misleading.
- BoorishBears 2y agoTitle is perfect. Their typical audience probably understands "memory" better than "context window", but then if you've actually deployed these systems it's not difficult to go the other way, from "memory" to "context window" since the context window specifically is known to take additional VRAM over the model itself
- solarkraft 2y agoAnd that’s what matters the most! To me, at small model sizes (1-8B), anyway. A few thousans tokens already bog my RAM down quite a lot and I’d love to have more - I’d go as far as saying that context greatly determines LLM capability at this point.
- danielbln 2y agoYes, pretraining and post-training is nice and important, but in-context learning turns LLMs from toys into tools.
- erikmalkavian 2y ago[dead]
- richwater 2y agoThis is for inference right? Not training?
- pizza 2y agoIt's for KV caching. In most conversations that will mean inference. But you can do reinforcement learning using sampled sequences, and you could use KV caching to speed that up too, so that would be an instance where training could get a slight boost.
- lawlessone 2y agodoesn't training require inference? so i guess it would help there too?
- fzzzy 2y agoTraining doesn't require inference. It uses back-propagation, a different algorithm.
- bitvoid 2y agoBackpropagation happens after some number of inferences. You need to infer to calculate a loss function to then backprop from.
- boringg 2y agoYeah but training requires the larger memory deployment data center infra
- dboreham 2y agoTFP: https://arxiv.org/abs/2410.13166 https://arxiv.org/abs/2410.13166
- vlovich123 2y agoTFS: https://github.com/SakanaAI/evo-memory https://github.com/SakanaAI/evo-memory
- vlovich123 2y agoWonder how this compares with Microsoft's HeadKV paper [1] which claims a 98% percent reduction in memory while retaining 97% of the performance. [1] https://arxiv.org/html/2410.19258v3 https://arxiv.org/html/2410.19258v3
- benatkin 2y agoSeems like a different thing. That paper appears to be memory reduction in caching while article appears to be memory reduction in content.
- vlovich123 2y agoThey’re both exploring the same space of optimizing the memory needed by the KV cache which is essentially another name for the context window (no one elides the KV cache as otherwise you’re doing N^2 math to do attention). They’re exploring different approaches to achieve the same goal and they may be both possible to apply simultaneously to reduce the attention mechanism to almost 0 memory usage which would be really cool, but I’m curious how they compare against each other individually.
- benatkin 2y agoThat sounds like a stretch to me. If not, I’m impressed how the articles can describe such similar things in such different terms.
- vlovich123 2y agoThe only memory mechanism within an LLM as far as I know is the attention mechanism where it compares all previous tokens to generate a probability distribution for the next token to generate. The attention mechanism has a thing called a KV cache to take the O(n^2) matrix math down to O(n) by caching and reusing the results of some math from previous tokens. The size of how many tokens the context will cover is called the context window (e.g. 128k for Llama). The articles use very similar verbiage. > The context window can be considered the model’s working memory Snip > Universal transformer memory optimizes prompts using neural attention memory models (NAMMs), simple neural networks that decide whether to “remember” or “forget” each given token stored in the LLM’s memory. snip > Meanwhile, by discarding unnecessary tokens, NAMM enabled the LLM model to save up to 75% of its cache memory while performing the tasks. You just have to be familiar with the wording in the space and read enough literature. Here’s more direct wording from the NAMM paper: > NAMMs use evolution to optimize the performance of LMs by pruning their KV cache memory. Evolved NAMMs can be zero-shot transferred to other transformers, even across input modalities and task domains. This is all related work about shrinking the size of the KV cache as the context grows both due to memory and it also has a speed up effect since you’re not having to attend all the tokens (O(n) -> sublinear with the size of the context). Context is critical in the LLM answering correctly and remembering all the information given to it + everything it said. Typical limits for open models these days are 128k but with techniques like this it could scale even further allowing better performance on thing like code completion.
- asdev 2y ago[flagged]
- popinman322 2y agoGoogle Trends make it seem like we're out of the exponential growth phase for LLMs-- search interest is possibly plateauing. A decline in search interest outside of academia makes sense. The groups who can get by on APIs don't care so much how the sausage is made and just want to see prices come down. Interested parties have likely already found tools that work for them. There's definitely some academic interest outside of CS in producing tools using LLMs. I know plenty of astro folks working to build domain specific tools with open models as their backbone. They're typically not interested in more operational work, I guess because they operate under the assumption that relevant optimizations will eventually make their way into public inference engines. And CS interest in these models will probably sustain for at least 5-10 more years, even if performance plateaus, as work continues into how LLMs function. All that to say, maybe we're just seeing the trend die for laypeople?
- mewpmewp2 2y agoBut "Large language model" as a topic in google trends is still in its peak. Maybe just everyone who would be the audience is already knowledgeable about LLMs so why would Google Search trends be able to keep rising? ChatGPT is at it's peak, and something like Claude is still rising.
- Agentus 2y agowell gary marcus a non lay person is helping spread word that ai winter is again upon us. but maybe statistical learning from pretraining is near its limit. not enough data or not enough juice to squeeze more performance out of averages. though with all the narrow ais it does seem plausible you might be able to cram all what these narrow ais can do in on big goliath model. wonder if reinforcement learning and reasoning can manage to keep the exponential curve of ai going if there are still hiccups in the short term. the difficulty in just shoehorning llms as they are in any and every day task without a hitch might be behind the temporary hype-dying down trend.
- yishanchuan 2y agointeresting
- lawlessone 2y agoSo it's like a garbage collector for prompts?
- cowsandmilk 2y agoMore of lossy compression
- tharmas 2y agoDoes this mean us plebs can run LLMs on gimped VRAM Nvidia lower end cards?
- swifthesitation 2y agoI don't think so. It seems to just lower the ram needed for the context window. Not for loading the model on the vram.
- voltagex_ 2y ago[flagged]
- hackernewds 2y agoLike what?
- h0l0cube 2y agoOh, I don’t know, how about reducing the search space/accelerating the search speed for potential room temperature superconductors? Or how about the same for viable battery chemistries?
- iandanforth 2y agoBoringness classifier! Pretty cool because this implies the large models already know what is useless and what isn't.
- lxgr 2y ago> NAMMs are trained separately from the LLM and are combined with the pre-trained model at inference time […]
- odyssey7 2y agoIs it possible that after 3-4 years of performance optimizations, both algorithmic and in hardware efficiency, it will turn out that we didn’t really need all of the nuclear plants we’re currently in the process of setting up to satisfy the power demands of AI data centers?
- lm28469 2y agoIt's called the rebound effect, at no point in modern history efficiency reduced our energy needs, we just use the extra energy to either run more of the same thing or run other things
- ndr 2y agoWirth's law: software is getting slower more rapidly than hardware is becoming faster. I think there's the energy parallel: Software is becoming more energy-hungry faster than algorithms are becoming efficient. So we'll still need the energy.
- thom 2y agoWhat if it's a big hoax and we create a better world for nothing?
- huijzer 2y agoI guess power demands will slowly grow. The same happened with compute in general. Compared to 1960, we have several orders of magnitude more compute but also several orders of magnitude more efficient compute. Data centers are currently about 0.4% of total energy use (electricity is about 20% of total energy use and of the electricity about 2% goes to data centers, so 20% * 2% = 0.4%).
- amelius 2y agoNo idea, but it may also turn out that OpenAI has no moat, which is more interesting.
- hoseja 2y agoThen you can simply have more AI in the data centres.
- 2y ago
- ironfootnz 2y agoI'm a big fan of their papers, this one didn't disappoint
- gcanyon 2y agoGiven that the algorithms powering present LLM models hadn't been invented ten years ago, I have to think that they are (potentially) far from optimal. Brains have gone through millions of iterations where being efficient was a huge driver of success. We should not be surprised if someone finds a new ML method that is both wildly more efficient and wildly more effective.
- mycall 2y agoPerhaps LLM++ will start iterating the algorithms via synthetic data until they are far more optimal
- solarkraft 2y agoIt’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.
- baq 2y agoYou have a sota multi-modal LLM running in your head at 20W, shared with best in class sensor package and top performing robotics control unit. There’s soooo much more to optimize.
- fecal_henge 2y agoBut can it know love?
- baq 2y agoWord on the street is researchers looked at the weights and weights looked back. You’ll have to ask the weights.
- myrmidon 2y agoI would argue that our sensor package is losing its lead very quickly-- audio performance is already on par with current tech and image processing is closing the gap very quickly as well (it helps a lot that silicon-based technology is much less constrained on bandwidth). Tactile sensing is still lightyears ahead, and I don't see that situation improving anytime soon...
- IWeldMelons 2y agochemical sensing is still quite good though.
- deleted 2y ago[deleted]
- 2y ago
- bamboozled 2y agoReally exciting news.