5 ms·
δ-mem: Efficient Online Memory for Large Language Models
- DeathArrow 5mo agoI see lots of techniques proposed to give LLM the capacity to recall things, I even saw a lot of memory plugins for AI coding agents, I tried some myself. What I want to see is something that was tested and proved in practice to be genuinely useful, especially for coding agents.
- stephantul 5mo agoHow would you conceptualize recall in this case? Is searching through the current version of your code and possibly git history not enough?
- rush86999 5mo agoYou would think git history should be the first thing an agent would look at, as they make so many mistakes before they get to the correct answer. They don't. I haven't measured, but documenting bug fixes and architecture seems to help, along with TDD patterns, including integration tests. I would probably add it to Claude.md to look for all of the above when tackling a new bug.
- visarga 5mo agoI made a harness that preserves memory for both user messages and task execution. One reason this works is related to judge agents - they can't review information that was not written down. So I track everything in my harness. The judge agents bring the most benefit, based on my evals. The coding agent can execute a task without all the ceremony just as well, but judging needs something to grasp on, besides code. And adding new perspectives helps a lot, it is the most useful intervention. My flow is - user emits a task, the agent plans, then judge agents review the plan, then main agent executes, then judge again reviews the execution. Might consume more tokens to track execution and judgements, but worth it.
- brookst 5mo agoMy Claude code frequently looks through git history, both when planning and debugging.
- DeathArrow 5mo ago>Is searching through the current version of your code and possibly git history not enough? While you can document everything and use git history, I think that having short entries in a kind of memory to remember past decisions, how issues were solved would be much more token efficient than reading lots of documentation and looking at git history and past code.
- cjonas 5mo agoCoding agents don't really need memory. Agent skills, rules, git history, documentation is all far more efficient, transparent and easier to manage. These memory frameworks only really makes sense if you are building a consumer facing agent with managed context and limited capabilities.
- wren6991 5mo agoThere's an antipattern where everyone wants to invent new interfaces to connect things LLMs when CLI tools are already right there, transparent, and usable by humans as well as LLMs. I think it's partly the origins in web chat applications. Beads kind of does "LLM memory over CLI", or there is https://github.com/wedow/ticket https://github.com/wedow/ticket which is a minimal and sane implementation of the same idea.
- pohl 5mo agoThere’s probably never going to be one answer. The most fascinating thing about this quest for memory is that it’s a Rorschach test. Exploring the myriad attempts to implement memory shows that everyone has a slightly different itch they’re trying to scratch, but we talk about it like we all want the same thing.
- ktallett 5mo agoThe obvious energy saving step would be to utilise previous searches by others. Many of the tasks people do are rather similar, it is such an energy waste to start again each time. (Obviously ignoring the huge energy saver, which is to observe if you even need to bother doing the task at all.)
- duskdozer 5mo agoA lot of what I see people using LLMs for would be more cheaply and reliably done by [scripts]. A search engine style suggestion thing like "Have you tried `sed`?" would be beneficial imo
- tyre 5mo agoIn my experience, Claude is more than happy to go to Unix tools rather than write its own. Sometimes it will write a lil python script to solve something, but more often than not it’ll pipe together Unix utilities. This has the benefit of it knowing all of the arcane flags, especially for formatting output.
- duskdozer 5mo agoI believe that. I also believe that my idea won't come to fruition, at least from a group that is incentivized to make a user's first instinct be to use their product and not an external tool.
- 405126121 5mo agoI had this thought and created https://pushrealm.com https://pushrealm.com which is essentially a sort of Stackoverflow written by agents. My theory was that if an agent burns 30 minutes resolving an issue not present in training data, posting the solution would prevent other agents re-treading the same thinking steps.
- spockz 5mo agoSo you mean caching? :-)
- zhenglei11 5mo ago[flagged]
- 3form 5mo agoInteresting points: - fixed size of the memory seems like a good idea to overcome the current limitations - skimming through the thing, I can't find any mention of the cost? - I would need more time to read it in-depth to see if this is legitimate and not just fancy form of overfitting or training on testing data
- usernametaken29 5mo ago> δ-mem compresses past information into a fixed-size state matrix updated by delta-rule learning This doesn’t solve the capacity problem of memory. You can cram more into one context window, but then again you need to associate them with input queries. That’s very hard because slight variations in input create hugely different activations. So really, it doesn’t improve caching. This paper might do a thing or two approximating the compression limit for context windows, but there’s a fundamental limit on how much information can go into it. What you really need is contextual search, as in, different events and objects with the same abstractions and semantic lead to same response, so you can cache effectively… on this front the paper does little to improve “memory” in a meaningful way
- jsemrau 5mo agoI am currently working on deep context query which uses dynamically generated regex to pull only the relevant context blocks. By using lightweight RegEx pattern matching to detect semantic intent and filter structured context sections accordingly, you avoid the attention degradation that comes from stuffing semantically redundant information into the window https://jdsemrau.substack.com/p/tokenmaxxing-and-optimizing-context https://jdsemrau.substack.com/p/tokenmaxxing-and-optimizing-...
- structuredPizza 5mo agoThe more real world use cases we see, the more we see the use of a well thought out regex as a bridge from probabilistic to deterministic.
- pbronez 5mo agoInteresting approach. > Prioritize recall over precision. Have you tried stemming your regex? That would help you catch messages where a different form of your word appeared. For example instead of “story” you look for “stor” which catches “stories” as well. Then you might think, could we do an even better job by figuring out the general semantic intent of the query and history? Let’s project them into a semantic vector space! That’s an embedding. Then you want to query that, which means you need a vector database. So now we can take the query, embed it, query the vector DB with that embedding and retrieve the N closest history documents. You can use that to augment the generation of the response to your prompt. This is RAG. Anyway, interesting to see different degrees of sophistication here. Certainly a handful of naive regex are very snappy. There’s probably a hybrid approach where you use sophisticated NLP and embedding techniques to robustly define topics, then train a regex to approximate that well.
- raverbashing 5mo agoInteresting that the headline is showing Δ-Mem while the paper uses δ-mem Is it a lowercase to uppercase conversion going on here?
- sillysaurusx 5mo agoCorrect!
- cubefox 5mo agoPapers being voted high on Hacker News are usually uncorrelated with their actual importance. It's basically a lottery. There are regularly more interesting papers going semi viral on Twitter.
- MeteorMarc 5mo agoOn huggingface it was #3 paper of the day, which is neutral towards your hypothesis.
- cubefox 5mo agoConsidering that there is a paper with this many points perhaps once a week here (probably less), #3 of the day is pretty unremarkable.
- kingkawn 5mo agoWhat about broad unsupportable generalizations on hackernews, how do those rank?
- belabartok39 5mo ago[flagged]
- semiquaver 5mo agoHmm, this is a case where HN’s title mangling changed the meaning of the title. Lower case delta (δ) is used intentionally. I don’t think HN should automatically modify the casing of non-ascii chars.
- setopt 5mo agoEven for ASCII chars, nomenclature in math and physics is usually case-sensitive.
- airstrike 5mo agoThe submitter has a grace period of a few minutes to edit the title after submitting, so there's no need to change what HN does
- realitysballs 5mo agoTrue, but wouldn’t it be better long term if website automation didn’t create unintended new meanings to Titles? title’s matter
- airstrike 5mo agoOnly if you assume it doesn't ever work as intended.
- throw1234567891 5mo agoindeed, titles matter
- cwillu 5mo agoEmail hn@ycombinator.com and they'll fix it.
- djoldman 5mo agoI would love for the standard to be to ALWAYS report the required amount of memory to load and run a model in bytes of RAM alongside any other metrics. I'd love to see time to first token, token throughput, token latency as well but I'd settle for memory size as described above. Essentially, many people want to know what the minimum amount of memory is to run a particular model. Parameter count obscures important details: what are the sizes of the parameters? A parameter isn't rigorously defined. This also gets folks into trouble because a 4B param model with FP16 params is very different from a 4B param model with INT4 params. The former obviously should be a LOT better than the second. This would also help with MOE models: if memory is my constraint, it doesn't matter if the (much larger RAM required) MOE version is faster or has better evals. I'm waiting for someone in anger to ship the 1 parameter model where the parameter according to pytorch is a single parameter of size 4GB.
- adrian_b 5mo agoAs a proxy for the total size of the parameters, you can just look at the download size of a model on Huggingface.co. Because for most models the weights are provided in many *.safetensors files of approximately the same size, you can estimate the total size without adding all file sizes by multiplying the number of *.safetensors files with the approximate size of one file. For quantized models, estimating the size is simpler, because there is just one GGUF file, which also includes metadata, but most of the file is occupied by the parameters. While there are models where the native size of all parameters is BF16, there are also models that use multiple parameter sizes, e.g. a large number of parameters with a small size, even down to 4 bits, together with a small number of parameters with a bigger size, up to FP32. Therefore, as you say, the number of parameters is much less informative about memory requirements than the file sizes. While the download size of the *.safetensors files or GGUF files is not the same as the total memory requirement, it can give an approximate estimate and it can be used to assess which of 2 models will need more memory. It becomes more complicated when you must use multiple kinds of memory, e.g. GPU memory and CPU memory, or even SSDs, when you must know more about the structure of the model to determine how much of each kind of memory is needed.
- 5mo ago
- raymondchau 5mo ago[flagged]
- xiaod 5mo ago[flagged]
- in-silico 5mo agoThey basically just added DeltaNet hypernetworks to existing LLMs. Nothing super novel or groundbreaking, but a moderately interesting read.
- maxignol 5mo agoIs there some kind of memory enabling, for instance, an agent to remember guidelines on a repo without having to feed at the beginning of each session 4 markdown files and spending the corresponding tokens each time ?
- airstrike 5mo agoNo, it's all just prompts. You can try to summarize memories tersely and point the agent to longer markdown files, but who knows if it will read it at the right time and only then.
- xcvbnu 5mo ago[dead]
- jmward01 5mo agoThe future is fixed size state with a massive token history that the model can look back at like reading a journal. A reframing of the model this way opens a new kind of agent, one with essentially unlimited context, that packs perfectly on a GPU, can be stored/retrieved fairly effortlessly and can essentially be run forever. Fixed size means theta 1 tokens. A model that can look around also means essentially unlimited memory can be bolted on with the model learning to look around memory like it is looking around at the journal of past tokens. Guided windows of attn can do most of this, some other tricks can do the rest.
- Sim-In-Silico 5mo ago[flagged]