4 ms·
They can have memory, it's just limited to their context window.
by dom96 2mo ago
They can have memory, it's just limited to their context window.
- rrr_oh_man 2mo agoIt's not memory in any sense of the word.
- nomel 2mo agohttps://en.wikipedia.org/wiki/Working_memory https://en.wikipedia.org/wiki/Working_memory
- rrr_oh_man 2mo agoAnd... now?
- nomel 2mo agoLLM context window is the same concept as working memory. An LLM loading its working memory from reference a reference text or note you made is similar to how you might, if you forgot something. Context is a type of memory, by all definitions of "memory", with the "past experiences" bit of it being the previous messages sent. You're perspective isn't clear, but if you're arguing that place this context is stored between sessions is too far away from the LLM, in a technical rather than functional sense, then I don't think that small implementation detail of a larger system (that's not even true with modern hosting) is a useful thing to critique.
- rrr_oh_man 2mo agoLLMs are fancy statistical word predictors. There is no functioning concept of memory in the context of (pun intended) LLMs. There are no past experiences. They cannot be referenced. Anything that is being "referenced" by an LLM is another word prediction that makes you anthropomorphize a statistical model.
- nomel 2mo agoSure, but nobody uses an LLM in isolation. They use systems that are built around an LLM. In many of those systems, like a chat interface, the context is almost always used a form of memory. I think I was correct that your disagreements based on the physical location of where the previous context is stored. At the functional system level, with nearly any tool you or I use, the context/RAG/etc is used as memory, loaded with relevant/past symbols. Say I created a new LLM architecture that integrated this concept of only appending to previous context, with no way to clear it. With it now being physically closer/integrated with the bit of code you call an "LLM", would you then agree it was memory? I don't see that implementation detail being all that important.
- rrr_oh_man 2mo agoDude, you’re just going down a rabbit hole with imaginary arguments. Nobody’s talking about your weird location argument.
- nomel 2mo agoI think you miss the simple point I'm trying to make: context can be a practical form of working memory. You said > There are no past experiences. I'm saying that if the context contains something added in the past, it's functionally no different than a "past experience". For example, a context that is a chat log you had with the LLM. If you tell the LLM to remember a word, continue your conversation, and then tell it to recall that word, the system can recall that past experience. That "memory" that is the context feed to the LLM is a form of memory, in some cases. Now you say, "but that chat log storage that you send to the LLM is not part of the LLM!" and then I say "see above".