4 ms·
But it's becoming increasingly clear that LLMs based on the transformer model will never be able to scale their context much further than the current frontier,
by bloppe 1y ago
But it's becoming increasingly clear that LLMs based on the transformer model will never be able to scale their context much further than the current frontier, due mainly to context rot. Taking advantage of greater context will require architectural breakthroughs.
- colordrops 1y agoWill it though? The human mind can hold less context at any one time than even a mediocre LLM. The problem isn't architecture. It's capturing context. Most of it is in a bunch of people's heads and encoded in the physical world. Once it's digitized and accessible through search, RAG, or whatever, the LLM will be able to use it effectively.
- prmph 1y agoHuman hold a lot of implicit context, I think far beyond any LLM. Context is not just what you consciously are thinking about in your head
- colordrops 1y agoSure, but so do LLMs models. They have a huge subconscious (the model itself). Recording every conversation a single person ever had, every book or text or site ever read, everything ever seen, is not a huge amount of data. Microsoft attempted this with a digital camera lanyard but they were too early.
- prmph 1y agoYeah, but the models are all based on explicit data. I'm saying humans have prior wiring that allows them to extract and keep context that LLMs do not have access to.
- colordrops 1y agoSo the suggestion here is that RAG, tools, LLM memory, fine tuning, context management etc are not enough to take advantage of all this context? Is there any evidence that these things aren't on a trajectory to be optimized enough to do the job?