3 ms·

Long context is a mistake. You don’t remember 30,000 words when you read the next word in a book. All of that has been summarised down aggressively to a tiny wo

Long context is a mistake. You don’t remember 30,000 words when you read the next word in a book. All of that has been summarised down aggressively to a tiny working memory. It should be a series of caches of increasing compression. The words before should be cached exactly as they are. Sentences before start focusing on essential phrases. Pages before are topics. Instead of training on 50,000 chars plus the current char it would be 5000 chars of cache. And if you do the cache right it will contain endless knowledge compared to just storing the entire text.

davidkunz 3y ago
> Pages before are topics. What if there's an important detail hidden in a topic?
xkcd-sucks 3y ago
Recently I've been doing some data modeling stuff that essentially boils down to aligning and identifying common concepts between several JSONSchema each of about 30000-50000 characters without whitespace. For example, "This field `$.run.acquisitionTime` in the first schema means the same thing as `$.operation.experiment_timestamp` in the second schema; but generally for more esoteric concepts than datetime. Naturally, I cannot remember most of the content and use text editors / home built data review tools to augment my very limited working memory. It is slow. I would love to chuck them all into ChatGPT4 and have that magically align and cross reference them, which is just not possible with limited context. There are perhaps some clever solutions involving progressive abstraction and stuff, which I do not have the time to figure out. So TL;DR maintaining a context much larger than human working memory is a value prop of computers in general, so it feels like it should be a value prop of LLMs too
amenhotep 3y ago
Aggressive summarisation to a tiny working memory does not seem like the right model to describe human recollection. Obviously we don't remember 30,000 words individually, but we're way better at remembering random seemingly insignificant stuff than a current LLM asked to summarise into a tiny context could possibly manage. Absent a good model for implementing human memory, increasing the context window seems a better approach than trying to fit literally everything the model should know into a summary and accepting that once any information at all falls out of the summary it's gone forever.

davidkunz 3y ago

> Pages before are topics. What if there's an important detail hidden in a topic?

xkcd-sucks 3y ago

Recently I've been doing some data modeling stuff that essentially boils down to aligning and identifying common concepts between several JSONSchema each of about 30000-50000 characters without whitespace. For example, "This field `$.run.acquisitionTime` in the first schema means the same thing as `$.operation.experiment_timestamp` in the second schema; but generally for more esoteric concepts than datetime. Naturally, I cannot remember most of the content and use text editors / home built data review tools to augment my very limited working memory. It is slow. I would love to chuck them all into ChatGPT4 and have that magically align and cross reference them, which is just not possible with limited context. There are perhaps some clever solutions involving progressive abstraction and stuff, which I do not have the time to figure out. So TL;DR maintaining a context much larger than human working memory is a value prop of computers in general, so it feels like it should be a value prop of LLMs too

amenhotep 3y ago

Aggressive summarisation to a tiny working memory does not seem like the right model to describe human recollection. Obviously we don't remember 30,000 words individually, but we're way better at remembering random seemingly insignificant stuff than a current LLM asked to summarise into a tiny context could possibly manage. Absent a good model for implementing human memory, increasing the context window seems a better approach than trying to fit literally everything the model should know into a summary and accepting that once any information at all falls out of the summary it's gone forever.