4 ms·

Increase GPT4's context size by asking it to compress your prompts

40 points by stolenmerch 4y ago

daydream 4y ago
A promising and potentially useful approach for adding additional context to your prompts. How likely is it that context size will greatly increase in the coming year or two? Are there fundamental limits, or could we reasonably expect greatly increased context size in the future?
- addisonl 4y ago
  We saw an almost 10x jump between GPT-3.5-turbo and gpt-4 (4k to 32k tokens).
  - daydream 4y ago
    But are we likely to get another 10x jump with say GPT4.5, and 10x over that with GPT5? Or is context likely to stay fixed around 32k for the foreseeable future? That is the question.
danShumway 4y ago
If you look into this into more detail: A) people have had a hard time reproducing it, and B) more damning, the "compressed" version uses more tokens than the original (https://gist.github.com/VictorTaelin/d293328f75291b23e203e9d9db9bd136 https://gist.github.com/VictorTaelin/d293328f75291b23e203e9d...) > In practice, plain, well-designed summaries should be optimal to fit larger documents in the context. > This concept has potential, though; building lookup tables seems to outperform long text summarization. It's a clever idea, and I agree that lookup tables and external storage of memory is likely going to be important at some point, but I suspect that's going to come out of giving LLM more ability to externally reference "long-term" memory rather than compressing everything into immediate context.
- catchnear4321 4y ago
  People are smashing the duplos together too hard. Sessions are getting treated as more valuable than necessary. This should look more like functions. Needs lower prices and greater availability for that. But until then… smashing duplos together.
- gfodor 4y ago
  I spent some time on this, here is a better version: https://mobile.twitter.com/gfodor/status/1643415357615640577 https://mobile.twitter.com/gfodor/status/1643415357615640577
catchnear4321 4y ago
This is the most disappointing part about the frenzy. Obvious ideas being held up as something more than that. If you think this is incredible, give yourself some time to go play with LLMs. Try things. Clever things. Stupid things. It helps set a more reasonable scale for these sensational sounding snippets.
mileseva 4y ago
This post from @gfodor is also a really interesting read on the topic of compressing prompts, not sure what the main use case would be though! https://twitter.com/gfodor/status/1643444605332099072?s=20 https://twitter.com/gfodor/status/1643444605332099072?s=20
Alex_Bell 4y ago
ChatGPT 3.5 in response to a brief analysis of the community gave out: "It seems like the conversation you provided is discussing the idea of compressing prompts or summaries of larger documents in order to fit them into a smaller context. While the idea has potential, it appears that people have had difficulty reproducing it and that lookup tables and external storage of memory may be more effective in the long term. Additionally, there is discussion about the possibility of greatly increasing context size in the future, with the example given that there was an almost 10x jump between GPT-3.5-turbo and GPT-4 in terms of token capacity."
dr_kiszonka 4y ago
ZipGPT is a novel, LLM-based non-deterministic compression algorithm. Give our beta version a try. The results may surprise you!
ftxbro 4y ago
The compressed one is shorter number of characters but does it have fewer tokens? https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
sippeangelo 4y ago
One reply points out that even though the text length is 1/4 of the original, the “compressed” output is actually LONGER in token count! https://twitter.com/dogeofcoin/status/1642918892602290179 https://twitter.com/dogeofcoin/status/1642918892602290179
Imnimo 4y ago
A more accurate title would be "decrease GPT4's context size by asking it to obfuscate your prompts". You're getting a prompt that consumes more tokens and isn't particularly faithful to the original.
getpost 4y ago
The context length includes all of GPT4's replies, is that correct? If I want to minimize the context length, should I be asking GPT4 to limit its replies to just the fix that is requested? GPT4 seems to prefer long-winded replies, i.e., when I ask for what amounts to a one-line fix, it repeats the entire block of code with the one line correction. In contrast, Replit's Ghostwriter often gives a concise reply showing only the one-line fix, and I have to ask for the entire block when it isn't clear where the fix is applied.

40 points by stolenmerch 4y ago

A promising and potentially useful approach for adding additional context to your prompts. How likely is it that context size will greatly increase in the coming year or two? Are there fundamental limits, or could we reasonably expect greatly increased context size in the future?

addisonl 4y ago
We saw an almost 10x jump between GPT-3.5-turbo and gpt-4 (4k to 32k tokens).
- daydream 4y ago
  But are we likely to get another 10x jump with say GPT4.5, and 10x over that with GPT5? Or is context likely to stay fixed around 32k for the foreseeable future? That is the question.

danShumway 4y ago

If you look into this into more detail: A) people have had a hard time reproducing it, and B) more damning, the "compressed" version uses more tokens than the original (https://gist.github.com/VictorTaelin/d293328f75291b23e203e9d9db9bd136 https://gist.github.com/VictorTaelin/d293328f75291b23e203e9d...) > In practice, plain, well-designed summaries should be optimal to fit larger documents in the context. > This concept has potential, though; building lookup tables seems to outperform long text summarization. It's a clever idea, and I agree that lookup tables and external storage of memory is likely going to be important at some point, but I suspect that's going to come out of giving LLM more ability to externally reference "long-term" memory rather than compressing everything into immediate context.

catchnear4321 4y ago
People are smashing the duplos together too hard. Sessions are getting treated as more valuable than necessary. This should look more like functions. Needs lower prices and greater availability for that. But until then… smashing duplos together.
gfodor 4y ago
I spent some time on this, here is a better version: https://mobile.twitter.com/gfodor/status/1643415357615640577 https://mobile.twitter.com/gfodor/status/1643415357615640577

catchnear4321 4y ago

This is the most disappointing part about the frenzy. Obvious ideas being held up as something more than that. If you think this is incredible, give yourself some time to go play with LLMs. Try things. Clever things. Stupid things. It helps set a more reasonable scale for these sensational sounding snippets.

mileseva 4y ago

This post from @gfodor is also a really interesting read on the topic of compressing prompts, not sure what the main use case would be though! https://twitter.com/gfodor/status/1643444605332099072?s=20 https://twitter.com/gfodor/status/1643444605332099072?s=20

Alex_Bell 4y ago

ChatGPT 3.5 in response to a brief analysis of the community gave out: "It seems like the conversation you provided is discussing the idea of compressing prompts or summaries of larger documents in order to fit them into a smaller context. While the idea has potential, it appears that people have had difficulty reproducing it and that lookup tables and external storage of memory may be more effective in the long term. Additionally, there is discussion about the possibility of greatly increasing context size in the future, with the example given that there was an almost 10x jump between GPT-3.5-turbo and GPT-4 in terms of token capacity."

dr_kiszonka 4y ago

ZipGPT is a novel, LLM-based non-deterministic compression algorithm. Give our beta version a try. The results may surprise you!

ftxbro 4y ago

The compressed one is shorter number of characters but does it have fewer tokens? https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer

sippeangelo 4y ago

One reply points out that even though the text length is 1/4 of the original, the “compressed” output is actually LONGER in token count! https://twitter.com/dogeofcoin/status/1642918892602290179 https://twitter.com/dogeofcoin/status/1642918892602290179

Imnimo 4y ago

A more accurate title would be "decrease GPT4's context size by asking it to obfuscate your prompts". You're getting a prompt that consumes more tokens and isn't particularly faithful to the original.

getpost 4y ago

The context length includes all of GPT4's replies, is that correct? If I want to minimize the context length, should I be asking GPT4 to limit its replies to just the fix that is requested? GPT4 seems to prefer long-winded replies, i.e., when I ask for what amounts to a one-line fix, it repeats the entire block of code with the one line correction. In contrast, Replit's Ghostwriter often gives a concise reply showing only the one-line fix, and I have to ask for the entire block when it isn't clear where the fix is applied.