2 ms·

I agree with summary. When I first wanted to tackle a hard problem I thought to reach for fine-tuning with lots of input and output pairs, but it wasn't needed.

I agree with summary. When I first wanted to tackle a hard problem I thought to reach for fine-tuning with lots of input and output pairs, but it wasn't needed. Past few shot and RAG, you can overcome context window limits if you find ways to break a single request into many, each with specific context and then roll them up somehow. This can help get past context window limits. Claude 2 has a large context window, but if you are actually giving that much in prompt examples, to cover tricky edge cases, I've found its better to break things down into multiple steps. And if you can break things up that way, and costs isn't at issue, GPT-4, with lots of few shot examples, and chain of thought seems to give me the best results. Or this is what I found writing a code translator for a language the LLM didn't know. I wrote it down in more details here: https://earthly.dev/blog/build-transpose/ https://earthly.dev/blog/build-transpose/

gk1 3y ago
How much do you make use of Claude's large context window, and has it helped? Or did you basically revert back to small context size + multiple steps?
- saliagato 3y ago
  Not OP but, I finally got access to GPT-4 32k and I thought that for a thing as simple as a summary it would be better than the smaller context-window model. But then I realized that now I have (practically) infinite context in input but what the LLM can output is always between 1-1.5k tokens. That is, I think, because of the samples in its training, not because it can't output a more lengthy output. In short, I think that multiple steps are better, for example in summarization.
- adamgordonbell 3y ago
  I am just using GPT4-8k and breaking tasks up. When I tested with Claude 1 ( 2 wasn't out yet) you could feed it lots of context, but it didn't always seem like it paid attention to it all. It would go off the rails. For example, if I put instructions first, then a lot of context, it would forget the task, so instead put a small explanation, then context, then question, but still it would seem to sometimes loose the plot. For feedback on prose though, I found it Claude very strong. It's nice to be able to give it half of a book, in text format and have it guide you to important parts and summarize sections. Some things, like what are the themes of this work, what's surprising finding, don't work well with vector DBs.
  - gk1 3y ago
    Interesting, thanks for sharing. We also found[1] that context stuffing leads (generally) to worse quality results, as did a recent study.[2] But that's for question-answering use cases and not summarization. [1] https://www.pinecone.io/blog/why-use-retrieval-instead-of-larger-context/ https://www.pinecone.io/blog/why-use-retrieval-instead-of-la... [2] https://arxiv.org/pdf/2307.03172.pdf https://arxiv.org/pdf/2307.03172.pdf
    - adamgordonbell 3y ago
      Great work! Matches my findings but much more rigorous. The tasks I'm doing might be uncommon. Here is a build script in some language you (the llm) understand, and I want to get it translated to a language you don't. And so the context is example conversions and documentation. I've also had some luck with things that boil down to "Here is a very large style guide. now how would you improve this code?". Or "here is a number of examples of feedback on writing to conform to a style. now generate the same on this new input." I found the large context windows and Claude to work quite well in those examples. But, if its possible, breaking it down into multiple steps with less context somehow and using GPT4 even better (though more work).
- ttul 3y ago
  In my experience, Claude 2 is marvellously good at sucking in massive documents and accurately generating responses to questions. I gave it the entire Georgia indictment (minus a few pages of crap at the start that was irrelevant to reduce token count) and it wrote a NYTimes article based on the indictment that could be favourably compared to the actual NYTimes piece summarizing the same. There were no factual errors in its output. I imagine OpenAI is not far behind in expanding the context window of its models. The LLM companies have access to the same techniques and - in my estimation - are just choosing to focus on one aspect or another to address different market needs. For instance, Claude 2 clearly focuses on maximum context window size at the cost of speedy inference and, presumably, inference cost. By contrast, OpenAI seems to be focused on speed and low cost (GPT-3.5) and accuracy (GPT-4) rather than maximum token length.

gk1 3y ago

How much do you make use of Claude's large context window, and has it helped? Or did you basically revert back to small context size + multiple steps?

saliagato 3y ago
Not OP but, I finally got access to GPT-4 32k and I thought that for a thing as simple as a summary it would be better than the smaller context-window model. But then I realized that now I have (practically) infinite context in input but what the LLM can output is always between 1-1.5k tokens. That is, I think, because of the samples in its training, not because it can't output a more lengthy output. In short, I think that multiple steps are better, for example in summarization.
adamgordonbell 3y ago
I am just using GPT4-8k and breaking tasks up. When I tested with Claude 1 ( 2 wasn't out yet) you could feed it lots of context, but it didn't always seem like it paid attention to it all. It would go off the rails. For example, if I put instructions first, then a lot of context, it would forget the task, so instead put a small explanation, then context, then question, but still it would seem to sometimes loose the plot. For feedback on prose though, I found it Claude very strong. It's nice to be able to give it half of a book, in text format and have it guide you to important parts and summarize sections. Some things, like what are the themes of this work, what's surprising finding, don't work well with vector DBs.
- gk1 3y ago
  Interesting, thanks for sharing. We also found[1] that context stuffing leads (generally) to worse quality results, as did a recent study.[2] But that's for question-answering use cases and not summarization. [1] https://www.pinecone.io/blog/why-use-retrieval-instead-of-larger-context/ https://www.pinecone.io/blog/why-use-retrieval-instead-of-la... [2] https://arxiv.org/pdf/2307.03172.pdf https://arxiv.org/pdf/2307.03172.pdf
  - adamgordonbell 3y ago
    Great work! Matches my findings but much more rigorous. The tasks I'm doing might be uncommon. Here is a build script in some language you (the llm) understand, and I want to get it translated to a language you don't. And so the context is example conversions and documentation. I've also had some luck with things that boil down to "Here is a very large style guide. now how would you improve this code?". Or "here is a number of examples of feedback on writing to conform to a style. now generate the same on this new input." I found the large context windows and Claude to work quite well in those examples. But, if its possible, breaking it down into multiple steps with less context somehow and using GPT4 even better (though more work).
ttul 3y ago
In my experience, Claude 2 is marvellously good at sucking in massive documents and accurately generating responses to questions. I gave it the entire Georgia indictment (minus a few pages of crap at the start that was irrelevant to reduce token count) and it wrote a NYTimes article based on the indictment that could be favourably compared to the actual NYTimes piece summarizing the same. There were no factual errors in its output. I imagine OpenAI is not far behind in expanding the context window of its models. The LLM companies have access to the same techniques and - in my estimation - are just choosing to focus on one aspect or another to address different market needs. For instance, Claude 2 clearly focuses on maximum context window size at the cost of speedy inference and, presumably, inference cost. By contrast, OpenAI seems to be focused on speed and low cost (GPT-3.5) and accuracy (GPT-4) rather than maximum token length.