3 ms·

> If you want to summarize a document and it's less than 16,000 tokens, you pipe it into gpt-3.5-turbo-16k. If it could fit within the context length then you

> If you want to summarize a document and it's less than 16,000 tokens, you pipe it into gpt-3.5-turbo-16k. If it could fit within the context length then you wouldn’t need to search it to find relevant portions at all. I’m talking about situations where you want to ask questions about a very long document, using the search approach, and one of those questions might be “please summarize chapters 2 through 8”. The search method will never accommodate that, but a model fine tuned on your data might plausibly.

bbor 3y ago
please summarize chapters 2 through 8 This really isn't related to search since it's "summarize this whole text" instead of any specific subset, but in general, I find the old ML advice of "how would a human expert do this task?" to be very helpful. This would be simple in a number of ways - consider two naive ones off the top of my head: - Iterative: Read and summarize one `MAX_INPUT_TOKENS - SUMMARY_LENGTH` section at a time, adding to your summary as you go along. The sections could be meaningful like chapters, or just arbitrary sets of paragraphs, depending on the context. - Intentful: Intuit a context-appropriate set of questions to ask about the text, use semantic search to find and summarize the answers to those, then clean up all the separate answers into one cohesive one. If your summary is getting too long to consider at once for meta-summaries, just add more layers of the above algos. I.e. combine summaries in a bracket of sorts rather than adding on to a single heap, or search for the relevant parts of the summary to tweak based on a given block of new content. Ultimately, yeah, one inference will only be able to understand as much information as it can fit in its context window. So if you want it to write a novel while holding every word in the novel up to that point in its head at once with equal weight, that's going to be hard. But a) context windows are getting longer b) I have never once seen fine-tuning give that level of context-specific accuracy - it is a language model after all, not a knowledge model! and c) that's kinda a lot to ask. Could a human even do that? Hope that rant has some interesting parts :)

bbor 3y ago

please summarize chapters 2 through 8 This really isn't related to search since it's "summarize this whole text" instead of any specific subset, but in general, I find the old ML advice of "how would a human expert do this task?" to be very helpful. This would be simple in a number of ways - consider two naive ones off the top of my head: - Iterative: Read and summarize one `MAX_INPUT_TOKENS - SUMMARY_LENGTH` section at a time, adding to your summary as you go along. The sections could be meaningful like chapters, or just arbitrary sets of paragraphs, depending on the context. - Intentful: Intuit a context-appropriate set of questions to ask about the text, use semantic search to find and summarize the answers to those, then clean up all the separate answers into one cohesive one. If your summary is getting too long to consider at once for meta-summaries, just add more layers of the above algos. I.e. combine summaries in a bracket of sorts rather than adding on to a single heap, or search for the relevant parts of the summary to tweak based on a given block of new content. Ultimately, yeah, one inference will only be able to understand as much information as it can fit in its context window. So if you want it to write a novel while holding every word in the novel up to that point in its head at once with equal weight, that's going to be hard. But a) context windows are getting longer b) I have never once seen fine-tuning give that level of context-specific accuracy - it is a language model after all, not a knowledge model! and c) that's kinda a lot to ask. Could a human even do that? Hope that rant has some interesting parts :)