3 ms·

I’m not trying to come across as a hater or anything, but I’m very curious what “next generation AI apps” will look like. The killer app I can see is a “person

I’m not trying to come across as a hater or anything, but I’m very curious what “next generation AI apps” will look like. The killer app I can see is a “personal copilot” where you train a model on some set of documents or code or both and it can give you a ChatGPT-like interface to your domain. But as far as I know, that’s going to be insanely expensive or time consuming if done at a personal level. The UX of selecting the code and documents doesn’t seem straightforward either and seems like something that would need to be done at the enterprise level. But after that, my imagination sort of dries up.

YetAnotherNick 3y ago
> But as far as I know, that’s going to be insanely expensive or time consuming if done at a personal level. rewind.ai is already doing this. Also I think we can personal LoRA(low parameter diff to model) that keeps evolving as we use computer.
simonw 3y ago
"But as far as I know, that’s going to be insanely expensive or time consuming if done at a personal level" I think it's going to be really cheap. Don't fall for the common trap of assuming that you need to fine-tune a model in order to implement Q&A etc against a large corpus of documents. It turns out it's much more effective (and much easier / less expensive) to do the Bing/Bard/etc trick, where you use an off-the-shelf LLM and run searches (including fancy embeddings vector-database searches) against your content, then feed the top matches into the LLM as part of a prompt that includes the user's question. I wrote about one way of doing that back in January: https://simonwillison.net/2023/Jan/13/semantic-search-answers/ https://simonwillison.net/2023/Jan/13/semantic-search-answer... - it's got even easier to implement since then. ChatGPT functions and ChatGPT 3.5 getting cheaper (and gaining a 16k token context) makes this even more accessible today than it was yesterday.
- mrtranscendence 3y ago
  It’s much easier to run searches and insert snippets from high-ranking results into the query. There are limitations to this approach, though. For example, how would you summarize a document this way? You’ll get snippets but not the whole picture. I’ve also found that GPT-4 tends to hallucinate when I ask a question but the search process doesn’t result in a good match. Like making up an author or title for a document I’m asking about.
  - beoberha 3y ago
    I have a very surface level knowledge of all this stuff, but this is what I would think would happen as well. Seems like a fairly lossy process that might work for “point lookup” questions but not “range scans” like summarizing.
  - simonw 3y ago
    Not sure I understand your point about summarizing a document. If you want to summarize a document and it's less than 16,000 tokens, you pipe it into gpt-3.5-turbo-16k. If it's less than 32,000 and you can afford the cost, you pipe it into gpt-4-32k. Hallucination continues to be a problem, though GPT-4 is massively less prone to it than 3.5. My instinct right now is that you can resolve a lot of that problem for the Q&A case using additional careful prompt engineering, but I don't yet have a recipe I'm 100% confident in myself.
    - mrtranscendence 3y ago
      > If you want to summarize a document and it's less than 16,000 tokens, you pipe it into gpt-3.5-turbo-16k. If it could fit within the context length then you wouldn’t need to search it to find relevant portions at all. I’m talking about situations where you want to ask questions about a very long document, using the search approach, and one of those questions might be “please summarize chapters 2 through 8”. The search method will never accommodate that, but a model fine tuned on your data might plausibly.
      - bbor 3y ago
        please summarize chapters 2 through 8 This really isn't related to search since it's "summarize this whole text" instead of any specific subset, but in general, I find the old ML advice of "how would a human expert do this task?" to be very helpful. This would be simple in a number of ways - consider two naive ones off the top of my head: - Iterative: Read and summarize one `MAX_INPUT_TOKENS - SUMMARY_LENGTH` section at a time, adding to your summary as you go along. The sections could be meaningful like chapters, or just arbitrary sets of paragraphs, depending on the context. - Intentful: Intuit a context-appropriate set of questions to ask about the text, use semantic search to find and summarize the answers to those, then clean up all the separate answers into one cohesive one. If your summary is getting too long to consider at once for meta-summaries, just add more layers of the above algos. I.e. combine summaries in a bracket of sorts rather than adding on to a single heap, or search for the relevant parts of the summary to tweak based on a given block of new content. Ultimately, yeah, one inference will only be able to understand as much information as it can fit in its context window. So if you want it to write a novel while holding every word in the novel up to that point in its head at once with equal weight, that's going to be hard. But a) context windows are getting longer b) I have never once seen fine-tuning give that level of context-specific accuracy - it is a language model after all, not a knowledge model! and c) that's kinda a lot to ask. Could a human even do that? Hope that rant has some interesting parts :)
bbor 3y ago
I think it might help to consider which of today's apps would be better if they understand you on an intuitive level. Thinking of apps where users directly run inferences (AKA ChatGPT-but-for-X) is missing the forest for the trees, IMO. For the latest example I've been excited about: the Doordash app sucks, and we spent 10 minutes the other day trying to convince it to let us order from the Boba franchise in the next town over rather than the one close by (ahh, the wonders of Silicon Valley, a true Eden). If the app was powered by an LLM with access to a series of API endpoints rather-than/in-addition-to the traditional visual interface, one can easily imagine how such a problem could be solved in seconds. Suddenly, it becomes possible to design interfaces that fit niche/edge/unexpected usecases much more reliably.

YetAnotherNick 3y ago

> But as far as I know, that’s going to be insanely expensive or time consuming if done at a personal level. rewind.ai is already doing this. Also I think we can personal LoRA(low parameter diff to model) that keeps evolving as we use computer.

simonw 3y ago

"But as far as I know, that’s going to be insanely expensive or time consuming if done at a personal level" I think it's going to be really cheap. Don't fall for the common trap of assuming that you need to fine-tune a model in order to implement Q&A etc against a large corpus of documents. It turns out it's much more effective (and much easier / less expensive) to do the Bing/Bard/etc trick, where you use an off-the-shelf LLM and run searches (including fancy embeddings vector-database searches) against your content, then feed the top matches into the LLM as part of a prompt that includes the user's question. I wrote about one way of doing that back in January: https://simonwillison.net/2023/Jan/13/semantic-search-answers/ https://simonwillison.net/2023/Jan/13/semantic-search-answer... - it's got even easier to implement since then. ChatGPT functions and ChatGPT 3.5 getting cheaper (and gaining a 16k token context) makes this even more accessible today than it was yesterday.

mrtranscendence 3y ago
It’s much easier to run searches and insert snippets from high-ranking results into the query. There are limitations to this approach, though. For example, how would you summarize a document this way? You’ll get snippets but not the whole picture. I’ve also found that GPT-4 tends to hallucinate when I ask a question but the search process doesn’t result in a good match. Like making up an author or title for a document I’m asking about.
- beoberha 3y ago
  I have a very surface level knowledge of all this stuff, but this is what I would think would happen as well. Seems like a fairly lossy process that might work for “point lookup” questions but not “range scans” like summarizing.
- simonw 3y ago
  Not sure I understand your point about summarizing a document. If you want to summarize a document and it's less than 16,000 tokens, you pipe it into gpt-3.5-turbo-16k. If it's less than 32,000 and you can afford the cost, you pipe it into gpt-4-32k. Hallucination continues to be a problem, though GPT-4 is massively less prone to it than 3.5. My instinct right now is that you can resolve a lot of that problem for the Q&A case using additional careful prompt engineering, but I don't yet have a recipe I'm 100% confident in myself.
  - mrtranscendence 3y ago
    > If you want to summarize a document and it's less than 16,000 tokens, you pipe it into gpt-3.5-turbo-16k. If it could fit within the context length then you wouldn’t need to search it to find relevant portions at all. I’m talking about situations where you want to ask questions about a very long document, using the search approach, and one of those questions might be “please summarize chapters 2 through 8”. The search method will never accommodate that, but a model fine tuned on your data might plausibly.
    - bbor 3y ago
      please summarize chapters 2 through 8 This really isn't related to search since it's "summarize this whole text" instead of any specific subset, but in general, I find the old ML advice of "how would a human expert do this task?" to be very helpful. This would be simple in a number of ways - consider two naive ones off the top of my head: - Iterative: Read and summarize one `MAX_INPUT_TOKENS - SUMMARY_LENGTH` section at a time, adding to your summary as you go along. The sections could be meaningful like chapters, or just arbitrary sets of paragraphs, depending on the context. - Intentful: Intuit a context-appropriate set of questions to ask about the text, use semantic search to find and summarize the answers to those, then clean up all the separate answers into one cohesive one. If your summary is getting too long to consider at once for meta-summaries, just add more layers of the above algos. I.e. combine summaries in a bracket of sorts rather than adding on to a single heap, or search for the relevant parts of the summary to tweak based on a given block of new content. Ultimately, yeah, one inference will only be able to understand as much information as it can fit in its context window. So if you want it to write a novel while holding every word in the novel up to that point in its head at once with equal weight, that's going to be hard. But a) context windows are getting longer b) I have never once seen fine-tuning give that level of context-specific accuracy - it is a language model after all, not a knowledge model! and c) that's kinda a lot to ask. Could a human even do that? Hope that rant has some interesting parts :)

bbor 3y ago

I think it might help to consider which of today's apps would be better if they understand you on an intuitive level. Thinking of apps where users directly run inferences (AKA ChatGPT-but-for-X) is missing the forest for the trees, IMO. For the latest example I've been excited about: the Doordash app sucks, and we spent 10 minutes the other day trying to convince it to let us order from the Boba franchise in the next town over rather than the one close by (ahh, the wonders of Silicon Valley, a true Eden). If the app was powered by an LLM with access to a series of API endpoints rather-than/in-addition-to the traditional visual interface, one can easily imagine how such a problem could be solved in seconds. Suddenly, it becomes possible to design interfaces that fit niche/edge/unexpected usecases much more reliably.