4 ms·

Even 100k still seems very limiting for many applications naturally suited to LLMs. What if I want an AI assistant that is specifically trained on a large code

Even 100k still seems very limiting for many applications naturally suited to LLMs. What if I want an AI assistant that is specifically trained on a large codebase? Or all my product’s docs (which might easily exceed 100k characters for a big project). Or one that knows the exact details of the entire tax code? Or one that knows every line of Dostoyevsky’s novels so I can have angsty existential conversations with it? Or that can fully remember conversations with me that stretch on for years? It seems like you’d need fine tuning for these kinds of use cases? Or am I missing something?

treprinum 3y ago
You use a vector database with a query similarity search to find portions of documents that might be relevant to your prompt, rank them, take the top n, put them into the context and run through your LLM to generate an answer to your question.
- fzliu 3y ago
  This paradigm applies for much more than just answering questions - I've found the in-context learning capabilities to be very relevant as well. For example, you can query a vector database and input that information into an LLM and ask it to make a prediction across the input data. https://zilliz.com/blog/ChatGPT-VectorDB-Prompt-as-code https://zilliz.com/blog/ChatGPT-VectorDB-Prompt-as-code
  - treprinum 3y ago
    That's pretty cool, thanks for sharing!
  - alach11 3y ago
    Can you elaborate? I don’t understand how the article you linked is different from normal question answering with context.
dragonwriter 3y ago
You use a (probably vector, ideally) database and a framework of behind-the-scenes prompting that uses it to do a “research loop” to gather info to answer questions. You might also fine tune it on the source you want it to incorporate, but training (including fine tuning) is most useful (as I understand) to provide “understanding”, not exact recall, whereas a searchable auxiliary memory and a framework which accesses it as part of the process of generating user responses is a way to provide exact recall over a larger dataset.
- danenania 3y ago
  Ok, I see what you're saying, but at least for the big codebase example, and I'd imagine for many other applications as well, are you really going to get good output by just loading in bits and pieces, even if they are intelligently selected? It seems like holistic "understanding" is exactly what you want in that case, so that the model can take into account the full architecture and know how every component and sub-component fits together. I wouldn't look forward to a PR for a significant feature from a human developer that just skimmed through all of a project's files and only read a handful of them in depth, so I guess I'm also skeptical this would lead to good results from an LLM.
  - taberiand 3y ago
    I think the suggested process is to ingest all of the code base into a vector database and provide prompts and APIs for the model to search and focus on relevant areas while developing a feature, which is analogous (identical, even) to how a human would approach the process - with long term and short term memory and applying general and specific experience.
    - XenophileJKO 3y ago
      You can also create hierarchical summaries that help the model select the right starting place in the code base. Additionally fine tuning isn't a great fit for mutable knowledge like a code base.
  - 3y ago
SanderNL 3y ago
You basically can't. We haven't developed that kind of AI. Some of us hope that just-in-time retrieval of data from an external source (usually a vector db) is going to work. Like, say, "retrieve chapter 3 of Dostoyevsky's CP and make an interesting comment about it". For the record, you can have plenty of interesting angsty existential conversations with both GPTs about D right now if you so desire. Just preface your dialogue with something slightly more sophisticated than "let us have an angsty existential conversation about Dostoyevsky". This JIT retrieval might work for some cases, but my guess is it won't for large holistic pieces of work where you have to integrate and "understand" the entire edifice at once. I'm not completely sure if codebases merit such a distinction though, but you can imagine a domain like "law" might. We can push the context limit and this might bring some temporary relief, but I'm not convinced. The recent pushes to 100K seem to rely on "dropping" attention in an intelligent way. It sounds like cheating and while it might work OK for some cases, I think it'll drop the ball eventually. The integrated understanding that results from, somehow, internalizing the relation between all the hundreds of thousands of seemingly unrelated datapoints is what makes us interesting. That's also what makes an LLM interesting. It's just that an LLM only has access to that level of development when it's training. If I were to guess I think we somehow need to enable "always-training" and I'm not sure anyone has the faintest idea how. You can only go so far if you step out of school and never learn anything ever again, no matter how brilliant you are. GPT4 is quite the scholar, but we're stretching it.
- danenania 3y ago
  > I think we somehow need to enable "always-training" and I'm not sure anyone has the faintest idea how. To ask the naive question, why can't we just keep continually training it with new stuff? If it takes, say, a year to ingest approximately the whole internet, tacking on a single large codebase should probably take something like an additional few seconds if we consider what fraction that codebase is of the full training set. I know it's not that simple for a lot of reasons, but it doesn't seem obvious that fundamentally new methods are needed to do this.
  - SanderNL 3y ago
    This new piece of information can impact all previous information in unknown ways. To give a dramatic and superficial example, let's say you just found out you have been living inside the Truman Show. This impacts, if not everything, then a whole lot of what you know about your world. Instantly and completely. This cannot be "tacked on". This has to be integrated somehow. Current AI doesn't work like that. Instead of building a tower of knowledge it sort of creates abstract cognitive landscapes filled with predetermined averaged out paths that, when followed, give good answers. Useful for a lot of tasks, but it doesn't strike me as the type of structure that can be manipulated with any kind of precision. Maybe that will change and/or I am wrong. Certainly the last one is very probable.
zbyforgotp 3y ago
Finetuning is not useful for teaching new facts, you need RAG for that: https://zzbbyy.substack.com/p/why-you-need-rag-not-finetuning https://zzbbyy.substack.com/p/why-you-need-rag-not-finetunin...

treprinum 3y ago

You use a vector database with a query similarity search to find portions of documents that might be relevant to your prompt, rank them, take the top n, put them into the context and run through your LLM to generate an answer to your question.

fzliu 3y ago
This paradigm applies for much more than just answering questions - I've found the in-context learning capabilities to be very relevant as well. For example, you can query a vector database and input that information into an LLM and ask it to make a prediction across the input data. https://zilliz.com/blog/ChatGPT-VectorDB-Prompt-as-code https://zilliz.com/blog/ChatGPT-VectorDB-Prompt-as-code
- treprinum 3y ago
  That's pretty cool, thanks for sharing!
- alach11 3y ago
  Can you elaborate? I don’t understand how the article you linked is different from normal question answering with context.

dragonwriter 3y ago

You use a (probably vector, ideally) database and a framework of behind-the-scenes prompting that uses it to do a “research loop” to gather info to answer questions. You might also fine tune it on the source you want it to incorporate, but training (including fine tuning) is most useful (as I understand) to provide “understanding”, not exact recall, whereas a searchable auxiliary memory and a framework which accesses it as part of the process of generating user responses is a way to provide exact recall over a larger dataset.

danenania 3y ago
Ok, I see what you're saying, but at least for the big codebase example, and I'd imagine for many other applications as well, are you really going to get good output by just loading in bits and pieces, even if they are intelligently selected? It seems like holistic "understanding" is exactly what you want in that case, so that the model can take into account the full architecture and know how every component and sub-component fits together. I wouldn't look forward to a PR for a significant feature from a human developer that just skimmed through all of a project's files and only read a handful of them in depth, so I guess I'm also skeptical this would lead to good results from an LLM.
- taberiand 3y ago
  I think the suggested process is to ingest all of the code base into a vector database and provide prompts and APIs for the model to search and focus on relevant areas while developing a feature, which is analogous (identical, even) to how a human would approach the process - with long term and short term memory and applying general and specific experience.
  - XenophileJKO 3y ago
    You can also create hierarchical summaries that help the model select the right starting place in the code base. Additionally fine tuning isn't a great fit for mutable knowledge like a code base.
- 3y ago

SanderNL 3y ago

You basically can't. We haven't developed that kind of AI. Some of us hope that just-in-time retrieval of data from an external source (usually a vector db) is going to work. Like, say, "retrieve chapter 3 of Dostoyevsky's CP and make an interesting comment about it". For the record, you can have plenty of interesting angsty existential conversations with both GPTs about D right now if you so desire. Just preface your dialogue with something slightly more sophisticated than "let us have an angsty existential conversation about Dostoyevsky". This JIT retrieval might work for some cases, but my guess is it won't for large holistic pieces of work where you have to integrate and "understand" the entire edifice at once. I'm not completely sure if codebases merit such a distinction though, but you can imagine a domain like "law" might. We can push the context limit and this might bring some temporary relief, but I'm not convinced. The recent pushes to 100K seem to rely on "dropping" attention in an intelligent way. It sounds like cheating and while it might work OK for some cases, I think it'll drop the ball eventually. The integrated understanding that results from, somehow, internalizing the relation between all the hundreds of thousands of seemingly unrelated datapoints is what makes us interesting. That's also what makes an LLM interesting. It's just that an LLM only has access to that level of development when it's training. If I were to guess I think we somehow need to enable "always-training" and I'm not sure anyone has the faintest idea how. You can only go so far if you step out of school and never learn anything ever again, no matter how brilliant you are. GPT4 is quite the scholar, but we're stretching it.

danenania 3y ago
> I think we somehow need to enable "always-training" and I'm not sure anyone has the faintest idea how. To ask the naive question, why can't we just keep continually training it with new stuff? If it takes, say, a year to ingest approximately the whole internet, tacking on a single large codebase should probably take something like an additional few seconds if we consider what fraction that codebase is of the full training set. I know it's not that simple for a lot of reasons, but it doesn't seem obvious that fundamentally new methods are needed to do this.
- SanderNL 3y ago
  This new piece of information can impact all previous information in unknown ways. To give a dramatic and superficial example, let's say you just found out you have been living inside the Truman Show. This impacts, if not everything, then a whole lot of what you know about your world. Instantly and completely. This cannot be "tacked on". This has to be integrated somehow. Current AI doesn't work like that. Instead of building a tower of knowledge it sort of creates abstract cognitive landscapes filled with predetermined averaged out paths that, when followed, give good answers. Useful for a lot of tasks, but it doesn't strike me as the type of structure that can be manipulated with any kind of precision. Maybe that will change and/or I am wrong. Certainly the last one is very probable.

zbyforgotp 3y ago

Finetuning is not useful for teaching new facts, you need RAG for that: https://zzbbyy.substack.com/p/why-you-need-rag-not-finetuning https://zzbbyy.substack.com/p/why-you-need-rag-not-finetunin...