3 ms·

Ok, I see what you're saying, but at least for the big codebase example, and I'd imagine for many other applications as well, are you really going to get good o

Ok, I see what you're saying, but at least for the big codebase example, and I'd imagine for many other applications as well, are you really going to get good output by just loading in bits and pieces, even if they are intelligently selected? It seems like holistic "understanding" is exactly what you want in that case, so that the model can take into account the full architecture and know how every component and sub-component fits together. I wouldn't look forward to a PR for a significant feature from a human developer that just skimmed through all of a project's files and only read a handful of them in depth, so I guess I'm also skeptical this would lead to good results from an LLM.

taberiand 3y ago
I think the suggested process is to ingest all of the code base into a vector database and provide prompts and APIs for the model to search and focus on relevant areas while developing a feature, which is analogous (identical, even) to how a human would approach the process - with long term and short term memory and applying general and specific experience.
- XenophileJKO 3y ago
  You can also create hierarchical summaries that help the model select the right starting place in the code base. Additionally fine tuning isn't a great fit for mutable knowledge like a code base.
rolisz 3y ago
But finetuning wouldn't give the LLM a holistic understanding. LLMs are finetuned to predict the next token. At best you give them a full file at a time, but more likely, you create batches of similar length sequences. So while finetuning will lead to the LLM seeing all the codebase, it won't necessarily know how all the files fit together (for example I'm pretty sure it won't see folder structure, unless you somehow include it via preprocessing)
- danenania 3y ago
  While I grant that it doesn't have the ability to go into single-file or line-level granularity, GPT-4 can have a highly informed discussion on the architectures of big projects in its training set like PostgreSQL or the Linux kernel. That's what I mean by "holistic understanding". If you're a contributor for one of those projects (or want to be), it could definitely help to evaluate the tradeoffs of various design approaches for new features or modifications. Getting into the nitty gritty of files and functions would be a step further, but it seems to me that just having this general knowledge of how a big project is structured would vastly improve any output related to that project compared to just loading in comparatively tiny pieces of it. Even if those tiny pieces are the most relevant to the prompt, it will be missing all the background context that it has for projects in its training set.
  - rolisz 3y ago
    I expect that for Linux or PostgreSQL it was trained not just on the codebase, but also on mailing lists and tons of other documents that discuss various things about it and that's what give it these reasoning powers.

taberiand 3y ago

I think the suggested process is to ingest all of the code base into a vector database and provide prompts and APIs for the model to search and focus on relevant areas while developing a feature, which is analogous (identical, even) to how a human would approach the process - with long term and short term memory and applying general and specific experience.

XenophileJKO 3y ago
You can also create hierarchical summaries that help the model select the right starting place in the code base. Additionally fine tuning isn't a great fit for mutable knowledge like a code base.

rolisz 3y ago

But finetuning wouldn't give the LLM a holistic understanding. LLMs are finetuned to predict the next token. At best you give them a full file at a time, but more likely, you create batches of similar length sequences. So while finetuning will lead to the LLM seeing all the codebase, it won't necessarily know how all the files fit together (for example I'm pretty sure it won't see folder structure, unless you somehow include it via preprocessing)

danenania 3y ago
While I grant that it doesn't have the ability to go into single-file or line-level granularity, GPT-4 can have a highly informed discussion on the architectures of big projects in its training set like PostgreSQL or the Linux kernel. That's what I mean by "holistic understanding". If you're a contributor for one of those projects (or want to be), it could definitely help to evaluate the tradeoffs of various design approaches for new features or modifications. Getting into the nitty gritty of files and functions would be a step further, but it seems to me that just having this general knowledge of how a big project is structured would vastly improve any output related to that project compared to just loading in comparatively tiny pieces of it. Even if those tiny pieces are the most relevant to the prompt, it will be missing all the background context that it has for projects in its training set.
- rolisz 3y ago
  I expect that for Linux or PostgreSQL it was trained not just on the codebase, but also on mailing lists and tons of other documents that discuss various things about it and that's what give it these reasoning powers.