5 ms·

+1; Ideally I want a tool I don't have to specify the context for. If I can point it via config files at my medium-sized codebase once (~2000 py files; 300k LOC

+1; Ideally I want a tool I don't have to specify the context for. If I can point it via config files at my medium-sized codebase once (~2000 py files; 300k LOC according to `cloc`) then it starts to get actually usable. Cursor Composer doesn't handle that and seems geared towards a small handful of handpicked files. Would codebuff be able to handle a proper sized codebase? Or do the models fundamentally not handle that much context?

jahooma 2y ago
Yes. Natively, the models are limited to 200k tokens which is on the order of dozens of files, which is way too small. But Codebuff has a whole preliminary step where it searches your codebase to find relevant files to your query, and only those get added to the coding agent's context. That's why I think it should work up to medium-large codebases. If the codebase is too large, then our file-finding step will also start to fail. I would give it a shot on your codebase. I think it should work.
- cratermoon 2y ago
  RAG is a well-known technique now, and to paraphrase Emily Bender[1], here are some reasons why it's not a solution. The code extruded from the LLM is still synthetic code, and likely to contain errors both in the form of extra tokens motivated by the pre-training data for the LLM rather than the input texts AND in the form of omission. It's difficult to detect when the summary you are relying on is actually missing critical information. Even if the set up includes the links to the retrieved documents, the presence of the generated code discourages users from actually drilling down and reading them. This is still a framing that says: Your question has an answer, and the computer can give it to you. 1 https://buttondown.com/maiht3k/archive/information-literacy-and-chatbots-as-search/ https://buttondown.com/maiht3k/archive/information-literacy-...
  - jahooma 2y ago
    We actually don't use RAG! It's not that good as you say. We build a description of the codebase including the file tree and parsed function names and class names, and then just ask Haiku which files are relevant! This works much better and doesn't require slowly creating an index. You can just run Codebuff in any directory and it works.
    - michaelmior 2y ago
      It sounds like it's arguably still a form of RAG, just where the retrieval is very different. I'm not saying that to knock your approach, just saying that it sounds like it's still the case where you're retrieving some context and then using that context to augment further generation. (I get that's definitely not what people think of when you say RAG though.)
      - brandonchen 2y ago
        Genuine question: at what point does the term RAG lose its meaning? Seems like LLMs work best when they have the right context, and that context must be pulled from somewhere for the LLM. But if that's RAG, then what isn't? Do you have a take on this? Been struggling to frame all this in my head, so would love some insight.
        petesergeant 2y ago
        Not RAG: asking the LLM to generate using its internal weights only RAG: providing the LLM with contextual data you’ve pulled from outside its weights that you believe relate to a query
        brandonchen 2y ago
        Nice, super simple. We're definitely fitting into this definition of RAG then!
        parsimo2010 2y ago
        RAG is a search step in an attempt to put relevant context into a prompt before performing inference. You are “augmenting” the prompt by “retrieving” information from a data set before giving it to an LLM to “generate” a response. The data set may be the internet, or a code base, or text files. The typical examples online uses an embedding model and a vector database for the search step, but doing a web query before inference is also RAG. Perplexity.ai is a RAG (but fairly good quality). I would argue that Codebuff’s directory tree search to find relevant files is a search step. It’s not the same as a similarity search on vector embeddings, and it’s not PageRank, but it is a search step. Things that aren’t RAG, but are also ways to get a LLM to “know” things that it didn’t know prior: 1. Fine-tuning with your custom training data, since it modifies the model weights instead of adding context. 2. LoRA with your custom training data, since it adds a few layers on top of a foundation model. 3. Stuffing all your context into the prompt, since there is no search step being performed.
- asattarmd 2y ago
  What's the fundamental limitation to context size here? Why can't a model be fine-tuned per codebase, taking the entire code into context (and be continuously trained as it's updated)? Forgive my naivety, I don't now anything about LLMs.
- craigds 2y ago
  I'll need to get approval to use this on that codebase. I've tried it out on a smaller open-source codebase as a first step. For anyone interested: - here's the Codebuff session: https://gist.github.com/craigds/b51bbd1aa19f2725c8276c5ad36947e2 - The result was this PR: https://github.com/koordinates/kart/pull/1011 It required a bit of back and forth to produce a relatively small change, and I think it was a bit too narrow with the files it selected (it missed updating the implementations of a method in some subclasses, since it didn't look at those files) So I'm not sure if this saved me time, but it's nevertheless promising! I'm looking forward to what it will be capable of in 6mo.

jahooma 2y ago

Yes. Natively, the models are limited to 200k tokens which is on the order of dozens of files, which is way too small. But Codebuff has a whole preliminary step where it searches your codebase to find relevant files to your query, and only those get added to the coding agent's context. That's why I think it should work up to medium-large codebases. If the codebase is too large, then our file-finding step will also start to fail. I would give it a shot on your codebase. I think it should work.

cratermoon 2y ago
RAG is a well-known technique now, and to paraphrase Emily Bender[1], here are some reasons why it's not a solution. The code extruded from the LLM is still synthetic code, and likely to contain errors both in the form of extra tokens motivated by the pre-training data for the LLM rather than the input texts AND in the form of omission. It's difficult to detect when the summary you are relying on is actually missing critical information. Even if the set up includes the links to the retrieved documents, the presence of the generated code discourages users from actually drilling down and reading them. This is still a framing that says: Your question has an answer, and the computer can give it to you. 1 https://buttondown.com/maiht3k/archive/information-literacy-and-chatbots-as-search/ https://buttondown.com/maiht3k/archive/information-literacy-...
- jahooma 2y ago
  We actually don't use RAG! It's not that good as you say. We build a description of the codebase including the file tree and parsed function names and class names, and then just ask Haiku which files are relevant! This works much better and doesn't require slowly creating an index. You can just run Codebuff in any directory and it works.
  - michaelmior 2y ago
    It sounds like it's arguably still a form of RAG, just where the retrieval is very different. I'm not saying that to knock your approach, just saying that it sounds like it's still the case where you're retrieving some context and then using that context to augment further generation. (I get that's definitely not what people think of when you say RAG though.)
    - brandonchen 2y ago
      Genuine question: at what point does the term RAG lose its meaning? Seems like LLMs work best when they have the right context, and that context must be pulled from somewhere for the LLM. But if that's RAG, then what isn't? Do you have a take on this? Been struggling to frame all this in my head, so would love some insight.
      - petesergeant 2y ago
        Not RAG: asking the LLM to generate using its internal weights only RAG: providing the LLM with contextual data you’ve pulled from outside its weights that you believe relate to a query
        brandonchen 2y ago
        Nice, super simple. We're definitely fitting into this definition of RAG then!
      - parsimo2010 2y ago
        RAG is a search step in an attempt to put relevant context into a prompt before performing inference. You are “augmenting” the prompt by “retrieving” information from a data set before giving it to an LLM to “generate” a response. The data set may be the internet, or a code base, or text files. The typical examples online uses an embedding model and a vector database for the search step, but doing a web query before inference is also RAG. Perplexity.ai is a RAG (but fairly good quality). I would argue that Codebuff’s directory tree search to find relevant files is a search step. It’s not the same as a similarity search on vector embeddings, and it’s not PageRank, but it is a search step. Things that aren’t RAG, but are also ways to get a LLM to “know” things that it didn’t know prior: 1. Fine-tuning with your custom training data, since it modifies the model weights instead of adding context. 2. LoRA with your custom training data, since it adds a few layers on top of a foundation model. 3. Stuffing all your context into the prompt, since there is no search step being performed.
asattarmd 2y ago
What's the fundamental limitation to context size here? Why can't a model be fine-tuned per codebase, taking the entire code into context (and be continuously trained as it's updated)? Forgive my naivety, I don't now anything about LLMs.
craigds 2y ago
I'll need to get approval to use this on that codebase. I've tried it out on a smaller open-source codebase as a first step. For anyone interested: - here's the Codebuff session: https://gist.github.com/craigds/b51bbd1aa19f2725c8276c5ad36947e2 - The result was this PR: https://github.com/koordinates/kart/pull/1011 It required a bit of back and forth to produce a relatively small change, and I think it was a bit too narrow with the files it selected (it missed updating the implementations of a method in some subclasses, since it didn't look at those files) So I'm not sure if this saved me time, but it's nevertheless promising! I'm looking forward to what it will be capable of in 6mo.