3 ms·
Consider an alternative approach: chunk up and embed your entire codebase (dramatically cheaper) and insert it into a vector store. when you go to send a query,
by jasonjmcghee 4y ago
Consider an alternative approach: chunk up and embed your entire codebase (dramatically cheaper) and insert it into a vector store. when you go to send a query, search the store and retrieve the most relevant chunks, and those are what you send with your query.
Optionally send other specific code alongside it.
This should be similar in quality and dramatically cheaper. It's quite doable with langchain, and you can use Chroma + DuckDB to avoid having to pay for Pinecone.
You could even maintain there vector store using git diffs to only update chunks that have changed.
- deathmonger5000 4y agoGreat ideas thank you!
- swader999 4y agoYeah this is the way!
- asteroidz 4y agoThe problem with your suggested approach is the resulting lack of holistic context. The problem of OP's approach (direct parsing) is cost and context-window-limits. There has to be a better way.
- jasonjmcghee 4y agoIt's searching by semantic meaning, so it should be able to find all relevant pieces. Using overlap during chunking should help too. Using the "give it everything" method will cause it to forget most of what you're feeding it if you have a large repo anyway, right?
- deathmonger5000 4y agoI don't think it will forget anything as long as everything fits in the context window, but I could totally be wrong. That's the big problem with the "give it everything" approach: if your codebase doesn't fit then it's game over. I've had success limiting what I give it to the relevant files.
- jasonjmcghee 4y agoRight- "forgetting" assuming a rolling context window maxed at the models max token count. "If it doesn't fit then it's game over" - assuming 8k tokens with each token being ~4 characters, that's a pretty small repo. And that's a motivator behind the similarity search approach.