3 ms·

https://www.anthropic.com/engineering/contextual-retrieval https://www.anthropic.com/engineering/contextual-retrieval Anthropic found embeddings + BM25 (keywor

by andai 1y ago

https://www.anthropic.com/engineering/contextual-retrieval https://www.anthropic.com/engineering/contextual-retrieval Anthropic found embeddings + BM25 (keyword search) gave the best results. (Well, after contextual summarization, and fusion, and reranking, and shoving the whole thing into an LLM...) But sadly they didn't say how BM25 did on its own, which is the really interesting part to me. In my own (small scale) tests with embeddings, I found that I'd be looking right at the page that contained the literal words in my query and embeddings would fail to find it... Ctrl+F wins again!

noobcoder 1y ago
No cross encoders?
bredren 1y ago
FWIW, the org decided against vector embeddings for Claude Code due in part to maintenance. See 41:05 here: https://youtu.be/IDSAMqip6ms https://youtu.be/IDSAMqip6ms
- mips_avatar 1y ago
  It would also blow up the price/latency of Claude code if every chunk of every file had to be read into haiku->summarized->sent to an embedding model ->reindexed into a project index and that index stored somewhere. Since there’s a lot of context inherent in things like the file structure, storing the central context in Claude.md is a lot simpler. I don’t think them not using vector embeddings in the project space is anything other than an indication that it’s hard to manage embeddings in Claude code.
  - andai 1y ago
    Some agents integrate with code intelligence tools which do use embeddings, right? (As well as "mechanical" solutions like LSPs, I imagine.) I think it's just a case of "this isn't something we need to solve, other companies solve it already and then our thing can integrate with that." Or maybe it's really just marginal gains compared with iterative grepping. I don't know. (Still amazed how well that works!)
    - mips_avatar 1y ago
      I think your last point captures it, for various reasons (RL, inherent structure of code) iterative grepping is unreasonably effective. Interestingly Cursor does use embedding vectors for codebase indexing: https://cursor.com/docs/context/codebase-indexing https://cursor.com/docs/context/codebase-indexing Seems like sometimes Cursor has a better understanding of the vibe of my codebase than Claude code, maybe this is part of it. Or maybe it’s just really marginally important in codebase indexing. Vector dbs still have a huge benefit in less verifiable domains.
- skrebbel 1y ago
  who's "the org"?

by andai 1y ago

noobcoder 1y ago

No cross encoders?

bredren 1y ago

FWIW, the org decided against vector embeddings for Claude Code due in part to maintenance. See 41:05 here: https://youtu.be/IDSAMqip6ms https://youtu.be/IDSAMqip6ms

mips_avatar 1y ago
It would also blow up the price/latency of Claude code if every chunk of every file had to be read into haiku->summarized->sent to an embedding model ->reindexed into a project index and that index stored somewhere. Since there’s a lot of context inherent in things like the file structure, storing the central context in Claude.md is a lot simpler. I don’t think them not using vector embeddings in the project space is anything other than an indication that it’s hard to manage embeddings in Claude code.
- andai 1y ago
  Some agents integrate with code intelligence tools which do use embeddings, right? (As well as "mechanical" solutions like LSPs, I imagine.) I think it's just a case of "this isn't something we need to solve, other companies solve it already and then our thing can integrate with that." Or maybe it's really just marginal gains compared with iterative grepping. I don't know. (Still amazed how well that works!)
  - mips_avatar 1y ago
    I think your last point captures it, for various reasons (RL, inherent structure of code) iterative grepping is unreasonably effective. Interestingly Cursor does use embedding vectors for codebase indexing: https://cursor.com/docs/context/codebase-indexing https://cursor.com/docs/context/codebase-indexing Seems like sometimes Cursor has a better understanding of the vibe of my codebase than Claude code, maybe this is part of it. Or maybe it’s just really marginally important in codebase indexing. Vector dbs still have a huge benefit in less verifiable domains.
skrebbel 1y ago
who's "the org"?