11 ms·

Show HN: Semble – Code search for agents that uses 98% fewer tokens than grep

445 points by Bibabomas 5mo ago

Hey HN! We (Stephan and Thomas) recently open-sourced Semble. We kept running into the same problem while using Claude Code on large codebases: when the agent can't find something directly, it falls back to grep, reading full files or launching subagents. This uses a lot of tokens, and often still misses the relevant code. There are existing tools for this, but they were either too slow to index on demand, needed API keys, or had poor retrieval quality. Semble is our solution for this. It combines static Model2Vec embeddings (using our latest static model: potion-code-16M) with BM25, fused via RRF and reranked with code-aware signals. Everything runs on CPU since there's no transformers involved. On our benchmark of ~1250 query/document pairs across 63 repos and 19 languages, it uses 98% fewer tokens than grep+read and reaches 99% of the retrieval quality of a 137M-parameter code-trained transformer, while being ~200x faster. Main features: - Token-efficient: 98% fewer tokens than grep+read - Fast: ~250ms to index a typical repo on our benchmark, ~1.5ms per query on CPU (very large repos may take longer) - Accurate: 0.854 NDCG@10, 99% of the best transformer setup we tested - MCP server: drop-in for Claude Code, Cursor, Codex, OpenCode - Zero config: no API keys, no GPU, no external services Install in Claude Code with: claude mcp add semble -s user -- uvx --from "semble[mcp]" semble Or check our README for other installation instructions, benchmarks, and methodology: Semble: https://github.com/MinishLab/semble https://github.com/MinishLab/semble Benchmarks: https://github.com/MinishLab/semble/tree/main/benchmarks https://github.com/MinishLab/semble/tree/main/benchmarks Model: https://huggingface.co/minishlab/potion-code-16M https://huggingface.co/minishlab/potion-code-16M Let us know if you have any feedback or questions!

eddyaipt 5mo ago
[flagged]
esafranchik 5mo ago
Is the benchmark measuring one-shot retrieval accuracy, or Coding agent response accuracy?
- stephantul 5mo ago
  Hey! Co-author here. The benchmark currently only measures retrieval accuracy. We’re interested in measuring it end to end and also optimizing, e.g. the prompt and tools, for this, but we just haven’t gotten around to it.
  - esafranchik 5mo ago
    Two follow-ups: 1) How do you compare accuracy? by checking if the answer is in any of the returned grep/bm25/semble snippets? 2) How do you measure token use without the agent, prompt, and tools?
    - stephantul 5mo ago
      1) yes! It’s not accuracy, but ndcg 2) we assume that if the agent gets the correct answer in the returned snippets it does not need to read further
      - esafranchik 5mo ago
        Wouldn't NDCG/token results vary wildly depending on the agent's query and the number of returned items? e.g. agents often run `grep -m 5 "QUERY"` with different queries, instead of one big grep for all items.
        stephantul 5mo ago
        The same holds for semble: the agent can fire off many different semble queries with different k/parameters. I guess the point we’re trying to make is that you need fewer semble queries to achieve the same outcome, compared to grep+readfile calls.
ludicrousdispla 5mo ago
grep doesn't need tokens, so what is 98% fewer than zero?
- stephantul 5mo ago
  You need readfile to do something with those tokens. Grep only gives you the matching lines, not the context.
  - djaboss 5mo ago
    `grep -C $NUM` ? ;)
    - stephantul 5mo ago
      Even so. Take a look at the NDCG numbers for grep. It's not pretty
      - hparadiz 5mo ago
        ripgrep exists though
        stephantul 5mo ago
        The comparison is with ripgrep, see the benchmarks.
- mrweasel 5mo ago
  We might not be AI/agent enough for this product. I wondered about that as well, but it's not actually a grep problem, it's an ingest problem for agents. Apparently some agents use grep to navigate code and it's this "operation" that consume tokens, grep does in fact consume zero tokens.
mrpf1ster 5mo ago
Does this work well for non-coding documents as well? Say api docs or AI memory files?
- stephantul 5mo ago
  Hey, this is something we're actively investigating. We recently added a flag, `--include-text-files`, which, when set, also makes Semble index regular documents (i.e., markdown, text, json). This should also work relatively well.
jerezzprime 5mo ago
I'd be interested in seeing actual agent benchmarks (eg CC or Copilot CLI with grep removed and this tool instead). For example, I have explored RTK and various LSP implementations and find that the models are so heavily RL'd with grep that they do not trust results in other forms and will continually retry or reread, and all token savings are lost because the model does not trust the results of the other tools.
- stephantul 5mo ago
  Yeah we're also interested in doing this, it's on the roadmap together with optimization of the prompt and descriptions so that models have an easier time using it. Perhaps anecdotally: we do use this tool ourselves of course, and it's been working pretty well so far. Anthropic models call it and seem to trust the results.
- giancarlostoro 5mo ago
  I forced Claude to have a global memory for RTK and my own AI memory system (GuardRails) which it happily uses both, the only times it doesnt use GuardRails is if I dont mention it at all, otherwise it always uses RTK unless RTK falls apart running a tool it does not support.
- deleted 5mo ago
  [deleted]
- nextaccountic 5mo ago
  Codex CLI is quite happy running RTK. Well with GPT 5.5 xhigh anyway One thing that irks me is that when it doesn't support eg. a cli flag of find, it gives an error message rather than sending the full output of the command instead. Then the agent wastes tokens retrying, or worse, doesn't even try because the prompting may make them afraid to not run commands without rtk
  - aleksiy123 5mo ago
    how effective is RTK for you? worth using?
singpolyma3 5mo ago
Semantic code search seems like a useful tool for a human too. Not just for agents.
- stephantul 5mo ago
  Yeah I agree. I have used semble to quickly index a large monorepo and just ask a question about it, it surfaced the right files pretty quickly. Although without an IDE, it's difficult to display them in nice way
vikeri 5mo ago
very curious to give it a spin but why write a cli in python? would surely be faster and more portable with go or rust?
- skeledrew 5mo ago
  Perhaps Python is their main language (they seem to be ML peeps, which would make that most likely), which means it's easier for them to do manual reviews even if they're using AI for implementing, etc.
  - stephantul 5mo ago
    Yes, this is the main reason. We've released some rust stuff in the past, but Python is our main language
smcleod 5mo ago
How does it compare to context-mode or serina that are both well established now?
- Bibabomas 5mo ago
  Serena does a lot more than semble (I actually used serena before building this and didn't like how much it does by default). That also made it hard to see if it was actually working well with how many moving parts there are. Semble only does 1 thing: very quick code search, that's it. Context-mode I have not used before though, I will have a look at that, thanks for sharing!
porker 5mo ago
Congratulations on the release! Could you add fff to the benchmarks?
- stephantul 5mo ago
  We hadn't found that one yet. Will do!
abcdefg12 5mo ago
Shouldn’t it be a part of the harness at least for local codebase? I wonder how many harnesses are doing that already.
- dopidopHN2 5mo ago
  I'm playing with PI as a custom harness ( for Claude code because that what is provided to me ) I will try that ! It make sense and I'm curious to see results, for this or any similar projects mentioned in the thread
  - molszanski 5mo ago
    allegedly this one is good for PI https://pi.dev/packages/@ff-labs/pi-fff https://pi.dev/packages/@ff-labs/pi-fff
- Bibabomas 5mo ago
  Afaik many harnesses ship the "default" which is grep+read (like Claude Code). But I agree, IMO it's a weird gap. To be fair I don't think providers are that incentivised to reduce token burn at the moment, but my guess is that that will change and tools like this will become at least an natively supported option in some harnesses.
  - sincerely 5mo ago
    I would be surprised if, in the "mature" future of AI tools/products, the labs building the models are also selling/building to end users like they are now.
    - losvedir 5mo ago
      Would or wouldn't be? I think it makes sense that they will be because of the feedback loop on training. The lab generates tons of example text as part of the training run and if they have that using their own tools then the models will tend to prefer those tools.
nextaccountic 5mo ago
How does this compare with colgrep? https://github.com/lightonai/next-plaid/tree/main/colgrep https://github.com/lightonai/next-plaid/tree/main/colgrep
- stephantul 5mo ago
  The comparison is in the benchmarks, see the README
AussieWog93 5mo ago
Better than grep obviously, but how does this compare to existing LSPs?
- CharlesW 5mo ago
  Or tools like `ck`: https://beaconbay.github.io/ck/ https://beaconbay.github.io/ck/
  - cormacrelf 5mo ago
    Try running both on the CK codebase. CK takes like 15 minutes to index itself and gives hundreds of completely irrelevant doc comments as results for “run model on CPU” query. Semble indexes for like 3 seconds and prints out the actual code that runs the model on the CPU.
    - CharlesW 5mo ago
      You didn’t use `ck` directly, you instructed Claude Code to use `ck`, right?
      - cormacrelf 5mo ago
        No? CK is better than I gave it credit for, didn’t take 15 minutes, took 2, somehow a lot faster than before, probably system busy. I was using hybrid which is wrong for this query. Still semble is a few orders of magnitude faster and gave better results against ck —-sem. I am running both on rust-lang/rust and CK is going to take hours at least, extrapolating from current stats probably 3 days? Semble: 26 seconds without any caching. The thing doesn’t have a cache and it’s still massively faster. I added caching support and watchman integration and got it down to 1.4 seconds. 3 days is basically not good enough for this use case. It’s slow enough that indexing is going to lag your code changes. Semble is fast enough that it’s not going to be behind.
      - xeyownt 5mo ago
        Tried both right now. Tried against a 84K loc C project. ck took at least 5 minutes to index, but replies are indeed fast. semble indexing (if any) took no noticeable time (except for the first download of HF model, which took a couple seconds), and replied in a couple of seconds. Unrelated but ck was a pain to install / compile (install instructions do not say you have to lock the build / you have to have latest libc).
_ink_ 5mo ago
Would this replace something like codebase-memory-mcp[1] or improve when both is being used? [1] - https://github.com/DeusData/codebase-memory-mcp https://github.com/DeusData/codebase-memory-mcp
- Bibabomas 5mo ago
  [dead]
jahala 5mo ago
This looks great! I built a tool in the same space- and I found that the biggest challenge was often to get the agent to prefer to use the tool over bash tools. What’s your experience with that?
- Escapade5160 5mo ago
  Setup hooks. Hooks are how your harness forces compliance with your own rules.
- Bibabomas 5mo ago
  This is mostly done with the prompt. To be fair, we are still evaluating and improving this as well!
wrxd 5mo ago
I also like the index feature form https://maki.sh https://maki.sh Source code has a lot of structure, using a real parser instead of grepping and reading files can potentially save a lot of tokens
- Bibabomas 5mo ago
  Interesting, will have a look at this, thanks for sharing.
ramsono 5mo ago
Very useful thanks for sharing!
esperent 5mo ago
I did some evals with pi and GPT 5.5. I tested RTK on / headroom on / both on / both off (all with the standard pi system instructions and no AGENTS.md). I forget the exact tests I used (a couple of the standard agent evals that people use, one python and one typescript because those are what I use). I don't claim it was an exhaustive test, or even a good one. It's possible I could have spent a day or so tuning my AGENTS.md and the pi system prompt/tool instructions and gotten better results, because if there's one thing running evals taught me it's that subtle differences there can change the results a lot. However, I got clearly better results with both off, enough to convince me to stop the tests immediately after 3 rounds. The problem was that while context use did go down (sometimes), the number of turns to complete went up so the overall cost of the conversation was higher. It's made me very aware of one thing: so many people are sharing these kind of tools, but either with zero evals (or suspiciously hard to reproduce), or in the case of this one, extensive benchmarks testing the wrong thing. I'm sure this tool does use fewer tokens than grep, and the benchmarks prove it, but that's not what matters here. What matters is, does an agent using it get the same quality of work done more quickly and for lower cost?
- zobzu 5mo ago
  with AI the "they could so they never wondered if they should" will be a very frequent thing.
  - jack_pp 5mo ago
    yeah I think I'm prone to do the same, it is so easy to create and we get too excited by it instead of first doing the research necessary which is much more boring than actually producing something.
  - stephantul 5mo ago
    This is a bit rude. We didn't generate this project, we wrote it, a lot of it manually, and trained custom models. We'd been working in the real-time retrieval space for a while, and we thought coding was a good fit for this specific technology.
onoesworkacct 5mo ago
fantastic token savings and performance... but unlike grep it's probabilistic search on search terms. is that an issue? the tiny model might not surface something important
- stephantul 5mo ago
  It's not probabilistic, and exact matches will always be preferred over non-exact. So if you search for a function name this will surface it.
jasonli0226 5mo ago
thanks for sharing!
andai 5mo ago
Nice, this sounds great. I want to mention a related issue here, which is that on small codebases, Claude spends a lot of time looking for stuff when it could have just dumped the whole codebase into the context in one go and used very little tokens. I found a nice workaround which is that you can just dump the whole directory into context, as a startup hook. So then Claude skips the "fumble around blindly in the dark" portion of every task. (I've also seen a great project that worked on bigger repos where it'll give the model an outline with stubs, though I forget what it was called.)
- mackenney 5mo ago
  Maybe aider? https://aider.chat/2023/10/22/repomap.html https://aider.chat/2023/10/22/repomap.html
  - Imanari 5mo ago
    Good old aider ahead of its time
  - andai 5mo ago
    I actually made a custom harness based on Aider's edit format (the find/replace thing). (I think most AI stuff ended up using a very similar format.) It just does what I need and no more: load code into context, append my question or instruction, call LLM, apply patches. Repeat. I haven't used Aider itself though, maybe it does that too. My harness was nice relative to Claude Code and Codex because, it doesn't need to poke around the filesystem (cause I have small repos and dump the whole thing), and it makes all edits simultaneously (doesn't need to edit one file at a time). It reads all files and edits all necessary files in a single round trip. The really nice thing is that when you're making many small fine grained changes like that, you can use a much smaller, faster, cheaper model. And if it's fast enough, it actually becomes a real-time activity. It's not "prompt, wait..." but "prompt, immediately get the result." It's interactive. You stay active and engaged. It's great.
- stephantul 5mo ago
  This is true, agents just don't know a lot about the things they're looking at, e.g., the number of files, file sizes, etc. Although for small codebases it also holds that whatever you would like to find it easy to find, so search still might help you with cost
aadishv 5mo ago
Seems like a cool idea so I decided to play with it a bit. The test I ran was in the browsercode (https://github.com/browser-use/browsercode https://github.com/browser-use/browsercode) repo with the following prompt: "Answer this question by only using the `semble` CLI (docs below): > What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work --- [the AGENTS.md snippet provided from https://github.com/MinishLab/semble#bash-integration https://github.com/MinishLab/semble#bash-integration]" And the equivalent for the non-Semble test: "Answer this question by only using the `rg` and `fd` CLIs: > What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work" In both cases, I used Pi with gpt-5.4 medium and a very minimal setup otherwise. (And yes, I did verify that either instance only used rg & fd, or only used semble.) Without Semble, it used 10.9% of the model context and used $0.144 of API credits (or, at least, that's what Pi reported - I used this with a Codex sub so cannot be sure). With Semble, it used 9.8% of the model context and $0.172 of API credits. The resulting responses were also about the same. Very close! I tried one more test in the OpenCode repo. The question was > Trace the path from 1) the OPENCODE_EXPERIMENTAL_EXA env var being set to to 1 to 2) the resulting effects in the system prompt or tool provided to the OpenCode agent. And I included the same instructions/docs as above. The non-Semble version was a bit more detailed -- it went into whether the tool call path invoked Exa based on whether Exa or Parallel was enabled for the web search provider -- but w.r.t. actually answering the question, both versions were accurate. The Semble version used 14.7% context / $0.282 API cost, while the non-Semble version used 19.0% / $0.352. Clearly a win for Semble for context efficiency, but note that the non-Semble version finished about twice as fast as the Semble version. Of course this is just me messing around. ymmv.
- stephantul 5mo ago
  Wow awesome, thanks for sharing! This is really useful and very much like the experiments we want to be doing in the near future
vemulasukrit 5mo ago
Nice!
gslepak 5mo ago
Does this support any language or is it limited to a specific set of languages?
- stephantul 5mo ago
  For chunking Semble supports all languages supported by tree-sitter-language-pack. The models we train are trained on 6 languages, but can handle way more.
deleted 5mo ago
[deleted]
ind-igo 5mo ago
[dead]
boyter 5mo ago
Interesting. I too have been working in this space, though I took a different approach. Rather than building an index, I worked on making a "smarter grep" by offering search over codebases (and any text content really) with ranking and some structural awareness of the code. Most of my time was spend dealing with performance, and as a result it runs extremely quickly. I will have to add this as a comparison to https://github.com/boyter/cs https://github.com/boyter/cs and see what my LLMs prefer for the sort of questions I ask. It too ships with MCP, but does NOT build an index for its search. I am very curious to see how it would rank seeing as it does not do basic BM25 but a code semantic variant of it. This seems to work better for the "how does auth work" style of queries, while cs does "authenticate --only-declarations" and then weighs results based on content of the files, IE where matches are, in code, comments and the overall complexity of the file. Have starred and will be watching.
- Bibabomas 5mo ago
  Nice! Let us know if you have any feedback or results to share, would be happy to do the same.
- zikani_03 5mo ago
  I was going to share a link to this. Thank you for making `cs`, I use it both with LLMs and directly in the terminal, despite not performing indexing it's pretty fast for my needs. Also definitely planning to try out semble.
  - boyter 5mo ago
    You are welcome. Glad to hear its working for you. I have a few ideas I am working on to improve its relevance too that I hope pan out.
freakynit 5mo ago
What I have personally observed with such tools is that they make the AI's dumb, similar to how it makes coders dumb when relying more on AI tools. These agentic AI's are already smart enough to figure out a highly optimized path to code exploration or search. But, with these tools, they just go very aggressive, partly because the search results from these tools almost in 100% of the cases do not furnish full details, but, just the pointers. To confirm this behaviour, I did a small test run. This is in no way conclusive, but, the results do align with what I been observing: --- Task: trace full ingestion and search paths in some okayish complex project. Harness is Pi. 1. With "codebase-memory-mcp": 85k/4.4k (input/output tokens). 2. With my own regular setup: 67k/3.2k. 3. Without any of these: 80k/3.2k. As we see, such a tool made it worse (not by much, but, still). The outputs were same in quality and informational content. --- Now, what my "regular setup" mentioned above is?: Just one line in AGENTS.md and CLAUDE.md: "Start by reading PROJECT.md" . And PROJECT.md contains just following: 2-3 line description of the project, all relevant files and their one-line description, any nuiances, and finally, ends with this line: ## To LLM Update this file if the changes you have done are worth updating here. The intent of this file is to give you a rough idea of the project, from where you can explore further, if needed.
- Bibabomas 5mo ago
  Hey, codebase-memory-mcp and semble are not exactly the same, but it's an interesting comparison, I'll put it on the todolist to check that out and add it to our benchmarks if feasible. If you ever get a chance to do this same comparison with semble it would be super useful feedback since these "real" scenarios are hard to benchmark/replicate.
  - freakynit 5mo ago
    So, I just tested with semble. Your MCP integration did not work, and kept throwing error (Failed to connect to "semble": MCP error -32000: Connection closed) though I installed using documented manner (tried both: pip and ux methods). Anyways, I made it work by making it generate relevant doc (using semble init), and then copying this into AGENTS.md, and then prompting it with this line: """ Start by reading AGENTS.md in current folder. Now, the task::: `Explore the ingestion and search paths. Do not read README.md at all`. Prefer to use `semble` search for code search. Do not do new installation. semble is already available at `/Users/nitinbansal/.local/bin/semble` . """ The results are much better. Even better than my own setup, but, vary a lot. I did 4 runs: 95k/2.9k 25k/2.7k 71k/2.9k 37k/4.0k
digitaltrees 5mo ago
Cool project. I built a custome IDE and coding agent harness and will integrate this into it. If you’re interested in a collaboration, I’d be happy to share revenue to sponsor your open source repo. https://calendly.com/ryanwmartin/open-office-hours https://calendly.com/ryanwmartin/open-office-hours
zhxiaoliang 5mo ago
The instructions on how to install and use it could use some work. I did eventually install it. Will try it later and report back.
- stephantul 5mo ago
  Oh sorry that happened. Feel free to open an issue or report it here
billetkit 5mo ago
[flagged]
therealdrag0 5mo ago
How does that compare to Cursors’s workspace indexing? Also curious what the authors think about Claude team explicitly trying out indexing and deciding against it.
- Bibabomas 5mo ago
  The first is hard to test for us unfortunately since we don't use Cursor. But the Claude thing is interesting. I think that providers (especially the ones that directly sell LLM calls like Anthropic) are not incentivised per se to think about token efficiency vs performance, so if you're chasing pure performance, just loading the full codebase into memory might still be the "benchmark topping" way to go. I think the dust hasn't really settled yet and we'll likely see a lot of changes in the coming year about what's the "correct" way to solve it. It might be different based on your harness/budget/model as well.
handonam 5mo ago
The one thing I'm a bit nervous about: security. Thoughts of supply-chain "what-ifs" gives me a bit of pause here. Would like to hear security-minded folks give their thoughts on this.
- Bibabomas 5mo ago
  Hey, we do a couple of things specifically to prevent supply-chain attacks. We use trusted publishing on PyPI, and --exclude newer for uv's package resolution. We also try to use the least amount of dependencies possible. A transitive dependency could in theory still be problematic though, e.g. if there's a supply-chain attack on numpy. The tool itself is fully local though, so there's no real security risks there, there are no outbound network calls or anything like that.
PufPufPuf 5mo ago
The savings are calculated against the assumption that agents read the matched files in their entirety. In my experience, they are smart enough to use grep to bring up a few lines of surrounding context, then read in full only the files that look relevant.
- Bibabomas 5mo ago
  In practice this rarely happens though, at least in practice I rarely see agents "grep -C N" or something like that on files it didn't read yet. I use Claude Code and OpenCode extensively, and especially during the first pass through a codebase that is not well understood the agent often just does "cat file" or something similar and gets the entire file in context first, and only then starts doing more finegrained searches, but at that point you already have a lot of irrelevant context in memory. I think the whole value proposition of semble is that you don't have to do that initial read at all and can instead get the right (small) context bits. If you experience is different, would you mind sharing what your setup is like, e.g. how do you get the agent to read less?
cagz 5mo ago
Would be nice to see the actual % token/time saving across end-to-end coding sessions over these time periods https://github.com/MinishLab/semble#savings https://github.com/MinishLab/semble#savings My observation is that greps and the processing of grep outputs account for only a small portion of overall consumption; I haven't measured this scientifically though.
- Bibabomas 5mo ago
  Yeah this is a good point, but it's complicated to do well since semble then has to be aware of everything else that happens in a session, which would then make it more intrusive (and it's deliberately designed as a local, non-intrusive alternative to other solutions). I'm thinking that perhaps we can do an isolated opt-in benchmark for this perhaps.
derrickrburns 5mo ago
Have you considered solving a different problem? What are agents trying to achieve when searching a code base? Finding seams. How about indexing seams instead?
- Bibabomas 5mo ago
  What do you mean by this exactly?
RyanJohn 5mo ago
[dead]
florians 5mo ago
“> Claude: Semble surfaced things grep missed — here are the additions to the earlier answer.” Nice!
sonink 5mo ago
The bigger problem with solutions like these is that most AI already know how to use grep and search really well because of their training. Any such new tool that you handle to the AI, takes away from the cognitive capability of the AI. Humans would normally 'learn' how to operate tools like this - but the learning in LLM's is frozen and they already with a very strong depth in existing tools like grep. For example, an AI would already use linux commands like tree to traverse the code base. And again it already has good training in this. The other problem is that it is easy to cook up examples which demonstrate the efficacy of tools like these - but actually proving that the cognitive deficit that such tools result it, is surmounted by their efficacy in long horizon runs. My first contact instinct is that this will result in a net negative 'deployable intelligence' over long horizon runs - make the agent perform worse than using existing tools. Proving the opposite is a non-trivial problem - but maybe it might be something you want to take up.
- Bibabomas 5mo ago
  [dead]
xiaosong001 5mo ago
[flagged]
pu_pe 5mo ago
> uses 98% fewer tokens than grep So are we supposed to believe that grep is so wasteful that models are reading 98% useless garbage every time they call it? Either this claim is not representative, or you're missing something else when you throw away the vast majority of context for the model.
- boyter 5mo ago
  Grep prints out every matching line. For some searches a LLM might do it will get a lot of noise, and it might have to make that search because it cannot be specific. Targeted search can reduce the number of tokens. I suspect this comparison is against reading the whole codebase though compared to just getting the bits you need.
- pas 5mo ago
  I had problems with Claude reading hundreds of kilobytes of outputs because grep found things in node_modules. (ripgrep helps, so it makes sense to add a line about it into some memory file.)
- Bibabomas 5mo ago
  The 98% is vs the grep+read loop, not grep output alone. When an agent hits an unfamiliar codebase it typically does "cat file" or reads the whole thing first, at least in my experience. If you're reliably getting agents to do "grep -C N" and stop there I'd genuinely be curious what your setup looks like, because I think the quality of the results is just too low to serve as useful context.
  - ac29 5mo ago
    > When an agent hits an unfamiliar codebase it typically does "cat file" or reads the whole thing first, at least in my experience. Depends on the size of the project and specific files. I have definitely seen agents make smart use of pi's "read" tool, which can take an offset and line limit (or defaults to a max 2000 lines/50KiB if the model doesn't specify). The bash tool also has the same max output, so if a model decides to cat instead of using the read tool it still wont blow out its context window with a single large file read. But this sort of thing is going to vary with harness, model, project, and whatever the RNG delivers for the day.
Aleesha_hacker 5mo ago
[flagged]
shermantanktop 5mo ago
Do inefficient grep/sed behaviors have a secondary benefit in seeding the context with breadcrumbs of code from irrelevant matches?
- Bibabomas 5mo ago
  In theory maybe, but in practice it hurts more than it helps I think. Irrelevant context makes the model more likely to reason from the wrong code (and it's slower and more expensive).
getmarketingai 5mo ago
[dead]
yuanyanorz 5mo ago
[flagged]
cityofdelusion 5mo ago
Feedback: codex-cli hangs when calling this through the MCP. The semble process even sticks around as a zombie, forever stalled out. No idea why, logs have nothing. When called through a skill via CLI style calling, GPT 5.5 loves to give a ton of search terms like it is used to doing with ripgrep. Not sure how effective this is, the short docs in the github and the instructions the agent has isn't clear on what is optimal. Lastly, I got some errors with external connections to github when I was installing it for bash use. Maybe its related to the hanging? No idea. edit: My agent also loves to follow-on with ripgrep, which seems redundant. Acts like it has trust issues. I think a more extensive agent skill description could guide the agent into proper use.
- Bibabomas 5mo ago
  Hey, thanks for the detailed feedback. For the bug, would you mind opening an issue with your setup details? This is definitely something we want to investigate and fix. The multiple queries thing is really good feedback, thanks for that, we'll update the prompt/instructions to prevent this from happening and we'll try to add some tests for this. The external connection errors during install are uv fetching deps from PyPI I think, those should not be the reason it's hanging.
michal_lola2 5mo ago
Exciting. I've been playing with AI dev pipelines and the "give the agent the full codebase vs. let it search" trade-off is what I keep running into - both have pros and cons depending on the task. This looks like the latter pushed harder than I've seen before. Looking forward to trying it
Oxlamarr 5mo ago
[flagged]
adelks 5mo ago
I know this tool was meant for AI, but I am more interested in using it myself when exploring new code bases or even my own, when I want to refactor something and want an overview of where to change stuff. LSPs do that, but this tool sounds like it can go one step further.
luodaint 5mo ago
"The problem is that there are several bottlenecks internally," which include the requirements, specs, and testing. Another one not mentioned in the article. Before you had faster implementation times, something would take six weeks to implement. Feedback from the client about how far off target you were came through in the same amount of time: a help desk ticket, a post-call check-in, a quarter end review. The price you paid for being off target was proportional to how long it took to figure out. Now, when you can ship features in an afternoon, the customer feedback loop remains the same speed. Surveys, help desk tickets, and churn analysis come back days, even weeks later, by which point you've shipped five new features going the same way. You can fix the internal bottlenecks easily enough: write better specs, have faster test cycles, deploy continuously. The customer feedback loop bottleneck is built into the system. It won't get any faster just because implementation did. Today most organizations are busy fixing the internal bottleneck, but not the external one.
luodaint 5mo ago
Metric that measures the quality beyond simple tokens count: correction loop frequency. When grep does not find a file of interest, the agent does not fail; it will continue working on an incomplete context. For a monolingual code base, the miss rate is okay. In case of polylingual code (Python backend code and TypeScript frontend code), the problems emerge when it comes to querying for cross-file dependencies. Grep will return a route from the backend API. However, there is an interface in TypeScript that needs to be matched. Agent generates a response that does not fit the type. Correction cycle is one; two if the type conflict is ambiguous. Combining grep with the understanding of semantic relations between files is a solution. Number of tokens saved is real but underestimates the actual benefit since fewer correction cycles are more valuable than tokens themselves.
flossly 5mo ago
I feel this should be a mode on ripgrep. a disk search-lib in python? really? Burntsushi (author of ripgrep), please chime in!
- Bibabomas 5mo ago
  Hey, what's the issue with a disk search-lib in python specifically? The library is extremely fast. Yes, we could probably squeeze some more performance in Rust, but that's not our native programming language, so we opted for doing it correctly rather than use a language that we don't understand well enough.
cold_harbor 5mo ago
[flagged]
jelder 5mo ago
LSPs are already miles better than grep-like tools. This was true for humans as much as it is for LLMs. It's a shame that Claude still treats it as a second-class citizen (both in the app itself and in the training). A simple "remember to use LSP instead of grep" is usually enough to get it on the right track.
psyplanner 5mo ago
[dead]
joka88xj 5mo ago
[flagged]
paolomon 5mo ago
Interesting approach, gonna leave a gh star clap
JasonBuildAI 5mo ago
amazing jobs
Waffle2180 5mo ago
[flagged]
finnley 5mo ago
[dead]
abelzentric 5mo ago
[flagged]
Mattykry 5mo ago
[flagged]
aminekhd 5mo ago
[flagged]
kcarriedo 5mo ago
[flagged]
parweb 5mo ago
[flagged]
cush 5mo ago
These almost always come with the cost of worse adherence
- Bibabomas 5mo ago
  Fair, many tools trade off cost for performance/adherence. So far our experience is good with Semble (at least with Anthropic/OpenAI models), but if you have any feedback feel free to reach out. We're working on evaluations for this as well, though it will take some time as this is much harder than benchmarking retrieval quality.
jrflo 5mo ago
Interesting for sure, but I think I need to be convinced this is necessary. How many tokens does Grep use on the average response anyways? Does this reduce input token usage from 100k to 2k per query or from 1k to 20? How does this effect output code quality?
- Bibabomas 5mo ago
  There's a "semble savings" command you can run to see token savings. It varies heavily based on the repo: in general, the larger the project the larger the savings. Output code quality is something we're still trying to measure, but it's much harder than measuring retrieval quality.
  - jrflo 5mo ago
    Gotcha. Might be interesting to show some plots of average/median/max input token savings per query as a function of repo size if you're going to do some more comparison testing. The efficiency gains are compelling, but I'd want to see the magnitude of gains as well to get a full picture. Regardless, cool project and I'll check it out for myself
deleted 5mo ago
[deleted]
bluejay2387 5mo ago
So about a year ago I wrote my own attempt at something like this using vector indexing and BM25 (the latest version uses CocoIndex, I had a custom coded solution using ChromaDB before). I wrote a comprehensive enough test set that showed performance increases on the quality of search results and reduction in token usage versus grep and rg. I haven't had time to really polish it but it worked well enough, particularly for one project where I have around 250k documentation files and docs out number code files 1000 to 1 (about 50% reduction in tokens and 30% increase in successful searches). Yesterday for grins I tried this project and was fairly disappointed to see it blow away my kludged solution particularly given that it doesn't have a lengthy indexing process. I haven't tested it on the 250k doc project yet, but in another project that I have a test suite for semantic search on it outperformed my solution by about 20% even on documentation in terms of successful search results (which I didn't expect given that it seems to only be tuned for code). I haven't gone through the code to see what its doing differently than what I tried, but what ever its doing it seems to have potential.
- Bibabomas 5mo ago
  Wow, thanks for sharing, and cool that you're working on similar things! Feel free to drop any feedback on the repo if you want!
martinloop 5mo ago
[flagged]
atlasagentsuite 5mo ago
[dead]
te0006 5mo ago
Related: https://github.com/jahala/tilth https://github.com/jahala/tilth Edit: recent HN discussion: https://news.ycombinator.com/item?id=46952321 https://news.ycombinator.com/item?id=46952321
adnasalk 5mo ago
[flagged]
ajaystream 5mo ago
[flagged]
economyballoon 5mo ago
[flagged]
kurtextrem 5mo ago
Related read (not from me): https://entire.io/blog/improving-agentic-search-in-coding-agents https://entire.io/blog/improving-agentic-search-in-coding-ag... > The clearest result was that faster search alone only modestly helps, while better-ranked results improve first-query retrieval and help agents find the right code sooner. Their tool "pgr" is a research preview only, so it'd be interesting to see semble vs pgr. I'm also collecting other tools that are similar, most notably is probably Morph's WarpGrep (has a free tier too). Apart from that, there is codemogger (https://github.com/glommer/codemogger https://github.com/glommer/codemogger), cs (the author also commented in this HN post). In the similar area, but not fully related, the author of fff is also pretty involved in any thread that goes into that direction (see e.g. https://x.com/neogoose_btw/status/2052161471296225710 https://x.com/neogoose_btw/status/2052161471296225710). Similar to colGREP is also mgrep (by mixedbread) and osgrep (but they seem to predate colGREP). I also found codedb on X (https://codegraff.com/blog/codedb-code-intelligence https://codegraff.com/blog/codedb-code-intelligence), the post reads well, but haven't tried.

445 points by Bibabomas 5mo ago

eddyaipt 5mo ago

[flagged]

esafranchik 5mo ago

Is the benchmark measuring one-shot retrieval accuracy, or Coding agent response accuracy?

stephantul 5mo ago
Hey! Co-author here. The benchmark currently only measures retrieval accuracy. We’re interested in measuring it end to end and also optimizing, e.g. the prompt and tools, for this, but we just haven’t gotten around to it.
- esafranchik 5mo ago
  Two follow-ups: 1) How do you compare accuracy? by checking if the answer is in any of the returned grep/bm25/semble snippets? 2) How do you measure token use without the agent, prompt, and tools?
  - stephantul 5mo ago
    1) yes! It’s not accuracy, but ndcg 2) we assume that if the agent gets the correct answer in the returned snippets it does not need to read further
    - esafranchik 5mo ago
      Wouldn't NDCG/token results vary wildly depending on the agent's query and the number of returned items? e.g. agents often run `grep -m 5 "QUERY"` with different queries, instead of one big grep for all items.
      - stephantul 5mo ago
        The same holds for semble: the agent can fire off many different semble queries with different k/parameters. I guess the point we’re trying to make is that you need fewer semble queries to achieve the same outcome, compared to grep+readfile calls.

ludicrousdispla 5mo ago

grep doesn't need tokens, so what is 98% fewer than zero?

stephantul 5mo ago
You need readfile to do something with those tokens. Grep only gives you the matching lines, not the context.
- djaboss 5mo ago
  `grep -C $NUM` ? ;)
  - stephantul 5mo ago
    Even so. Take a look at the NDCG numbers for grep. It's not pretty
    - hparadiz 5mo ago
      ripgrep exists though
      - stephantul 5mo ago
        The comparison is with ripgrep, see the benchmarks.
mrweasel 5mo ago
We might not be AI/agent enough for this product. I wondered about that as well, but it's not actually a grep problem, it's an ingest problem for agents. Apparently some agents use grep to navigate code and it's this "operation" that consume tokens, grep does in fact consume zero tokens.

mrpf1ster 5mo ago

Does this work well for non-coding documents as well? Say api docs or AI memory files?

stephantul 5mo ago
Hey, this is something we're actively investigating. We recently added a flag, `--include-text-files`, which, when set, also makes Semble index regular documents (i.e., markdown, text, json). This should also work relatively well.

jerezzprime 5mo ago

I'd be interested in seeing actual agent benchmarks (eg CC or Copilot CLI with grep removed and this tool instead). For example, I have explored RTK and various LSP implementations and find that the models are so heavily RL'd with grep that they do not trust results in other forms and will continually retry or reread, and all token savings are lost because the model does not trust the results of the other tools.

stephantul 5mo ago
Yeah we're also interested in doing this, it's on the roadmap together with optimization of the prompt and descriptions so that models have an easier time using it. Perhaps anecdotally: we do use this tool ourselves of course, and it's been working pretty well so far. Anthropic models call it and seem to trust the results.
giancarlostoro 5mo ago
I forced Claude to have a global memory for RTK and my own AI memory system (GuardRails) which it happily uses both, the only times it doesnt use GuardRails is if I dont mention it at all, otherwise it always uses RTK unless RTK falls apart running a tool it does not support.
deleted 5mo ago
[deleted]
nextaccountic 5mo ago
Codex CLI is quite happy running RTK. Well with GPT 5.5 xhigh anyway One thing that irks me is that when it doesn't support eg. a cli flag of find, it gives an error message rather than sending the full output of the command instead. Then the agent wastes tokens retrying, or worse, doesn't even try because the prompting may make them afraid to not run commands without rtk
- aleksiy123 5mo ago
  how effective is RTK for you? worth using?

singpolyma3 5mo ago

Semantic code search seems like a useful tool for a human too. Not just for agents.

stephantul 5mo ago
Yeah I agree. I have used semble to quickly index a large monorepo and just ask a question about it, it surfaced the right files pretty quickly. Although without an IDE, it's difficult to display them in nice way

vikeri 5mo ago

very curious to give it a spin but why write a cli in python? would surely be faster and more portable with go or rust?

skeledrew 5mo ago
Perhaps Python is their main language (they seem to be ML peeps, which would make that most likely), which means it's easier for them to do manual reviews even if they're using AI for implementing, etc.
- stephantul 5mo ago
  Yes, this is the main reason. We've released some rust stuff in the past, but Python is our main language

smcleod 5mo ago

How does it compare to context-mode or serina that are both well established now?

Bibabomas 5mo ago
Serena does a lot more than semble (I actually used serena before building this and didn't like how much it does by default). That also made it hard to see if it was actually working well with how many moving parts there are. Semble only does 1 thing: very quick code search, that's it. Context-mode I have not used before though, I will have a look at that, thanks for sharing!

porker 5mo ago

Congratulations on the release! Could you add fff to the benchmarks?

stephantul 5mo ago
We hadn't found that one yet. Will do!

abcdefg12 5mo ago

Shouldn’t it be a part of the harness at least for local codebase? I wonder how many harnesses are doing that already.

dopidopHN2 5mo ago
I'm playing with PI as a custom harness ( for Claude code because that what is provided to me ) I will try that ! It make sense and I'm curious to see results, for this or any similar projects mentioned in the thread
- molszanski 5mo ago
  allegedly this one is good for PI https://pi.dev/packages/@ff-labs/pi-fff https://pi.dev/packages/@ff-labs/pi-fff
Bibabomas 5mo ago
Afaik many harnesses ship the "default" which is grep+read (like Claude Code). But I agree, IMO it's a weird gap. To be fair I don't think providers are that incentivised to reduce token burn at the moment, but my guess is that that will change and tools like this will become at least an natively supported option in some harnesses.
- sincerely 5mo ago
  I would be surprised if, in the "mature" future of AI tools/products, the labs building the models are also selling/building to end users like they are now.
  - losvedir 5mo ago
    Would or wouldn't be? I think it makes sense that they will be because of the feedback loop on training. The lab generates tons of example text as part of the training run and if they have that using their own tools then the models will tend to prefer those tools.

nextaccountic 5mo ago

How does this compare with colgrep? https://github.com/lightonai/next-plaid/tree/main/colgrep https://github.com/lightonai/next-plaid/tree/main/colgrep

stephantul 5mo ago
The comparison is in the benchmarks, see the README

AussieWog93 5mo ago

Better than grep obviously, but how does this compare to existing LSPs?

CharlesW 5mo ago
Or tools like `ck`: https://beaconbay.github.io/ck/ https://beaconbay.github.io/ck/
- cormacrelf 5mo ago
  Try running both on the CK codebase. CK takes like 15 minutes to index itself and gives hundreds of completely irrelevant doc comments as results for “run model on CPU” query. Semble indexes for like 3 seconds and prints out the actual code that runs the model on the CPU.
  - CharlesW 5mo ago
    You didn’t use `ck` directly, you instructed Claude Code to use `ck`, right?
    - cormacrelf 5mo ago
      No? CK is better than I gave it credit for, didn’t take 15 minutes, took 2, somehow a lot faster than before, probably system busy. I was using hybrid which is wrong for this query. Still semble is a few orders of magnitude faster and gave better results against ck —-sem. I am running both on rust-lang/rust and CK is going to take hours at least, extrapolating from current stats probably 3 days? Semble: 26 seconds without any caching. The thing doesn’t have a cache and it’s still massively faster. I added caching support and watchman integration and got it down to 1.4 seconds. 3 days is basically not good enough for this use case. It’s slow enough that indexing is going to lag your code changes. Semble is fast enough that it’s not going to be behind.
    - xeyownt 5mo ago
      Tried both right now. Tried against a 84K loc C project. ck took at least 5 minutes to index, but replies are indeed fast. semble indexing (if any) took no noticeable time (except for the first download of HF model, which took a couple seconds), and replied in a couple of seconds. Unrelated but ck was a pain to install / compile (install instructions do not say you have to lock the build / you have to have latest libc).

_ink_ 5mo ago

Would this replace something like codebase-memory-mcp[1] or improve when both is being used? [1] - https://github.com/DeusData/codebase-memory-mcp https://github.com/DeusData/codebase-memory-mcp

Bibabomas 5mo ago
[dead]

jahala 5mo ago

This looks great! I built a tool in the same space- and I found that the biggest challenge was often to get the agent to prefer to use the tool over bash tools. What’s your experience with that?

Escapade5160 5mo ago
Setup hooks. Hooks are how your harness forces compliance with your own rules.
Bibabomas 5mo ago
This is mostly done with the prompt. To be fair, we are still evaluating and improving this as well!

wrxd 5mo ago

I also like the index feature form https://maki.sh https://maki.sh Source code has a lot of structure, using a real parser instead of grepping and reading files can potentially save a lot of tokens

Bibabomas 5mo ago
Interesting, will have a look at this, thanks for sharing.

ramsono 5mo ago

Very useful thanks for sharing!

esperent 5mo ago

I did some evals with pi and GPT 5.5. I tested RTK on / headroom on / both on / both off (all with the standard pi system instructions and no AGENTS.md). I forget the exact tests I used (a couple of the standard agent evals that people use, one python and one typescript because those are what I use). I don't claim it was an exhaustive test, or even a good one. It's possible I could have spent a day or so tuning my AGENTS.md and the pi system prompt/tool instructions and gotten better results, because if there's one thing running evals taught me it's that subtle differences there can change the results a lot. However, I got clearly better results with both off, enough to convince me to stop the tests immediately after 3 rounds. The problem was that while context use did go down (sometimes), the number of turns to complete went up so the overall cost of the conversation was higher. It's made me very aware of one thing: so many people are sharing these kind of tools, but either with zero evals (or suspiciously hard to reproduce), or in the case of this one, extensive benchmarks testing the wrong thing. I'm sure this tool does use fewer tokens than grep, and the benchmarks prove it, but that's not what matters here. What matters is, does an agent using it get the same quality of work done more quickly and for lower cost?

zobzu 5mo ago
with AI the "they could so they never wondered if they should" will be a very frequent thing.
- jack_pp 5mo ago
  yeah I think I'm prone to do the same, it is so easy to create and we get too excited by it instead of first doing the research necessary which is much more boring than actually producing something.
- stephantul 5mo ago
  This is a bit rude. We didn't generate this project, we wrote it, a lot of it manually, and trained custom models. We'd been working in the real-time retrieval space for a while, and we thought coding was a good fit for this specific technology.

onoesworkacct 5mo ago

fantastic token savings and performance... but unlike grep it's probabilistic search on search terms. is that an issue? the tiny model might not surface something important

stephantul 5mo ago
It's not probabilistic, and exact matches will always be preferred over non-exact. So if you search for a function name this will surface it.

jasonli0226 5mo ago

thanks for sharing!

andai 5mo ago

Nice, this sounds great. I want to mention a related issue here, which is that on small codebases, Claude spends a lot of time looking for stuff when it could have just dumped the whole codebase into the context in one go and used very little tokens. I found a nice workaround which is that you can just dump the whole directory into context, as a startup hook. So then Claude skips the "fumble around blindly in the dark" portion of every task. (I've also seen a great project that worked on bigger repos where it'll give the model an outline with stubs, though I forget what it was called.)

mackenney 5mo ago
Maybe aider? https://aider.chat/2023/10/22/repomap.html https://aider.chat/2023/10/22/repomap.html
- Imanari 5mo ago
  Good old aider ahead of its time
- andai 5mo ago
  I actually made a custom harness based on Aider's edit format (the find/replace thing). (I think most AI stuff ended up using a very similar format.) It just does what I need and no more: load code into context, append my question or instruction, call LLM, apply patches. Repeat. I haven't used Aider itself though, maybe it does that too. My harness was nice relative to Claude Code and Codex because, it doesn't need to poke around the filesystem (cause I have small repos and dump the whole thing), and it makes all edits simultaneously (doesn't need to edit one file at a time). It reads all files and edits all necessary files in a single round trip. The really nice thing is that when you're making many small fine grained changes like that, you can use a much smaller, faster, cheaper model. And if it's fast enough, it actually becomes a real-time activity. It's not "prompt, wait..." but "prompt, immediately get the result." It's interactive. You stay active and engaged. It's great.
stephantul 5mo ago
This is true, agents just don't know a lot about the things they're looking at, e.g., the number of files, file sizes, etc. Although for small codebases it also holds that whatever you would like to find it easy to find, so search still might help you with cost

aadishv 5mo ago

Seems like a cool idea so I decided to play with it a bit. The test I ran was in the browsercode (https://github.com/browser-use/browsercode https://github.com/browser-use/browsercode) repo with the following prompt: "Answer this question by only using the `semble` CLI (docs below): > What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work --- [the AGENTS.md snippet provided from https://github.com/MinishLab/semble#bash-integration https://github.com/MinishLab/semble#bash-integration]" And the equivalent for the non-Semble test: "Answer this question by only using the `rg` and `fd` CLIs: > What tools does Browsercode provide to the agent other than the base OpenCode tools? Provide the exact schema for tool input and tool output and briefly summarize what they do and how they work" In both cases, I used Pi with gpt-5.4 medium and a very minimal setup otherwise. (And yes, I did verify that either instance only used rg & fd, or only used semble.) Without Semble, it used 10.9% of the model context and used $0.144 of API credits (or, at least, that's what Pi reported - I used this with a Codex sub so cannot be sure). With Semble, it used 9.8% of the model context and $0.172 of API credits. The resulting responses were also about the same. Very close! I tried one more test in the OpenCode repo. The question was > Trace the path from 1) the OPENCODE_EXPERIMENTAL_EXA env var being set to to 1 to 2) the resulting effects in the system prompt or tool provided to the OpenCode agent. And I included the same instructions/docs as above. The non-Semble version was a bit more detailed -- it went into whether the tool call path invoked Exa based on whether Exa or Parallel was enabled for the web search provider -- but w.r.t. actually answering the question, both versions were accurate. The Semble version used 14.7% context / $0.282 API cost, while the non-Semble version used 19.0% / $0.352. Clearly a win for Semble for context efficiency, but note that the non-Semble version finished about twice as fast as the Semble version. Of course this is just me messing around. ymmv.

stephantul 5mo ago
Wow awesome, thanks for sharing! This is really useful and very much like the experiments we want to be doing in the near future

vemulasukrit 5mo ago

Nice!

gslepak 5mo ago

Does this support any language or is it limited to a specific set of languages?

stephantul 5mo ago
For chunking Semble supports all languages supported by tree-sitter-language-pack. The models we train are trained on 6 languages, but can handle way more.

deleted 5mo ago

[deleted]

ind-igo 5mo ago

[dead]

boyter 5mo ago

Interesting. I too have been working in this space, though I took a different approach. Rather than building an index, I worked on making a "smarter grep" by offering search over codebases (and any text content really) with ranking and some structural awareness of the code. Most of my time was spend dealing with performance, and as a result it runs extremely quickly. I will have to add this as a comparison to https://github.com/boyter/cs https://github.com/boyter/cs and see what my LLMs prefer for the sort of questions I ask. It too ships with MCP, but does NOT build an index for its search. I am very curious to see how it would rank seeing as it does not do basic BM25 but a code semantic variant of it. This seems to work better for the "how does auth work" style of queries, while cs does "authenticate --only-declarations" and then weighs results based on content of the files, IE where matches are, in code, comments and the overall complexity of the file. Have starred and will be watching.

Bibabomas 5mo ago
Nice! Let us know if you have any feedback or results to share, would be happy to do the same.
zikani_03 5mo ago
I was going to share a link to this. Thank you for making `cs`, I use it both with LLMs and directly in the terminal, despite not performing indexing it's pretty fast for my needs. Also definitely planning to try out semble.
- boyter 5mo ago
  You are welcome. Glad to hear its working for you. I have a few ideas I am working on to improve its relevance too that I hope pan out.

freakynit 5mo ago

What I have personally observed with such tools is that they make the AI's dumb, similar to how it makes coders dumb when relying more on AI tools. These agentic AI's are already smart enough to figure out a highly optimized path to code exploration or search. But, with these tools, they just go very aggressive, partly because the search results from these tools almost in 100% of the cases do not furnish full details, but, just the pointers. To confirm this behaviour, I did a small test run. This is in no way conclusive, but, the results do align with what I been observing: --- Task: trace full ingestion and search paths in some okayish complex project. Harness is Pi. 1. With "codebase-memory-mcp": 85k/4.4k (input/output tokens). 2. With my own regular setup: 67k/3.2k. 3. Without any of these: 80k/3.2k. As we see, such a tool made it worse (not by much, but, still). The outputs were same in quality and informational content. --- Now, what my "regular setup" mentioned above is?: Just one line in AGENTS.md and CLAUDE.md: "Start by reading PROJECT.md" . And PROJECT.md contains just following: 2-3 line description of the project, all relevant files and their one-line description, any nuiances, and finally, ends with this line: ## To LLM Update this file if the changes you have done are worth updating here. The intent of this file is to give you a rough idea of the project, from where you can explore further, if needed.

Bibabomas 5mo ago
Hey, codebase-memory-mcp and semble are not exactly the same, but it's an interesting comparison, I'll put it on the todolist to check that out and add it to our benchmarks if feasible. If you ever get a chance to do this same comparison with semble it would be super useful feedback since these "real" scenarios are hard to benchmark/replicate.
- freakynit 5mo ago
  So, I just tested with semble. Your MCP integration did not work, and kept throwing error (Failed to connect to "semble": MCP error -32000: Connection closed) though I installed using documented manner (tried both: pip and ux methods). Anyways, I made it work by making it generate relevant doc (using semble init), and then copying this into AGENTS.md, and then prompting it with this line: """ Start by reading AGENTS.md in current folder. Now, the task::: `Explore the ingestion and search paths. Do not read README.md at all`. Prefer to use `semble` search for code search. Do not do new installation. semble is already available at `/Users/nitinbansal/.local/bin/semble` . """ The results are much better. Even better than my own setup, but, vary a lot. I did 4 runs: 95k/2.9k 25k/2.7k 71k/2.9k 37k/4.0k

digitaltrees 5mo ago

Cool project. I built a custome IDE and coding agent harness and will integrate this into it. If you’re interested in a collaboration, I’d be happy to share revenue to sponsor your open source repo. https://calendly.com/ryanwmartin/open-office-hours https://calendly.com/ryanwmartin/open-office-hours

zhxiaoliang 5mo ago

The instructions on how to install and use it could use some work. I did eventually install it. Will try it later and report back.

stephantul 5mo ago
Oh sorry that happened. Feel free to open an issue or report it here

billetkit 5mo ago

[flagged]

therealdrag0 5mo ago

How does that compare to Cursors’s workspace indexing? Also curious what the authors think about Claude team explicitly trying out indexing and deciding against it.

Bibabomas 5mo ago
The first is hard to test for us unfortunately since we don't use Cursor. But the Claude thing is interesting. I think that providers (especially the ones that directly sell LLM calls like Anthropic) are not incentivised per se to think about token efficiency vs performance, so if you're chasing pure performance, just loading the full codebase into memory might still be the "benchmark topping" way to go. I think the dust hasn't really settled yet and we'll likely see a lot of changes in the coming year about what's the "correct" way to solve it. It might be different based on your harness/budget/model as well.

handonam 5mo ago

The one thing I'm a bit nervous about: security. Thoughts of supply-chain "what-ifs" gives me a bit of pause here. Would like to hear security-minded folks give their thoughts on this.

Bibabomas 5mo ago
Hey, we do a couple of things specifically to prevent supply-chain attacks. We use trusted publishing on PyPI, and --exclude newer for uv's package resolution. We also try to use the least amount of dependencies possible. A transitive dependency could in theory still be problematic though, e.g. if there's a supply-chain attack on numpy. The tool itself is fully local though, so there's no real security risks there, there are no outbound network calls or anything like that.

PufPufPuf 5mo ago

The savings are calculated against the assumption that agents read the matched files in their entirety. In my experience, they are smart enough to use grep to bring up a few lines of surrounding context, then read in full only the files that look relevant.

Bibabomas 5mo ago
In practice this rarely happens though, at least in practice I rarely see agents "grep -C N" or something like that on files it didn't read yet. I use Claude Code and OpenCode extensively, and especially during the first pass through a codebase that is not well understood the agent often just does "cat file" or something similar and gets the entire file in context first, and only then starts doing more finegrained searches, but at that point you already have a lot of irrelevant context in memory. I think the whole value proposition of semble is that you don't have to do that initial read at all and can instead get the right (small) context bits. If you experience is different, would you mind sharing what your setup is like, e.g. how do you get the agent to read less?

cagz 5mo ago

Would be nice to see the actual % token/time saving across end-to-end coding sessions over these time periods https://github.com/MinishLab/semble#savings https://github.com/MinishLab/semble#savings My observation is that greps and the processing of grep outputs account for only a small portion of overall consumption; I haven't measured this scientifically though.

Bibabomas 5mo ago
Yeah this is a good point, but it's complicated to do well since semble then has to be aware of everything else that happens in a session, which would then make it more intrusive (and it's deliberately designed as a local, non-intrusive alternative to other solutions). I'm thinking that perhaps we can do an isolated opt-in benchmark for this perhaps.

derrickrburns 5mo ago

Have you considered solving a different problem? What are agents trying to achieve when searching a code base? Finding seams. How about indexing seams instead?

Bibabomas 5mo ago
What do you mean by this exactly?

RyanJohn 5mo ago

[dead]

florians 5mo ago

“> Claude: Semble surfaced things grep missed — here are the additions to the earlier answer.” Nice!

sonink 5mo ago

The bigger problem with solutions like these is that most AI already know how to use grep and search really well because of their training. Any such new tool that you handle to the AI, takes away from the cognitive capability of the AI. Humans would normally 'learn' how to operate tools like this - but the learning in LLM's is frozen and they already with a very strong depth in existing tools like grep. For example, an AI would already use linux commands like tree to traverse the code base. And again it already has good training in this. The other problem is that it is easy to cook up examples which demonstrate the efficacy of tools like these - but actually proving that the cognitive deficit that such tools result it, is surmounted by their efficacy in long horizon runs. My first contact instinct is that this will result in a net negative 'deployable intelligence' over long horizon runs - make the agent perform worse than using existing tools. Proving the opposite is a non-trivial problem - but maybe it might be something you want to take up.

Bibabomas 5mo ago
[dead]

xiaosong001 5mo ago

[flagged]

pu_pe 5mo ago

> uses 98% fewer tokens than grep So are we supposed to believe that grep is so wasteful that models are reading 98% useless garbage every time they call it? Either this claim is not representative, or you're missing something else when you throw away the vast majority of context for the model.

boyter 5mo ago
Grep prints out every matching line. For some searches a LLM might do it will get a lot of noise, and it might have to make that search because it cannot be specific. Targeted search can reduce the number of tokens. I suspect this comparison is against reading the whole codebase though compared to just getting the bits you need.
pas 5mo ago
I had problems with Claude reading hundreds of kilobytes of outputs because grep found things in node_modules. (ripgrep helps, so it makes sense to add a line about it into some memory file.)
Bibabomas 5mo ago
The 98% is vs the grep+read loop, not grep output alone. When an agent hits an unfamiliar codebase it typically does "cat file" or reads the whole thing first, at least in my experience. If you're reliably getting agents to do "grep -C N" and stop there I'd genuinely be curious what your setup looks like, because I think the quality of the results is just too low to serve as useful context.
- ac29 5mo ago
  > When an agent hits an unfamiliar codebase it typically does "cat file" or reads the whole thing first, at least in my experience. Depends on the size of the project and specific files. I have definitely seen agents make smart use of pi's "read" tool, which can take an offset and line limit (or defaults to a max 2000 lines/50KiB if the model doesn't specify). The bash tool also has the same max output, so if a model decides to cat instead of using the read tool it still wont blow out its context window with a single large file read. But this sort of thing is going to vary with harness, model, project, and whatever the RNG delivers for the day.

Aleesha_hacker 5mo ago

[flagged]

shermantanktop 5mo ago

Do inefficient grep/sed behaviors have a secondary benefit in seeding the context with breadcrumbs of code from irrelevant matches?

Bibabomas 5mo ago
In theory maybe, but in practice it hurts more than it helps I think. Irrelevant context makes the model more likely to reason from the wrong code (and it's slower and more expensive).

getmarketingai 5mo ago

[dead]

yuanyanorz 5mo ago

[flagged]

cityofdelusion 5mo ago

Feedback: codex-cli hangs when calling this through the MCP. The semble process even sticks around as a zombie, forever stalled out. No idea why, logs have nothing. When called through a skill via CLI style calling, GPT 5.5 loves to give a ton of search terms like it is used to doing with ripgrep. Not sure how effective this is, the short docs in the github and the instructions the agent has isn't clear on what is optimal. Lastly, I got some errors with external connections to github when I was installing it for bash use. Maybe its related to the hanging? No idea. edit: My agent also loves to follow-on with ripgrep, which seems redundant. Acts like it has trust issues. I think a more extensive agent skill description could guide the agent into proper use.

Bibabomas 5mo ago
Hey, thanks for the detailed feedback. For the bug, would you mind opening an issue with your setup details? This is definitely something we want to investigate and fix. The multiple queries thing is really good feedback, thanks for that, we'll update the prompt/instructions to prevent this from happening and we'll try to add some tests for this. The external connection errors during install are uv fetching deps from PyPI I think, those should not be the reason it's hanging.

michal_lola2 5mo ago

Exciting. I've been playing with AI dev pipelines and the "give the agent the full codebase vs. let it search" trade-off is what I keep running into - both have pros and cons depending on the task. This looks like the latter pushed harder than I've seen before. Looking forward to trying it

Oxlamarr 5mo ago

[flagged]

adelks 5mo ago

I know this tool was meant for AI, but I am more interested in using it myself when exploring new code bases or even my own, when I want to refactor something and want an overview of where to change stuff. LSPs do that, but this tool sounds like it can go one step further.

luodaint 5mo ago

"The problem is that there are several bottlenecks internally," which include the requirements, specs, and testing. Another one not mentioned in the article. Before you had faster implementation times, something would take six weeks to implement. Feedback from the client about how far off target you were came through in the same amount of time: a help desk ticket, a post-call check-in, a quarter end review. The price you paid for being off target was proportional to how long it took to figure out. Now, when you can ship features in an afternoon, the customer feedback loop remains the same speed. Surveys, help desk tickets, and churn analysis come back days, even weeks later, by which point you've shipped five new features going the same way. You can fix the internal bottlenecks easily enough: write better specs, have faster test cycles, deploy continuously. The customer feedback loop bottleneck is built into the system. It won't get any faster just because implementation did. Today most organizations are busy fixing the internal bottleneck, but not the external one.

luodaint 5mo ago

Metric that measures the quality beyond simple tokens count: correction loop frequency. When grep does not find a file of interest, the agent does not fail; it will continue working on an incomplete context. For a monolingual code base, the miss rate is okay. In case of polylingual code (Python backend code and TypeScript frontend code), the problems emerge when it comes to querying for cross-file dependencies. Grep will return a route from the backend API. However, there is an interface in TypeScript that needs to be matched. Agent generates a response that does not fit the type. Correction cycle is one; two if the type conflict is ambiguous. Combining grep with the understanding of semantic relations between files is a solution. Number of tokens saved is real but underestimates the actual benefit since fewer correction cycles are more valuable than tokens themselves.

flossly 5mo ago

I feel this should be a mode on ripgrep. a disk search-lib in python? really? Burntsushi (author of ripgrep), please chime in!

Bibabomas 5mo ago
Hey, what's the issue with a disk search-lib in python specifically? The library is extremely fast. Yes, we could probably squeeze some more performance in Rust, but that's not our native programming language, so we opted for doing it correctly rather than use a language that we don't understand well enough.

cold_harbor 5mo ago

[flagged]

jelder 5mo ago

LSPs are already miles better than grep-like tools. This was true for humans as much as it is for LLMs. It's a shame that Claude still treats it as a second-class citizen (both in the app itself and in the training). A simple "remember to use LSP instead of grep" is usually enough to get it on the right track.

psyplanner 5mo ago

[dead]

joka88xj 5mo ago

[flagged]

paolomon 5mo ago

Interesting approach, gonna leave a gh star clap

JasonBuildAI 5mo ago

amazing jobs

Waffle2180 5mo ago

[flagged]

finnley 5mo ago

[dead]

abelzentric 5mo ago

[flagged]

Mattykry 5mo ago

[flagged]

aminekhd 5mo ago

[flagged]

kcarriedo 5mo ago

[flagged]

parweb 5mo ago

[flagged]

cush 5mo ago

These almost always come with the cost of worse adherence

Bibabomas 5mo ago
Fair, many tools trade off cost for performance/adherence. So far our experience is good with Semble (at least with Anthropic/OpenAI models), but if you have any feedback feel free to reach out. We're working on evaluations for this as well, though it will take some time as this is much harder than benchmarking retrieval quality.

jrflo 5mo ago

Interesting for sure, but I think I need to be convinced this is necessary. How many tokens does Grep use on the average response anyways? Does this reduce input token usage from 100k to 2k per query or from 1k to 20? How does this effect output code quality?

Bibabomas 5mo ago
There's a "semble savings" command you can run to see token savings. It varies heavily based on the repo: in general, the larger the project the larger the savings. Output code quality is something we're still trying to measure, but it's much harder than measuring retrieval quality.
- jrflo 5mo ago
  Gotcha. Might be interesting to show some plots of average/median/max input token savings per query as a function of repo size if you're going to do some more comparison testing. The efficiency gains are compelling, but I'd want to see the magnitude of gains as well to get a full picture. Regardless, cool project and I'll check it out for myself

deleted 5mo ago

[deleted]

bluejay2387 5mo ago

So about a year ago I wrote my own attempt at something like this using vector indexing and BM25 (the latest version uses CocoIndex, I had a custom coded solution using ChromaDB before). I wrote a comprehensive enough test set that showed performance increases on the quality of search results and reduction in token usage versus grep and rg. I haven't had time to really polish it but it worked well enough, particularly for one project where I have around 250k documentation files and docs out number code files 1000 to 1 (about 50% reduction in tokens and 30% increase in successful searches). Yesterday for grins I tried this project and was fairly disappointed to see it blow away my kludged solution particularly given that it doesn't have a lengthy indexing process. I haven't tested it on the 250k doc project yet, but in another project that I have a test suite for semantic search on it outperformed my solution by about 20% even on documentation in terms of successful search results (which I didn't expect given that it seems to only be tuned for code). I haven't gone through the code to see what its doing differently than what I tried, but what ever its doing it seems to have potential.

Bibabomas 5mo ago
Wow, thanks for sharing, and cool that you're working on similar things! Feel free to drop any feedback on the repo if you want!

martinloop 5mo ago

[flagged]

atlasagentsuite 5mo ago

[dead]

te0006 5mo ago

Related: https://github.com/jahala/tilth https://github.com/jahala/tilth Edit: recent HN discussion: https://news.ycombinator.com/item?id=46952321 https://news.ycombinator.com/item?id=46952321

adnasalk 5mo ago

[flagged]

ajaystream 5mo ago

[flagged]

economyballoon 5mo ago

[flagged]

kurtextrem 5mo ago

Related read (not from me): https://entire.io/blog/improving-agentic-search-in-coding-agents https://entire.io/blog/improving-agentic-search-in-coding-ag... > The clearest result was that faster search alone only modestly helps, while better-ranked results improve first-query retrieval and help agents find the right code sooner. Their tool "pgr" is a research preview only, so it'd be interesting to see semble vs pgr. I'm also collecting other tools that are similar, most notably is probably Morph's WarpGrep (has a free tier too). Apart from that, there is codemogger (https://github.com/glommer/codemogger https://github.com/glommer/codemogger), cs (the author also commented in this HN post). In the similar area, but not fully related, the author of fff is also pretty involved in any thread that goes into that direction (see e.g. https://x.com/neogoose_btw/status/2052161471296225710 https://x.com/neogoose_btw/status/2052161471296225710). Similar to colGREP is also mgrep (by mixedbread) and osgrep (but they seem to predate colGREP). I also found codedb on X (https://codegraff.com/blog/codedb-code-intelligence https://codegraff.com/blog/codedb-code-intelligence), the post reads well, but haven't tried.