stephantul

6 ms·

61.
by stephantul 5mo ago
I was also at the event and was pretty disappointed. Most of the talks were pretty low on information. I was at the “build” stage, which supposedly was the technical stage, but the talks there didn’t really go into technical specifics. The
62.
Why scikit learn's fit transform is probably not for you(stephantul.github.io)
1 points by stephantul 5mo ago|0 comments
63.
by stephantul 5mo ago
It was directed at the parent who implied that we didn’t think about this. I agree with your point about the evals and how you can get discontinuities: good search can be worse than bad search when agents can do many searches. We’re working
64.
by stephantul 5mo ago
It's not probabilistic, and exact matches will always be preferred over non-exact. So if you search for a function name this will surface it.
65.
by stephantul 5mo ago
This is a bit rude. We didn't generate this project, we wrote it, a lot of it manually, and trained custom models. We'd been working in the real-time retrieval space for a while, and we thought coding was a good fit for this speci
66.
by stephantul 5mo ago
Oh sorry that happened. Feel free to open an issue or report it here
67.
by stephantul 5mo ago
Wow awesome, thanks for sharing! This is really useful and very much like the experiments we want to be doing in the near future
68.
by stephantul 5mo ago
This is true, agents just don't know a lot about the things they're looking at, e.g., the number of files, file sizes, etc. Although for small codebases it also holds that whatever you would like to find it easy to find, so search
69.
by stephantul 5mo ago
Yeah I agree. I have used semble to quickly index a large monorepo and just ask a question about it, it surfaced the right files pretty quickly. Although without an IDE, it's difficult to display them in nice way
70.
by stephantul 5mo ago
The comparison is in the benchmarks, see the README
71.
by stephantul 5mo ago
For chunking Semble supports all languages supported by tree-sitter-language-pack. The models we train are trained on 6 languages, but can handle way more.
72.
by stephantul 5mo ago
We hadn't found that one yet. Will do!
73.
by stephantul 5mo ago
Yes, this is the main reason. We've released some rust stuff in the past, but Python is our main language
74.
by stephantul 5mo ago
The comparison is with ripgrep, see the benchmarks.
75.
by stephantul 5mo ago
Yeah we're also interested in doing this, it's on the roadmap together with optimization of the prompt and descriptions so that models have an easier time using it. Perhaps anecdotally: we do use this tool ourselves of course, and
76.
by stephantul 5mo ago
Even so. Take a look at the NDCG numbers for grep. It's not pretty
77.
by stephantul 5mo ago
Hey, this is something we're actively investigating. We recently added a flag, `--include-text-files`, which, when set, also makes Semble index regular documents (i.e., markdown, text, json). This should also work relatively well.
78.
by stephantul 5mo ago
The same holds for semble: the agent can fire off many different semble queries with different k/parameters. I guess the point we’re trying to make is that you need fewer semble queries to achieve the same outcome, compared to grep+rea
79.
by stephantul 5mo ago
You need readfile to do something with those tokens. Grep only gives you the matching lines, not the context.
80.
by stephantul 5mo ago
1) yes! It’s not accuracy, but ndcg 2) we assume that if the agent gets the correct answer in the returned snippets it does not need to read further
81.
by stephantul 5mo ago
Hey! Co-author here. The benchmark currently only measures retrieval accuracy. We’re interested in measuring it end to end and also optimizing, e.g. the prompt and tools, for this, but we just haven’t gotten around to it.
82.
by stephantul 5mo ago
How would you conceptualize recall in this case? Is searching through the current version of your code and possibly git history not enough?
83.
by stephantul 5mo ago
Really cool! I was investigating PCA on retrieval, thanks for the references.
84.
Show HN: Semble – Code search for agents that uses 98% fewer tokens than grep(github.com)
8 points by stephantul 5mo ago|0 comments
85.
Show HN: Semble – Fast code search for agents with near-transformer accuracy(github.com)
7 points by stephantul 6mo ago|0 comments
86.
by stephantul 6mo ago
The idea that a random uuid == anonymous, and would protect users from having entire bash commands piped through is preposterous, and you know it.
87.
by stephantul 7mo ago
It’s a serious comment, posted solely with the intent to sow doubt.
88.
by stephantul 7mo ago
Ah ok sorry, it just looked like you were speaking on his behalf.
89.
by stephantul 7mo ago
I’m not saying it invalidates the result. I am saying that they knew the headline and comparison was not correct and they still decided to roll with it. It’s an incorrect representation of what happened, designed to get eyeballs and possibl
90.
by stephantul 7mo ago
Yes it matters, because it’s not a measurement of whether it accomplishes the task if a human tells it how to solve it.

More ›