5 ms·
If coding agents are the new entry point to your library, how sure are you that they’re using it well? I asked this question to about 50 library maintainers an
by richardblythman 1y ago
If coding agents are the new entry point to your library, how sure are you that they’re using it well?
I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know.
Existing code generation benchmarks focus mainly on self-contained code snippets and compare models not agents. Almost none focus on library-specific generation.
So we built a simple app to test how well coding agents interact with libraries:
• Takes your library’s docs
• Automatically extracts usage examples
• Tasks AI agents (like Claude Code) with generating those examples from scratch
• Logs mistakes and analyzes performance
We’re testing libraries now, but it’s early days. If you're interested: Input your library, see what breaks, spot patterns, and share the results below.
We plan to expand to more coding agents, more library-specific tasks, and new metrics. Let us know what we should prioritize next.
- dotancohen 1y agoNote that this comment is not hijacking. The author of this comment is also the author of the post.
- add-sub-mul-div 1y agoThat's the more likely assumption. Accounts with only self-promotion spam activity have become more of a rule here than an exception.
- justonceokay 1y agoIf making dev tooling is selling shovels to the miners, then this is like selling sheet metal to the shovel makers.
- grim_io 1y agoYeah. Feels like a data mining operation for training data. I could be wrong.
- bdhcuidbebe 1y ago> If coding agents are the new entry point to your library, how sure are you that they’re using it well? > I asked this question to about 50 library maintainers and dev tool builders, and the majority didn't really know. Why should they even bother to answer such a loaded and hypothetical question?
- richardblythman 1y agoim paraphrasing. the questions i asked to dev tool builders were more neutral.
- spankalee 1y agoWhy do we need to log in?
- richardblythman 1y agowe send out an email when the tests are finished (takes about 30 mins)
- grim_io 1y agoThat makes you sound like you are dodging the question.
- richardblythman 1y agoi mean that we wanted an email address to send the results to when they finish. based on comments here, i do think we should allow users to run the audit first (and provide an email address if they want us to follow up with results later).
- weitendorf 1y agoLet’s meet and see if it might make sense for us to team up. We’re working on this from the agent/library-specific-task side, and we might be better than chatgpt at marketing your product :)
- mxkopy 1y agoIMO a tool like this doesn’t make sense until the hallucination problem is fixed