4 ms·

As someone who has done a lot of work with agentic coding I am not sure specialized agents are the best solution. I think standardizing knowledge packs would b

As someone who has done a lot of work with agentic coding I am not sure specialized agents are the best solution. I think standardizing knowledge packs would be better that any agent can read to understand a domain or library is more useful. In particular this allows for an agent to know multiple domains at the same time. Basically knowledge packs could be specified in each npm package.json or similar. And we should view a knowledge pack as just a cache in a way. Because agents these days are capable of discovering that knowledge themselves, via web browsing and running tests, it is just costly to do so on every agent run or for every library they don't know. I sort of view specialized agents as akin to micro services, great if you have perfect domain decomposition, but likely to introduce artificial barriers and become inconvenient as the problem domain shifts from the original decomposition design. I guess I should write this up as blogpost or something similar. EDIT: Newly written blog post here: https://benhouston3d.com/blog/crafting-readmes-for-ai https://benhouston3d.com/blog/crafting-readmes-for-ai

sestinj 2y ago
We think about this a lot, and I think there are merits to the viewpoint. If I were to write a rule that said "make sure all code you write uses best practices", it should already be obvious enough to a good language model that this is always the case. It's "common knowledge". In some cases today there might be "common knowledge" that is a bit more rare, and the language model doesn't quite know this. I might agree that this could be obviated as well. A situation to think about: if I were to write a rule that said "I am using Tailwind CSS for styling", then this is actually information that can't be just known. It's not "common knowledge", but instead "preference" or "personal knowledge". I do think it's a fair response to say "can't it just read my package.json"? Probably this works in a handful of cases, but I've come to find that even so there are a few benefits to custom rules that I expect to hold true regardless of LLM progress: - It's more efficient to read a rule than to call a tool to read package.json on every request - Especially in large enterprise codebases, the majority of knowledge is highly implicit (oftentimes in detrimental ways, but so the world works) But yeah this is a majorly important and interesting question in my mind. What types of customization will last, and which won't? A blog post would be amazing
r_singh 2y ago
> As someone who has done a lot of work with agentic coding Can you please share what are your favourite tools and for what exactly? Would be helpful I've been using Cline a lot with the PLAN + ACT modes and Cursor for the Inline Edits but I've noticed that for anything much larger than Claude 3.7's context window things get less reliable and it's not worth it anymore. Have you found a way to share knowledge packs? Any conventions? How do you manage chat histories / old tasks and do you create documentation from it for future work?
- bhouston 2y ago
  > Can you please share what are your favourite tools and for what exactly? Would be helpful I wrote my own open source one here: https://github.com/drivecore/mycoder https://github.com/drivecore/mycoder. Covered on hacker news here: https://news.ycombinator.com/item?id=43177117 https://news.ycombinator.com/item?id=43177117 I've also studied coding with it and wrote a lot about my findings here: - https://benhouston3d.com/blog/lean-into-agentic-coding-mistakes https://benhouston3d.com/blog/lean-into-agentic-coding-mista... - https://benhouston3d.com/blog/building-an-agentic-code-from-scratch https://benhouston3d.com/blog/building-an-agentic-code-from-... - https://benhouston3d.com/blog/agentic-coder-automation https://benhouston3d.com/blog/agentic-coder-automation - https://news.ycombinator.com/item?id=43177117 https://news.ycombinator.com/item?id=43177117 - https://benhouston3d.com/blog/the-rise-of-test-theater https://benhouston3d.com/blog/the-rise-of-test-theater My findings are generally that agentic coders are relatively interchangeable and the reason they work is primarily because of the LLM's intelligence and that is a result of the training they are undergoing on agentic coding tasks. I think that both LLMs and agentic coding tools are converging quite quickly in terms of capabilities. > Have you found a way to share knowledge packs? Any conventions? How do you manage chat histories / old tasks and do you create documentation from it for future work? I've run into this wall as well. I am working on it right now. :) Here is a hint of the direction I am exploring: https://benhouston3d.com/blog/ephemeral-software-in-the-era-of-ai https://benhouston3d.com/blog/ephemeral-software-in-the-era-... But using Github as external memory is a near term solution: https://benhouston3d.com/blog/github-mode-for-agentic-coding https://benhouston3d.com/blog/github-mode-for-agentic-coding
  - 2y ago
sestinj 2y ago
On another note, this rang true to me: > Basically knowledge packs should be specified in each npm package.json or similar. Our YAML-based file format for assistants (https://docs.continue.dev/reference https://docs.continue.dev/reference) is hoping to do just this by allowing you "import knowledge packs". Does it need to be decoupled from package.json, etc.? One of the most interesting reasons we decided not to go that route was it can be cumbersome for all of your many dependencies to taken into account at once. Another is the question of how the ecosystem will evolve. I definitely think that each package author should take the time to encode the rules and best practices of using their library, however it might be difficult for community to help out if this is gated by getting in a pull request. At the same time, one of the soon-to-be-released features we are working on is the ability to auto-generate or suggest rules (based on package.json, etc.).
- bhouston 2y ago
  > Does it need to be decoupled from package.json, etc.? Knowledge packs should be decoupled from package.json just as eslint rules or commit-lint rules are. You can include them in package.json or in separate files. But including pointers to the main files in a package.json helps with discovery. All packages across all languages should support AI friendly knowledge packages so a level of decoupling is required. EDIT: After thinking about it I think README.md should just be written with Agentic Coders in mind. I wrote up my thoughts on that here: https://benhouston3d.com/blog/crafting-readmes-for-ai https://benhouston3d.com/blog/crafting-readmes-for-ai
- mentalgear 2y ago
  how are "knowledge packs" different than just the package's README ? (if present and well written, it should be as usable to devs as to an LLM, if not, maybe consider letting the LLM write it's own "Readme" for a package on the hub be scanning source/types of the package)
  - sestinj 2y ago
    I'd say rules are quite similar to a README, just tailored to LLMs, which often benefit from slightly different information than a human would. One way to think about the difference is that we as developers have the chance to build up memory/context over time, whereas LLMs are "memoryless" so you want to efficiently load all of the necessary high-level understanding. > consider letting the LLM write it's own "Readme" for a package on the hub be scanning source/types of the package This is something we're looking to ship soon
deleted 2y ago
[deleted]
collingreen 2y ago
I was expecting more than "have a readme" tbh

sestinj 2y ago

We think about this a lot, and I think there are merits to the viewpoint. If I were to write a rule that said "make sure all code you write uses best practices", it should already be obvious enough to a good language model that this is always the case. It's "common knowledge". In some cases today there might be "common knowledge" that is a bit more rare, and the language model doesn't quite know this. I might agree that this could be obviated as well. A situation to think about: if I were to write a rule that said "I am using Tailwind CSS for styling", then this is actually information that can't be just known. It's not "common knowledge", but instead "preference" or "personal knowledge". I do think it's a fair response to say "can't it just read my package.json"? Probably this works in a handful of cases, but I've come to find that even so there are a few benefits to custom rules that I expect to hold true regardless of LLM progress: - It's more efficient to read a rule than to call a tool to read package.json on every request - Especially in large enterprise codebases, the majority of knowledge is highly implicit (oftentimes in detrimental ways, but so the world works) But yeah this is a majorly important and interesting question in my mind. What types of customization will last, and which won't? A blog post would be amazing

r_singh 2y ago

> As someone who has done a lot of work with agentic coding Can you please share what are your favourite tools and for what exactly? Would be helpful I've been using Cline a lot with the PLAN + ACT modes and Cursor for the Inline Edits but I've noticed that for anything much larger than Claude 3.7's context window things get less reliable and it's not worth it anymore. Have you found a way to share knowledge packs? Any conventions? How do you manage chat histories / old tasks and do you create documentation from it for future work?

bhouston 2y ago
> Can you please share what are your favourite tools and for what exactly? Would be helpful I wrote my own open source one here: https://github.com/drivecore/mycoder https://github.com/drivecore/mycoder. Covered on hacker news here: https://news.ycombinator.com/item?id=43177117 https://news.ycombinator.com/item?id=43177117 I've also studied coding with it and wrote a lot about my findings here: - https://benhouston3d.com/blog/lean-into-agentic-coding-mistakes https://benhouston3d.com/blog/lean-into-agentic-coding-mista... - https://benhouston3d.com/blog/building-an-agentic-code-from-scratch https://benhouston3d.com/blog/building-an-agentic-code-from-... - https://benhouston3d.com/blog/agentic-coder-automation https://benhouston3d.com/blog/agentic-coder-automation - https://news.ycombinator.com/item?id=43177117 https://news.ycombinator.com/item?id=43177117 - https://benhouston3d.com/blog/the-rise-of-test-theater https://benhouston3d.com/blog/the-rise-of-test-theater My findings are generally that agentic coders are relatively interchangeable and the reason they work is primarily because of the LLM's intelligence and that is a result of the training they are undergoing on agentic coding tasks. I think that both LLMs and agentic coding tools are converging quite quickly in terms of capabilities. > Have you found a way to share knowledge packs? Any conventions? How do you manage chat histories / old tasks and do you create documentation from it for future work? I've run into this wall as well. I am working on it right now. :) Here is a hint of the direction I am exploring: https://benhouston3d.com/blog/ephemeral-software-in-the-era-of-ai https://benhouston3d.com/blog/ephemeral-software-in-the-era-... But using Github as external memory is a near term solution: https://benhouston3d.com/blog/github-mode-for-agentic-coding https://benhouston3d.com/blog/github-mode-for-agentic-coding
- 2y ago

sestinj 2y ago

On another note, this rang true to me: > Basically knowledge packs should be specified in each npm package.json or similar. Our YAML-based file format for assistants (https://docs.continue.dev/reference https://docs.continue.dev/reference) is hoping to do just this by allowing you "import knowledge packs". Does it need to be decoupled from package.json, etc.? One of the most interesting reasons we decided not to go that route was it can be cumbersome for all of your many dependencies to taken into account at once. Another is the question of how the ecosystem will evolve. I definitely think that each package author should take the time to encode the rules and best practices of using their library, however it might be difficult for community to help out if this is gated by getting in a pull request. At the same time, one of the soon-to-be-released features we are working on is the ability to auto-generate or suggest rules (based on package.json, etc.).

bhouston 2y ago
> Does it need to be decoupled from package.json, etc.? Knowledge packs should be decoupled from package.json just as eslint rules or commit-lint rules are. You can include them in package.json or in separate files. But including pointers to the main files in a package.json helps with discovery. All packages across all languages should support AI friendly knowledge packages so a level of decoupling is required. EDIT: After thinking about it I think README.md should just be written with Agentic Coders in mind. I wrote up my thoughts on that here: https://benhouston3d.com/blog/crafting-readmes-for-ai https://benhouston3d.com/blog/crafting-readmes-for-ai
mentalgear 2y ago
how are "knowledge packs" different than just the package's README ? (if present and well written, it should be as usable to devs as to an LLM, if not, maybe consider letting the LLM write it's own "Readme" for a package on the hub be scanning source/types of the package)
- sestinj 2y ago
  I'd say rules are quite similar to a README, just tailored to LLMs, which often benefit from slightly different information than a human would. One way to think about the difference is that we as developers have the chance to build up memory/context over time, whereas LLMs are "memoryless" so you want to efficiently load all of the necessary high-level understanding. > consider letting the LLM write it's own "Readme" for a package on the hub be scanning source/types of the package This is something we're looking to ship soon

deleted 2y ago

[deleted]

collingreen 2y ago

I was expecting more than "have a readme" tbh