7 ms·

We're in a transition phase today where agents need special guidance to understand a codebase that go beyond what humans need. Before long, I don't think they w

We're in a transition phase today where agents need special guidance to understand a codebase that go beyond what humans need. Before long, I don't think they will. I think we should focus on our own project documentation being comprehensive (e.g. the contents of this AGENTS.md are appropriate to live somewhere in our documentation), but we should always write for humans. The LLM's whole shtick is that it can read and comprehend our writing, so let's architect for it at that level.

tptacek 1y ago
It's not just understanding the codebase, it's also stylistic things, like "use this assert library to write tests", or "never write comments", or "use structured logging". It's just as useful --- more so even --- on fresh projects without much code.
- naniwaduni 1y ago
  ... most of which would also be valuable information to communicate when onboarding new devs.
  - ameliaquining 1y ago
    If there were already a universal convention on where to put that stuff, then probably the agents would have just looked there. But there's not, so it was necessary to invent one.
    - naniwaduni 1y ago
      Reality is just that people neglected onboarding docs until LLM-based coding agents put them in a position to directly benefit from having more knowledge of the codebase explicitly written down.
      - deleted 1y ago
        [deleted]
    - Mtinie 1y ago
      Common sense takes time to sink in.
  - jaggederest 1y ago
    Yeah I agree. I think the best place for all this lives in CONTRIBUTING.md which is already a standard-ish thing. I've started adding it even to my private projects that only I work on - when I have to come back in 3 or 4 months, I always appreciate it.
    - andyferris 1y ago
      I agree. My current thought is that (human) contributors should be encouraged to `ln -s CONTRIBUTING.md CLAUDE.local.md` or whatever in their local checkout for their agent of choice, have that .gitignored, and all contributors (human and LLM) will read and write to the same file. The "new" thing would be putting CONTRIBUTING.md into subfolders as appropriate - which could often be quite useful for humans anyway.
      - jaggederest 1y ago
        Yeah I think having a docs/contributing folder or equivalent, essentially referenced/linked in the CONTRIBUTING.md makes a bunch of sense, but I'd leave that kind of thing more or less up to the project
- time0ut 1y ago
  Honestly, everything I have written in markdown files as AI context fodder is stuff that I write down for human contributors anyway. Or at least stuff I want to always write down, but maybe only halfway do. The difference now is it is actually being read, seemingly understood, and often followed!
  - clbrmbr 1y ago
    So true. I find myself doing a lot more documentation these days as it is actually having a direct visible benefit. There’s a bit of a mirage here, but hey it’s getting me to document so shhh.
- deleted 1y ago
  [deleted]
- nfw2 1y ago
  Stylistic preferences could usually be inferred by just looking at the code. Perhaps if code is mid-refactor there may be inconsistencies, but ideal ai coding agent could also look through git history
vFunct 1y ago
> We're in a transition phase today where agents need special guidance to understand a codebase that go beyond what humans need. Before long, I don't think they will. This isn't guaranteed. Just like we will never have fully self-driving cars, we likely won't have fully human quality coders. Right now AI coders are going to be another tool in the tool bucket.
- CuriouslyC 1y ago
  Have you taken a Waymo?
  - vFunct 1y ago
    The limited self-driving cars, with a remote human operator? no, I never have.
    - bix6 1y ago
      This is a rather dismissive response considering the progress they’ve made over the past few years. The other commenter is correct that they use highly detailed maps but you are incorrect as they do not have a remote human operator. I find them more enjoyable than Uber. They’ve already surpassed Lyft in SF ridership and soon they will take the crown from Uber.
      - lemming 1y ago
        you are incorrect as they do not have a remote human operator Yes, they do, the term to search is “remote assistance operator”. e.g. https://philkoopman.substack.com/p/all-robotaxis-have-remote-drivers https://philkoopman.substack.com/p/all-robotaxis-have-remote...
        bix6 1y ago
        That’s phone a friend not someone remotely driving the car
deleted 1y ago
[deleted]
TZubiri 1y ago
This applies to mcp too
rr808 1y ago
One of the most common usages I see from colleagues is to get agents to write the comments so you can go full circle. :)
joegibbs 1y ago
I think they'll always need special guidance for things like business logic. They'll never know exactly what it is that you're building and why, what the end goal of the project is without you telling them. Architectural stuff is also a matter of human preference: if you have it mapped out in your head where things should go and how they should be done, it will be better for you when reading the changes, which will be the real bottleneck.
- fritzo 1y ago
  Indeed I have observed that my coworkers "never know exactly what it is that [we]'re building and why, what the end goal of the project is without [me] telling them"
goosejuice 1y ago
I suspect machine readable practices will become standard as AI is incorporated more into society. A good example is autonomous driving and local laws / context. "No turn on red. School days 7am-9am". So you need: where am I, when are school days for this specific school, and what datetime it is. You could attempt to gather that through search. Though more realistically I think the municipality will make the laws require less context, or some machine readable (e.g. qrcode) transfer of information will be on the sign. If they don't there's going to be a lot of rule breaking.
- username223 1y ago
  Very strong "reverse centaur" vibes here, in the sense of humans becoming servants to machines, instead of vice versa. Not that I think making things more machine-readable is a waste of time, but you have to keep in mind the amount of human time sacrificed.
  - oblio 1y ago
    Well, it wouldn't even be the first time. We've completely redesigned society around cars - making the most human populated environments largely worse for humans along the way. Universal sidewalks (not really needed with slow moving traffic like horses and carts - though nice even back then), traffic lights, stop signs, street crossing, interchanges, etc.
    - username223 1y ago
      As a cyclist, I’m with you 100%. Unfortunately we’re probably going to do it again with self-driving cars, with segregated lanes, special markers, etc.
      - oblio 1y ago
        A pessimistic look at self driving cars: https://www.youtube.com/watch?v=040ejWnFkj0&t=3148s https://www.youtube.com/watch?v=040ejWnFkj0&t=3148s If we end up where the video presents, humans don't deserve technology of any kind.
4b11b4 1y ago
I agree with this general sentiment, but there might be some things you want to force into the context every time via a specific agent file.
mvkel 1y ago
This is mostly true if the existing codebase is largely self documented, which is rare
theletterf 1y ago
Yes! That was precisely my point here: https://news.ycombinator.com/item?id=44837875 https://news.ycombinator.com/item?id=44837875
seb1204 1y ago
Unless we write down on the what we often consider implicit the LLM will not know it. There might be the option to deduce some implicit requirements from the code but unlikely 100%. Thus making the requirements explicit is the go.
jillesvangurp 1y ago
Here's a prompt I wrote a few days ago for codex: Analyze the repository and add a suitable agents.md It did a decent job. I didn't really have much to add to that. I guess, having this file is a nice optimization but obviously it doesn't contain anything it wasn't able to figure out by itself. What's really needed is a per repository learning base that gets populated with facts the agents discovers during it's many experiments with the repository over the course of many conversations. It's a performance optimization. The core problem is that every conversation is like ground hog day. You always start from scratch. Agents.md is a stop gap solution for that problem. Chatgpt actually has some notional memory that works across conversations. But it's a bit flaky, slow, and limited. It doesn't really learn across conversations. That btw. is a big missing piece on the path to AGIs. There are some imperfect workarounds but a lot of knowledge is lost in between conversations. And the trick of just growing the amount of context we give to our prompts doesn't seem like it's the solution.
- athrowaway3z 1y ago
  I see the groundhog day problem as a feature, not a bug. It's an organizational challenge, requiring a top level overview and easy to find sub documentation - and clear directives to use them when the AI starts architecting on a fresh start. Overall, it's a good sign when a project is understandable in small independent chunks that don't demand a programmer/llm take in more context than was referenced. I think the sweet spot would be all agents agree on a MUST-READ reference syntax for inside comments & docs that through simple scanning forces the file into the context. eg // See @{../docs/payment-flow.md} for the overall design.
- criley2 1y ago
  Your prompt is pretty basic. Both Claude Code and Github Copilot having similar features. Claude Code has `init` which has a lot of special sauce in the prompt to improve the CLAUDE.md. And github copilot added a self-documenting prompt as well that runs on new repos, and you can see their prompt here https://docs.github.com/en/copilot/how-tos/configure-custom-instructions/add-repository-instructions https://docs.github.com/en/copilot/how-tos/configure-custom-... Reading their prompt gives ideas on how you can improve yours.
kissgyorgy 1y ago
Not at all. Good documentation for humans are working well for models too, but they need so much more details and context to be reliable than humans that it needs a different style of description. This needs to contain things that you would never write for humans. They also do stupid things which need to be adjusted by these descriptions.
criley2 1y ago
Better to work with the tools we have instead of the tools we might one day have. If you want agents to work well today, you need to build for the agents we have today. We may never achieve your future where context is unlimited, models are trained on your codebase specifically, and tokens are cheap enough to use all of this. We might have a bubble pop and in a few years we could all be paying 5-10X current prices (read: the actual cost) for similar functionality to today. In that reality, how many years of inferior agent behavior do you tolerate before you give up hoping that it will evolve past needing the tweaks?

tptacek 1y ago

It's not just understanding the codebase, it's also stylistic things, like "use this assert library to write tests", or "never write comments", or "use structured logging". It's just as useful --- more so even --- on fresh projects without much code.

naniwaduni 1y ago
... most of which would also be valuable information to communicate when onboarding new devs.
- ameliaquining 1y ago
  If there were already a universal convention on where to put that stuff, then probably the agents would have just looked there. But there's not, so it was necessary to invent one.
  - naniwaduni 1y ago
    Reality is just that people neglected onboarding docs until LLM-based coding agents put them in a position to directly benefit from having more knowledge of the codebase explicitly written down.
    - deleted 1y ago
      [deleted]
  - Mtinie 1y ago
    Common sense takes time to sink in.
- jaggederest 1y ago
  Yeah I agree. I think the best place for all this lives in CONTRIBUTING.md which is already a standard-ish thing. I've started adding it even to my private projects that only I work on - when I have to come back in 3 or 4 months, I always appreciate it.
  - andyferris 1y ago
    I agree. My current thought is that (human) contributors should be encouraged to `ln -s CONTRIBUTING.md CLAUDE.local.md` or whatever in their local checkout for their agent of choice, have that .gitignored, and all contributors (human and LLM) will read and write to the same file. The "new" thing would be putting CONTRIBUTING.md into subfolders as appropriate - which could often be quite useful for humans anyway.
    - jaggederest 1y ago
      Yeah I think having a docs/contributing folder or equivalent, essentially referenced/linked in the CONTRIBUTING.md makes a bunch of sense, but I'd leave that kind of thing more or less up to the project
time0ut 1y ago
Honestly, everything I have written in markdown files as AI context fodder is stuff that I write down for human contributors anyway. Or at least stuff I want to always write down, but maybe only halfway do. The difference now is it is actually being read, seemingly understood, and often followed!
- clbrmbr 1y ago
  So true. I find myself doing a lot more documentation these days as it is actually having a direct visible benefit. There’s a bit of a mirage here, but hey it’s getting me to document so shhh.
deleted 1y ago
[deleted]
nfw2 1y ago
Stylistic preferences could usually be inferred by just looking at the code. Perhaps if code is mid-refactor there may be inconsistencies, but ideal ai coding agent could also look through git history

vFunct 1y ago

> We're in a transition phase today where agents need special guidance to understand a codebase that go beyond what humans need. Before long, I don't think they will. This isn't guaranteed. Just like we will never have fully self-driving cars, we likely won't have fully human quality coders. Right now AI coders are going to be another tool in the tool bucket.

CuriouslyC 1y ago
Have you taken a Waymo?
- vFunct 1y ago
  The limited self-driving cars, with a remote human operator? no, I never have.
  - bix6 1y ago
    This is a rather dismissive response considering the progress they’ve made over the past few years. The other commenter is correct that they use highly detailed maps but you are incorrect as they do not have a remote human operator. I find them more enjoyable than Uber. They’ve already surpassed Lyft in SF ridership and soon they will take the crown from Uber.
    - lemming 1y ago
      you are incorrect as they do not have a remote human operator Yes, they do, the term to search is “remote assistance operator”. e.g. https://philkoopman.substack.com/p/all-robotaxis-have-remote-drivers https://philkoopman.substack.com/p/all-robotaxis-have-remote...
      - bix6 1y ago
        That’s phone a friend not someone remotely driving the car

deleted 1y ago

[deleted]

TZubiri 1y ago

This applies to mcp too

rr808 1y ago

One of the most common usages I see from colleagues is to get agents to write the comments so you can go full circle. :)

joegibbs 1y ago

I think they'll always need special guidance for things like business logic. They'll never know exactly what it is that you're building and why, what the end goal of the project is without you telling them. Architectural stuff is also a matter of human preference: if you have it mapped out in your head where things should go and how they should be done, it will be better for you when reading the changes, which will be the real bottleneck.

fritzo 1y ago
Indeed I have observed that my coworkers "never know exactly what it is that [we]'re building and why, what the end goal of the project is without [me] telling them"

goosejuice 1y ago

I suspect machine readable practices will become standard as AI is incorporated more into society. A good example is autonomous driving and local laws / context. "No turn on red. School days 7am-9am". So you need: where am I, when are school days for this specific school, and what datetime it is. You could attempt to gather that through search. Though more realistically I think the municipality will make the laws require less context, or some machine readable (e.g. qrcode) transfer of information will be on the sign. If they don't there's going to be a lot of rule breaking.

username223 1y ago
Very strong "reverse centaur" vibes here, in the sense of humans becoming servants to machines, instead of vice versa. Not that I think making things more machine-readable is a waste of time, but you have to keep in mind the amount of human time sacrificed.
- oblio 1y ago
  Well, it wouldn't even be the first time. We've completely redesigned society around cars - making the most human populated environments largely worse for humans along the way. Universal sidewalks (not really needed with slow moving traffic like horses and carts - though nice even back then), traffic lights, stop signs, street crossing, interchanges, etc.
  - username223 1y ago
    As a cyclist, I’m with you 100%. Unfortunately we’re probably going to do it again with self-driving cars, with segregated lanes, special markers, etc.
    - oblio 1y ago
      A pessimistic look at self driving cars: https://www.youtube.com/watch?v=040ejWnFkj0&t=3148s https://www.youtube.com/watch?v=040ejWnFkj0&t=3148s If we end up where the video presents, humans don't deserve technology of any kind.

4b11b4 1y ago

I agree with this general sentiment, but there might be some things you want to force into the context every time via a specific agent file.

mvkel 1y ago

This is mostly true if the existing codebase is largely self documented, which is rare

theletterf 1y ago

Yes! That was precisely my point here: https://news.ycombinator.com/item?id=44837875 https://news.ycombinator.com/item?id=44837875

seb1204 1y ago

Unless we write down on the what we often consider implicit the LLM will not know it. There might be the option to deduce some implicit requirements from the code but unlikely 100%. Thus making the requirements explicit is the go.

jillesvangurp 1y ago

Here's a prompt I wrote a few days ago for codex: Analyze the repository and add a suitable agents.md It did a decent job. I didn't really have much to add to that. I guess, having this file is a nice optimization but obviously it doesn't contain anything it wasn't able to figure out by itself. What's really needed is a per repository learning base that gets populated with facts the agents discovers during it's many experiments with the repository over the course of many conversations. It's a performance optimization. The core problem is that every conversation is like ground hog day. You always start from scratch. Agents.md is a stop gap solution for that problem. Chatgpt actually has some notional memory that works across conversations. But it's a bit flaky, slow, and limited. It doesn't really learn across conversations. That btw. is a big missing piece on the path to AGIs. There are some imperfect workarounds but a lot of knowledge is lost in between conversations. And the trick of just growing the amount of context we give to our prompts doesn't seem like it's the solution.

athrowaway3z 1y ago
I see the groundhog day problem as a feature, not a bug. It's an organizational challenge, requiring a top level overview and easy to find sub documentation - and clear directives to use them when the AI starts architecting on a fresh start. Overall, it's a good sign when a project is understandable in small independent chunks that don't demand a programmer/llm take in more context than was referenced. I think the sweet spot would be all agents agree on a MUST-READ reference syntax for inside comments & docs that through simple scanning forces the file into the context. eg // See @{../docs/payment-flow.md} for the overall design.
criley2 1y ago
Your prompt is pretty basic. Both Claude Code and Github Copilot having similar features. Claude Code has `init` which has a lot of special sauce in the prompt to improve the CLAUDE.md. And github copilot added a self-documenting prompt as well that runs on new repos, and you can see their prompt here https://docs.github.com/en/copilot/how-tos/configure-custom-instructions/add-repository-instructions https://docs.github.com/en/copilot/how-tos/configure-custom-... Reading their prompt gives ideas on how you can improve yours.

kissgyorgy 1y ago

Not at all. Good documentation for humans are working well for models too, but they need so much more details and context to be reliable than humans that it needs a different style of description. This needs to contain things that you would never write for humans. They also do stupid things which need to be adjusted by these descriptions.

criley2 1y ago

Better to work with the tools we have instead of the tools we might one day have. If you want agents to work well today, you need to build for the agents we have today. We may never achieve your future where context is unlimited, models are trained on your codebase specifically, and tokens are cheap enough to use all of this. We might have a bubble pop and in a few years we could all be paying 5-10X current prices (read: the actual cost) for similar functionality to today. In that reality, how many years of inferior agent behavior do you tolerate before you give up hoping that it will evolve past needing the tweaks?