4 ms·

> Our hypothesis is simple: session logs are now the most important artifact in software development, and should be stored alongside the code itself in the repo

> Our hypothesis is simple: session logs are now the most important artifact in software development, and should be stored alongside the code itself in the repository. I don’t think this scales. We recently have been doing “spec driven development” and we are committing the specs and prompts to our repo, alongside the generated code. At the beginning it seems fine: you wanna change something, you update the spec and ask the machine to regenerate the code. Easy. Over time, though, you have hundreds if not thousands if spec files in MD. It’s all English prose. There is duplication and subtle inconsistencies. It’s difficult to search for sections of a spec. Do you create a new file for this new requirement or update an existing one? What level of detail is enough here? Should I hint the machine about using the “saga” pattern or just let it know that we are dealing with non atomic transactions distributed across services? Etc. When a colleague opens a PR updating a spec, it’s hard to suggest objective changes (at least with code, you can demonstrate the presence of bugs… not so much with English prose. Sometimes I feel like a lawyer) All in all, it seems as if maintenance of english prose is way worse than maintenance of actual code in big enough systems. You not only need to review the spec but also review the generated code. It’s painful

aabhay 3mo ago
Exactly. The spec should go in a git comment, NOT vcs. Or, you keep the document in history for the duration of the PR and remove it post review or pre merge
- crazygringo 3mo ago
  Disagree completely. The spec is vital so that future changes continue to conform to it. Specs absolutely need to live in the VCS, because they continue to be needed to keep the code conformant. They essentially are a form of code now. And code goes in the VCS.
  - staticshock 3mo ago
    At scale, specs can only be vital to the degree to which their conformance testing is automated. Good specs should use a formal, runnable verification language. Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't change this dynamic, it amplifies it.
    - seanmcdirmid 3mo ago
      But conformance testing is where agents really excel at if you set things up right: * Black-box testing: the agent writing the tests cannot see implementation and the agent writing implementation cannot see tests, they only agree on a spec and an interface (the minimum needed to write tests). * Evaluate test coverage using code coverage, but when gaps are found communicate those gaps in terms of the specification. Good specs should be grounded (complete and not ambiguous), they don't need to be formal. You should be able to re-run your agents when the spec changes on diffs to the spec, and if a change happens out of bad, you should have agents that go in and propose fixes to the spec. Since agents are doing deterministic codegen like a compiler would, this is all pretty straightforward. You also need to consider public and internal specifications (the public specification being for reuse of the component), and you might test your integrating component with a test-double (built from the public specification alone) rather than the real component itself.
      - sdevonoes 3mo ago
        > Good specs should be grounded (complete and not ambiguous), they don't need to be formal IMHO, this is a mistake. I guess we play with it because there’s isn’t anything better nowadays. Writing and maintaining “specs” in plain english is painful.
        seanmcdirmid 3mo ago
        Natural language is incredibly expressive and fairly easy to read and write. Pick your favorite formal specification language...and you can just express some properties in them, and they are mostly niche at that. If you increase the expressiveness of your formal language too much, it just becomes code, then you are back to square one. LLMs are also incredibly proficient in processing natural language; i.e. writing, maintaining, and using a spec written in plain english is actually fairly viable with a modern LLM.
        crazygringo 3mo ago
        > Writing and maintaining “specs” in plain english is painful. The subject of this entire post is development with agents. Writing specs in English is how you do that. If you don't like it, then this is probably not the right article for you to be commenting on.
- onel 3mo ago
  The problem I see is that git commit meaaages are not as flexible at storing context information as files. Or easier to retrieve. We do have --grep to search them but it's definitely not as poweful as all the tools we have to handle files at the OS level. As verbose as storing specs in files is, I also prefer it
williamcotton 3mo ago
I’ve had good luck with this approach: https://github.com/williamcotton/algraf/tree/main/docs https://github.com/williamcotton/algraf/tree/main/docs There’s also some tests in place to make sure some things from the master spec are up to date, eg, error codes.
frizlab 3mo ago
If only we had a way of describing exactly and in great detail to the machine what to do! Some sort of language, maybe, idk… /s

aabhay 3mo ago

Exactly. The spec should go in a git comment, NOT vcs. Or, you keep the document in history for the duration of the PR and remove it post review or pre merge

crazygringo 3mo ago
Disagree completely. The spec is vital so that future changes continue to conform to it. Specs absolutely need to live in the VCS, because they continue to be needed to keep the code conformant. They essentially are a form of code now. And code goes in the VCS.
- staticshock 3mo ago
  At scale, specs can only be vital to the degree to which their conformance testing is automated. Good specs should use a formal, runnable verification language. Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't change this dynamic, it amplifies it.
  - seanmcdirmid 3mo ago
    But conformance testing is where agents really excel at if you set things up right: * Black-box testing: the agent writing the tests cannot see implementation and the agent writing implementation cannot see tests, they only agree on a spec and an interface (the minimum needed to write tests). * Evaluate test coverage using code coverage, but when gaps are found communicate those gaps in terms of the specification. Good specs should be grounded (complete and not ambiguous), they don't need to be formal. You should be able to re-run your agents when the spec changes on diffs to the spec, and if a change happens out of bad, you should have agents that go in and propose fixes to the spec. Since agents are doing deterministic codegen like a compiler would, this is all pretty straightforward. You also need to consider public and internal specifications (the public specification being for reuse of the component), and you might test your integrating component with a test-double (built from the public specification alone) rather than the real component itself.
    - sdevonoes 3mo ago
      > Good specs should be grounded (complete and not ambiguous), they don't need to be formal IMHO, this is a mistake. I guess we play with it because there’s isn’t anything better nowadays. Writing and maintaining “specs” in plain english is painful.
      - seanmcdirmid 3mo ago
        Natural language is incredibly expressive and fairly easy to read and write. Pick your favorite formal specification language...and you can just express some properties in them, and they are mostly niche at that. If you increase the expressiveness of your formal language too much, it just becomes code, then you are back to square one. LLMs are also incredibly proficient in processing natural language; i.e. writing, maintaining, and using a spec written in plain english is actually fairly viable with a modern LLM.
      - crazygringo 3mo ago
        > Writing and maintaining “specs” in plain english is painful. The subject of this entire post is development with agents. Writing specs in English is how you do that. If you don't like it, then this is probably not the right article for you to be commenting on.
onel 3mo ago
The problem I see is that git commit meaaages are not as flexible at storing context information as files. Or easier to retrieve. We do have --grep to search them but it's definitely not as poweful as all the tools we have to handle files at the OS level. As verbose as storing specs in files is, I also prefer it

williamcotton 3mo ago

I’ve had good luck with this approach: https://github.com/williamcotton/algraf/tree/main/docs https://github.com/williamcotton/algraf/tree/main/docs There’s also some tests in place to make sure some things from the master spec are up to date, eg, error codes.

frizlab 3mo ago

If only we had a way of describing exactly and in great detail to the machine what to do! Some sort of language, maybe, idk… /s