3 ms·

I've used OpenSpec extensively on two large pieces of software which were worked solo over 6-9 months. Recently, I've completely ditched the specification part

by mafro 17d ago

I've used OpenSpec extensively on two large pieces of software which were worked solo over 6-9 months. Recently, I've completely ditched the specification part. I found they just weren't useful over the longer term. I used an LLM to assess in both directions whether the code matched the specs and whether the specs matched the code. On both software projects this came out with huge divergence from spec to code. Basically the old theory is true - the code IS the specification. What I did find very useful and have retained is the process flow. Create a proposal, review the proposal, implement, review the code. Also useful was building and maintaining ADRs and invariant logs for where a unit test cannot be made to verify behaviour. The process and the ADRs, unit test, invariant log all help the software stay coherent as the LLM churns on it over many unconnected contexts.

polycaster 17d ago
The code is only part of the specification. It does rarely document the actual requirements to a degree you can rely on for decision making. Sure, the code should speak for itself, but it mostly speaks about the WHYs of the implementation, not the reasoning behind the actual requirement. I found that OpenSpec actually helps a lot in this regard.
- mafro 17d ago
  I challenge you to use a frontier LLM to analyse your code and specs and verify if they are meaningfully aligned. Code almost never speaks to the WHY; ADRs will help there. Code never captures requirements, but it does reflect actual system behaviour, which is a specification.
mactavish88 17d ago
Agreed. Having used OpenSpec extensively for a few months now, long-lived specs in the repo are useless, but the OpenSpec change process is super valuable for keeping agents on track across a single complex project (e.g. developing a single complex new feature in a large codebase).
sieve 17d ago
> Basically the old theory is true - the code IS the specification. The spec is whatever I write by hand. The code is what the LLM writes for me. The spec could be anything depending on how much detail you want. The problem with the "code IS the spec" in the age of LLMs is that they will change stuff without telling you while hitting their immediate goal. Six months ago, I used to review every single change. Now I get the LLM to audit the code to compare against the spec. Any divergence means one of two things: - either I have to update the spec, or - the LLM has to update the code.
- threatofrain 17d ago
  I don't think your dichotomy works. When an LLM is reaching into agents.md it is absolutely modifying the specs. Who cares about original providence when it ends up in agents.md?
- mafro 17d ago
  I can appreciate your approach, but I'm not hand writing 100s to 1000s of specs by hand - at that point I'll just write the code myself
cloverich 17d ago
The reason the code is the specification is because people don't take specification seriously. Usually for good reason, but sometimes not. Real, long-lived RFC's can exist, and can have directional and corrective impacts on your LLM generated code. The trick is the RFC needs to be human written and maintained. The moment the org allows the LLM themselves to modify the spec... then yes the spec is no longer useful; or rather, "why" comments in code + a sliver of high level directional / summary content is probably all that is valuable. A fun example can be having the LLM implement a well known spec (which it cannot edit), and then to use the spec in review to find mistakes and corrections. That's helped it really click for me. All these tools that let LLMs generate spec, even with iterative planning... I haven't found it useful for very long. It's great for building something complex in the very short term (e.g. days). But i haven't found keeping them afterwards to provide any benefit.
gbckf97ke 17d ago
[dead]
cherry_tree 17d ago
Are you maintaining a single long lived spec for an entire repo/project and trying to have AI implement against that, or are you creating a spec for each feature addition or piece of work you want to AI to do? What you say I hear commonly from folks who are trying to maintain a single spec for a repo and finding it falls over as that spec gets complex and what’s being asked of the AI is muddied by you asking it to figure out the diff between what was already in the spec/previously implemented and what it’s being asked to do.

by mafro 17d ago

polycaster 17d ago

The code is only part of the specification. It does rarely document the actual requirements to a degree you can rely on for decision making. Sure, the code should speak for itself, but it mostly speaks about the WHYs of the implementation, not the reasoning behind the actual requirement. I found that OpenSpec actually helps a lot in this regard.

mafro 17d ago
I challenge you to use a frontier LLM to analyse your code and specs and verify if they are meaningfully aligned. Code almost never speaks to the WHY; ADRs will help there. Code never captures requirements, but it does reflect actual system behaviour, which is a specification.

mactavish88 17d ago

Agreed. Having used OpenSpec extensively for a few months now, long-lived specs in the repo are useless, but the OpenSpec change process is super valuable for keeping agents on track across a single complex project (e.g. developing a single complex new feature in a large codebase).

sieve 17d ago

> Basically the old theory is true - the code IS the specification. The spec is whatever I write by hand. The code is what the LLM writes for me. The spec could be anything depending on how much detail you want. The problem with the "code IS the spec" in the age of LLMs is that they will change stuff without telling you while hitting their immediate goal. Six months ago, I used to review every single change. Now I get the LLM to audit the code to compare against the spec. Any divergence means one of two things: - either I have to update the spec, or - the LLM has to update the code.

threatofrain 17d ago
I don't think your dichotomy works. When an LLM is reaching into agents.md it is absolutely modifying the specs. Who cares about original providence when it ends up in agents.md?
mafro 17d ago
I can appreciate your approach, but I'm not hand writing 100s to 1000s of specs by hand - at that point I'll just write the code myself

cloverich 17d ago

The reason the code is the specification is because people don't take specification seriously. Usually for good reason, but sometimes not. Real, long-lived RFC's can exist, and can have directional and corrective impacts on your LLM generated code. The trick is the RFC needs to be human written and maintained. The moment the org allows the LLM themselves to modify the spec... then yes the spec is no longer useful; or rather, "why" comments in code + a sliver of high level directional / summary content is probably all that is valuable. A fun example can be having the LLM implement a well known spec (which it cannot edit), and then to use the spec in review to find mistakes and corrections. That's helped it really click for me. All these tools that let LLMs generate spec, even with iterative planning... I haven't found it useful for very long. It's great for building something complex in the very short term (e.g. days). But i haven't found keeping them afterwards to provide any benefit.

gbckf97ke 17d ago

[dead]

cherry_tree 17d ago

Are you maintaining a single long lived spec for an entire repo/project and trying to have AI implement against that, or are you creating a spec for each feature addition or piece of work you want to AI to do? What you say I hear commonly from folks who are trying to maintain a single spec for a repo and finding it falls over as that spec gets complex and what’s being asked of the AI is muddied by you asking it to figure out the diff between what was already in the spec/previously implemented and what it’s being asked to do.