3 ms·

Very cool! Yep, a constantly updated spec is the key. Wrote about this here: https://lukebechtel.com/blog/vibe-speccing https://lukebechtel.com/blog/vibe-spec

Very cool! Yep, a constantly updated spec is the key. Wrote about this here: https://lukebechtel.com/blog/vibe-speccing https://lukebechtel.com/blog/vibe-speccing I've also found it's helpful to have it keep an "experiment log" at the bottom of the original spec, or in another document, which it must update whenever things take "a surprising turn"

ctoth 9mo ago
Honest question: what do you do when your spec has grown to over a megabyte? Some things I've been doing: - Move as much actual data into YML as possible. - Use CEL? - Ask Claude to rewrite pseudocode in specs into RFC-style constrained language? How do you sync your spec and code both directions? I have some slash commands that do this but I'm not thrilled with them? I tend to have to use Gemini for actually juggling the whole spec. Of course it's nice and chunked as much as it can be? but still. There's gonna need to be a whole new way of doing this. If programming languages can have spooky language at a distance wait until we get into "but paragraph 7, subsection 5 of section G clearly defines asshole as..." What does a structured language look like when it doesn't need mechanical sympathy? YML + CEL is really powerful and underexplored but it's still just ... not what I'm actually wanting.
- lukebechtel 9mo ago
  Sharding or compaction, both possible with LLMs. Sharding: Make well-named sub-documents for parts of work. LLM will be happy to create these and maintain cross references for you. Compaction: Ask the LLM to compact parts of the spec, or changelog, which are over specified or redundant.
  - ctoth 9mo ago
    My question was something like: what is the right representation for program semantics when the consumer is an LLM and the artifact exceeds context limits? "Make sub-documents with cross-references" is just... recreating the problem of programming languages but worse. Now we have implicit dependencies between prose documents with no tooling to track them, no way to know if a change in document A invalidates assumptions in document B, no refactoring support, no tests for the spec. To make things specific: https://github.com/ctoth/polyarray-spec https://github.com/ctoth/polyarray-spec
    - lukebechtel 9mo ago
      Ah, I see your point more clearly now. At some level you have to do semantic compression... To your point on non-explicitness -- the dependencies between the specs and sub-specs can be explicit (i.e. file:// links, etc). But your overall point on assumption invalidation remains... Reminds me of a startup some time ago that was doing "Automated UX Testing" where user personas (i.e. prosumer, avg joe, etc) were created, and Goals/ Implicit UX flows through the UI were described (i.e. "I want to see my dashboard", etc). Then, an LLM could pretend to be each persona, and test each day whether that user type could achieve the goals behind their user flow. This doesn't fully solve your problem, but it hints at a solution perhaps. Some of what you're looking for is found by adding strict linter / tests. But your repo looks like something in an entirely different paradigm and I'm curious to dig into it more.
- vidarh 9mo ago
  Telling it to maintain a list of areas that needs work, with references to specs for those specific areas has worked well for me.
daliusd 9mo ago
Looks like default OpenCode / Claude Code behavior with Claude models. Why the extra prompt ?
- lukebechtel 9mo ago
  Good question! 1. The post was written before this was common :) 2. If using Cursor (as I usually am), this isn't what it always does by default, though you can invoke something like it using "plan" mode. It's default is to keep todo items in a little nice todo list, but that isn't the same thing as a spec. 3. I've found that Claude Code doesn't always do this, for reasons unknown to me. 4. The prompt is completely fungible! It's really just an example of the idea.
celadin 9mo ago
I'm still sharing this post in the internal org trainings I run for those new to LLMs. Thanks for it - really great overview of the concept! I saw in your other comment you've made accommodations for the newer generation, and I will confess than in Cursor (with plan mode) I've found an abbreviated form works just as well as the extremely explicit example found in the post. If you ever had a followup, I imagine it'd be just as well received!
anonzzzies 9mo ago
We found, especially with Opus and recent claude code that it is better/more precise at reading existing code for figuring out what the current status is than reading specs. It seems (for us) it is less precise at 'comprehending' the spec English than it is the code and that will sometimes reflect in wrong assumptions for new tasks which will result in incorrect implementations of those tasks. So we dropped this. Because of caching, it doesn't seem too bad on the tokens either.
- nonethewiser 9mo ago
  Specs with agents seem destined for drift. It'll randomly change something you dont know about and it will go too fast for you to really keep it updated. I went from using claude code totally naively to using little project management frameworks to now just using it by itself again. Im gettin the best results like this, and usually start in planning mode (unless the issue is quite small/clear). My experience has been that it gets worse with more structure. You misinform it and heavily bias it's results in ways you dont intend. Maybe there are AI wizards out there with the perfect system of markdown artifacts but I found it increased the trouble a lot and made the results worse. It's a non deterministic system. Knock yourself out tryin to micromanage it.

ctoth 9mo ago

Honest question: what do you do when your spec has grown to over a megabyte? Some things I've been doing: - Move as much actual data into YML as possible. - Use CEL? - Ask Claude to rewrite pseudocode in specs into RFC-style constrained language? How do you sync your spec and code both directions? I have some slash commands that do this but I'm not thrilled with them? I tend to have to use Gemini for actually juggling the whole spec. Of course it's nice and chunked as much as it can be? but still. There's gonna need to be a whole new way of doing this. If programming languages can have spooky language at a distance wait until we get into "but paragraph 7, subsection 5 of section G clearly defines asshole as..." What does a structured language look like when it doesn't need mechanical sympathy? YML + CEL is really powerful and underexplored but it's still just ... not what I'm actually wanting.

lukebechtel 9mo ago
Sharding or compaction, both possible with LLMs. Sharding: Make well-named sub-documents for parts of work. LLM will be happy to create these and maintain cross references for you. Compaction: Ask the LLM to compact parts of the spec, or changelog, which are over specified or redundant.
- ctoth 9mo ago
  My question was something like: what is the right representation for program semantics when the consumer is an LLM and the artifact exceeds context limits? "Make sub-documents with cross-references" is just... recreating the problem of programming languages but worse. Now we have implicit dependencies between prose documents with no tooling to track them, no way to know if a change in document A invalidates assumptions in document B, no refactoring support, no tests for the spec. To make things specific: https://github.com/ctoth/polyarray-spec https://github.com/ctoth/polyarray-spec
  - lukebechtel 9mo ago
    Ah, I see your point more clearly now. At some level you have to do semantic compression... To your point on non-explicitness -- the dependencies between the specs and sub-specs can be explicit (i.e. file:// links, etc). But your overall point on assumption invalidation remains... Reminds me of a startup some time ago that was doing "Automated UX Testing" where user personas (i.e. prosumer, avg joe, etc) were created, and Goals/ Implicit UX flows through the UI were described (i.e. "I want to see my dashboard", etc). Then, an LLM could pretend to be each persona, and test each day whether that user type could achieve the goals behind their user flow. This doesn't fully solve your problem, but it hints at a solution perhaps. Some of what you're looking for is found by adding strict linter / tests. But your repo looks like something in an entirely different paradigm and I'm curious to dig into it more.
vidarh 9mo ago
Telling it to maintain a list of areas that needs work, with references to specs for those specific areas has worked well for me.

daliusd 9mo ago

Looks like default OpenCode / Claude Code behavior with Claude models. Why the extra prompt ?

lukebechtel 9mo ago
Good question! 1. The post was written before this was common :) 2. If using Cursor (as I usually am), this isn't what it always does by default, though you can invoke something like it using "plan" mode. It's default is to keep todo items in a little nice todo list, but that isn't the same thing as a spec. 3. I've found that Claude Code doesn't always do this, for reasons unknown to me. 4. The prompt is completely fungible! It's really just an example of the idea.

celadin 9mo ago

I'm still sharing this post in the internal org trainings I run for those new to LLMs. Thanks for it - really great overview of the concept! I saw in your other comment you've made accommodations for the newer generation, and I will confess than in Cursor (with plan mode) I've found an abbreviated form works just as well as the extremely explicit example found in the post. If you ever had a followup, I imagine it'd be just as well received!

anonzzzies 9mo ago

We found, especially with Opus and recent claude code that it is better/more precise at reading existing code for figuring out what the current status is than reading specs. It seems (for us) it is less precise at 'comprehending' the spec English than it is the code and that will sometimes reflect in wrong assumptions for new tasks which will result in incorrect implementations of those tasks. So we dropped this. Because of caching, it doesn't seem too bad on the tokens either.

nonethewiser 9mo ago
Specs with agents seem destined for drift. It'll randomly change something you dont know about and it will go too fast for you to really keep it updated. I went from using claude code totally naively to using little project management frameworks to now just using it by itself again. Im gettin the best results like this, and usually start in planning mode (unless the issue is quite small/clear). My experience has been that it gets worse with more structure. You misinform it and heavily bias it's results in ways you dont intend. Maybe there are AI wizards out there with the perfect system of markdown artifacts but I found it increased the trouble a lot and made the results worse. It's a non deterministic system. Knock yourself out tryin to micromanage it.