4 ms·

The spec rarely has enough detail to deterministically create a product, so current vibecoding is a lottery. So we generate one or many changesets (in series o

The spec rarely has enough detail to deterministically create a product, so current vibecoding is a lottery. So we generate one or many changesets (in series or in parallel) then iterate on one. We force the “chosen one” to be the one true codification of the spec + the other stuff we didn’t write down anywhere. Call it luck driven development. But there’s another way. If we keep starting fresh from the spec, but keep adding detail after detail, regenerating from scratch each time.. and the LLM has enough room in context to handle a detailed spec AND produce output, and the result is reasonably close to deterministic because the LLM makes “reasonable choices” for everything underspecified.. that’s a paradigm shift.

tjr 10mo ago
At that level of detail, how far removed are we from “programming”?
- cadamsdotcom 10mo ago
  Far! But without the need to “program” you can focus on the end user and better understand their needs - which is super exciting.
- Vegenoid 10mo ago
  Without understanding the level of detail required, which we do not yet know, we cannot say. When I think of English specifications that (generally) aim to be very precise, I think of laws. Laws do not read like plain, common language, because plain common language is bad at being specific. Interpreting and creating laws requires an education on par with that required of an engineer, often greater.
  - Muromec 10mo ago
    Laws being unreadable is largely an Enlish-language problem zo. I have no problem reading them in my native language. Not requiring massive context size of case law makes things easier still. Big part of being a lawyer is having the same context with all the other lawyers and knowing what was already decided and what possible new interpretation is likely to be accepted by everyone else.
    - Vegenoid 10mo ago
      > Big part of being a lawyer is having the same context with all the other lawyers and knowing what was already decided and what possible new interpretation is likely to be accepted by everyone else. And to create software specifications with language, the same thing will need to happen. You’ll need shared terminology and context that the LLM will correctly and consistently interpret, and that other engineers will understand. This means that very specific meanings become attached to certain words and phrases. Without this, you aren’t making precise specifications. To create and interpret these specifications will require learning the language of the specs. It may well still be easier than code - but then it would also be less precise.
      - jimbokun 10mo ago
        And this could end up looking more like mathematics notation than English. For the same reason mathematicians opt to use specialized notation to communicate with greater precision than natural language.
      - Muromec 10mo ago
        >And to create software specifications with language, the same thing will need to happen. You’ll need shared terminology and context that the LLM will correctly and consistently interpret, and that other engineers will understand. That sounds awfully similar to... software development.
        est31 10mo ago
        Yeah many programming languages have been advertised to fulfil precisely this goal, that people can program computers via natural language instead of having to think hard and too much about details. Usually programming languages intend to make editing as easy as possible, but also understanding what the program does, as well as reasoning about performance, with different languages putting different emphasis on the various aspects.
        Muromec 10mo ago
seanmcdirmid 10mo ago
> The spec rarely has enough detail to deterministically create a product, so current vibecoding is a lottery. How is that different from how it worked without LLMs? The only difference is that we can now get a failing product faster and iterate. > If we keep starting fresh from the spec, but keep adding detail after detail, regenerating from scratch each time.. This sounds like the worst way to use AI. LLMs can work existing code, whether it was generated by an LLM or written by human. It can even work on code that has been edited by a human, there is no good reason to not be iterative when using an LLM to develop code, and plenty of good reasons to be iterative.
- dizlexic 10mo ago
  >How is that different from how it worked without LLMs? I won't lie and say "That's a great idea" when it isn't.
- dwringer 10mo ago
  For me it just depends. If the response to my prompt shows the model misunderstood something, then I go back and retry the previous prompt again. Otherwise the "wrong ideas" that it comes up with persist in the context and seem to sabotage all future results. The most of this sort of coding I've done was in Google's AI studio, and I often do have a context that spans dozens of messages, but I always rewind if something goes off-track. Basically any time I'm about to make a difficult request, I clone the entire context/app to a new one so I can roll back [cleanly] whenever necessary.
  - seanmcdirmid 10mo ago
    If you fix something it sticks, the AI won't keep making the same mistake, it won't change the code that already exists if you ask it not to. It actually ONLY works well when you are doing iterative changes and not used as a pure code generator, actually, AI's one-shot performance is kind of crap. A mistake happens, you point it out to the LLM and ask it to update the code and the instructions used to create the code in tandem. Or you just ask it to fix the code once. You add tests, partially generated by the AI and curated by a human, the AI runs the tests and fixes the code if they fail (or fixes the tests).
emodendroket 10mo ago
Well, it’s really a return to the old-fashioned role of an analyst coming up with a data dictionary and a detailed spec. But in practice how often did that work as intended?
mungoman2 10mo ago
Yes, I believe the paradigm shift will be to not treat the code as particularly valuable, just like binaries today. Instead the value is in the input that can generate the code.
DelightOne 10mo ago
In what environment do you run such tests? Do you have a script for it, or do you have a UI that manages the process?
hu3 10mo ago
This is interesting. It's like the nix philosophy. When changes are needed, improve the spec and you can nuke the entire thing and start over. something like immutable code development. One major problem is: how do you not break existing data on the database when code changes? Maybe include current database structure in the spec.

tjr 10mo ago

At that level of detail, how far removed are we from “programming”?

cadamsdotcom 10mo ago
Far! But without the need to “program” you can focus on the end user and better understand their needs - which is super exciting.
Vegenoid 10mo ago
Without understanding the level of detail required, which we do not yet know, we cannot say. When I think of English specifications that (generally) aim to be very precise, I think of laws. Laws do not read like plain, common language, because plain common language is bad at being specific. Interpreting and creating laws requires an education on par with that required of an engineer, often greater.
- Muromec 10mo ago
  Laws being unreadable is largely an Enlish-language problem zo. I have no problem reading them in my native language. Not requiring massive context size of case law makes things easier still. Big part of being a lawyer is having the same context with all the other lawyers and knowing what was already decided and what possible new interpretation is likely to be accepted by everyone else.
  - Vegenoid 10mo ago
    > Big part of being a lawyer is having the same context with all the other lawyers and knowing what was already decided and what possible new interpretation is likely to be accepted by everyone else. And to create software specifications with language, the same thing will need to happen. You’ll need shared terminology and context that the LLM will correctly and consistently interpret, and that other engineers will understand. This means that very specific meanings become attached to certain words and phrases. Without this, you aren’t making precise specifications. To create and interpret these specifications will require learning the language of the specs. It may well still be easier than code - but then it would also be less precise.
    - jimbokun 10mo ago
      And this could end up looking more like mathematics notation than English. For the same reason mathematicians opt to use specialized notation to communicate with greater precision than natural language.
    - Muromec 10mo ago
      >And to create software specifications with language, the same thing will need to happen. You’ll need shared terminology and context that the LLM will correctly and consistently interpret, and that other engineers will understand. That sounds awfully similar to... software development.
      - est31 10mo ago
        Yeah many programming languages have been advertised to fulfil precisely this goal, that people can program computers via natural language instead of having to think hard and too much about details. Usually programming languages intend to make editing as easy as possible, but also understanding what the program does, as well as reasoning about performance, with different languages putting different emphasis on the various aspects.
        Muromec 10mo ago

seanmcdirmid 10mo ago

> The spec rarely has enough detail to deterministically create a product, so current vibecoding is a lottery. How is that different from how it worked without LLMs? The only difference is that we can now get a failing product faster and iterate. > If we keep starting fresh from the spec, but keep adding detail after detail, regenerating from scratch each time.. This sounds like the worst way to use AI. LLMs can work existing code, whether it was generated by an LLM or written by human. It can even work on code that has been edited by a human, there is no good reason to not be iterative when using an LLM to develop code, and plenty of good reasons to be iterative.

dizlexic 10mo ago
>How is that different from how it worked without LLMs? I won't lie and say "That's a great idea" when it isn't.
dwringer 10mo ago
For me it just depends. If the response to my prompt shows the model misunderstood something, then I go back and retry the previous prompt again. Otherwise the "wrong ideas" that it comes up with persist in the context and seem to sabotage all future results. The most of this sort of coding I've done was in Google's AI studio, and I often do have a context that spans dozens of messages, but I always rewind if something goes off-track. Basically any time I'm about to make a difficult request, I clone the entire context/app to a new one so I can roll back [cleanly] whenever necessary.
- seanmcdirmid 10mo ago
  If you fix something it sticks, the AI won't keep making the same mistake, it won't change the code that already exists if you ask it not to. It actually ONLY works well when you are doing iterative changes and not used as a pure code generator, actually, AI's one-shot performance is kind of crap. A mistake happens, you point it out to the LLM and ask it to update the code and the instructions used to create the code in tandem. Or you just ask it to fix the code once. You add tests, partially generated by the AI and curated by a human, the AI runs the tests and fixes the code if they fail (or fixes the tests).

emodendroket 10mo ago

Well, it’s really a return to the old-fashioned role of an analyst coming up with a data dictionary and a detailed spec. But in practice how often did that work as intended?

mungoman2 10mo ago

Yes, I believe the paradigm shift will be to not treat the code as particularly valuable, just like binaries today. Instead the value is in the input that can generate the code.

DelightOne 10mo ago

In what environment do you run such tests? Do you have a script for it, or do you have a UI that manages the process?

hu3 10mo ago

This is interesting. It's like the nix philosophy. When changes are needed, improve the spec and you can nuke the entire thing and start over. something like immutable code development. One major problem is: how do you not break existing data on the database when code changes? Maybe include current database structure in the spec.