3 ms·

A spec is an envelope that contains all programs that comply. Creating this spec is often going to be harder than writing a single compliant program. Since eve

A spec is an envelope that contains all programs that comply. Creating this spec is often going to be harder than writing a single compliant program. Since every invocation of an LLM may create a different program, just like people, we will see that the spec will leave much room for good and bad implementations, and highlight the imprecision in the spec. Once we start using a particular implementation it often becomes the spec for subsequent versions, because it's interfaces expose surface texture that other programs and people will begin to rely on. I'm not sure how well LLMs will fare are brownfield software development. There is no longer a clean specification. Regenerating the code from scratch isn't acceptable. You need TPS reports.

abainbridge 7mo ago
> A spec is an envelope that contains all programs that comply. Creating this spec is often going to be harder than writing a single compliant program. This perfectly explains the feeling I had when, 20 years into my career, I had to start writing specs. I could never quite put my finger on why it was harder than coding. My greater familiarity with coding didn't seem a sufficient explanation. When writing a line of spec, I had to consider how it might be interpreted in the context of different implementations of other lines of spec - combinatorial nightmare. But code is just a spec as far as, say, a C compiler is concerned. The compiler is free to implement the assembly however it likes. Writing that spec is definitely easier than writing the assembly (Fred Brookes said this, so it must be true). So why the difference?
- skydhash 7mo ago
  > But code is just a spec as far as, say, a C compiler is concerned. The compiler is free to implement the assembly however it likes. Not at all. Code is formal, and going from C to assembly os deterministic. While the rules may be complex, they are still rules, and the compiler can’t stray away from them. Writing C code is easier than writing assembly because it’s an easier notation for thinking. It’s like when hammering a nail. Doing it with a rock is hard. But using a hammer is better. You’re still hitting the nail with a hard surface, but the handle, which is more suitable for a human hand, makes the process easier. So programming languages are not about the machine performance, they are about easier reasoning. And good programmers can get you to a level above that with proper abstraction and architecture, and give you concepts that directly map to the problem space (Gui frameworks instead of directly rendering to a framebuffer, OS syscall instead of messing with hardware,…).
  - abainbridge 7mo ago
    > Code is formal, and going from C to assembly is deterministic. OK, this is the main thing. Going from C to assembly is not deterministic in a sense because different compilers can produce different output. But the behaviour of the generated assembly is always the same. This isn't true of a spec.
    - skydhash 7mo ago
      > Going from C to assembly is not deterministic in a sense because different compilers can produce different output. That’s a weird definition of determinism.
      - derrak 7mo ago
        Why is this a weird definition of determinism? Could you please define what you mean when you say deterministic? A C program does not identify a single assembly program. It identifies a set of assembly programs. This fits the pretty standard definition of non-determinism. A difference between natural language and C code is that natural language does not have a formal semantics. Having no formal semantics is a very different problem from having a semantics that admits a well-defined set of interpretations.
        skydhash 7mo ago
        Determinism implies that the same input will result in the same output. I agree with you that for a single C program there’s a set of assembly code that satisfies it. But by choosing a compiler, an architecture and a set of flags,… you will always get the same assembly code. If you decide to randomize them, then you can no longer guarantee a specific result, but you can still guarantee sets of result. Which is the definition of non-determinism. Formalism is orthogonal as its about having well defined sets and transformation. LLMs are formal because it’s a finite set of weights and tokens ad the operations are well defined. But the prompt -> tokens -> tokens -> code transformation is non- deterministic in most tools (claude, chatgpt). And the relation between the input and the output os a mathematical one, not a semantic one.
- angry_octet 7mo ago
  C has a simpler mapping to assembly than most languages, so you are doing most of the high level translation when writing C. But even C compilers have considerable scope for weirdness, hence projects like CompCert. But much of the code we run today is JIT executed, and that leaves ample room for exploiting with weird machines. Eg the TOCTOU in the Corina exploit. Even at this very low level, full coverage specs require years of careful formal methods work. We have no hope of doing it at for vibe coding, everything will be iterative, and if TDD helps then good, but specs are by no means easier than code.
riteshkew1001 7mo ago
[flagged]
- derrak 7mo ago
  > two programs can both satisfy the same spec and have completely different The spec should express all relevant constraints. If your spec admits two things and only one is admissible in your mind, your spec is incomplete. > has a massive envelope The size of the envelope is less relevant than the expressivity of the language used to express subsets of that envelope. But almost always there is some logic which is more succinct for expressing the spec than the programming language used to express the implementation. > your security posture is now a function of how exhaustive your spec is The alternative is that your security posture is a function of unstated intentions living in somebody’s brain. This alternative seems strictly worse. > You'd need to enumerate what NOT to do This is equivalent to declaring what you must do and usually there is a succinct way to do this that does not involve listing every negative example
derrak 7mo ago
> Creating this spec is often going to be harder than writing a single compliant program. I’m not sure if I agree. A good spec will specify what the program is supposed to do rather than how the program should do that thing. It should be easier to specify what rather than how. Moreover, it should be standard practice to express what before you start writing how. I see some people pushing back on this by saying two programs that satisfy the same spec might have different performance or security properties. That is correct and if you care about those things, you should specify them. Writing down these properties, e.g., “the program is O(n) where n is blah” should be much easier than implementing a non-trivial linear time algorithm founded in deep ideas.

abainbridge 7mo ago

> A spec is an envelope that contains all programs that comply. Creating this spec is often going to be harder than writing a single compliant program. This perfectly explains the feeling I had when, 20 years into my career, I had to start writing specs. I could never quite put my finger on why it was harder than coding. My greater familiarity with coding didn't seem a sufficient explanation. When writing a line of spec, I had to consider how it might be interpreted in the context of different implementations of other lines of spec - combinatorial nightmare. But code is just a spec as far as, say, a C compiler is concerned. The compiler is free to implement the assembly however it likes. Writing that spec is definitely easier than writing the assembly (Fred Brookes said this, so it must be true). So why the difference?

skydhash 7mo ago
> But code is just a spec as far as, say, a C compiler is concerned. The compiler is free to implement the assembly however it likes. Not at all. Code is formal, and going from C to assembly os deterministic. While the rules may be complex, they are still rules, and the compiler can’t stray away from them. Writing C code is easier than writing assembly because it’s an easier notation for thinking. It’s like when hammering a nail. Doing it with a rock is hard. But using a hammer is better. You’re still hitting the nail with a hard surface, but the handle, which is more suitable for a human hand, makes the process easier. So programming languages are not about the machine performance, they are about easier reasoning. And good programmers can get you to a level above that with proper abstraction and architecture, and give you concepts that directly map to the problem space (Gui frameworks instead of directly rendering to a framebuffer, OS syscall instead of messing with hardware,…).
- abainbridge 7mo ago
  > Code is formal, and going from C to assembly is deterministic. OK, this is the main thing. Going from C to assembly is not deterministic in a sense because different compilers can produce different output. But the behaviour of the generated assembly is always the same. This isn't true of a spec.
  - skydhash 7mo ago
    > Going from C to assembly is not deterministic in a sense because different compilers can produce different output. That’s a weird definition of determinism.
    - derrak 7mo ago
      Why is this a weird definition of determinism? Could you please define what you mean when you say deterministic? A C program does not identify a single assembly program. It identifies a set of assembly programs. This fits the pretty standard definition of non-determinism. A difference between natural language and C code is that natural language does not have a formal semantics. Having no formal semantics is a very different problem from having a semantics that admits a well-defined set of interpretations.
      - skydhash 7mo ago
        Determinism implies that the same input will result in the same output. I agree with you that for a single C program there’s a set of assembly code that satisfies it. But by choosing a compiler, an architecture and a set of flags,… you will always get the same assembly code. If you decide to randomize them, then you can no longer guarantee a specific result, but you can still guarantee sets of result. Which is the definition of non-determinism. Formalism is orthogonal as its about having well defined sets and transformation. LLMs are formal because it’s a finite set of weights and tokens ad the operations are well defined. But the prompt -> tokens -> tokens -> code transformation is non- deterministic in most tools (claude, chatgpt). And the relation between the input and the output os a mathematical one, not a semantic one.
angry_octet 7mo ago
C has a simpler mapping to assembly than most languages, so you are doing most of the high level translation when writing C. But even C compilers have considerable scope for weirdness, hence projects like CompCert. But much of the code we run today is JIT executed, and that leaves ample room for exploiting with weird machines. Eg the TOCTOU in the Corina exploit. Even at this very low level, full coverage specs require years of careful formal methods work. We have no hope of doing it at for vibe coding, everything will be iterative, and if TDD helps then good, but specs are by no means easier than code.

riteshkew1001 7mo ago

[flagged]

derrak 7mo ago
> two programs can both satisfy the same spec and have completely different The spec should express all relevant constraints. If your spec admits two things and only one is admissible in your mind, your spec is incomplete. > has a massive envelope The size of the envelope is less relevant than the expressivity of the language used to express subsets of that envelope. But almost always there is some logic which is more succinct for expressing the spec than the programming language used to express the implementation. > your security posture is now a function of how exhaustive your spec is The alternative is that your security posture is a function of unstated intentions living in somebody’s brain. This alternative seems strictly worse. > You'd need to enumerate what NOT to do This is equivalent to declaring what you must do and usually there is a succinct way to do this that does not involve listing every negative example

derrak 7mo ago

> Creating this spec is often going to be harder than writing a single compliant program. I’m not sure if I agree. A good spec will specify what the program is supposed to do rather than how the program should do that thing. It should be easier to specify what rather than how. Moreover, it should be standard practice to express what before you start writing how. I see some people pushing back on this by saying two programs that satisfy the same spec might have different performance or security properties. That is correct and if you care about those things, you should specify them. Writing down these properties, e.g., “the program is O(n) where n is blah” should be much easier than implementing a non-trivial linear time algorithm founded in deep ideas.