4 ms·

The author presents a false dichotomy when discussing "Why Not AI". ... there are some serious costs and reasonable reservations to AI development. Let's

The author presents a false dichotomy when discussing "Why Not AI". ... there are some serious costs and reasonable reservations to AI development. Let's start by listing those concerns These are super-valid concerns. They're also concerns that I suspect came around when we developed compilers and people stopped writing assembly by hand, instead trusting programs like gcc ... Compilers are deterministic, making their generated assembly code verifiable (for those compilers which produce assembly code). "AI", such as "Claude Code (or Cursor)" referenced in the article, is nondeterministic in their output and therefore incomparable to a program compiler. One might as well equate the predictability of a Fibonacci sequence[0] to that of a PRNG[1] since both involve numbers. 0 - https://en.wikipedia.org/wiki/Fibonacci_sequence https://en.wikipedia.org/wiki/Fibonacci_sequence 1 - https://en.wikipedia.org/wiki/Pseudorandom_number_generator https://en.wikipedia.org/wiki/Pseudorandom_number_generator

llmslave3 9mo ago
I suspect the argument is that both AI and a compiler enables building software at a higher level of abstraction.
- Retric 9mo ago
  Abstraction is only useful when it involves a consistent mapping between A and B, LLM’s don’t provide that. In most contexts you can abstract the earth as a sphere and it works fine ex:aligning solar panels etc. Until you enter the realm of precision where treating the earth as a sphere utterly fails. There’s no realistic set of tests you can right where an unsupervised LLM’s output can be trusted to generate a complex system which works if it’s constantly being recreated. Actual compilers don’t have that issue.
nl 9mo ago
> Compilers are deterministic, making their generated assembly code verifiable People keep saying this like it is an absolute fact, whereas in reality it is a scale. Compilers are more deterministic than LLMs in general, but no they are not completely deterministic. That's why making reproducible builds is hard! https://stackoverflow.com/questions/52974259/what-are-some-examples-of-non-determinism-in-the-c-compiler https://stackoverflow.com/questions/52974259/what-are-some-e... and https://github.com/mgrang/non-determinism https://github.com/mgrang/non-determinism give some good examples of this. This leads to the point: in general do we care about this non-determinism? Most of the time, no we don't. Once you accept that the next stage is accepting that most of the time the non-deterministic output of an LLM is good enough! This leads to how do I verify it is good enough which leads to testing and then suddenly you have a working agentic loop....
- AdieuToLogic 9mo ago
  >> Compilers are deterministic, making their generated assembly code verifiable > People keep saying this like it is an absolute fact, whereas in reality it is a scale. My statement is of course a generalization due to its terseness and focuses on the expectation of repeatable results given constant input, excluding pathological definitions of nondeterminism such as compiler-defined macro values or implementation defects. Modern compilers are complex systems and not really my point. > This leads to the point: in general do we care about this non-determinism? > Most of the time, no we don't. Not generally the type of nondeterminism I described, no. Nor the nondeterministic value of the `__DATE__` macro referenced in the StackOverflow link you provided. > Once you accept that the next stage is accepting that most of the time the non-deterministic output of an LLM is good enough! This is where the wheels fall off. First, "most of the time" only makes sense when there is another disjoint group of "other times." Second, the preferred group defined is "non-deterministic [sic] output of an LLM is good enough", which means the "other times" are when LLM use is not good enough. Third, and finally, when use of an approach (or a tool) is unpredictable (again, excluding pathological cases) given the same input, it requires an open set of tests to verify correctness over time. That last point may not be obvious, so I will extrapolate as to why it holds. Assuming the LLM in use has, or is reasonably expected to have, model evolution, documents generated by same will diverge unpredictably given a constant prompt. This implies prompt evolution will also be required at a frequency almost certainly different than unpredictable document generation intrinsic to LLMs. This in turn implies test expectations and/or definitions having to evolve over time with nothing changing other than undetectable model evolution. Which means any testing which exists at one point in time cannot be relied upon to provide the same verifications at a later point in time. Thus the requirement of an open set of tests to verify correctness over time. Finally, to answer your question of: how do I verify it is good enough You can't, because what you describe is a multi-story brick house built on a sand dune.
  - nl 9mo ago
    > Assuming the LLM in use has, or is reasonably expected to have, model evolution, documents generated by same will diverge unpredictably given a constant prompt. So what? You tell it once. It writes code. You test that code, not the prompt.
enraged_camel 9mo ago
>> Compilers are deterministic, making their generated assembly code verifiable This is true (to an extent), but the generated LLM code is also verifiable. We use automated tests to do it.
- vrighter 9mo ago
  automated tests are not verification. The "llm as a compiler" provides zero guarantees about the code. A compiler offers absolute guarantees that what you write is semantically preserved, barring bugs in the compiler itself. An llm provides zero guarantees even with zero bugs in the llm's code.
  - enraged_camel 9mo ago
    >> A compiler offers absolute guarantees I think one of the sibling comments addresses this myth rather neatly: https://news.ycombinator.com/item?id=46563383 https://news.ycombinator.com/item?id=46563383 tl;dr compilers are not fully deterministic either.
    - vrighter 9mo ago
      please point out where i said "deterministic". I said guarantees that semantics are preserved.
      - enraged_camel 9mo ago
        I don't know what you are arguing, or why. Please follow the thread in its full context. Specifically, the argument the article author is making is that moving to a higher level of abstraction also cost developers the benefit of understanding the internals. Ultimately, that ended up not mattering very much. The OP pushed back on this, saying compilers are deterministic and LLMs are not, and that lack of determinism makes LLM output unverifiable. I said the latter is not true because you can perform verification using tests. You claimed tests are not verification because LLMs don't preserve the semantics. I'm not sure why semantics matter. LLMs providing no guarantees regarding the preservation of semantics is not important because you can guarantee the behavior of the generated code using tests. In most domains, this is sufficient. You tell the LLM to write code that does X, Y and Z, and then verify X, Y and Z using a test. That's it.
hmry 9mo ago
If LLMs were like compilers, you could put src/ into .gitignore and only upload the prompt. Even the earliest compilers didn't work by the programmer writing code, copying the assembly output into their source tree, and throwing away the code. This is not a value judgement, they simply aren't the same thing at all.
- dist-epoch 9mo ago
  here you go, a prompt only library: https://github.com/dbreunig/whenwords https://github.com/dbreunig/whenwords
  - jncraton 9mo ago
    That's great. Here's "me" implementing a JS version of that library in one shot using Github Copilot and a 1 sentence prompt: > Implement when.js as a simple, zero-dependency js library following SPEC.md exactly. https://github.com/jncraton/whenwords/pulls https://github.com/jncraton/whenwords/pulls

llmslave3 9mo ago

I suspect the argument is that both AI and a compiler enables building software at a higher level of abstraction.

Retric 9mo ago
Abstraction is only useful when it involves a consistent mapping between A and B, LLM’s don’t provide that. In most contexts you can abstract the earth as a sphere and it works fine ex:aligning solar panels etc. Until you enter the realm of precision where treating the earth as a sphere utterly fails. There’s no realistic set of tests you can right where an unsupervised LLM’s output can be trusted to generate a complex system which works if it’s constantly being recreated. Actual compilers don’t have that issue.

nl 9mo ago

> Compilers are deterministic, making their generated assembly code verifiable People keep saying this like it is an absolute fact, whereas in reality it is a scale. Compilers are more deterministic than LLMs in general, but no they are not completely deterministic. That's why making reproducible builds is hard! https://stackoverflow.com/questions/52974259/what-are-some-examples-of-non-determinism-in-the-c-compiler https://stackoverflow.com/questions/52974259/what-are-some-e... and https://github.com/mgrang/non-determinism https://github.com/mgrang/non-determinism give some good examples of this. This leads to the point: in general do we care about this non-determinism? Most of the time, no we don't. Once you accept that the next stage is accepting that most of the time the non-deterministic output of an LLM is good enough! This leads to how do I verify it is good enough which leads to testing and then suddenly you have a working agentic loop....

AdieuToLogic 9mo ago
>> Compilers are deterministic, making their generated assembly code verifiable > People keep saying this like it is an absolute fact, whereas in reality it is a scale. My statement is of course a generalization due to its terseness and focuses on the expectation of repeatable results given constant input, excluding pathological definitions of nondeterminism such as compiler-defined macro values or implementation defects. Modern compilers are complex systems and not really my point. > This leads to the point: in general do we care about this non-determinism? > Most of the time, no we don't. Not generally the type of nondeterminism I described, no. Nor the nondeterministic value of the `__DATE__` macro referenced in the StackOverflow link you provided. > Once you accept that the next stage is accepting that most of the time the non-deterministic output of an LLM is good enough! This is where the wheels fall off. First, "most of the time" only makes sense when there is another disjoint group of "other times." Second, the preferred group defined is "non-deterministic [sic] output of an LLM is good enough", which means the "other times" are when LLM use is not good enough. Third, and finally, when use of an approach (or a tool) is unpredictable (again, excluding pathological cases) given the same input, it requires an open set of tests to verify correctness over time. That last point may not be obvious, so I will extrapolate as to why it holds. Assuming the LLM in use has, or is reasonably expected to have, model evolution, documents generated by same will diverge unpredictably given a constant prompt. This implies prompt evolution will also be required at a frequency almost certainly different than unpredictable document generation intrinsic to LLMs. This in turn implies test expectations and/or definitions having to evolve over time with nothing changing other than undetectable model evolution. Which means any testing which exists at one point in time cannot be relied upon to provide the same verifications at a later point in time. Thus the requirement of an open set of tests to verify correctness over time. Finally, to answer your question of: how do I verify it is good enough You can't, because what you describe is a multi-story brick house built on a sand dune.
- nl 9mo ago
  > Assuming the LLM in use has, or is reasonably expected to have, model evolution, documents generated by same will diverge unpredictably given a constant prompt. So what? You tell it once. It writes code. You test that code, not the prompt.

enraged_camel 9mo ago

>> Compilers are deterministic, making their generated assembly code verifiable This is true (to an extent), but the generated LLM code is also verifiable. We use automated tests to do it.

vrighter 9mo ago
automated tests are not verification. The "llm as a compiler" provides zero guarantees about the code. A compiler offers absolute guarantees that what you write is semantically preserved, barring bugs in the compiler itself. An llm provides zero guarantees even with zero bugs in the llm's code.
- enraged_camel 9mo ago
  >> A compiler offers absolute guarantees I think one of the sibling comments addresses this myth rather neatly: https://news.ycombinator.com/item?id=46563383 https://news.ycombinator.com/item?id=46563383 tl;dr compilers are not fully deterministic either.
  - vrighter 9mo ago
    please point out where i said "deterministic". I said guarantees that semantics are preserved.
    - enraged_camel 9mo ago
      I don't know what you are arguing, or why. Please follow the thread in its full context. Specifically, the argument the article author is making is that moving to a higher level of abstraction also cost developers the benefit of understanding the internals. Ultimately, that ended up not mattering very much. The OP pushed back on this, saying compilers are deterministic and LLMs are not, and that lack of determinism makes LLM output unverifiable. I said the latter is not true because you can perform verification using tests. You claimed tests are not verification because LLMs don't preserve the semantics. I'm not sure why semantics matter. LLMs providing no guarantees regarding the preservation of semantics is not important because you can guarantee the behavior of the generated code using tests. In most domains, this is sufficient. You tell the LLM to write code that does X, Y and Z, and then verify X, Y and Z using a test. That's it.

hmry 9mo ago

If LLMs were like compilers, you could put src/ into .gitignore and only upload the prompt. Even the earliest compilers didn't work by the programmer writing code, copying the assembly output into their source tree, and throwing away the code. This is not a value judgement, they simply aren't the same thing at all.

dist-epoch 9mo ago
here you go, a prompt only library: https://github.com/dbreunig/whenwords https://github.com/dbreunig/whenwords
- jncraton 9mo ago
  That's great. Here's "me" implementing a JS version of that library in one shot using Github Copilot and a 1 sentence prompt: > Implement when.js as a simple, zero-dependency js library following SPEC.md exactly. https://github.com/jncraton/whenwords/pulls https://github.com/jncraton/whenwords/pulls