4 ms·

Yeah prompting doesn't work for this problem because the entire point of an LLM is you give it the what and it outputs the how. The more how that you have to co

Yeah prompting doesn't work for this problem because the entire point of an LLM is you give it the what and it outputs the how. The more how that you have to condition it with in the prompt, the less profitable the interaction will be. A few hints is OK, but doing all the work for the LLM tends to lead to negative productivity. Writing prompts and writing code takes about the same amount of time, for the same amount of text, plus there's the extra time that the LLM takes to accomplish the task, and review time afterwards. So you might as well just write the code yourself if you have to specify every tiny implementation detail in the prompt.

kqr 8mo ago
Makes me think of this commitstrip comic: https://i.xkqr.org/itscalledcode.jpg https://i.xkqr.org/itscalledcode.jpg (mirrored from the original due to TLS issues with the original domain.) A guy with a mug comes up to a person standing with their laptop on a small table. The mug guy says, "Some day we won't even need coders any more. We'll be able to just write the specification and the program will write itself." Guy with laptop looks up. "Oh, wow, you're right! We'll be able to write a comprehensive and precise spec and bam, we won't need programmers any more!" Guy with mug takes a sip. "Exactly!" Guy with laptop says, "And do you know the industry term for a project specification that is comprehensive and precise enough to generate a program?" "Uh... no..." "Code. It's called code."
- datastoat 8mo ago
  Goodhart's Law of Specification: When a spec reaches a state where it's comprehensive and precise enough to generate code, it has fallen out of alignment with the original intent. Of course there are some systems where correctness is vital, and for those I'd like a precise spec and proof of correctness. But I think there's a huge bulk of code where formal specification impedes what should be a process of learning and adapting.
- keybored 8mo ago
  My dream antiprogram is a specification compiler that interprets any natural language and compiles it to a strict specification. But on any possible ambiguity it gives an error. ? This terse error was found to be necessary as to not overwhelm the user with pages and pages of decision trees enumerating the ambiguities.
  - newswasboring 8mo ago
    Openspec does this. But instead of "?" it has a separate Open Questions section in the design document. In codex cli, if you first go in plan mode it will ask you open questions before it proceeds with the rest. The UX is there, for small things it does work for me, but there is still something left for LLMs to truly capture major issues.
    - keybored 8mo ago
      Bless our interesting times.
- Sophira 8mo ago
  You know, this makes me wonder... is anybody actually prompting LLMs with pseudocode rather than an English specification? Could doing so result in code that that's more true to the original pseudocode?
  - zarzavat 8mo ago
    You can give the macro-structure using stubs then ask the LLM to fill in the blanks. The problem is that it doesn't work too well for the meso-structure. Models tend to be quite good at the micro-structure because they've seen a lot of it already, and the macro-structure can easily be promoted, but the levels in between are what distinguishes a good vs bad model (or human!).
  - dyarosla 8mo ago
    I’m not sure if it went anywhere but I remember there was this attempt at one point called Sudolang: https://medium.com/javascript-scene/sudolang-a-powerful-pseudocode-programming-language-for-llms-d64d42aa719b https://medium.com/javascript-scene/sudolang-a-powerful-pseu...
FeepingCreature 8mo ago
the goal would be to write it a reusable prompt. this is what AGENT.md is for.
pricechild 8mo ago
> the entire point of an LLM is you give it the what and it outputs the how I'm still struggling to move past the magic trick of guessing what characters come next to ascribe understanding of "how" and implying understanding?

kqr 8mo ago

Makes me think of this commitstrip comic: https://i.xkqr.org/itscalledcode.jpg https://i.xkqr.org/itscalledcode.jpg (mirrored from the original due to TLS issues with the original domain.) A guy with a mug comes up to a person standing with their laptop on a small table. The mug guy says, "Some day we won't even need coders any more. We'll be able to just write the specification and the program will write itself." Guy with laptop looks up. "Oh, wow, you're right! We'll be able to write a comprehensive and precise spec and bam, we won't need programmers any more!" Guy with mug takes a sip. "Exactly!" Guy with laptop says, "And do you know the industry term for a project specification that is comprehensive and precise enough to generate a program?" "Uh... no..." "Code. It's called code."

datastoat 8mo ago
Goodhart's Law of Specification: When a spec reaches a state where it's comprehensive and precise enough to generate code, it has fallen out of alignment with the original intent. Of course there are some systems where correctness is vital, and for those I'd like a precise spec and proof of correctness. But I think there's a huge bulk of code where formal specification impedes what should be a process of learning and adapting.
keybored 8mo ago
My dream antiprogram is a specification compiler that interprets any natural language and compiles it to a strict specification. But on any possible ambiguity it gives an error. ? This terse error was found to be necessary as to not overwhelm the user with pages and pages of decision trees enumerating the ambiguities.
- newswasboring 8mo ago
  Openspec does this. But instead of "?" it has a separate Open Questions section in the design document. In codex cli, if you first go in plan mode it will ask you open questions before it proceeds with the rest. The UX is there, for small things it does work for me, but there is still something left for LLMs to truly capture major issues.
  - keybored 8mo ago
    Bless our interesting times.
Sophira 8mo ago
You know, this makes me wonder... is anybody actually prompting LLMs with pseudocode rather than an English specification? Could doing so result in code that that's more true to the original pseudocode?
- zarzavat 8mo ago
  You can give the macro-structure using stubs then ask the LLM to fill in the blanks. The problem is that it doesn't work too well for the meso-structure. Models tend to be quite good at the micro-structure because they've seen a lot of it already, and the macro-structure can easily be promoted, but the levels in between are what distinguishes a good vs bad model (or human!).
- dyarosla 8mo ago
  I’m not sure if it went anywhere but I remember there was this attempt at one point called Sudolang: https://medium.com/javascript-scene/sudolang-a-powerful-pseudocode-programming-language-for-llms-d64d42aa719b https://medium.com/javascript-scene/sudolang-a-powerful-pseu...

FeepingCreature 8mo ago

the goal would be to write it a reusable prompt. this is what AGENT.md is for.

pricechild 8mo ago

> the entire point of an LLM is you give it the what and it outputs the how I'm still struggling to move past the magic trick of guessing what characters come next to ascribe understanding of "how" and implying understanding?