3 ms·

> If I prompt "Create a function with two arguments, a and b, which returns adding those two together", I'll get exactly what I specify. If I feel like it using

> If I prompt "Create a function with two arguments, a and b, which returns adding those two together", I'll get exactly what I specify. If I feel like it using u8 instead of u32 was wrong, I add "two arguments which are both u8", then you now get this. This is actually a good example of how your spec will progress: First pass: "Create a function [in language $X] with two arguments, a and b, which returns adding those two together" Second pass: "It must take u8 types, not u32 types" Third pass: "You are not handling overflows. It must return a u8 type." Fourth pass: "Don't clamp the output, and you're still not handling overflows" Fifth pass: "Don't panic if the addition overflows, return an error" (depending on the language, this could be "throw an exception" or return a tuple with an error field, or use an out parameter for the result or error) For just a simple "add two numbers" function, the specification can easily exceed the actual code. So you can probably understand the skepticism when the task is not trivial, and depends on a lot of existing code.

embedding-shape 10mo ago
So you do know how the general "writing specification" part is working, you just have the wrong process. Instead of iterating and adding more context on top, restructure your initial prompt to include the context. DONT DO: First pass: "Create a function [in language $X] with two arguments, a and b, which returns adding those two together" Second pass: "It must take u8 types, not u32 types" INSTEAD DO: First pass: "Create a function [in language $X] with two arguments, a and b, which returns adding those two together" Second pass: "Create a function [in language $X] with two arguments, a and b, both using u8, which returns adding those two together" ---- What you don't want to do, is adding additional messages/context on top of "known bad" context, so instead you should take the clue that the LLM didn't understand correctly as "I need to edit my prompt" not "I need to now after their reply, add more context to correct what was wrong". The goal should be to completely avoid anything bad, not correct it. Together with this, you build up a system/developer prompt you can reuse across projects/scopes, that follows how you code. In that, you add stuff as you discover what's needed to be added, like "Make sure to always handle Exceptions in X way" or similar. > > For just a simple "add two numbers" function, the specification can easily exceed the actual code. So you can probably understand the skepticism when the task is not trivial, and depends on a lot of existing code. Yes, please be skeptical, I am as well, which I guess is why I am seemingly more effective at using LLMs than others who are less skeptical. It's a benefit here to be skeptical, not a drawback. And yes, it isn't trivial to verify work that others have done for you, when you have a concrete idea of how it should be exactly. But as I managed to work with outsourced/contracting developers before, or even collaborate with developers in the same company as me, I also learned to use LLMs in a similar way where you have to review and ensure code follow the architecture/design you intended.
- troupo 10mo ago
  > INSTEAD DO: > First pass: "Create a function [in language $X] with two arguments, a and b, which returns adding those two together" > Second pass: "Create a function [in language $X] with two arguments, a and b, both using u8, which returns adding those two together" So it will create two different functions (and LLMs do love to ignore anything that came before and create a lot of stuff from scratch again and again). Now what.
  - embedding-shape 10mo ago
    What? No, I think you fundamentally misunderstand what workflow I'm suggesting here. You ask: "Do X". The LLM obliges, gives you something you don't want. At this point, don't accept/approve it, so nothing has changed, you still have an empty directory, or whatever. Then you start a brand new context, with iteration on the prompt: "Do X with Y", and the LLM again tries to do it. If something is wrong, repeat until you get what you're happy with, extract what you can into reusable system/developer prompts, then accept/approve the change. Then you end up with one change, and one function, exactly as you specified it. Then if you want, you can re-run the exact same prompt, with the exact same context (nothing!) and you'll get the same results. "LLMs do love to ignore anything that came before" literally cannot happen in this workflow, because there is nothing that "came before".
    - troupo 10mo ago
      > No, I think you fundamentally misunderstand what workflow I'm suggesting here. Ah. Basically meaningless monkey work of baby sitting an eager junior developer. And this is for a simple thing like adding two numbers. See how it doesn't scale at all with anything remotely complex? > "LLMs do love to ignore anything that came before" literally cannot happen in this workflow, because there is nothing that "came before". Of course it can. Because what came before is the project you're working on. Unless of course you end up specifying every single utility function and every single library call in your specs. Which, once again, doesn't scale.
      - 10mo ago

embedding-shape 10mo ago

So you do know how the general "writing specification" part is working, you just have the wrong process. Instead of iterating and adding more context on top, restructure your initial prompt to include the context. DONT DO: First pass: "Create a function [in language $X] with two arguments, a and b, which returns adding those two together" Second pass: "It must take u8 types, not u32 types" INSTEAD DO: First pass: "Create a function [in language $X] with two arguments, a and b, which returns adding those two together" Second pass: "Create a function [in language $X] with two arguments, a and b, both using u8, which returns adding those two together" ---- What you don't want to do, is adding additional messages/context on top of "known bad" context, so instead you should take the clue that the LLM didn't understand correctly as "I need to edit my prompt" not "I need to now after their reply, add more context to correct what was wrong". The goal should be to completely avoid anything bad, not correct it. Together with this, you build up a system/developer prompt you can reuse across projects/scopes, that follows how you code. In that, you add stuff as you discover what's needed to be added, like "Make sure to always handle Exceptions in X way" or similar. > > For just a simple "add two numbers" function, the specification can easily exceed the actual code. So you can probably understand the skepticism when the task is not trivial, and depends on a lot of existing code. Yes, please be skeptical, I am as well, which I guess is why I am seemingly more effective at using LLMs than others who are less skeptical. It's a benefit here to be skeptical, not a drawback. And yes, it isn't trivial to verify work that others have done for you, when you have a concrete idea of how it should be exactly. But as I managed to work with outsourced/contracting developers before, or even collaborate with developers in the same company as me, I also learned to use LLMs in a similar way where you have to review and ensure code follow the architecture/design you intended.

troupo 10mo ago
> INSTEAD DO: > First pass: "Create a function [in language $X] with two arguments, a and b, which returns adding those two together" > Second pass: "Create a function [in language $X] with two arguments, a and b, both using u8, which returns adding those two together" So it will create two different functions (and LLMs do love to ignore anything that came before and create a lot of stuff from scratch again and again). Now what.
- embedding-shape 10mo ago
  What? No, I think you fundamentally misunderstand what workflow I'm suggesting here. You ask: "Do X". The LLM obliges, gives you something you don't want. At this point, don't accept/approve it, so nothing has changed, you still have an empty directory, or whatever. Then you start a brand new context, with iteration on the prompt: "Do X with Y", and the LLM again tries to do it. If something is wrong, repeat until you get what you're happy with, extract what you can into reusable system/developer prompts, then accept/approve the change. Then you end up with one change, and one function, exactly as you specified it. Then if you want, you can re-run the exact same prompt, with the exact same context (nothing!) and you'll get the same results. "LLMs do love to ignore anything that came before" literally cannot happen in this workflow, because there is nothing that "came before".
  - troupo 10mo ago
    > No, I think you fundamentally misunderstand what workflow I'm suggesting here. Ah. Basically meaningless monkey work of baby sitting an eager junior developer. And this is for a simple thing like adding two numbers. See how it doesn't scale at all with anything remotely complex? > "LLMs do love to ignore anything that came before" literally cannot happen in this workflow, because there is nothing that "came before". Of course it can. Because what came before is the project you're working on. Unless of course you end up specifying every single utility function and every single library call in your specs. Which, once again, doesn't scale.
    - 10mo ago