4 ms·

I think it is more like googling: when the search engine appeared, everybody had to learn how to write a good query, even though the expectation was that everyb

I think it is more like googling: when the search engine appeared, everybody had to learn how to write a good query, even though the expectation was that everybody could use them. With LLMs, it's quite similar: you have to learn how to use them. Yes, they are non-deterministic, but if you know how to use them, you can increase your chances of getting a good result dramatically. Often, this not only means articulating a task, but also looking at the bigger picture and asking yourself what tasks you should assign in the first place. For example, I can ask the LLM to write software directly, or I can ask it to write user stories or prototypes and then take a multi-step approach to develop the software. This can make a huge difference in reliability. And to be clear, I don't mean that every bad result is caused by not correctly handling the LLM (some models are simply poor at specific tasks), but rather that it is a significant factor to consider when evaluating results.

leptons 1y ago
>Yes, they are non-deterministic The LLM is more like a Ouija board than a reliable tool. >I can ask it to write user stories or prototypes By the time I write enough to explain thoroughly to an LLM what to write in "user stories" or "prototypes", I could have just written it myself, without the middleman(bot), and without the LLM hallucinating. If half the time I spend with an LLM is telling it what to do, and then another half is correcting what it did, then I'm not really saving any time at all by using it.
- selfhoster11 1y ago
  Writing the prompt takes time, but in many cases it's like rubber-duck debugging - it helps you to figure out the actual requirements, flow, or even incorrect/missing assumptions. Is it worth the time? Subjective.
- arendtio 1y ago
  For me, it is a different experience. Yes, it takes time to provide the LLM with the necessary context, but overall it can save me 3-4 times the time I would have invested. Maybe I am just slower at software development myself ;-) However, the work you do is indeed more that of a product owner than that of a developer. To avoid hallucinations, providing extensive documentation and allowing the LLM to perform test-driven development can be a game-changer. When you do that, generation time is highly correlated (negatively) with code quality. So when the AI solves the tasks quickly and easily, you have a good chance of it generating good code. As soon as the AI has to try and try again to build something working, you should be very skeptical of the result. Over the past months and years, I have used this method, and because it is somewhat reproducible, you can see the progress that LLMs are making. Sessions where the model does stupid things are becoming fewer, and sessions where the model finds a good solution become more frequent.

leptons 1y ago

>Yes, they are non-deterministic The LLM is more like a Ouija board than a reliable tool. >I can ask it to write user stories or prototypes By the time I write enough to explain thoroughly to an LLM what to write in "user stories" or "prototypes", I could have just written it myself, without the middleman(bot), and without the LLM hallucinating. If half the time I spend with an LLM is telling it what to do, and then another half is correcting what it did, then I'm not really saving any time at all by using it.

selfhoster11 1y ago
Writing the prompt takes time, but in many cases it's like rubber-duck debugging - it helps you to figure out the actual requirements, flow, or even incorrect/missing assumptions. Is it worth the time? Subjective.
arendtio 1y ago
For me, it is a different experience. Yes, it takes time to provide the LLM with the necessary context, but overall it can save me 3-4 times the time I would have invested. Maybe I am just slower at software development myself ;-) However, the work you do is indeed more that of a product owner than that of a developer. To avoid hallucinations, providing extensive documentation and allowing the LLM to perform test-driven development can be a game-changer. When you do that, generation time is highly correlated (negatively) with code quality. So when the AI solves the tasks quickly and easily, you have a good chance of it generating good code. As soon as the AI has to try and try again to build something working, you should be very skeptical of the result. Over the past months and years, I have used this method, and because it is somewhat reproducible, you can see the progress that LLMs are making. Sessions where the model does stupid things are becoming fewer, and sessions where the model finds a good solution become more frequent.