4 ms·
For now, my mind is still made up. I leave the door open to be shown any serious piece of software that is built primarily through agentic workflows. Having tri
by BowBun 1y ago
For now, my mind is still made up. I leave the door open to be shown any serious piece of software that is built primarily through agentic workflows. Having tried to use these tools over the past month to build a critical piece of infrastructure for my company, I agree with OP. I spent so much time wrangling back unnecessary garbage that the LLM found was important, that I wondered if just writing it in one shot would have been actually faster. Simple things like 'test all this workflow logic' resulted in the LLM inserting a non-sensical mock at the top of the test file that took me an hour or two to unwind.
Other than that, I keep hearing the same arguments - "LLMs free up more time for me to think about the 'important things'." Son, your system is not durable, your tests are misleading, and you can't reason about what's happening because you didn't write it. What important things are left to think about??
- ukuina 1y ago> Son, your system is not durable, your tests are misleading, and you can't reason about what's happening You just described the majority of pre-LLM enterprise software.
- ares623 1y agoSo let’s make things worse?
- djeastm 1y ago>resulted in the LLM inserting a non-sensical mock at the top of the test file that took me an hour or two to unwind Why didn't you just not accept the change?
- BowBun 1y agoLet's step through the flow: 1. Write up a detailed spec as a prompt, including a bunch of detail about the actual functionality. Add in some details about wanting to test everything. At this point I did not specifically say "Hey LLM, in my tests, do not mock one of the key idempotency functions in this system". This is obvious to me since 1/4 of the system was concerned with idempotency, and this was super clear in my original spec 2. LLM works for 30 minutes, spits out 2.5K lines 3. I review the product functionality and the test cases at a glance, and then start running tests and iterating on the software. If I have to mentally split every line of the LLM output and decide whether it's correct or not (remember, I didn't write the rest of the system), then I lose all the productivity gains that the LLM offers. You don't do this, no one does this unless they're modifying a few lines at a time. My point is that you cannot build robust software from scratch with LLMs right now. They walk right into footguns we've spent decades learning about and building solutions for.