5 ms·
Only if they are supremely lazy. It’s possible to use these tools in a diligent way, where you maintain understanding and control of the system but outsource th
by Dig1t 8mo ago
Only if they are supremely lazy. It’s possible to use these tools in a diligent way, where you maintain understanding and control of the system but outsource the implementation of tasks to the LLM.
An engineer should be code reviewing every line written by an LLM, in the same way that every line is normally code reviewed when written by a human.
Maybe this changes the original argument from software being “free”, but we could just change that to mean “super cheap”.
- collinvandyck76 8mo agoThere's a pretty big difference between the understanding that comes with reviewing code versus writing it, for most people I think.
- deleted 8mo ago[deleted]
- macintux 8mo agoDefinitely true for me. What’s particularly problematic is code I need to review but can’t effectively test due to environmental challenges.
- mapontosevenths 8mo agoThats a tough situation. How do you handle the testing with human code?
- macintux 8mo agoJust as poorly.
- mapontosevenths 8mo ago> An engineer should be code reviewing every line written by an LLM, I disagree. Instead, a human should be reviewing the LLM generated unit tests to ensure that they test for the right thing. Beyond that, YOLO. If your architecture makes testing hard build a better one. If your tests arent good enough make the AI write better ones.
- jddj 8mo agoThe venn diagram for "bad things an LLM could decide are a good idea" and "things you'll think to check that it tests for" has very little overlap. The first circle includes, roughly, every possible action. And the second is tiny. Just read the code.
- kavok 8mo agoIt’s amazing how often an LLM mocks or stubs some code and then writes a test that only checks the mock, which ends up testing nothing.
- mapontosevenths 8mo agoYou really do have to verify and validate the tests. Worse you have to constantly battle the thing trying to cheat at the tests or bypass them completely. But once you figure that out, it's pretty effective.
- Dig1t 8mo agoI have seen junior engineers do this on multiple occasions. This is why all code should be reviewed by experienced engineers, whether written by a human or an LLM.
- sarchertech 8mo agoThere’s no way you or the AI wrote tests to cover everything you care about. If you did, the tests would be at least as complicated as the code (almost certainly much more so), so looking at the tests isn’t meaningfully easier than looking at the code. If you didn’t, any functionality you didn’t test is subject to change every time the AI does any work at all. As long as AIs are either non-deterministic or chaotic (suffer from prompt instability, the code is the spec. Non determinism is probably solvable, but prompt instability is a much harder problem.
- mapontosevenths 8mo ago> As long as AIs are either non-deterministic or chaotic You just hit the nail on the head. LLM's are stochastic. We want deterministic code. The way you do that is with is by bolting on deterministic linting, unit tests, AST pattern checks, etc. You can transform it into a deterministic system by validating and constraining output. One day we will look back on the days before we validated output the same way we now look at ancient code that didn't validate input.
- vips7L 8mo agoThe majority of devs I meet are extremely lazy. It’s why so many people are outsourcing their jobs to Claude.