5 ms·
I didn't take a look at the code, but to me it sounds quite dangerous to take an implementation AND the unit tests straight from an LLM, commit and move on. Is
by planb 3y ago
I didn't take a look at the code, but to me it sounds quite dangerous to take an implementation AND the unit tests straight from an LLM, commit and move on.
Is this the new normal now?
- Xenoamorphous 3y agoI guess most people would review the code as if it had been written by a colleague?
- DougBTX 3y agoYes, a great way to think of it is as a widely read intern: https://www.oneusefulthing.org/p/on-boarding-your-ai-intern https://www.oneusefulthing.org/p/on-boarding-your-ai-intern You’ve still got to avoid prompting for questionable code in the first place, eg, splitting SQL statements on semicolons with an ad-hoc regex is going to fail in edge cases, but may be sufficient for a specific task.
- afiodorov 3y ago>but may be sufficient for a specific task Yes more than sufficient for an internal tool - we can assume good intentions of the users of the tool since people want for this to actually work and have no intention of hacking.
- phanimahesh 3y agoExcept now it's a vector if anyone gets access to this internal tool. I would be fine with this for one off scripts but absolutely can not consider anything less than full sql parsing or something equally robust if it is exposed over the network, even if only internally and behind authn and authz.
- docmars 3y agoFor this reason, I tend to ask LLMs additional questions like: "show me another way to do this" or specifically "how would someone with a higher need for security write this?"... knowing that I'm likely to get a more refined answer from different sources that have probably discussed deeper security implications around the same goals, for instance.
- mattlutze 3y agoIf someone uses an LLM to produce the code, I'd guess they'll use it to evaluate the code as well.
- draxil 3y agoThis is the part I actually want from an LLM, I write the code and it spots the problems. A mega linter. Unfortunately it's not very good at this yet.
- willvarfar 3y agoYeap, I want a code-review bot that just says "this is very improbable; are you sure you didn't mean x instead?" The old Coverity used to achieve similar results in a different way, spotting probable mistakes based on patterns its heuristics found in the rest of the same codebase.
- m_fayer 3y agoRight on. These days my llm-assisted workflow feels very similar to the 20% of my day that I used to devote to code review, just now it’s more like 60% of my day.
- clbrmbr 3y agoI’m finding it’s more effective (and pleasurable) to write using GitHub CoPilot and CMD-RIGHT (accept next word). I put a detailed doc comment above and write in tandem with copilot. I’ve written the structure and I review as I write jointly with the model. This way I don’t need to review a block of code I didn’t write. <aside>I had an experience yesterday where CoPilot correctly freed all the memory in correct order at the end of a rather complicated C algorithm, even where there was nested mallocs.</aside>
- swman 3y agoIt’s the new boot camp dev. It is still the same as copy pasting SO solutions lol
- draxil 3y agoWhat as in something you should know not to do pretty quickly?
- tietjens 3y agoMean-spirited, gatekeeping comment unless I’ve misunderstood. Reference to AI is frequently used to punch down like this I’ve noticed.
- docmars 3y agoI take it to mean that the code quality deserves more scrutiny because you can't guarantee what it has provided is quality code, without reviewing it first. The same applies to brand new devs — it's normal to apply a little more scrutiny because they simply don't have the experience to make the right decisions as confidently (or frequently) as someone more senior. It's an analogy and the natural fact that output reflects experience and practice over time.
- taneq 3y agoReminds me of a Facebook thread I saw a few days ago, on the topic of 3D printing houses. All the comments were angry dismissive "hurr durr that's clearly poor quality work" with no further justification of their position, and it struck me how similar the overall energy was to the "all AI image generation is bad and shit and is also heinous immoral theft and you're literally the worst person in the world and yous should feel bad" sort of raging that you see any time someone posts some SD or Midjourney or whatever pic of a cute puppy riding a tricycle. These comments originate from people who've spent their lives learning skills that are now largely replaceable by a few gigs of download and a Python tutorial. No wonder they're upset.
- OOPMan 3y agoRight...because requiring developers to actually grasp the code they're outputting is gatekeeping. If you want to pretend the rush to AI won't lead to more incompetent chefs in the kitchen than we already have (which is too many as it stands) then feel free, but acting like it's some kind of "party" people are being kept out of is daft. Standards exist for a reason, not just to make people feel bad for not meeting them.
- ugh123 3y agoPresumably people look at things before committing the code. And code reviews and pull requests are still normal. Blindly copying code from any source and running it or committing it to your main branch without even the slightest critical glance is foolish.
- ogrisel 3y agoArguably the tests should be easier to review than the implementation. But if there non-trivial logic in the code of the tests, I agree this is probably a risky approach.
- fileyfood500 3y agoIt's very powerful, I can enter implementations for any algorithm by typing 5 words and clicking tab. If I want the AI to use a hashmap to solve my problem in O(n), I just say that. If I need to rewrite a bunch of poorly written code to get rid of dead code, add constants, etc I do that. If I need to convert files between languages or formats, I do that. I have to do a lot more code review than before, and a lot less writing. It saves a huge amount of time, it's pretty easy to measure. Personally, the order of consultation is Github Copilot -> GPT4 -> Grimoire -> Me. If it's going to me, there is a high probability that I'm trying to do too many things at once in an over-complicated function. That or I'm using a relatively niche library and the AI doesn't know the methods.
- RamblingCTO 3y agoHopefully not, I feel it's a waste of time. The time spent on stupid minor mistakes by github copilot I didn't catch probably doesn't really compare to the time I would've spent typing on my own. (I only use that stuff for fancy code completion, nothing more. Every LLM is absolutely moronic. Yesterday I asked chatgpt to convert gohtml to templ, to no avail ...)