4 ms·

Seems like a reasonable move to me. I’ve seen folks use ChatGPT to generate code and review it for security flaws. It does often solve the tasks. And it leaves

Seems like a reasonable move to me. I’ve seen folks use ChatGPT to generate code and review it for security flaws. It does often solve the tasks. And it leaves behind many kinds of vulnerabilities: injections, overruns, etc. Based on the little empirical evidence we have about informal code review [0], it seems that we ought to limit or outright ban generated code. A Human can only read so much code before their impact on catching errors significantly drops. OSS project maintainers have enough on their plate and we don’t need to exhaust them with trying to maintain AI generated code. [0] https://sail.cs.queensu.ca/data/pdfs/EMSE_AnEmpiricalStudyOfTheImpactOfModernCodeReviewPracticesOnSoftwareQuality.pdf https://sail.cs.queensu.ca/data/pdfs/EMSE_AnEmpiricalStudyOf... Update spelling

cik 3y ago
Absolutely - though for me it's a personal anecdote. I remember within the first week of ChatGPT becoming generally available, I spent 2 hours crafting a response to a pull request, detailing a "security vulnerability". It didn't matter that this wasn't a vulnerability; it didn't matter that the reporter didn't know of what they were speaking. What mattered was eliminating the time I had that morning, to do meaningful work. Responding to that (and since, several) PRs was not the desired outcome. It also kicked off internal, processes - which in turn added more time. If you think about it, it's an interesting hack at least!