3 ms·

> Let's pick one of the best possible examples of AI-generated code changes. An example so good, that the Cursor Editor uses it to advertise on their front page

> Let's pick one of the best possible examples of AI-generated code changes. An example so good, that the Cursor Editor uses it to advertise on their front page. I can see where the author is coming from, but “marketing image is dumb” is by far the least interesting critique possible for these tools. Give me an argument for why SWE bench is flawed, or an analysis of what areas didn’t improve in Claude4. Meanwhile I’m vibe coding with Claude and having a great time. Sure, I wouldn’t use it for anything high stakes, but vs. Claude 3.7 I’m seeing a much higher success rate on tens to hundreds of lines of code. It’s really, really easy to get a better code sample than the one shown here.

deivid 1y ago
> It’s really, really easy to get a better code sample than the one shown here. The problem is that in a team setting, you can't control the type of "code sample" that your teammates send for review. It's frequently the case that some teammates will request review for this kind of slop, while outsourcing all the "thinking" to the reviewer. As a revwiewer, I now have to review much more code (LLMs absolutely increase the amount of code per hour you can put out), which is often unfiltered LLM output, so it needs more careful and thorough review. Essentially, a careless developer with access to an LLM can now perform a cheap DoS on reviewers.
- imiric 1y ago
  > Essentially, a careless developer with access to an LLM can now perform a cheap DoS on reviewers. Exactly. And not just a careless developer, but one intentionally doing so. Recently [this project][1] made it to the front page and amassed quite a few upvotes, positive feedback, and GitHub stars. Yet anyone giving it a cursory look would be able to tell that it's AI-generated slop, that is completely unnecessary at best, and possible malware at worst. Not only that, but the author[2] fired off several AI slop PRs to popular projects in a single day (May 12th), which is exactly the cheap DoS you mention. The author of cURL wrote about AI-generated bogus security reports last year[3], and the work they add to open source maintainers who are already stretched thin. So this is a real problem that most AI proponents are ignoring. [1]: https://news.ycombinator.com/item?id=44009321 https://news.ycombinator.com/item?id=44009321 [2]: https://github.com/dipampaul17 https://github.com/dipampaul17 [3]: https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...

deivid 1y ago

> It’s really, really easy to get a better code sample than the one shown here. The problem is that in a team setting, you can't control the type of "code sample" that your teammates send for review. It's frequently the case that some teammates will request review for this kind of slop, while outsourcing all the "thinking" to the reviewer. As a revwiewer, I now have to review much more code (LLMs absolutely increase the amount of code per hour you can put out), which is often unfiltered LLM output, so it needs more careful and thorough review. Essentially, a careless developer with access to an LLM can now perform a cheap DoS on reviewers.

imiric 1y ago
> Essentially, a careless developer with access to an LLM can now perform a cheap DoS on reviewers. Exactly. And not just a careless developer, but one intentionally doing so. Recently [this project][1] made it to the front page and amassed quite a few upvotes, positive feedback, and GitHub stars. Yet anyone giving it a cursory look would be able to tell that it's AI-generated slop, that is completely unnecessary at best, and possible malware at worst. Not only that, but the author[2] fired off several AI slop PRs to popular projects in a single day (May 12th), which is exactly the cheap DoS you mention. The author of cURL wrote about AI-generated bogus security reports last year[3], and the work they add to open source maintainers who are already stretched thin. So this is a real problem that most AI proponents are ignoring. [1]: https://news.ycombinator.com/item?id=44009321 https://news.ycombinator.com/item?id=44009321 [2]: https://github.com/dipampaul17 https://github.com/dipampaul17 [3]: https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/ https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...