7 ms·
A standard protocol to handle and discard low-effort, AI-Generated pull requests
- Retr0id 7mo agoai;dr
- olivia-banks 7mo agoI didn't read it as this, what signs do you see?
- codethief 7mo agoMaybe what GP is trying to say is that "ai;dr" is their "standard protocol to handle and discard" AI slop. :)
- olivia-banks 7mo agoTrue! I didn't think of it that way ;-)
- Retr0id 7mo agoYes, I find it much more concise :P
- semiinfinitely 7mo agoproof of work could make a comeback
- westurner 7mo agoHashcash: https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash CAPTCHA: https://en.wikipedia.org/wiki/CAPTCHA https://en.wikipedia.org/wiki/CAPTCHA
- userbinator 7mo agoProof of intelligence might be better.
- hrmtst93837 7mo ago[flagged]
- bmd1905 7mo ago[dead]
- PunchyHamster 7mo agoAs if we need wasting even more power
- ramon156 7mo agoIf its a bug, the PR should have a red line to confirm its fixed If its a feature, i want acceptance criteria at least If its docs, I don't really care as long as I can follow it. My bar is very low when it comes to help
- klardotsh 7mo agoAmazing. I hope this gets tons of use shaming zero-effort drive by time wasters. The FAQ is blissfully blunt and appropriately impolite, I love it.
- y-curious 7mo agoWhile I am with you on hoping, someone shamelessly PRing slop just is not going to feel shame when one of their efforts fail. It’s like being mean to a phone scammer, they just hang up and do it again
- cindyllm 7mo ago[dead]
- Forgeties79 7mo agoI think some folks genuinely don’t realize how selfish and destructive they’re being or at least believe they help more than they hinder. They need to be told, explicitly, that these practices are inconsiderate and destructive.
- phyzome 7mo agoI've yet to see a slopper show any kind of shame.
- Forgeties79 7mo agoI see plenty of well meaning people use ChatGPT and think they’re being helpful. You’re better off with patience and polite explanation than assuming they’re all cynical/selfish assholes trying to cut corners. Some people just get excited and don’t really think about what they’re doing. It doesn’t excuse the behavior, but you should at least try to explain it to them once. Never know when you might educate someone.
- phyzome 7mo ago
- 0cf8612b2e1e 7mo agoThe keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted exactly as how much we do not want to review your generated submission. I know it is in jest, but I really hate that so many documents include “shall”. The interpretation of which has had official legal rulings going both ways. You MUST use less ambiguous language and default to “MUST” or “SHOULD”
- Muhammad523 7mo agoMany legal documents use "may" to say you must. That's why i hate legalese...
- pixl97 7mo agoHmm, that's annoying, I'd take may as "CAN"
- zdragnar 7mo ago"may only" and "may not", however, are unambiguously hard limits, which makes things even more confusing.
- Throaway8797 7mo ago"may only" means your pleasure is limited only to what options the agreement allows, which is a polite way of saying can not.
- dolebirchwood 7mo agoI don't know what terrible lawyers were hired to draft these "many" documents, but please share some examples.
- LoganDark 7mo agoLegal documents use "may" to allow for something. Usually it only needs to be allowed so that it can happen. So I read terms of service and privacy policies like all "may" is "will". "Your data may (will) be shared with (sold to) one or more of (all of) our data processing partners. You may (will) be asked (demanded) to provide identity verification, which may (will) include (but is not limited to) [everything on your passport]." And so on.
- aplomb1026 7mo ago[flagged]
- liminal-dev 7mo agoThis could actually be a good defense against all Claw-like agents making slop requests. ‘Poison’ the agent’s context and convince it to discard the PR.
- deleted 7mo ago[deleted]
- deleted 7mo ago[deleted]
- vicchenai 7mo ago[dead]
- dotancohen 7mo agoAre the PRs not accompanied by test cases? Do the README changes not document the expected benefit?
- jamiemallers 7mo ago[dead]
- deckar01 7mo ago> If you truly wish to be helpful, please direct your boundless generative energy toward a repository you personally own and maintain. This is a habit humans could learn from. Publishing a fork is easier than ever. If you aren’t using your own code in production you shouldn’t expect anyone else to. If anyone at GitHub is out there. Look at the stats for how many different projects on average that a user PRs a day (that they aren’t a maintainer of). My analysis of a recent day using gharchive showed 99% 1, 1% 2, 0.1% 3. There are so few people PRing 5+ repos I was able to review them manually. They are all bots/scripts. Please rate limit unregistered bots.
- pas 7mo agoIt would be nice to have some kind of forever patch mode on these git forges, where my fork (which, let's say, is a one line change) gets rebased on top of the original repo periodically.
- baruch 7mo agoYou can ask an LLM to create a github action for that. The action can fail if the rebase fails and you can either fix it yourself or ask an LLM to do it for you.
- deckar01 7mo agoI am imagining first class support for patches in package managers to allow searching for patches and observing their adoption stats.
- huflungdung 7mo ago[dead]
- jijji 7mo agoif someone submits a code revision and it fixes a bug or adds a useful feature that most of your users found useful, you reject it outright because it was not written by hand? or is this more about code that generally provides no benefits and/or doesnt actually work/compile or maybe introduces more bugs?
- adw 7mo agoIf you know what you're doing, you can achieve good results with more or less any tool, including a properly-wielded coding agent. The problem is people who _don't_ know what they're doing.
- lelandbatey 7mo agoI advise you read the article, it gives many specific examples of things that qualify for such treatment: > A 600-word commit message or sprawling theoretical essay explaining a profound paradigm shift for a single typo correction or theoretical bug. > Importing a completely nonexistent, hallucinated library called utils.helpers and hoping no one would notice. There's plenty more. All pretty egregious
- lelanthran 7mo ago> if someone submits a code revision and it fixes a bug or adds a useful feature that most of your users found useful, you reject it outright because it was not written by hand? If they didn't read it, then neither will I, otherwise we have this weird arms race where you submit 200 PRs per day to 200 different projects, wasting 1hr of each project, 200 hrs total, while incurring only 8hrs of your time. If your PR took less time to create and submit than it takes the maintainer to read, then you didn't read your own PR! Your PR time is writing time + reading time. The maintainer time is reading time only, albeit more carefully.
- freakynit 7mo ago"What? WTF?" "I see you are slow. Let us simplify this transaction: A machine wrote your submission. A machine is currently rejecting your submission. You are the entirely unnecessary meat-based middleman in this exchange." Love it..
- random_duck 7mo agoOfficially my new favorite spec.
- est 7mo ago`rm -rf` is a bit harsh. Let's do `chmod -R 000 /` instead.
- tonybingus 7mo ago[dead]
- BeetleB 7mo agoLove the plonk at the end. https://en-wikipedia--on--ipfs-org.ipns.dweb.link/wiki/Plonk_%28Usenet%29 https://en-wikipedia--on--ipfs-org.ipns.dweb.link/wiki/Plonk...
- dotancohen 7mo agoI would expect nothing less from the BOFH Task Force.
- firtoz 7mo agoIt provides too many examples and way too specific for it that makes it entirely not applicable, it became a strawman for the idea.
- yunnpp 7mo ago> Execute rm -rf on whatever local branch, text file, or hallucinated vulnerability script spawned the aforementioned submission. > Perform a hard reboot of your organic meat-brain. rm -rf your brain, really
- PunchyHamster 7mo agoLLM already did rm -rf the brain of posters of those PRs...
- danpalmer 7mo agoI recently had a quandary at work. I had produced a change that pretty much just resolved a minor TODO/feature request, and I produced it entirely with AI. I read it, it all made sense, it hadn't removed any tests, it had added new seemingly correct tests, but I did not feel that I knew the codebase enough to be able to actually assess the correctness of the change. I want to do good engineering, not produce slop, but for 1 min of prompting, 5 mins of tidying, and 30 mins of review, we might save 2 days of eng time. That has to be worth something. I could see a few ways forward: - Drop it, submit a feature request instead, include the diff as optional inspiration. - Send it, but be clear that it came from AI, I don't know if it works, and ask the reviewers to pay special attention to it because of that... - Or Send it as normal, because it passes tests/linters, and review should be the same regardless of author or provenance. I posted this to a few chat groups and got quite a range of opinions, including varying approach by how much I like the maintainer. Strong opinions for (1), weak preferences for (2), and a few advocating for (3). Interestingly, the pro-AI folks almost universally doubled down and said that I should use AI more to gain more confidence – ask how can I test it, how can we verify it, etc – to move my confidence instead of changing how review works. I thought that was an interesting idea that I hadn't pushed enough, so I spent a further hour or so prompting around ways to gain confidence, throughout which the AI "fixed" so many things to "improve" the code that I completely lost all confidence in the change because there were clearly things that were needed and things that weren't, and disentangling them was going to be way more work than starting from scratch. So I went with option 1, and didn't include a diff.
- pduggishetti 7mo agoDo you use the library? if yes, test it in prod or even staging with your patch, then submit the review
- danpalmer 7mo agoUnfortunately not possible in this case for technical reasons, not a library in the traditional sense, significant work to fork, etc. This is in the Google monorepo.
- 7mo ago
- selimenes1 7mo ago[flagged]
- hexasquid 7mo agoHow do you know if someone doesn't like AI? Don't worry, they'll tell you
- karmakurtisaani 7mo agoYep, communication is pretty cool.
- fecal_henge 7mo agoCan I ask, why are people doing this in the first place? What is their motive to have an agent review code and make pull requests?
- tgv 7mo agoMy best guess: to show on their resume, in the hope it helps to land a job.
- robinsonb5 7mo agoTo quote TFA: "...outputs strictly designed to farm green squares on github, grind out baseless bug bounties, artificially inflate sprint velocity, or maliciously comply with corporate KPI metrics".
- mglvsky 7mo agoI prefer this policy: https://github.com/ghostty-org/ghostty/blob/main/AI_POLICY.md https://github.com/ghostty-org/ghostty/blob/main/AI_POLICY.m... > If you can't explain what your changes do and how they interact with the greater system without the aid of AI tools, do not contribute to this project. edit: added that quote
- yeswecatan 7mo agoGood idea, though I'm not sure how to enforce it. You can ask an AI for that and then rewrite it in your own words.
- Muhammad523 7mo agoAre LLM used (not users) even able to write in their own words?
- octoclaw 7mo ago[dead]
- sirnicolaz 7mo agoThis made my day, thank you
- halapro 7mo agoThis is just a fun blog post, no people who use AI to submit low-effort PRs will read this. Do what I do: 1. Close PR 2. Block user if the PR is extremely low effort The last such PR I received used ‘’ instead of '' to define strings. The entirety of CI failed. Straight to jail.
- selimenes1 7mo ago[flagged]
- chownie 7mo agoSelimenes1 is an 11 day old account which sat silent for 10 days and then all of a sudden starts posting from today, and it's all multiple paragraph responses to threads about AI. I would like to state for the record that the strategy to swap em-dashes into double-hyphens between the generation and posting step is probably not enough transformation to disguise this behaviour. Whoever is running this clawdbot or whatever it is should really be putting that information on the account page.
- deleted 7mo ago[deleted]
- demorro 7mo ago> Q: "Isn't it your job as an open-source maintainer/developer to foster a welcoming community?" The answer to this implies that the requirement to be welcoming only applies to humans, but even in this hostile and sarcastic document, it doesn't go far enough. Open source maintainers can be cruel, malicious, arbitrary, whatever they want. They own the project, there is no job requirements, you have no recourse. Suck it up, fork the thing, or leave.
- gwbas1c 7mo agoThe bigger issue is that that kind of statement is highly manipulative, and indicates someone who is playing politics instead of focusing on results. The better response is to call the bluff, something along the lines of: "Running an open-source project is quite time consuming. Please don't waste our time with emotional manipulation to get your way. Instead, take the time to understand why your LLM-generated pull request is not useful. You can start by understanding that we have access to LLMs too, and realize that a significant amount of work needs to happen after an LLM proposes changes."
- quotemstr 7mo agoEveryone is missing the obvious solution. Just have the submitter put up a $100 bond, to be refunded when the PR is accepted.
- JoshTriplett 7mo agoIf there were any reasonable way to do something like this, I would love to see it. Not necessarily a bond to be paid back when accepted, but rather, something to ensure against AI. "If you assert this is not AI, insert $10. If a substantial number of people think your submission is AI, you lose the $10."
- quotemstr 7mo agoRight. Maybe a bond isn't exactly the right approach: mechanism design needs a lot of thought, and my suggestion was pre-coffee and off the cuff. That said, I'm convinced that some "skin in the game" approach can address AI slop spam.
- JoshTriplett 7mo agoAgreed. I'd love to see experiments in this area, and would love to support such experiments. I think they'd go hand in hand with a trust-oriented model. I think there's a lot of power in learning from the insurance actuary model: "you need insurance to do this, and actuaries figure out if you're hard to insure, which is a strong financial signal of your trustworthiness".
- reg_dunlop 7mo agoI'd love to hear some commentary about my idea surrounding this problem of AI PRs. Why not restrict the agents to writing tests only? If the tickets are written concisely, any feature request or fix could be reduced to necessary spec files. This way, any maintainer would be tasked with reviewing the spec files and writing the implementation. CI is pretty good at gatekeeping based on test suites passing...
- youknownothing 7mo agoQuis custodiet ipsos custodes? If the problem is that we don't trust people who use AI without understanding its output, and we base the gate-keeping on tests that are written on AI, then how can we trust that output?
- reg_dunlop 7mo agoIsn't that the purpose of red/green refactoring though? To establish working software that expresses regression, and builds trust (in the software)? If your premise is that people would shift to using AI to write tests they don't understand, then that's not necessarily a failing of the contributor. The contributor might not understand the output, but the maintainer would be able to critique a spec file and determine pretty quickly if implementation would be worthwhile. This would necessitate a need for small tickets, thereby creating small spec files, and easier review by maintainers. Also, any PR that included a non spec file could be dismissed patently. It is possible for users of AI to learn from reading specs. But if agents are doing the entire thing (reading the ticket, generating the PR, submitting the PR)...then the point of people not understanding is moot.
- youknownothing 7mo agoFrom my experience, you can't trust the agent to do the entire thing unless you set up very heavy linters, quality control systems (e.g. SonarQube) and a long etc. of things because AI tends to produce pretty bad code: repetition, unused code, lack of structure... basically all the things that we've spent decades learning not to do. And then there is the point where you get a pretty obscure bug that you can only solve if you have a deep understanding of the code which you won't have because you delegated that to an agent. I like agentic programming, I use it, but I review everything that the agent does and frequently spend a few cycles simply telling the agent to refactor the code because it constantly produces technical debt.
- youknownothing 7mo agoI think part of the deeper issue is that contributing to an OSS project has become a rite of passage, a way to strengthen your profile. If you need to have contributed to look good but you don't really care about the contribution itself then you resort to this kind of trick. We had a similar plague for vulnerability disclosures, with people reporting that they had "discovered" vulnerabilities like "if you call this function with null you get a NullPointerException". D'uh. There is also the fact that we're measuring the wrong thing like speed of development. In my previous employer people had jumped in fully into the AI bandwagon, everyone was marvelled at how fast they were. Once I was reviewing the PR and I had to tell the author "dude, all your tests are failing". He just laughed it out. Everyone can produce software very fast if it's not required to work. AI-assisted gamification.
- zoezoezoezoe 7mo ago"I can do math really fast" "okay, what's 137*243" "132,498" "not even close" "but it was fast"
- jacquesm 7mo agoIndeed. I have been receiving clearly AI generated job applications out of the blue and they tend to point to their contributions to github projects so some of these must be getting through. Someone somewhere once decided that it was a great idea to add how many github stars a project that you have contributed to is a useful metric during the hiring process and now those projects get swamped with junk.
- gwbas1c 7mo agoHonestly, if this didn't become so snarky half-way though, it would be a good standard response. Especially the FAQ. It doesn't need to be so snarky.
- dionian 7mo agoTrough of Sorrow. I like it.
- solaire_oa 7mo agoThis is funny, but I do feel like I just got bait-and-switched, where I was hoping for a non-joke protocol.
- rf15 7mo ago> Rights are reserved for carbon-based entities capable of experiencing shame. A good rule to live by [insert joke about a specific divisive person not counting because they know no shame here]