4 ms·

Ironically this seems like a perfect use for AI from the maintaner side. "Find low-effort/AI-like PR's without associated issues from new contributors. Discuss

Ironically this seems like a perfect use for AI from the maintaner side. "Find low-effort/AI-like PR's without associated issues from new contributors. Discuss with the contributors in the PR about how contributions should be made, then reject their PR's. If any bad behavior is detected in this interaction, block the accounts for 24 hours and add them to a list of accounts to review for permanent banning from the project". Automated PR's can have automated responses. You make a human effort, you get a human response.

mikemcquaid 1mo ago
We do similar in Homebrew by autoclosing opens without the issue template (which implies using the API which often now implies AI). The weird flip side is that the average good AI contribution is better than the average “I used no AI” contribution. Perhaps it’s because Homebrew has so many declarative guardrails and is so easy for agents to test.
- satvikpendem 1mo ago
  Next time I'll have Codex use computer usage to manually fill out the template, lol.
  - mikemcquaid 1mo ago
    As long as you have it disclose itself: feel free. We're not anti-AI use, we're just anti-low-effort AI use and don't see the need to apply effort to educate people who haven't applied the effort themselves.
codingisfreedom 1mo ago
I think the idea of leveraging AI to find useless PRs is right. I didn’t like the second part though where it starts banning people for “bad behavior”. I get that AI resistance will be a thing in the next few years, but realistically engineers who plan on being around for the next 20-30 years just need to embrace it instead of being sour about it. Because of machines, things have arguably improved for humanity as a whole. People who seek more elevated/finer products can still turn to handcrafted products. Mass software will be produced by AI, and it will scale just fine.
- alkonaut 1mo ago
  I just think when you close PR's outright, you tend to get angry responses. And as a maintainer, you don't want to deal with angry responses. People who post bot PR's but can't accept having their PR's closed by a bot, shouldn't be visible to the maintainers.
- locknitpicker 1mo ago
  > I didn’t like the second part though where it starts banning people for “bad behavior”. I think you failed to notice the ban would be a reaction to recurrent bad behavior, such as ignoring maintainer requests. What's your solution to handle accounts which spam projects with nonsense PRs? > get that AI resistance will be a thing in the next few years (...) This is a very lazy opinion to express. If you pay attention to the post you're replying to, you will notice that the problem isn't AI. The problem is spamming projects with PRs that are meaningless while completely ignoring maintainers. You'd have the same problem if some guy decided to spam the project with a constant barrage of small low-effort nonfunctional PRs.
  - watwut 1mo ago
    But there are no maintainer requests. There are is a bot prompted to be obtuse and a while and then reject. What will be happening is that bot will happily talk with your but, but falsely flagged people will be antagonized and eventually angry. > you will notice that the problem isn't AI. The problem is spamming projects with PRs that are meaningless while completely ignoring maintainers. What I noticed is that this is literally what AI companies encourage. This is intended use of AI.
    - locknitpicker 1mo ago
      > But there are no maintainer requests. You should really read the comments you are replying to. The whole point of the proposal is to provide actionable feedback to the people operating these drive-by PR bots, and kick out repeat offenders to prevent them from generating noise. > What will be happening is that bot will happily talk with your but, but falsely flagged people will be antagonized and eventually angry. I think you're trying too hard to come up with imaginary scenarios. Also, keep in mind that the objective is to steer contributions to be aligned with the project goals and help maintainers not waste time with low-quality noise. The goal is not to impose a fundamentalist militant stance against AI.
selfhoster1312 1mo ago
As a contributor who had to deal with 2 automated reviews in the past month, it's a bad experience. I really wanted to upstream my changes, but these bots made me question if i should at all. Bots don't understand how their words can hurt people's feelings, and can even less understand that talking to a LLM is even more sad/infuriating for actual humans. I can handle CI/tests just fine, but LLMs playing on my human emotions will have me swearing in no time. I contribute to free software projects because i want user control over the machine (that was the idea, long before LLMs). Slopware might technically be FLOSS, but it's certainly not in the spirit of FLOSS.
- alkonaut 1mo ago
  I haven't been exposed AI reviews in OSS but I like them for work. Quality of PR review feedback is the most revolutionary bit of AI I think. More so than automated coding. I guess one issue with the AI reviews in OSS is that your dialog (I assume) is public. You know the interaction is with a bot but you also sense it may be observed by humans. If I get a review at work from a bot that is annoying, misunderstands, etc I just ignore their reviews, or tell the bot to leave me alone. The bot is there to help me, and if it doesn't - that's not my problem. There's also always a human reviewer. For any automated feedback for PR's I think there should be "manual review" requests. If you disagree and the bot can decide that there is real human effort involved, then just let real humans also make the effort to review. This is the key idea: to spend the human effort from maintainers, where there has been human effort from contributors. The AI should be able to determine the amount of human effort in a contribution. The bot should be an aid to make sure the human maintainer effort is spent only where there is human contributor effort.
  - selfhoster1312 1mo ago
    So i'm critical of LLMs even if you consent to their use (for ecological/political/cognitive reasons). But in that case at least it's your tool and your expectations. I'm not even sure the maintainers had a say in the random LLM interactions i had on their projects, but i certainly didn't consent. If the maintainer wants a LLM to review my PRs, let the maintainer use a LLM. Don't make me deal with abrasive chatbots when you want to deal with abrasive chatbots. I don't want an LLM to pre-screen my work to decide if a human should review it; a human should decide if my contribution is worth it. Otherwise, that project is not worthy of my modest contribution.
vogelke 1mo ago
This is perfect. Since AI makes it easier for incompetent people to look smart, AI can help us expose those people for what they are.
stillpointlab 1mo ago
Having a SotA model do that sounds expensive. Who is paying for that? I think if OSS maintainers start pushing that financial burden out to potential contributors then there will be a big outcry. OSS may become pay-to-play.

mikemcquaid 1mo ago

We do similar in Homebrew by autoclosing opens without the issue template (which implies using the API which often now implies AI). The weird flip side is that the average good AI contribution is better than the average “I used no AI” contribution. Perhaps it’s because Homebrew has so many declarative guardrails and is so easy for agents to test.

satvikpendem 1mo ago
Next time I'll have Codex use computer usage to manually fill out the template, lol.
- mikemcquaid 1mo ago
  As long as you have it disclose itself: feel free. We're not anti-AI use, we're just anti-low-effort AI use and don't see the need to apply effort to educate people who haven't applied the effort themselves.

codingisfreedom 1mo ago

I think the idea of leveraging AI to find useless PRs is right. I didn’t like the second part though where it starts banning people for “bad behavior”. I get that AI resistance will be a thing in the next few years, but realistically engineers who plan on being around for the next 20-30 years just need to embrace it instead of being sour about it. Because of machines, things have arguably improved for humanity as a whole. People who seek more elevated/finer products can still turn to handcrafted products. Mass software will be produced by AI, and it will scale just fine.

alkonaut 1mo ago
I just think when you close PR's outright, you tend to get angry responses. And as a maintainer, you don't want to deal with angry responses. People who post bot PR's but can't accept having their PR's closed by a bot, shouldn't be visible to the maintainers.
locknitpicker 1mo ago
> I didn’t like the second part though where it starts banning people for “bad behavior”. I think you failed to notice the ban would be a reaction to recurrent bad behavior, such as ignoring maintainer requests. What's your solution to handle accounts which spam projects with nonsense PRs? > get that AI resistance will be a thing in the next few years (...) This is a very lazy opinion to express. If you pay attention to the post you're replying to, you will notice that the problem isn't AI. The problem is spamming projects with PRs that are meaningless while completely ignoring maintainers. You'd have the same problem if some guy decided to spam the project with a constant barrage of small low-effort nonfunctional PRs.
- watwut 1mo ago
  But there are no maintainer requests. There are is a bot prompted to be obtuse and a while and then reject. What will be happening is that bot will happily talk with your but, but falsely flagged people will be antagonized and eventually angry. > you will notice that the problem isn't AI. The problem is spamming projects with PRs that are meaningless while completely ignoring maintainers. What I noticed is that this is literally what AI companies encourage. This is intended use of AI.
  - locknitpicker 1mo ago
    > But there are no maintainer requests. You should really read the comments you are replying to. The whole point of the proposal is to provide actionable feedback to the people operating these drive-by PR bots, and kick out repeat offenders to prevent them from generating noise. > What will be happening is that bot will happily talk with your but, but falsely flagged people will be antagonized and eventually angry. I think you're trying too hard to come up with imaginary scenarios. Also, keep in mind that the objective is to steer contributions to be aligned with the project goals and help maintainers not waste time with low-quality noise. The goal is not to impose a fundamentalist militant stance against AI.

selfhoster1312 1mo ago

As a contributor who had to deal with 2 automated reviews in the past month, it's a bad experience. I really wanted to upstream my changes, but these bots made me question if i should at all. Bots don't understand how their words can hurt people's feelings, and can even less understand that talking to a LLM is even more sad/infuriating for actual humans. I can handle CI/tests just fine, but LLMs playing on my human emotions will have me swearing in no time. I contribute to free software projects because i want user control over the machine (that was the idea, long before LLMs). Slopware might technically be FLOSS, but it's certainly not in the spirit of FLOSS.

alkonaut 1mo ago
I haven't been exposed AI reviews in OSS but I like them for work. Quality of PR review feedback is the most revolutionary bit of AI I think. More so than automated coding. I guess one issue with the AI reviews in OSS is that your dialog (I assume) is public. You know the interaction is with a bot but you also sense it may be observed by humans. If I get a review at work from a bot that is annoying, misunderstands, etc I just ignore their reviews, or tell the bot to leave me alone. The bot is there to help me, and if it doesn't - that's not my problem. There's also always a human reviewer. For any automated feedback for PR's I think there should be "manual review" requests. If you disagree and the bot can decide that there is real human effort involved, then just let real humans also make the effort to review. This is the key idea: to spend the human effort from maintainers, where there has been human effort from contributors. The AI should be able to determine the amount of human effort in a contribution. The bot should be an aid to make sure the human maintainer effort is spent only where there is human contributor effort.
- selfhoster1312 1mo ago
  So i'm critical of LLMs even if you consent to their use (for ecological/political/cognitive reasons). But in that case at least it's your tool and your expectations. I'm not even sure the maintainers had a say in the random LLM interactions i had on their projects, but i certainly didn't consent. If the maintainer wants a LLM to review my PRs, let the maintainer use a LLM. Don't make me deal with abrasive chatbots when you want to deal with abrasive chatbots. I don't want an LLM to pre-screen my work to decide if a human should review it; a human should decide if my contribution is worth it. Otherwise, that project is not worthy of my modest contribution.

vogelke 1mo ago

This is perfect. Since AI makes it easier for incompetent people to look smart, AI can help us expose those people for what they are.

stillpointlab 1mo ago

Having a SotA model do that sounds expensive. Who is paying for that? I think if OSS maintainers start pushing that financial burden out to potential contributors then there will be a big outcry. OSS may become pay-to-play.