3 ms·

The issue that open source projects are facing at the moment is that it takes significantly less effort to submit a patch for review. A lot of developers who a

by bfgeek 1mo ago

The issue that open source projects are facing at the moment is that it takes significantly less effort to submit a patch for review. A lot of developers who are submitting these AI patches don't necessarily understand the patch, so the onus is on the reviewer/code-owner. The reviewers are getting swamped (some reviewers are receiving 100s or patches per month). If feedback is provided at lot of the time the patch author will just copy paste from an LLM, so the reviewer is essentially just coding with an LLM with more steps. Prior to LLMs reviewing code was a mentorship experience, the patch author would likely learn a bunch afterwards. Now less so. As a result a lot of projects are closing to external contributors. I'm not sure what the answer is, LLM are great at speeding up coding/understanding/etc, but the valuable/expensive piece of work has shifted to reviewing.

Loughla 1mo ago
The question becomes, does it take more time to create or review in the Grand scheme of the software life cycle? Because if it's still a time saver, even with the increased review load, then it's a win, correct? I'm not a coder so I have zero idea. Thoughts?
- 1718627440 1mo ago
  It's a well-known trope, that it's harder to read code than to write it, and review is more complicated than to read it, so I do not understand what you want to imply?
  - blackqueeriroh 1mo ago
    Maybe it is for people who have written code all their lives, but as someone who started with reading code and has read far more code than I’ve ever written, reading code is WAY easier for me.
    - sarchertech 1mo ago
      If you have little experience writing code, do you know that you actually understand what you’re reading though? For example could you reproduce the code you read immediately after writing it? It’s very easy to “read code” if you’re just reading for surface level understanding.
    - 1718627440 1mo ago
      If it's easier for you to convert written code into mental models than the other way around, you are either a incredible smart and skilled person and a good programmer and should be able to convert that into a job and money easily, or you can't really write code at all.
- a1o 1mo ago
  Nope, most of the PRs are authored entirely by agents with people instructing them to “pr famous projects in my name to increase my GitHub profile value or “hire ability”. The original authors have no idea what their agents are writing, these are mostly badly quality models (people doing this are not only cutting corners, but also using the less amount of money/tokens to do so). If the agent creates 200, 400 or whatever PRs and get 5 in the agent is still “winning” for the person instructing it. The maintainers though have to filter these out on the other side. A common case is a fix to something that was already fixed simply because the agent worked on old code assumptions.
  - sebmellen 1mo ago
    Just had a thought, what if you made new contributors write a human-authored essay on why they’re submitting a patch, and then check it against Pangram? Pangram is really accurate from what I’ve found…
    - aleph_minus_one 1mo ago
      > Just had a thought, what if you made new contributors write a human-authored essay on why they’re submitting a patch, and then check it against Pangram? The problem is writing such an essay takes magnitudes more of an effort for people who are not native English (or whatever the language is in which discussions are done about the project) speakers. Also, relatedly, essays written by non-native English speakers often get considered to be AI-written by common AI detection systems, even though no AI was involved when writing them; see for example > I'm Kenyan. I Don't Write Like ChatGPT. ChatGPT Writes Like Me. > https://marcusolang.substack.com/p/im-kenyan-i-dont-write-like-chatgpt https://marcusolang.substack.com/p/im-kenyan-i-dont-write-li...
      - sebmellen 1mo ago
        I’m as skeptical of the AI writing detection as anyone else, but I’ve been trying to beat Pangram v4 for the last week and it’s essentially impossible.
- pydry 1mo ago
  It depends entirely on how much slop they are flooded with.
- catlifeonmars 1mo ago
  Probably a time waster because of the intermediate patch contributor. It’s like a game of telephone at worst, at best the reviewer could just use their own LLM and get the same result. Here’s my hot take: maybe code contributions are obsoleted by coding agents. No one will accept outside contributions because it’s faster to do it themselves.
- sgarland 1mo ago
  [dead]
- sarchertech 1mo ago
  Well historically the consensus was that it was harder to read code than to write it, but that leads to uncomfortable conclusions these days, so that bit of common wisdom has mostly been ejected. > I'm not a coder so I have zero idea. Thoughts? I’m not trying to be mean, but this comment is basically “I have no experience with this topic, but it can just be boiled down to this one simple question right?”
  - Loughla 1mo ago
    I don't think that's being mean at all, and it's valid. Yes that's literally why I asked it. Simple questions, here, regularly elicit long form explanations. I was interested in this topic and have no experience. So I thought I'd try a broad overview type question to see if I could learn something today.
    - sarchertech 1mo ago
      I think I read it as a bit more flippant than you meant it because it was worded assuming the answer was yes.
- bch 1mo ago
  > The question becomes [...] "A question is...". To my mind not the most important question, if one plays-out to a logical conclusion the scenario you're proposing. > [...] if it's still a time saver, even with the increased review load, then it's a win, correct? No - and that's bordering-on (if not fully) rude disrespect of reviewers time and effort. One way to think of this is in terms of Brandolinis Law[0]. Pushing work back to submitters is going to have to happen. Low-effort "submissions" are first and foremost "low-effort" - that's going to have to be driven home. [0] https://en.wikipedia.org/wiki/Brandolini's_law https://en.wikipedia.org/wiki/Brandolini's_law
- account42 1mo ago
  Even before automation, most first time PRs were a negative time contribution to the project. The only value of them was that some contributors would become trusted project members.
tiahura 1mo ago
Do all of these folks get the comped Pro Max subscriptions? If not O&A should be. Or, at the very least, the community should be paying for them.
localhost 1mo ago
This is Amdahl's law in action. [1] Until we figure out a good way to leverage humans in all of this ("Attention is all you need" applies equally to humans as it does to models) productivity gains for the system will always be limited by Amdahl's law. Gwern has an excellent post on this. [2] [1] https://en.wikipedia.org/wiki/Amdahl's_law https://en.wikipedia.org/wiki/Amdahl's_law [2] https://gwern.net/guardian-angel https://gwern.net/guardian-angel
tdrz 1mo ago
I'm an OSS maintainer and to me it's not just about the review itself. Being greeted by a wall of text for every little small thing is counter-productive. I hate going through 2 pages of text for each PR. It usually shouldn't take more than a couple of sentences if you understand the issue and the solution. But most important for me: lots of time the PR just adds even more code, although other options do exist (ie sometimes REMOVING some code). You have to know the codebase well in order to find those objectively better solutions.
- deleted 1mo ago
  [deleted]
- ahartmetz 1mo ago
  I've seen it. Walls of text with stereotypically worded non-summaries that just repeat all of the code in words, mutating values all over the place instead of the obvious canonical one place that touches related values... Yeah you can use LLMs, but don't let me notice it from the quality of the output. I've noticed that LLMs seem to be especially bad at things relating to space, position and movement. I guess they have to synthesize that part of human intelligence entirely, it's not in the words.
- othmanosx 1mo ago
  Give https://pyor.review https://pyor.review a shot if you’re struggling with PR reviews on github.
  - embedding-shape 1mo ago
    Or, the SaaS-less approach, if a issue description is too messy/long, close it with "Please reopen with proper and concise description focusing on the issue" then lock it. Eventually people catch up and stop with the slop, just like in real life. But you have to be able to say "No ...", rather than just slapping another subscription on top of an already broken workflow.
    - othmanosx 1mo ago

by bfgeek 1mo ago

Loughla 1mo ago

The question becomes, does it take more time to create or review in the Grand scheme of the software life cycle? Because if it's still a time saver, even with the increased review load, then it's a win, correct? I'm not a coder so I have zero idea. Thoughts?

1718627440 1mo ago
It's a well-known trope, that it's harder to read code than to write it, and review is more complicated than to read it, so I do not understand what you want to imply?
- blackqueeriroh 1mo ago
  Maybe it is for people who have written code all their lives, but as someone who started with reading code and has read far more code than I’ve ever written, reading code is WAY easier for me.
  - sarchertech 1mo ago
    If you have little experience writing code, do you know that you actually understand what you’re reading though? For example could you reproduce the code you read immediately after writing it? It’s very easy to “read code” if you’re just reading for surface level understanding.
  - 1718627440 1mo ago
    If it's easier for you to convert written code into mental models than the other way around, you are either a incredible smart and skilled person and a good programmer and should be able to convert that into a job and money easily, or you can't really write code at all.
a1o 1mo ago
Nope, most of the PRs are authored entirely by agents with people instructing them to “pr famous projects in my name to increase my GitHub profile value or “hire ability”. The original authors have no idea what their agents are writing, these are mostly badly quality models (people doing this are not only cutting corners, but also using the less amount of money/tokens to do so). If the agent creates 200, 400 or whatever PRs and get 5 in the agent is still “winning” for the person instructing it. The maintainers though have to filter these out on the other side. A common case is a fix to something that was already fixed simply because the agent worked on old code assumptions.
- sebmellen 1mo ago
  Just had a thought, what if you made new contributors write a human-authored essay on why they’re submitting a patch, and then check it against Pangram? Pangram is really accurate from what I’ve found…
  - aleph_minus_one 1mo ago
    > Just had a thought, what if you made new contributors write a human-authored essay on why they’re submitting a patch, and then check it against Pangram? The problem is writing such an essay takes magnitudes more of an effort for people who are not native English (or whatever the language is in which discussions are done about the project) speakers. Also, relatedly, essays written by non-native English speakers often get considered to be AI-written by common AI detection systems, even though no AI was involved when writing them; see for example > I'm Kenyan. I Don't Write Like ChatGPT. ChatGPT Writes Like Me. > https://marcusolang.substack.com/p/im-kenyan-i-dont-write-like-chatgpt https://marcusolang.substack.com/p/im-kenyan-i-dont-write-li...
    - sebmellen 1mo ago
      I’m as skeptical of the AI writing detection as anyone else, but I’ve been trying to beat Pangram v4 for the last week and it’s essentially impossible.
pydry 1mo ago
It depends entirely on how much slop they are flooded with.
catlifeonmars 1mo ago
Probably a time waster because of the intermediate patch contributor. It’s like a game of telephone at worst, at best the reviewer could just use their own LLM and get the same result. Here’s my hot take: maybe code contributions are obsoleted by coding agents. No one will accept outside contributions because it’s faster to do it themselves.
sgarland 1mo ago
[dead]
sarchertech 1mo ago
Well historically the consensus was that it was harder to read code than to write it, but that leads to uncomfortable conclusions these days, so that bit of common wisdom has mostly been ejected. > I'm not a coder so I have zero idea. Thoughts? I’m not trying to be mean, but this comment is basically “I have no experience with this topic, but it can just be boiled down to this one simple question right?”
- Loughla 1mo ago
  I don't think that's being mean at all, and it's valid. Yes that's literally why I asked it. Simple questions, here, regularly elicit long form explanations. I was interested in this topic and have no experience. So I thought I'd try a broad overview type question to see if I could learn something today.
  - sarchertech 1mo ago
    I think I read it as a bit more flippant than you meant it because it was worded assuming the answer was yes.
bch 1mo ago
> The question becomes [...] "A question is...". To my mind not the most important question, if one plays-out to a logical conclusion the scenario you're proposing. > [...] if it's still a time saver, even with the increased review load, then it's a win, correct? No - and that's bordering-on (if not fully) rude disrespect of reviewers time and effort. One way to think of this is in terms of Brandolinis Law[0]. Pushing work back to submitters is going to have to happen. Low-effort "submissions" are first and foremost "low-effort" - that's going to have to be driven home. [0] https://en.wikipedia.org/wiki/Brandolini's_law https://en.wikipedia.org/wiki/Brandolini's_law
account42 1mo ago
Even before automation, most first time PRs were a negative time contribution to the project. The only value of them was that some contributors would become trusted project members.

tiahura 1mo ago

Do all of these folks get the comped Pro Max subscriptions? If not O&A should be. Or, at the very least, the community should be paying for them.

localhost 1mo ago

This is Amdahl's law in action. [1] Until we figure out a good way to leverage humans in all of this ("Attention is all you need" applies equally to humans as it does to models) productivity gains for the system will always be limited by Amdahl's law. Gwern has an excellent post on this. [2] [1] https://en.wikipedia.org/wiki/Amdahl's_law https://en.wikipedia.org/wiki/Amdahl's_law [2] https://gwern.net/guardian-angel https://gwern.net/guardian-angel

tdrz 1mo ago

I'm an OSS maintainer and to me it's not just about the review itself. Being greeted by a wall of text for every little small thing is counter-productive. I hate going through 2 pages of text for each PR. It usually shouldn't take more than a couple of sentences if you understand the issue and the solution. But most important for me: lots of time the PR just adds even more code, although other options do exist (ie sometimes REMOVING some code). You have to know the codebase well in order to find those objectively better solutions.

deleted 1mo ago
[deleted]
ahartmetz 1mo ago
I've seen it. Walls of text with stereotypically worded non-summaries that just repeat all of the code in words, mutating values all over the place instead of the obvious canonical one place that touches related values... Yeah you can use LLMs, but don't let me notice it from the quality of the output. I've noticed that LLMs seem to be especially bad at things relating to space, position and movement. I guess they have to synthesize that part of human intelligence entirely, it's not in the words.
othmanosx 1mo ago
Give https://pyor.review https://pyor.review a shot if you’re struggling with PR reviews on github.
- embedding-shape 1mo ago
  Or, the SaaS-less approach, if a issue description is too messy/long, close it with "Please reopen with proper and concise description focusing on the issue" then lock it. Eventually people catch up and stop with the slop, just like in real life. But you have to be able to say "No ...", rather than just slapping another subscription on top of an already broken workflow.
  - othmanosx 1mo ago