4 ms·

(Disclaimer: I work on coding agents at GitHub) This data is great, and it is exciting to see the rapid growth of autonomous coding agents across GitHub. One

(Disclaimer: I work on coding agents at GitHub) This data is great, and it is exciting to see the rapid growth of autonomous coding agents across GitHub. One thing to keep in mind regarding merge rates is that each of these products creates the PR at a different phase of the work. So just tracking PR create to PR merge tells a different story for each product. In some cases, the work to iterate on the AI generated code (and potentially abandon it if not sufficiently good) is done in private, and only pushed to a GitHub PR once the user decides they are ready to share/merge. This is the case for Codex for example. The merge rates for product experiences like this will look good in the stats presented here, even if many AI generated code changes are being abandoned privately. For other product experiences, the Draft PR is generated immediately when a task is assigned, and users can iterate on this “in the open” with the coding agent. This creates more transparency into both the success and failure cases (including logs of the agent sessions for both). This is the case for GitHub Copilot coding agent for example. We believe this “learning in the open” is valuable for individuals, teams, and the industry. But it does lead to the merge rates reported here appearing worse - even if logically they are the same as “task assignment to merged PR” success rates for other tools. We’re looking forward to continuing to evolve the notion of Draft PR to be even more natural for these use cases. And to enabling all of these coding agents to benefit from open collaboration on GitHub.

soamv 1y ago
This is a great point! But there's an important tradeoff here about human engineering time versus the "learning in the open" benefits; a PR discarded privately consumes no human engineering time, a fact that the humans involved might appreciate. How do you balance that tradeoff? Is there such a thing as a diff that's "too bad" to iterate on with a human?
- ambicapter 1y ago
  Do people where you work spend time reviewing draft PRs? I wouldn’t do that unless asked to by the author.
  - drawnwren 1y ago
    It’s hard enough for me to get time to review actual PRs, who are these engineers trawling through the drafts?
- lukehoban 1y ago
  I do agree there is a balance here, and that the ideal point in the spectrum is likely in between the two product experiences that are currently being offered here. There are a lot of benefits to using PRs for the review and iteration - familiar diff UX, great comment/review feedback mechanisms, ability to run CI, visibility and auth tracked natively within GitHub, etc. But Draft PRs are also a little too visible by default in GitHub today, and there are times when you want a shareable PR link that isn't showing up by default on the Pull Requests list in GitHub for your repo. (I frankly want this even for human-authored Draft PRs, but its even more compelling for agent authored PRs). We are looking into paths where we can support this more personal/private kind of PR, which would provide the foundation within GitHub to support the best of both worlds here.
polskibus 1y ago
What is your team’s take on the copyright for commits generated by ai agent ? Would the copyright protect it? Current US stance seems to be: https://www.copyright.gov/newsnet/2025/1060.html https://www.copyright.gov/newsnet/2025/1060.html “It concludes that the outputs of generative AI can be protected by copyright only where a human author has determined sufficient expressive elements”. If entire commit is generated by AI then it is obvious what created it - it’s AI. Such commit might not be covered by the law. Is this something your team has already analysed?
- IanCal 1y ago
  >If entire commit is generated by AI then it is obvious what created it - it’s AI. Whether it's committed or not is irrelevant to the conclusion there, the question is what was the input.
  - chiph 1y ago
    For something like a compiler where the output is mostly deterministic[0] I agree. For an AI that was trained on an unknown corpus, and that corpus changes over time, the output is much less deterministic and I would say you lose the human element needed of copyright claims. If it can be shown that for the same prompt, run through the AI several times over perhaps a year, results in the same output - then I will change my mind. Or if the AI achieves personhood. [0] Allowances for register & loop optimization, etc.
- rustc 1y ago
  > “It concludes that the outputs of generative AI can be protected by copyright only where a human author has determined sufficient expressive elements” How would that work if it's a patch to a project with a copyleft license like GPL which requires all derivate work to be licensed the same?
  - anticensor 1y ago
    GPL is a copyright licence, not a ToS.

soamv 1y ago

This is a great point! But there's an important tradeoff here about human engineering time versus the "learning in the open" benefits; a PR discarded privately consumes no human engineering time, a fact that the humans involved might appreciate. How do you balance that tradeoff? Is there such a thing as a diff that's "too bad" to iterate on with a human?

ambicapter 1y ago
Do people where you work spend time reviewing draft PRs? I wouldn’t do that unless asked to by the author.
- drawnwren 1y ago
  It’s hard enough for me to get time to review actual PRs, who are these engineers trawling through the drafts?
lukehoban 1y ago
I do agree there is a balance here, and that the ideal point in the spectrum is likely in between the two product experiences that are currently being offered here. There are a lot of benefits to using PRs for the review and iteration - familiar diff UX, great comment/review feedback mechanisms, ability to run CI, visibility and auth tracked natively within GitHub, etc. But Draft PRs are also a little too visible by default in GitHub today, and there are times when you want a shareable PR link that isn't showing up by default on the Pull Requests list in GitHub for your repo. (I frankly want this even for human-authored Draft PRs, but its even more compelling for agent authored PRs). We are looking into paths where we can support this more personal/private kind of PR, which would provide the foundation within GitHub to support the best of both worlds here.

polskibus 1y ago

What is your team’s take on the copyright for commits generated by ai agent ? Would the copyright protect it? Current US stance seems to be: https://www.copyright.gov/newsnet/2025/1060.html https://www.copyright.gov/newsnet/2025/1060.html “It concludes that the outputs of generative AI can be protected by copyright only where a human author has determined sufficient expressive elements”. If entire commit is generated by AI then it is obvious what created it - it’s AI. Such commit might not be covered by the law. Is this something your team has already analysed?

IanCal 1y ago
>If entire commit is generated by AI then it is obvious what created it - it’s AI. Whether it's committed or not is irrelevant to the conclusion there, the question is what was the input.
- chiph 1y ago
  For something like a compiler where the output is mostly deterministic[0] I agree. For an AI that was trained on an unknown corpus, and that corpus changes over time, the output is much less deterministic and I would say you lose the human element needed of copyright claims. If it can be shown that for the same prompt, run through the AI several times over perhaps a year, results in the same output - then I will change my mind. Or if the AI achieves personhood. [0] Allowances for register & loop optimization, etc.
rustc 1y ago
> “It concludes that the outputs of generative AI can be protected by copyright only where a human author has determined sufficient expressive elements” How would that work if it's a patch to a project with a copyleft license like GPL which requires all derivate work to be licensed the same?
- anticensor 1y ago
  GPL is a copyright licence, not a ToS.