4 ms·

> we've merged almost 1,000 pull requests contributed by Copilot I'm curious to know how many Copilot PRs were not merged and/or required human take-overs.

sethammons 1y ago
textbook survivorship bias https://en.wikipedia.org/wiki/Survivorship_bias https://en.wikipedia.org/wiki/Survivorship_bias every bullet hole in that plane is the 1k PRs contributed by copilot. The missing dots, and whole missing planes, are unaccounted for. Ie, "ai ruined my morning"
- n2d4 1y ago
  It's not survivorship bias. Survivorship bias would be if you made any conclusions from the 1000 merged PRs (eg. "90% of all merged PRs did not get reverted"). But simply stating the number of PRs is not that.
  - tines 1y ago
    As with all good marketing, the conclusions omitted and implied, no?
    - n2d4 1y ago
      The implied conclusion ("Copilot made 1000 changes to the codebase") is also not survivorship bias. By that logic, literally every statement would be survivorship bias.
      - tines 1y ago
        That’s not the implied conclusion my guy. That’s the statement.
        n2d4 1y ago
        Then what do you claim the implied conclusion is?
        Jenk 1y ago
        That the number of successful (as in, merged and works) contributions are greater than those that did not.
  - krainboltgreene 1y ago
    Given that Github is continuing with the product and marketing to us it feels sufficient to count that as a conclusion.
- MoreQARespect 1y ago
  If they measured that too it would make it harder to justify a MSFT P/E ratio of 29.6.
literalAardvark 1y ago
"We need to get 1000 PRs merged from Copilot" "But that'll take more time" "Doesn't matter"
- Cthulhu_ 1y ago
  I feel the same about automated dependency updates, but if your tests and verifications are good, these become trivial.
  - skydhash 1y ago
    Sometimes there are some paradigms shift in the dependency that get past the current tests you have. So it’s always good to read the changelog and plan the update accordingly.
  - no_wizard 1y ago
    Strong automated tests and verifications seem to be nearly as rare as unicorns, at least if you take most of developers feelings on this. It seems places don't prioritize it, so you don't see it very often. Some developers are outright dismissive of the practice. Unfortunately, AI won't seemingly help with that
- worldsayshi 1y ago
  I do agree that some scepticism is due here but how can we tell if we're treading into "moving the goal posts" territory?
  - overfeed 1y ago
    I'd love to know where you think the starting position of the goal posts was. Everyone who has used AI coding tools interactively or as agents knows they're unpredictably hit or miss. The old, non-agent Copilot has a dashboard that shows org-wide rejection rates for for paying customers. I'm curious to learn what the equivalent rejection-rate for the agent is for the people who make the thing.
  - taurath 1y ago
philipwhiuk 1y ago
I'm curious how many were much more than Dependabot changes.
xeromal 1y ago
I see number of PRs as modern LOC, something that doesn't tell me anything about quality.

sethammons 1y ago

textbook survivorship bias https://en.wikipedia.org/wiki/Survivorship_bias https://en.wikipedia.org/wiki/Survivorship_bias every bullet hole in that plane is the 1k PRs contributed by copilot. The missing dots, and whole missing planes, are unaccounted for. Ie, "ai ruined my morning"

n2d4 1y ago
It's not survivorship bias. Survivorship bias would be if you made any conclusions from the 1000 merged PRs (eg. "90% of all merged PRs did not get reverted"). But simply stating the number of PRs is not that.
- tines 1y ago
  As with all good marketing, the conclusions omitted and implied, no?
  - n2d4 1y ago
    The implied conclusion ("Copilot made 1000 changes to the codebase") is also not survivorship bias. By that logic, literally every statement would be survivorship bias.
    - tines 1y ago
      That’s not the implied conclusion my guy. That’s the statement.
      - n2d4 1y ago
        Then what do you claim the implied conclusion is?
        Jenk 1y ago
        That the number of successful (as in, merged and works) contributions are greater than those that did not.
- krainboltgreene 1y ago
  Given that Github is continuing with the product and marketing to us it feels sufficient to count that as a conclusion.
MoreQARespect 1y ago
If they measured that too it would make it harder to justify a MSFT P/E ratio of 29.6.

literalAardvark 1y ago

"We need to get 1000 PRs merged from Copilot" "But that'll take more time" "Doesn't matter"

Cthulhu_ 1y ago
I feel the same about automated dependency updates, but if your tests and verifications are good, these become trivial.
- skydhash 1y ago
  Sometimes there are some paradigms shift in the dependency that get past the current tests you have. So it’s always good to read the changelog and plan the update accordingly.
- no_wizard 1y ago
  Strong automated tests and verifications seem to be nearly as rare as unicorns, at least if you take most of developers feelings on this. It seems places don't prioritize it, so you don't see it very often. Some developers are outright dismissive of the practice. Unfortunately, AI won't seemingly help with that
worldsayshi 1y ago
I do agree that some scepticism is due here but how can we tell if we're treading into "moving the goal posts" territory?
- overfeed 1y ago
  I'd love to know where you think the starting position of the goal posts was. Everyone who has used AI coding tools interactively or as agents knows they're unpredictably hit or miss. The old, non-agent Copilot has a dashboard that shows org-wide rejection rates for for paying customers. I'm curious to learn what the equivalent rejection-rate for the agent is for the people who make the thing.
- taurath 1y ago

philipwhiuk 1y ago

I'm curious how many were much more than Dependabot changes.

xeromal 1y ago

I see number of PRs as modern LOC, something that doesn't tell me anything about quality.