6 ms·

There is no way 33% of code is written with Copilot. I want to see the data backing that claim up.

rvense 5y ago
Maybe it's bots doing it? Someone might be trying to fill up Github with autocompleted code.
smoe 5y ago
That doesn't seem to be the claim. They say one third was written with AI help. Doesn't say how much of the new code was generated by Copilot, just that it was used in the process of writing it. edit: I read the article wrong, skimming it. I understood it as 30% of new code written by GitHub engineers uses AI help, which sounds much more reasonable.
- minimaxir 5y ago
  Even if you do the "% of users using Copilot" approach, Copilot was limited to both Visual Studio Code (until today) and an invitation-only program. The fact that the combo reached 30% of newly generated code doesn't make sense.
  - uberman 5y ago
    Ya, it is clearly a lie or the journalist was clueless
  - ghaff 5y ago
    The specific claim in the video was % of code being written by early adopters. I can't tell from the video how "early adopters" was defined or how they measured % of code so it's ambiguous. But story should have made it clear this isn't a percentage of all new code.
    - minimaxir 5y ago
      I hadn't seen the video quote until it was posted above: I assume "early adopters" is those who were accepted into the program and installed it, but yes it's still ambigious.
- lhorie 5y ago
  I mean, Google search uses AI, and developers use Google to find things, so technically you could say the vast majority of code is written with "AI help", but that's not exactly a noteworthy observation to make...
- Bancakes 5y ago
  Yep! I was just watching how clang-tidy can now automatically apply code fixes in realtime. Is this AI? https://www.youtube.com/watch?v=EfYMKOMc738 https://www.youtube.com/watch?v=EfYMKOMc738
sammorrowdrums 5y ago
I work for GitHub (although writing in an entirely personal capacity) and the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help. I don’t know about how the data was gathered, I was not involved, but the segment is 4 minutes in so you can see what the actual announcement claims here: https://youtu.be/etMvd9IKPH4 https://youtu.be/etMvd9IKPH4 I have used copilot and watched a colleague writing comments and frequently getting great go code suggestions without loads of edits. I personally am sincerely very impressed.
- gregshap 5y ago
  Thanks for sharing. So in other words it's the journalism that is wrong.
- uberman 5y ago
  So 30% of code written by users with the extension installed was written with some extension assistance? How exactly does that translate into the claim being made? Is github asserting the 100% of users have the extension installed? My guess is that some small number of trial users did their homework with it then ran to stack overflow when it did something other than what they hoped.
- jstx1 5y ago
  > the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help. The gap between this statement and what's in the OP title and text is too big. It's just false misleading bad journalism. I need to write an article titled "People who install Copilot only use it for 30% of the code".
- lsaferite 5y ago
  That video says: "Since we started, 30% of all code written by early adopters come from copilot suggestions." That is a far different headline than the one on this post. WTH is wrong with journalists now days? FFS.
- toastal 5y ago
superfrank 5y ago
Maybe they're just measuring lines committed (total guess)? If you count white space, comments, variable and function declarations, and boiler plate code like creating loops, then maybe I could see how they got there. Kind of related, but if I think about autocomplete, it probably writes about 10%-20% of the individual characters in my code. While it is totally misleading, it wouldn't be factually incorrect to say that autocomplete writes X percentage of my code. I wouldn't be surprised if GitHub is using some "misleading, but technically true" metric for this.
rattlesnakedave 5y ago
+1. >Nearly a third of new code on GitHub is written with AI help May be true if you count VSCode's IntelliCode, JetBrains IntelliSense, CoPilot, and other similar tools together. But no way copilot has that much marketshare yet.
deleted 5y ago
[deleted]
martinwoodward 5y ago
Martin from GitHub here. While Copilot is good it's not quite up to those numbers yet I'm afraid. It’s early days, but we’re already seeing that for some languages when people have it installed, we see about 30% of newly written code being suggested by Copilot. Vary’s between languages and the numbers of users in the technical preview is a LOT less than the 73m folks we have on GitHub, so a long way to go before it makes up a third of everything..
- martinwoodward 5y ago
  For more information, the interview with one of the creators of Copilot has been posted here https://www.githubuniverse.com/2021/session/707714/inside-github-how-copilot-took-flight https://www.githubuniverse.com/2021/session/707714/inside-gi...

Maybe it's bots doing it? Someone might be trying to fill up Github with autocompleted code.

smoe 5y ago

That doesn't seem to be the claim. They say one third was written with AI help. Doesn't say how much of the new code was generated by Copilot, just that it was used in the process of writing it. edit: I read the article wrong, skimming it. I understood it as 30% of new code written by GitHub engineers uses AI help, which sounds much more reasonable.

minimaxir 5y ago
Even if you do the "% of users using Copilot" approach, Copilot was limited to both Visual Studio Code (until today) and an invitation-only program. The fact that the combo reached 30% of newly generated code doesn't make sense.
- uberman 5y ago
  Ya, it is clearly a lie or the journalist was clueless
- ghaff 5y ago
  The specific claim in the video was % of code being written by early adopters. I can't tell from the video how "early adopters" was defined or how they measured % of code so it's ambiguous. But story should have made it clear this isn't a percentage of all new code.
  - minimaxir 5y ago
    I hadn't seen the video quote until it was posted above: I assume "early adopters" is those who were accepted into the program and installed it, but yes it's still ambigious.
lhorie 5y ago
I mean, Google search uses AI, and developers use Google to find things, so technically you could say the vast majority of code is written with "AI help", but that's not exactly a noteworthy observation to make...
Bancakes 5y ago
Yep! I was just watching how clang-tidy can now automatically apply code fixes in realtime. Is this AI? https://www.youtube.com/watch?v=EfYMKOMc738 https://www.youtube.com/watch?v=EfYMKOMc738

sammorrowdrums 5y ago

I work for GitHub (although writing in an entirely personal capacity) and the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help. I don’t know about how the data was gathered, I was not involved, but the segment is 4 minutes in so you can see what the actual announcement claims here: https://youtu.be/etMvd9IKPH4 https://youtu.be/etMvd9IKPH4 I have used copilot and watched a colleague writing comments and frequently getting great go code suggestions without loads of edits. I personally am sincerely very impressed.

gregshap 5y ago
Thanks for sharing. So in other words it's the journalism that is wrong.
uberman 5y ago
So 30% of code written by users with the extension installed was written with some extension assistance? How exactly does that translate into the claim being made? Is github asserting the 100% of users have the extension installed? My guess is that some small number of trial users did their homework with it then ran to stack overflow when it did something other than what they hoped.
jstx1 5y ago
> the segment in the GitHub Universe Keynote today said that 30% of Python code (written by users with copilot extension active) was written with copilot help. The gap between this statement and what's in the OP title and text is too big. It's just false misleading bad journalism. I need to write an article titled "People who install Copilot only use it for 30% of the code".
lsaferite 5y ago
That video says: "Since we started, 30% of all code written by early adopters come from copilot suggestions." That is a far different headline than the one on this post. WTH is wrong with journalists now days? FFS.
toastal 5y ago

superfrank 5y ago

Maybe they're just measuring lines committed (total guess)? If you count white space, comments, variable and function declarations, and boiler plate code like creating loops, then maybe I could see how they got there. Kind of related, but if I think about autocomplete, it probably writes about 10%-20% of the individual characters in my code. While it is totally misleading, it wouldn't be factually incorrect to say that autocomplete writes X percentage of my code. I wouldn't be surprised if GitHub is using some "misleading, but technically true" metric for this.

rattlesnakedave 5y ago

+1. >Nearly a third of new code on GitHub is written with AI help May be true if you count VSCode's IntelliCode, JetBrains IntelliSense, CoPilot, and other similar tools together. But no way copilot has that much marketshare yet.

deleted 5y ago

[deleted]

martinwoodward 5y ago

Martin from GitHub here. While Copilot is good it's not quite up to those numbers yet I'm afraid. It’s early days, but we’re already seeing that for some languages when people have it installed, we see about 30% of newly written code being suggested by Copilot. Vary’s between languages and the numbers of users in the technical preview is a LOT less than the 73m folks we have on GitHub, so a long way to go before it makes up a third of everything..

martinwoodward 5y ago
For more information, the interview with one of the creators of Copilot has been posted here https://www.githubuniverse.com/2021/session/707714/inside-github-how-copilot-took-flight https://www.githubuniverse.com/2021/session/707714/inside-gi...