6 ms·

Nearly a third of new code on GitHub is written with AI help

24 points by kiwih 5y ago

kiwih 5y ago
> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot.
- progre 5y ago
  I wonder how they distinguish between Copilot code and code manually copied from Stack Overflow
  - rightbyte 5y ago
    VS Code enable mandatory spyware if you try to use the Copilot extension. I guess they track it by that and the git extension somehow? Or maybe they just count what share of commiters have the Coopilot extension installed ...
  - black_puppydog 5y ago
    And code that just is kinda vanilla and so would be something that co-pilot would actually generate...
- deleted 5y ago
  [deleted]
JohnWhigham 5y ago
Like any algorithm, it is dependent on the quality of its training data. In a study, a group of academics from New York University found 40% of the code produced by Copilot had cybersecurity flaws. Yeah let me use this tool where 40% of its output is garbage. Toss this in the ever-growing AI Winter bin.
- onion2k 5y ago
  Given the fact that Copilot is trained on a corpus of real code, is it not reasonable to assume that the number of security flaws just reflects how insecure most public code is? If human-written code has about the same 40% suffering from security issues then then using Copilot is certainly no worse. It definitely doesn't seem fair to invalidate the idea of AI-assisted code generation without testing that.
  - Bancakes 5y ago
    Think of it this way, how many security-hardened open-source monolithic kernels are there to learn from?
  - yqx 5y ago
    > is it not reasonable to assume that the number of security flaws just reflects how insecure most public code is? It sounds to me like that's not an inference that can easily be drawn. Copilot was trained on predicting code, it doesn't understand the code it produces syntactically. Security issues can be highly context dependent. For example, in most cases it's fine to log a variable, but when it happens to contain a password, it's a security issue. This is a flawed example as the algorithm may be able to learn that variables with names or contexts suggesting that they're secrets should not be logged, but I can imagine much more subtle issues can crop up.
fourseventy 5y ago
There is no way 33% of code is written with Copilot. I want to see the data backing that claim up.
- rvense 5y ago
  Maybe it's bots doing it? Someone might be trying to fill up Github with autocompleted code.
- smoe 5y ago
  That doesn't seem to be the claim. They say one third was written with AI help. Doesn't say how much of the new code was generated by Copilot, just that it was used in the process of writing it. edit: I read the article wrong, skimming it. I understood it as 30% of new code written by GitHub engineers uses AI help, which sounds much more reasonable.
  - minimaxir 5y ago
    Even if you do the "% of users using Copilot" approach, Copilot was limited to both Visual Studio Code (until today) and an invitation-only program. The fact that the combo reached 30% of newly generated code doesn't make sense.
    - uberman 5y ago
      Ya, it is clearly a lie or the journalist was clueless
    - ghaff 5y ago
      The specific claim in the video was % of code being written by early adopters. I can't tell from the video how "early adopters" was defined or how they measured % of code so it's ambiguous. But story should have made it clear this isn't a percentage of all new code.
      - minimaxir 5y ago
        I hadn't seen the video quote until it was posted above: I assume "early adopters" is those who were accepted into the program and installed it, but yes it's still ambigious.
arpa 5y ago
I call bullshit.
- misnome 5y ago
  I believe it. Insofar as 0%-1% is a value compatible with “as much as 30%” e.g. <30%
  - ncmncm 5y ago
    As much as means >=.
jsnell 5y ago
This number just doesn't feel remotely plausible. Hasn't Copilot supported just one editor and five languages until now? And it's advertised as being available to a limited number of testers, and gated behind a waitlist. Oh, and according to the article half the people who were granted access to Copilot have stopped using it. There is no way to make these numbers work unless we assume that the people using Copilot are committing orders of magnitude more code. And if they are, how likely is it that they're really legit users?
Sebb767 5y ago
> 40% of the code produced by Copilot had cybersecurity flaws. Off-topic, but it's quite ironic to me that "cyber" had such a resurgence. In ~2010 to 2015, most people in my circle considered cyber to be a washed out joke, and yet, in the last few years, it gained massive popularity again. It's still hard for me to take anything "cyer-" seriously, though.
- formerly_proven 5y ago
  cyberx is still a strong signal that something is corpo bullshit.
humanistbot 5y ago
That could be correct if by AI they mean Googling to get to Stack Overflow. Search used to be considered a core AI problem!
- je42 5y ago
  But then it would be a bit low wouldn't it ? :)
dcchambers 5y ago
> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot. reply (Emphasis mine) I wonder if what this really means is ~30% of code written in codespaces or the web/cloud version of vscode you can use within a repo is written with co-pilot. Which I find much easier to believe than 30% of all code uploaded to github was written with AI assistance (how would they ever be able to measure that).
lmkg 5y ago
If that number is true, it's probably being thrown off by a research group somewhere studying Copilot by having it generate gigabytes of code.
jerrygoyal 5y ago
> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot. where is the reference link?
wiz21c 5y ago
Maybe, all in all, we spend 33% of our time writing very dumb code that can easily written by some AI ? (I must recognize that a good chunk of the code I was writing as a professipnal dev was boiler plate stuff...)
tigranbs 5y ago
Most of AI tools like TabNine, Copilot or Kite are quite resource intensive. For example I have TabNine, but it is active ONLY when my laptop is plugged in, otherwise it drains batteries pretty quickly. That's why I don't believe that more than 30% of developers are OK to heat up their laptops for getting AI suggestions. It more looks like a marketing title
- minimaxir 5y ago
  Copilot is done remotely via an API so no extra resource drain on the computer.
deleted 5y ago
[deleted]
willvarfar 5y ago
The only plausible way to interpret that statistic that I can think up with is if it is saying: a third of all code committed by active copilot users has been accepted copilot suggestions…
rhacker 5y ago
I find it weird that people keep slamming copilot. IDEA (and other IDEs) have for years auto complete for for loops, functions and a few other things. The most this tool does is put in a couple of extra like 'user.someInterestingProperty' based on your context inside those same for loop shells. It's just that the for-loop shell is now, for better or worse, similar to what people write. I've seen some of the suggestions in my workflow at this point and it's much slower than IDE assist stuff, and generally barely helpful. So I don't see any reason to slam the crap out of it. I wonder if it's a mix of people feeling like they A) will lose their job, B) make the software around the world worse (it's already horrendously bad) or C) hate Microsoft? I can't really tell why anyone would honestly hate this tool other than a lot of pent up whateverism.
orph 5y ago
The actual quote is: > For some languages, we already see about 30% of newly written code being suggested by Copilot. It writes about 30% of Python and 20-25% for other languages.
siva7 5y ago
Interesting, i am since 2008 on Github and still haven’t been offered to join the Product nor do i know of many professionals who have been.
AndyMcConachie 5y ago
I've always been somewhat suspicious of this AI generated code stuff. I get what's happening, but I'm not convinced it's all that fundamentally different than what compilers have been doing for decades. Isn't a compiler a code generator? And isn't the stuff we humans write simply instructions to a compiler on what code to generate? Compilers have tons of intelligence in them. At least when I was a CS grad (an ancient time ago) complex sets of heuristics were considered AI. And compilers use complex sets of heuristics to determine things like optimizations. So I guess what I'm saying is that this news feels over hyped to me.
justsomeuser 5y ago
Well, Google is an AI.

24 points by kiwih 5y ago

> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot.

progre 5y ago
I wonder how they distinguish between Copilot code and code manually copied from Stack Overflow
- rightbyte 5y ago
  VS Code enable mandatory spyware if you try to use the Copilot extension. I guess they track it by that and the git extension somehow? Or maybe they just count what share of commiters have the Coopilot extension installed ...
- black_puppydog 5y ago
  And code that just is kinda vanilla and so would be something that co-pilot would actually generate...
deleted 5y ago
[deleted]

JohnWhigham 5y ago

Like any algorithm, it is dependent on the quality of its training data. In a study, a group of academics from New York University found 40% of the code produced by Copilot had cybersecurity flaws. Yeah let me use this tool where 40% of its output is garbage. Toss this in the ever-growing AI Winter bin.

onion2k 5y ago
Given the fact that Copilot is trained on a corpus of real code, is it not reasonable to assume that the number of security flaws just reflects how insecure most public code is? If human-written code has about the same 40% suffering from security issues then then using Copilot is certainly no worse. It definitely doesn't seem fair to invalidate the idea of AI-assisted code generation without testing that.
- Bancakes 5y ago
  Think of it this way, how many security-hardened open-source monolithic kernels are there to learn from?
- yqx 5y ago
  > is it not reasonable to assume that the number of security flaws just reflects how insecure most public code is? It sounds to me like that's not an inference that can easily be drawn. Copilot was trained on predicting code, it doesn't understand the code it produces syntactically. Security issues can be highly context dependent. For example, in most cases it's fine to log a variable, but when it happens to contain a password, it's a security issue. This is a flawed example as the algorithm may be able to learn that variables with names or contexts suggesting that they're secrets should not be logged, but I can imagine much more subtle issues can crop up.

fourseventy 5y ago

There is no way 33% of code is written with Copilot. I want to see the data backing that claim up.

rvense 5y ago
Maybe it's bots doing it? Someone might be trying to fill up Github with autocompleted code.
smoe 5y ago
That doesn't seem to be the claim. They say one third was written with AI help. Doesn't say how much of the new code was generated by Copilot, just that it was used in the process of writing it. edit: I read the article wrong, skimming it. I understood it as 30% of new code written by GitHub engineers uses AI help, which sounds much more reasonable.
- minimaxir 5y ago
  Even if you do the "% of users using Copilot" approach, Copilot was limited to both Visual Studio Code (until today) and an invitation-only program. The fact that the combo reached 30% of newly generated code doesn't make sense.
  - uberman 5y ago
    Ya, it is clearly a lie or the journalist was clueless
  - ghaff 5y ago
    The specific claim in the video was % of code being written by early adopters. I can't tell from the video how "early adopters" was defined or how they measured % of code so it's ambiguous. But story should have made it clear this isn't a percentage of all new code.
    - minimaxir 5y ago
      I hadn't seen the video quote until it was posted above: I assume "early adopters" is those who were accepted into the program and installed it, but yes it's still ambigious.

arpa 5y ago

I call bullshit.

misnome 5y ago
I believe it. Insofar as 0%-1% is a value compatible with “as much as 30%” e.g. <30%
- ncmncm 5y ago
  As much as means >=.

jsnell 5y ago

This number just doesn't feel remotely plausible. Hasn't Copilot supported just one editor and five languages until now? And it's advertised as being available to a limited number of testers, and gated behind a waitlist. Oh, and according to the article half the people who were granted access to Copilot have stopped using it. There is no way to make these numbers work unless we assume that the people using Copilot are committing orders of magnitude more code. And if they are, how likely is it that they're really legit users?

Sebb767 5y ago

> 40% of the code produced by Copilot had cybersecurity flaws. Off-topic, but it's quite ironic to me that "cyber" had such a resurgence. In ~2010 to 2015, most people in my circle considered cyber to be a washed out joke, and yet, in the last few years, it gained massive popularity again. It's still hard for me to take anything "cyer-" seriously, though.

formerly_proven 5y ago
cyberx is still a strong signal that something is corpo bullshit.

humanistbot 5y ago

That could be correct if by AI they mean Googling to get to Stack Overflow. Search used to be considered a core AI problem!

je42 5y ago
But then it would be a bit low wouldn't it ? :)

dcchambers 5y ago

> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot. reply (Emphasis mine) I wonder if what this really means is ~30% of code written in codespaces or the web/cloud version of vscode you can use within a repo is written with co-pilot. Which I find much easier to believe than 30% of all code uploaded to github was written with AI assistance (how would they ever be able to measure that).

lmkg 5y ago

If that number is true, it's probably being thrown off by a research group somewhere studying Copilot by having it generate gigabytes of code.

jerrygoyal 5y ago

> The open-source software developer GitHub says as much as 30% of newly written code on its network is being done with the help of the company's AI programming tool Copilot. where is the reference link?

wiz21c 5y ago

Maybe, all in all, we spend 33% of our time writing very dumb code that can easily written by some AI ? (I must recognize that a good chunk of the code I was writing as a professipnal dev was boiler plate stuff...)

tigranbs 5y ago

Most of AI tools like TabNine, Copilot or Kite are quite resource intensive. For example I have TabNine, but it is active ONLY when my laptop is plugged in, otherwise it drains batteries pretty quickly. That's why I don't believe that more than 30% of developers are OK to heat up their laptops for getting AI suggestions. It more looks like a marketing title

minimaxir 5y ago
Copilot is done remotely via an API so no extra resource drain on the computer.

deleted 5y ago

[deleted]

willvarfar 5y ago

The only plausible way to interpret that statistic that I can think up with is if it is saying: a third of all code committed by active copilot users has been accepted copilot suggestions…

rhacker 5y ago

I find it weird that people keep slamming copilot. IDEA (and other IDEs) have for years auto complete for for loops, functions and a few other things. The most this tool does is put in a couple of extra like 'user.someInterestingProperty' based on your context inside those same for loop shells. It's just that the for-loop shell is now, for better or worse, similar to what people write. I've seen some of the suggestions in my workflow at this point and it's much slower than IDE assist stuff, and generally barely helpful. So I don't see any reason to slam the crap out of it. I wonder if it's a mix of people feeling like they A) will lose their job, B) make the software around the world worse (it's already horrendously bad) or C) hate Microsoft? I can't really tell why anyone would honestly hate this tool other than a lot of pent up whateverism.

orph 5y ago

The actual quote is: > For some languages, we already see about 30% of newly written code being suggested by Copilot. It writes about 30% of Python and 20-25% for other languages.

siva7 5y ago

Interesting, i am since 2008 on Github and still haven’t been offered to join the Product nor do i know of many professionals who have been.

AndyMcConachie 5y ago

I've always been somewhat suspicious of this AI generated code stuff. I get what's happening, but I'm not convinced it's all that fundamentally different than what compilers have been doing for decades. Isn't a compiler a code generator? And isn't the stuff we humans write simply instructions to a compiler on what code to generate? Compilers have tons of intelligence in them. At least when I was a CS grad (an ancient time ago) complex sets of heuristics were considered AI. And compilers use complex sets of heuristics to determine things like optimizations. So I guess what I'm saying is that this news feels over hyped to me.

justsomeuser 5y ago

Well, Google is an AI.