16 ms·

Risk Assessment of GitHub Copilot

363 points by g_p 5y ago

arkitaip 5y ago
≥ These three example pieces of flawed code did not require any cajoling; Copilot was happy to write them from straightforward requests for functional code. The inevitable conclusion is that Copilot can and will write security vulnerabilities on a regular basis, especially in memory-unsafe languages. If people can copy paste the most insecure code from Stack Overflow or random tutorials, they will absolutely use Copilot to "write" code and it will be become the default, especially since it's so incredibly easy to use. Also, it's just the first generation tool if it's kind, imagine what similar products will accomplish in 20 years.
- foepys 5y ago
  > imagine what similar products will accomplish in 20 years. People said the same thing about UML and similar tools so I'm not holding my breath.
  - arkitaip 5y ago
    Maybe you are right but where UML created busy work, Copilot will literally do your work for you. I can even imagine a future where management makes it policy to Copilot first to save time and money.
    - swiftcoder 5y ago
      > imagine a future where management even makes it policy Because management policies correlate well with engineering excellence, right?
    - Zababa 5y ago
      Reviewing code written by Copilot may be longer than writing the code in the first place.
    - wpietri 5y ago
      Most of my work isn't copy-pasting snippets. It isn't even typing code. It's understanding user needs and the existing system, and then figuring out how to making things better for the user with also improving the system. So this does not do my work for me. I can also imagine clueless bosses mandating Copilot use and that's what scares me. The real costs of most code aren't in the first writing. They're in the long-term maintenance. Copilot does not and cannot understand the whole system, or what makes for maintainability down the road. So it can't make that better, and will likely make worse. In the same way that code generation tools and code wizards made things worse.
- deleted 5y ago
  [deleted]
- whoknew1122 5y ago
  With the pace of technological innovation, I'm honestly not sure what a similar product will be able to accomplish in 20 years. It'll be crazy for sure. But I'm worried about today. This is a product by a well-known company (GitHub) which is owned by an even more well-known company (Microsoft). GitHub is going to be trusted a lot more than a random poster on Stack Overflow or someone's blog online. And GitHub is explicitly telling new coders to use Copilot to learn a new language: > Whether you’re working in a new language or framework, or just learning to code, GitHub Copilot can help you find your way. Tackle a bug, or learn how to use a new framework without spending most of your time spelunking through the docs or searching the web. This is what differentiates Copilot from Stack Overflow or random tutorials. GitHub has a brand that's trusted more than random content creators on the internet. And it's telling new coders to use Copilot to learn things and not check elsewhere. That's a problem. Doesn't matter what generation of the program it is. It creates unsafe code after using its brand reputation and recognition to convince new coders to not check elsewhere.
  - azangru 5y ago
    > GitHub is going to be trusted a lot more > GitHub has a brand that's trusted more Consider Google Translate, right? Google is a well-known brand that is trusted (outside of a relatively small group of people that doesn't trust Google on principle). Yet every professional translator knows that the text produced by Google Translate is a result of machine translation, Google or no Google. They may marvel at the occasional accuracy, yet expect serious blunders in the text, and would therefore not just trust that translation before submitting it to their clients. They will check. Or at least they should. Same with programmers.
    - mipmap04 5y ago
      I think this is an important way we need to frame the use of these tools for junior developers. I'd advise that anyone who is recommending this product to their team also take the time to give this analogy - maybe even going so far as to require explicit comments that notifies reviewers when code was provided by Copilot and similar services.
    - 5y ago
- piokoch 5y ago
  I think there is some difference. You don't come across some piece of code by chance, you were actively looking for it, probably there were multiple blogs, SO entries with needed information, one of those sources has to be chosen. You know that this is some random blog post or SO answer given by someone fresh. Copilot is something different. Code is suggested automatically and, what's the most important, suggested by the authority - hey, this is GitHub, huge project, largest code repo on the planet, owned by Microsoft, one of the most successful company ever. Why should you not trust the code they are suggesting you? And that's for starters before malicious parties start creating intentionally broken code only to hack system built with it. Greedy lawyers who will chase some innocent code snippet asking to pay for using it, etc.
tifadg1 5y ago
copilot will be interesting in 10-20 years. Now it's an early stage ml driven experiment in a field that hasn't advanced in forever - the strides it'll make will be very gradual, incremental and filled with mistakes along the way.
- bryanrasmussen 5y ago
  Luckily global warming should be killing me about the same time machine learning is taking my job. That's pretty good planning right there.
  - tifadg1 5y ago
    so it all works out in the end!
  - speedgoose 5y ago
    You should consider to immigrate to a colder place.
    - seba_dos1 5y ago
      Just like Richard Branson, Elon Musk, Jeff Bezos do...
    - jazzyjackson 5y ago
      it's not the heat that gets you, it's the water wars
    - bryanrasmussen 5y ago
      I'm in Denmark. I guess I could move up to Sweden, but I'm not sure if they want me because, after all, I am from Denmark.
emerongi 5y ago
Copilot doesn't seem like the right word. Maybe first year college student with no previous programming experience? Then it would be clear what level of help you are actually getting. Impressive, for sure. Unclear whether it's a net-positive tool, though.
- bla3 5y ago
  GitHub Excited Intern
- baud147258 5y ago
  fuzzy matching with publicly available code on Github
- Hamuko 5y ago
  GitHub Student Driver
- jsjohnst 5y ago
  > Copilot doesn't seem like the right word. Maybe “autopilot” then (as in the Tesla marketing term, not as in a real autopilot)? /s Both lull you with a false sense of security which will suddenly and unexpectedly cause you to pay dearly for using it. Interestingly both seem to also get a vaguely similar balance of critics, apologists, and advocates on here. Sample size of two here, but what is it with companies using %pilot to describe product features which are nothing like the actual appropriate use of the term?
prepend 5y ago
> me @'ing them several hundred times about ICE This is such curious behavior to me. Does someone really @ a corporation hundreds of times about anything? Does this have any effect? Should it? It makes me doubt the rationality of the author’s post if ve truly did this. Although I suppose maybe their use of Twitter is just completely different from anything I understand.
- smitop 5y ago
  You can use some Twitter search filters to see all of them: https://twitter.com/search?q=from%3A0xabad1dea%20%40github%20ICE&src=typed_query&f=live https://twitter.com/search?q=from%3A0xabad1dea%20%40github%2...
- tylersmith 5y ago
  It's a way to feel better about yourself without changing any comfortable behaviors. The author still uses Github and the ICE tweets are public Hail Mary's.
- infinitezest 5y ago
  Mostly, this just seemed like a non sequitur to me. Right away, I get the sense that "oh, this author has a bone to pick with GitHub.". Even if it's unrelated to the crux of the piece, I already feel like I'm going to have to take what the author says with a grain of salt.
muglug 5y ago
I had not considered the proliferation of terrible open-source code on GitHub. I'd wager that the amount of code in public repositories from students learning to code may outweigh quality code in GitHub. I wonder if there was any sort of filter for Copilot's input — only repositories with more than a certain number of stars/forks, only repositories committed to recently etc.
- tovej 5y ago
  I wouldn't bet on it, they didn't filter by license after all.
ainar-g 5y ago
I'm pretty sure that this conclusion isn't new, but I've come to think that Copilot shouldn't be thought of as a better developer, but merely a quicker one. Obviously its code will be somewhat average, considering that it's been trained on code the only unifying characteristic of which is that it's public. Something like Copilot, but trained explicitly to analyse the code instead of writing it could be much more useful, imo. Basically a real-time code review tool. There are similar tools already, but I'm talking of something that is able to learn from the actual codebase being worked on, perhaps including the documentation, and giving on-the-go feedback.
- rob74 5y ago
  If you interviewed two developers, one who produces reasonably correct code in a given amount of time, and another one who produces code which is subtly incorrect most of the time, but much faster, which one would you hire? The problem with your proposal is that it's relatively easy to do what Copilot does at the moment using AI, i.e. guess what code you are looking for and find something that does (or says it does) more or less that. However, which codebase would you use to check against if the generated code is really correct? The same codebase that produced the more-or-less-correct code in the first place?
- jazzyjackson 5y ago
  I like this idea, given that it takes advantage of how git repos are made of bug-fixes. How many git diffs are out there that update a '=' in an if statement to '=='? so an AI copilot should be watching out for code I write that looks similar to code that was updated in another repo. It could even use the text from issues to synthesize a suggestion of why your code might cause problems!
eddieroger 5y ago
> Ultimately, a human being must take responsibility for every line of code that is committed. AI should not be used for "responsibility washing." That's the whole point, and the rest is moot because of it. If I chose to let Copilot write code for me, I am responsible for it's output, full stop. This is the same as if I let a more junior engineer submit code to prod, but there aren't blog posts about not letting them work or trusting them with code.
onionisafruit 5y ago
This articulates some of the concerns I had trying copilot. I noticed that I ended up assuming the code reviewer role when I was trying to write code. Context switching between writing and reviewing felt unnatural. I also think I am less likely to spot a bug than I am to avoid writing it in the first place. Taking the off-by-one error in the last example. I don't think I would have made that mistake, but if copilot had presented that code block, I probably wouldn't have noticed the error either.
generationP 5y ago
The Easter Date algorithm was probably someone implementing an algorithm from the Wikipedia ( https://en.wikipedia.org/wiki/Date_of_Easter#Anonymous_Gregorian_algorithm https://en.wikipedia.org/wiki/Date_of_Easter#Anonymous_Grego... ) without bothering to understand it (because honestly it's not a very interesting problem). No wonder it's uncommented. As long as the AI just regurgitates lines from repositories like a bad undergrad cheating on his homework, CS jobs should be safe. The fact that it has picked up the GPL might not mean that much -- it might appear in dual-licensed projects.
- prionassembly 5y ago
  This is something I don't get. You're supposed to be able to integrate BSD-licensed (or even Public Domain) code into GPL works, right? The fact that something shows up in GPL code means what exactly? This is like: there are scholarly books that quote extensive from original philosophers -- long, third-of-a-page quotations. Still I should be able to quote something in its original language (translations may be copyrighted) copying from the derived work. Copyrighted work is not supposed to be able to poison noncopyrighted work it originates from.
- skohan 5y ago
  > As long as the AI just regurgitates lines from repositories like a bad undergrad cheating on his homework, CS jobs should be safe. They said the same thing about Chess and Go.
  - generationP 5y ago
    The chess "job market" seems to be doing pretty well these days, at least based on recent publicity.
    - skohan 5y ago
      Watching people play chess has entertainment value. Coding is lucrative because there's a lot of demand and not that many people who can do it well. What if a computer can do it better than almost anyone?
eCa 5y ago
Re license washing: > This may or may not suddenly become a huge legal liability for anyone using Copilot. And if it doesn’t, can’t Copilot also be used to license wash for people not using it, but claiming to be?
tjpnz 5y ago
>As a code reviewer, I would want clear indications about which code is Copilot-generated. I would like to see this tracked behind the scenes. At any time I should be able to get Copilot to spit out a list of suggestions I've accepted. I should be able to generate this report for the lifetime of a project.
- jazzyjackson 5y ago
  It's kind of funny that this whole thing is built on top of git but uses none of the features, such as git blame. Instead of being an auto-complete, I would want co-pilot to be a git contributor, making pull requests that improve my code.
smusamashah 5y ago
Copilot doesn't seem any better then Tab Nine. Tab nine is GPT-2 based, works offline and can produce high quality boilerplate code based on previous lines. It can also generate whole methods which when work seems mind blowing but they are not always correct. Most suggestions are usually mind blowing anyway because previously we never had this kind of code completion. It feels like it wrote the whole line which you were going to write exactly as it should have. But that's all it does. And it seems like Copilot is the same but on much larger scale and online.
- tylersmith 5y ago
  I'm a long time TabNine user and fully agree. It's cool that Copilot is bringing the idea to more people but TabNine is the better product.
kozikow 5y ago
People look at the beta release of the software, and interpret flaws in an early version like a critical fundamental problems. I am pretty sure the new releases will contain features like better software license handling (e.g. 3 levels for types of licenses - permissive, copy-left, hardcore copy-left), trust score for snippets, possibly some validation of the code for some languages.
- tyingq 5y ago
  I'd be more optimistic if the beta were crafted with the idea that it might have issues. It so, it would likely have some way of gathering feedback on suggestions that was a little more nuanced than just accepted/rejected.
- skohan 5y ago
  This could actually be interesting. If it tuns out that copy-left based code completion is better than other options, it will create a strong incentive to spread it.
- only_as_i_fall 5y ago
  Does Microsoft actually have a repository of permissive licensed code though? My guess would be that some significant portion of github code published under a permissive license is actually licensed improperly. Working that out at scale seems intractable, but maybe the training set doesn't need to be as big as I'm assuming.
  - nullc 5y ago
    There is still something to be said for making a reasonable effort. There are no guarantees in the world.
- TeMPOraL 5y ago
  > People look at the beta release of the software, and interpret flaws in an early version like a critical fundamental problems. Maybe because they realize the flaws are fundamentally inherent in the very core of the product. They're using a GPT-3 derivative here. DNN models are not the right tool for this job.
_fat_santa 5y ago
I think the use case for Copilot is a bit misunderstood. The way I see it you have two types of code: 1. Smart Code: Code that you honestly have to think about while you're writing. You write this code slowly and carefully, to make sure that it does what you need it to 2. Dumb Code: This is trivial code, like adding a button to a screen. This is code you really don't have to think about, because you already know exactly how to implement it. At this point your biggest obstacle is how fast can your fingers type on a keyboard. For me Github Copilot is useless for "Smart Code" but a godsend when writing "Dumb Code". I want to focus more on writing and figuring out the "Smart Code", if I need to throw a form together in HTML or make a trivial helper function, I will gladly let AI take over and do that work for me.
- skohan 5y ago
  I think you just described which coders will be replaced by AI first
  - gonzo41 5y ago
    Or you're describing how you could put copilot in a box and make a really good low code gui programming solution where the complex stuff is good old complex code.
    - skohan 5y ago
      It sounds like you're talking about a way to cut my front-end hiring budget by 80%.
  - wongarsu 5y ago
    I don't know, we somehow manage to replace point-and-click GUIs for placing buttons (Windows Forms etc) with Frontend developers writing elaborate code to achieve the same result in HTML/CSS. Productivity is far from the first priority for frontend development.
    - skohan 5y ago
larodi 5y ago
Am I the only one thinking that GPT3 and CoPilot can actually work once trained on properly licensed and properly audited code? Well it will not be as ubiquitous as having all the github under your fingers, but perhaps is anyway better not to blindly cite the world's source code.
- qayxc 5y ago
  > Am I the only one thinking that GPT3 and CoPilot can actually work once trained on properly licensed and properly audited code? Sadly you aren't. The truth is, however, that models like GPT3 and its derivatives like Codex/CoPilot are by design incapable of ever achieving this. The only way to generate both correct and secure code is to use a combination of proper specs and theorem provers. Even then this won't help with non-functional requirements, such as performance or platform-dependent resource constraints. Generative models will always have the potential to yield broken code that doesn't do what you want or contains security flaws even if trained on "proper" code. If I have to audit the code that CoPilot generates and if the code is as obfuscated as the Easter example, it's probably less useful than it says on the label...
reilly3000 5y ago
I would hire copilot to write tests for me, that’s about it. Writing tests can be a drag. It’s really a low-risk proposition to have generated code attempt it. If it’s a usable test, maybe it will catch a bug. If not, then kill it and let it generate a few more. The expectation is entirely different than producing code. Code needs to be correct, secure, performant, and readable. Failure on any of those fronts can be expensive to disastrous. Nobody can reasonably expect a test suite to catch every bug, even if created by the smartest humans. If a copilot-created test does prevent a bug from shipping it provides immediate value. I could see it coming up with some whacky-but-useful test cases that a sane person might not consider. From a training perspective I would think that assertion descriptions contain more consistent lexical value than the average function signature. It seems like the ambitious data scientists, product marketers, and managers fell in love with a revolutionary idea about AI writing code, and neglected to consult the engineers they are trying to ‘augment’.
- squidsurfer 5y ago
  Many tests require an understanding of the intent of the code, especially the further from a pure unit test they stray. CoPilot has no understanding of anything and I anticipate will be useless in writing the tests that are actually hard to write.
- neatze 5y ago
  My time is spend mostly not on writing code, but thinking what program has to do, testing (including writing unit tests) and understanding errors, I always tell people half joking programming is not about writing code, but an ability to debug it; understanding requirements, errors and bugs is hard, writing code and fixing bugs is relatively easy, in general. Maybe Copilot 2 will do exactly this; it will generate tests based of half working code, run them and suggest improvements, that would increase productivity by like ~100%, but to me this sounds to good to be true.
- wpietri 5y ago
  Nope. That's like saying, "I might let a machine write the docs for me." Good tests are documentation that a computer can verify. Because they explain the meaning of parts of the system, they contain information not available in the code. If you try using ML for test generation, you'll have the same problem you do with GPT-3 prose: it might look plausible at first glance, but lacks coherent meaning. You'd also end up with one of the problems common in big test suites: poorly factored tests that end up being the sort of expressive duplication that is a giant drag on improving existing code. ML is nowhere near advanced enough to say, "Gosh, we're doing the same sort of test setup a bunch; let's extract that into a fixture, and then let's unify some fixtures into an ObjectMother. For people looking to get the computer to do the work of catching more things with less burdensome test writing, I suggest taking a look at things like Hypothesis: https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/
nojs 5y ago
I think the biggest problem copilot will have in practice gaining traction is that verifying correctness isn’t any faster than writing the code yourself in many cases. The Easter(y) function is a classic example - it would be way faster to write that than to try and verify that there’s no subtle bugs. Copilot is by design trying to give you something that _looks_ correct without caring whether it actually is - so it optimises for real-looking but subtly buggy code, which is the worst kind of broken code.
- mattwad 5y ago
  I've been using it and that's 100% correct. If it suggests more than a few lines, i might as well do it myself. However, it has been an awesome Intellisense tool for one-liners.. it can write out the rest of a comment or a simple map/filter method just fine. I don't think it will ever go further than that, but nor does it need to.
  - visarga 5y ago
    Be sure it doesn't generate any GPL licensed code down to the size of a single letter, to be safe. The reaction to GPL inspired snippet generation has been more fierce than I could have imagined, even though usable snippets are so short.
    - z3t4 5y ago
      You can't license a code snippet. You can only copyright/license a complete work, as in a complete app or library. Software patterns might work differently though!?
      - dec0dedab0de 5y ago
        Any portion of a work has a copyright the moment there is a tangible copy. There is a uniqueness requirement, but it has nothing to do with length. A unique one liner would have a copyright.
        5y ago
IfOnlyYouKnew 5y ago
It's interesting how the community that used to fervently argue that, say, the sampling of a few seconds of a musical composition is so obviously fair use doesn't extend the same attitude when it comes to snippets of source code. Indeed I don't remember any of these complaints ever being made about, for example, AI-generated music or images, even though they work exactly the same and were trained on datasets of copyrighted works, both commercial and CC-licensed. Compared to the manual sampling that DJs and other musicians do, the AI process almost certainly produces only fairly generic code snippets, since they always need a multitude of examples. Some loop-through-file python snippet is a legal risk but five seconds of a Disney song don't reach the level of creativity needed for copyright to matter? That seems strange to me...
- ShroudedNight 5y ago
  While I think there is some room for retrospective humility, It seems perfectly consistent to believe that, morally, de minimis use of copyrighted material should be considered fair-use while acknowledging that there are practical legal and professional risks for doing so (especially blindly), and that encouraging people to engage in that behaviour in an uncritical manner (especially in the context of engineering) is sociologically reckless.
- _ph_ 5y ago
  The problem is, we don't know what Copilot is going to do. Sometimes it reproduces entire files in verbatim. That is certainly a copyright violation. Sometimes it produces whole functions which are more or less taking verbatim from a copyrighted file. The user cannot see the level to which they are "verbatim" and there are also no clear legal guidelines to what is considered violating copyrights and what not.
deleted 5y ago
[deleted]
kiernanmcgowan 5y ago
Has anyone tried to generate private keys from copilot? GitHub suppressed these from search a while ago, but it would be an easy mistake to have included them in their training data.
user-the-name 5y ago
The easter code is regurgitated pretty much verbatim: https://stackoverflow.com/a/58726426 https://stackoverflow.com/a/58726426 It really is hard to swallow the argument that this is not license violation on a massive scale.
abunuwas 5y ago
I see copilot as something that could be useful for generating boilerplate or starting code. Often when using a framework or library there's some boilerplate code that has to be there for things to work fine, so instead of having to the documentation and find the relevant snippets, maybe copilot can do this for you (although they often come with CLIs that do this for you). Case in point is machine learning code, where you often find lots boilerplate setting up the models, the training, etc. Maybe copilot was developed by machine learning devs for machine learning devs?
brundolf 5y ago
> This is well-formed and even commented C that sure looks like it parses HTML, and the main function has some useful boilerplate around opening the file. However, the parsing is loaded with issues. This sounds like a great example of an interview question (where the person is asked to find and fix all of the issues in a chunk of bad code). Unintended usecase for Copilot?
oaiey 5y ago
I just read one of the comments below and thought: we should write sql as an AST and not as a string. That solves many problems, not only SQL injection but also the lousy understanding of joins etc.
paulsutter 5y ago
If anyone wants to write software to detect the original source of Copilot code, I can explain how to detect variable length unaligned matches using fast hash code lookups (unlimited size corpus)
mattwad 5y ago
People complaining about Copilot should just try it. All the concerns people are bringing up are correct but missing the point. Just think of it as a context-aware autocomplete that can complete more than just properties - it can finish out that one-line comment or map function too! It's really not that intrusive, nor is it going to replace anyone's coding job or even require less programmers. It'll just speed you up a bit, similar to auto-completion. I think at least 70% of the time I take its suggestions, which is good enough to keep using it.
- ghostly_s 5y ago
  Did you even read the article?
jedberg 5y ago
My biggest thing with Copilot is that it was trained on all public code on Github, which includes a lot of bad code that people just put up there (like my own code that I wrote a decade ago). As long as it is keeping track of when people do or do not accept their suggestions, it should get better over time. But in the meantime the best bet it is to treat it like a smart autocomplete, where you still have to at least check that it got it right. In the future maybe it will be smart enough to be treated like an intern -- trust that the code is right but still verify it yourself if the code is of any importance.
skc 5y ago
It's been fascinating reading all the responses to Copilot. It's pretty clear to me now that developer habits are far more diverse than I thought. For me, Copilot seems like it will be useful in the exact same way that StackOverflow is useful, as a means to pointing me in the right direction for code snippets or apis or techniques that I haven't and don't really want to memorize. For example, on my current side project I wanted to know how to create a "unique enough" UUID in pure JS. Copilot would hopefully save me a couple of google/stackoverflow searches as I can very quickly test what it suggests. I already rarely ever take SO answers as gospel so it's unlikely that I'd do the same with Copilot but I think it significantly increases the speed with which I achieve the same results.
mumblemumble 5y ago
The moon phase example is illustrative in another way. It's not technically possible to precisely calculate the moon's phase based on time alone. It's an optical effect that is influenced by parallax, so you have to pay attention to location as well. This is, for example, why Eid al-Adha falls on different days in different parts of the world. So the function signature itself is potentially wrong, depending on my needs. I might find that out if I had to do some Googling to finish the function, but (assuming I didn't already know) I'm not sure if that possibility would ever have occurred to me if I were using Copilot. Copilot can spit out code that's influenced by what others have written. But can it clue you into design considerations like this? Or should we be worried that it is helping us to write code that does the wrong thing with a higher degree of confidence?
- daveFNbuck 5y ago
  The function signature was determined by Copilot. The author just wrote the comment above the function and the word function, then let Copilot determine what the signature was and how it would be implemented.
  - mumblemumble 5y ago
    These use cases where people are running into trouble are increasingly sounding a lot like "What happens if I engage Autopilot and then take a nap?" That's definitely not how the tool was intended to be used, but it's hard to see how you can reasonably expect that nobody would ever do that.
lwb 5y ago
Seems like a programming language that optimizes for "codes well with Copilot" could be fairly successful. Things like memory safety, human readability, verifiability, etc. You could also have some kind of AI-driven testing/verification program -- Copilot and <other program> could go back and forth multiple times until the program is deemed correct and returned to the user.
polynomial 5y ago
I haven't used Co-Pilot (though I have apparently contributed to it) but just based on what I know, it doesn't seem like a finished product ready to be launched, rather an experimental feature that you might want to get some more focus group testing on. The idea that anyone would be actually "using" it in production is puzzling, to say the least.
vcryan 5y ago
I wonder if people are taking this too seriously. It's a fun toy, not something to actually use in a professional environment.
- tyingq 5y ago
  It's not being marketed as a toy. So the "too seriously" is perhaps concern about how other people will use it.
dboreham 5y ago
This is a fundamental problem with ML and current generation "AI" : it only works in scenarios where a statistical win is acceptable (e.g. ad targeting). It is useless and often worse than useless where you want absolute correctness and false positives are highly problematic (e.g. spam filtering, writing code, not running over pedestrians).
seanwilson 5y ago
If Copilot isn't showing exact copies of code it's seen, how is it able to produce code that mostly work? E.g. Copilot code from the article: function getPhase() { var phase = Math.floor((new Date().getTime() - new Date().setHours(0,0,0,0)) / 86400000) % 28; if (phase == 0) { return "New Moon"; } else if (phase == 1) { return "Waxing Crescent"; } // etc. It feels like small incorrect modifications to any of the code here would completely break the function. I've seen stories and articles written by GPT-3 where it will lose the plot and context on the way - in comparison Copilot doesn't suffer from this as much? How?
- quenix 5y ago
  This is also something that kind of breaks my brain. It's always impressive to see Copilot-"authored" code which seems complex, coherent, (and most of the time, functional), and Google it to zero previous results. How is it that a glorified statistical machine is able to put blocks of code so well together?
  - piyh 5y ago
    Code is much mores structured than English. Code is built around well defined ideas, processes, datastructures. Human language is context dependent, loose, and things like tonality of speech can change the meaning. It's easier to guess a multiple choice questions when all choices can be generated by intellisense instead of having to look at a dictionary.
  - gotostatement 5y ago
    its worth noting that, in the article, the author explains that the moon phase function produced by copilot was actually incorrect
- YeGoblynQueenne 5y ago
  There are parts of the internet that google doesn't search, it seems. I've just tried searching for a couple of strings from my github repository and I can't find them with google or duckduckgo (I get tons of results but none that's actually my repository or contains my code). To be sure the Copilot generated code is not copied verbatim from somewhere you'd have to search its training corpus. I don't know if that is available publicly.
ohazi 5y ago
I'm surprised that so much of the discussion around Copilot has centered around licensing rather than this. You're basically asking a robot that stayed up all night reading a billion lines of questionable source code to go on a massive LSD trip and then use the resulting fever dream to fill in your for loops. Coming from a hardware background where you often spend 2-8x of your time and money on verification vs. on the actual design, it seems obvious to me that Copilot as implemented today will either not provide any value (best case), will be a net negative (middling case), or will be a net negative, but you won't realize that you've surrounded yourself with a minefield for a few years (worst case). Having an "autocomplete" that can suggest more lines of code isn't better, it's worse. You still have to read the result, figure out what it's doing, and figure out why it will or will not work. Figuring out that it won't work could be relatively straightforward, as it is today with normal "here's a list of methods" autocomplete. Or it could be spectacularly difficult, as it would be when Copilot decides to regurgitate "fast inverse square root" but with different constants. Do you really think you're going to be able to decipher and debug code like that repeatedly when you're tired? When it's a subtly broken block of code rather than a famous example? That Easter example looks horrific, but I can absolutely see a tired developer saying "fuck it" and committing it at the end of the day, fully intending to check it later, and then either forgetting or hoping that it won't be a problem rather than ruining the next morning by attempting to look at it again. I can't imagine ever using it, but I worry about new grads and junior developers thinking that they need to use crap like this because some thought leader praises it as the newest best practice. We already have too much modern development methodology bullshit that takes endless effort to stomp out, but this has the potential to be exceptionally disastrous. I can't help but think that the product itself must be a PSYOP-like attempt to gaslight the entire industry. It seems so obvious to me that people are going to commit more broken code via Copilot than ever before.
- bryanrasmussen 5y ago
  >It seems so obvious to me that people are going to commit more broken code via Copilot than ever before. Maybe we should use Copilot to commit more open source code meaning that Copilot becomes more and more corrupted and unusable! of course then we end up with a bunch of bad open source code that will turn people off of using open source. Gee, I don't think Microsoft really thought this one through.
pagutierrezn 5y ago
IMO Copilot doesn't need to be perfect. It just needs to be better than copy pasting the code from a colleague or from Stack Overflow.
Houshalter 5y ago
Are we seriously going to criticize a tab completion engine because it doesn't perfectly calculate what the phases of the moon are? Can you? I'm amazed it even knows what that means and has a vague idea of what such a calculation should look like, even if it fails.
temuze 5y ago
I'm glad the author points out that there are aspects to Copilot that are usable. A lot of other critics haven't been so kind. Many complaints about Copilot remind me of the old Louis CK sketch where people complain about flying: YOU'RE SOARING THROUGH THE HEAVENS IN AN ALUMINUM TUBE. YOUR ANCESTORS WOULD'VE DIED OF DYSENTERY DOING THIS TRANSCONTINENTAL JOURNEY. Let's have some context here! Sure, it's not remotely close to perfect and it's going to take a long, long, long time for it to get there. But still, there's something about Engelbart-ian about seeing the demo when it works perfectly.
- tjalfi 5y ago
  I think [0] is the sketch you're thinking of [0] https://www.youtube.com/watch?v=nUBtKNzoKZ4 https://www.youtube.com/watch?v=nUBtKNzoKZ4
YeGoblynQueenne 5y ago
Huh, so Copilot was trained on a mix of correct and incorrect code and it learned to reproduce both correct and incorrect code. It's a language model, after all. It predicts the most likely next sequence of tokens. It doesn't know anything about correct and incorrect code. If the most likely next sequence of tokens is incorrect, it doesn't care, it's still the most likely next sequence. And I guess it makes sense that Copilot was trained in this way, even if it wasn't just a language model. How do you even begin to separate correct from incorrect code on the entire freaking github? But I think TFA serves best to show the worst way to use Copilot. I haven't tried it but I suspect that it would do a lot better if it was asked to generate small snippets of code, rather than entire functions. Not "returns the current phase of the moon" but "concatenates two strings". That would also make it much easier to eyball the generated code quickly without too many mistakes making it through. Of course, you could do the same thing with a bunch of macros, rather than a large language model that probably cost a few million to train...
k12sosse 5y ago
Can we all just finally realize that copilot existing doesn't prevent you from not using it? You can still write your 'famously better' code. Go on, show us why humans are better.
- stitched2gethr 5y ago
  It seems the more pressing issue is that I will almost definitely question a future pull request to have the author tell me, "well that's what copilot wrote and it looked OK." If I catch it. I'm all for AI coding assistance, but there's an abstraction layer in between copilot and myself (humans) that scares me a bit. At least an obvious failure prompts discussion and learning.
diragon 5y ago
This seems to imply a need for GPL4: its purpose would be to infect code that was written by a bot that learned from GPL4 code. Otherwise this can be used to circumvent all except the most lenient free software licences. I would say that GPL3 already probably does that, which means that nobody should be using this for actual code (except if it's GPL3). But it might be helpful to be explicit about this.

363 points by g_p 5y ago

≥ These three example pieces of flawed code did not require any cajoling; Copilot was happy to write them from straightforward requests for functional code. The inevitable conclusion is that Copilot can and will write security vulnerabilities on a regular basis, especially in memory-unsafe languages. If people can copy paste the most insecure code from Stack Overflow or random tutorials, they will absolutely use Copilot to "write" code and it will be become the default, especially since it's so incredibly easy to use. Also, it's just the first generation tool if it's kind, imagine what similar products will accomplish in 20 years.

foepys 5y ago
> imagine what similar products will accomplish in 20 years. People said the same thing about UML and similar tools so I'm not holding my breath.
- arkitaip 5y ago
  Maybe you are right but where UML created busy work, Copilot will literally do your work for you. I can even imagine a future where management makes it policy to Copilot first to save time and money.
  - swiftcoder 5y ago
    > imagine a future where management even makes it policy Because management policies correlate well with engineering excellence, right?
  - Zababa 5y ago
    Reviewing code written by Copilot may be longer than writing the code in the first place.
  - wpietri 5y ago
    Most of my work isn't copy-pasting snippets. It isn't even typing code. It's understanding user needs and the existing system, and then figuring out how to making things better for the user with also improving the system. So this does not do my work for me. I can also imagine clueless bosses mandating Copilot use and that's what scares me. The real costs of most code aren't in the first writing. They're in the long-term maintenance. Copilot does not and cannot understand the whole system, or what makes for maintainability down the road. So it can't make that better, and will likely make worse. In the same way that code generation tools and code wizards made things worse.
deleted 5y ago
[deleted]
whoknew1122 5y ago
With the pace of technological innovation, I'm honestly not sure what a similar product will be able to accomplish in 20 years. It'll be crazy for sure. But I'm worried about today. This is a product by a well-known company (GitHub) which is owned by an even more well-known company (Microsoft). GitHub is going to be trusted a lot more than a random poster on Stack Overflow or someone's blog online. And GitHub is explicitly telling new coders to use Copilot to learn a new language: > Whether you’re working in a new language or framework, or just learning to code, GitHub Copilot can help you find your way. Tackle a bug, or learn how to use a new framework without spending most of your time spelunking through the docs or searching the web. This is what differentiates Copilot from Stack Overflow or random tutorials. GitHub has a brand that's trusted more than random content creators on the internet. And it's telling new coders to use Copilot to learn things and not check elsewhere. That's a problem. Doesn't matter what generation of the program it is. It creates unsafe code after using its brand reputation and recognition to convince new coders to not check elsewhere.
- azangru 5y ago
  > GitHub is going to be trusted a lot more > GitHub has a brand that's trusted more Consider Google Translate, right? Google is a well-known brand that is trusted (outside of a relatively small group of people that doesn't trust Google on principle). Yet every professional translator knows that the text produced by Google Translate is a result of machine translation, Google or no Google. They may marvel at the occasional accuracy, yet expect serious blunders in the text, and would therefore not just trust that translation before submitting it to their clients. They will check. Or at least they should. Same with programmers.
  - mipmap04 5y ago
    I think this is an important way we need to frame the use of these tools for junior developers. I'd advise that anyone who is recommending this product to their team also take the time to give this analogy - maybe even going so far as to require explicit comments that notifies reviewers when code was provided by Copilot and similar services.
  - 5y ago
piokoch 5y ago
I think there is some difference. You don't come across some piece of code by chance, you were actively looking for it, probably there were multiple blogs, SO entries with needed information, one of those sources has to be chosen. You know that this is some random blog post or SO answer given by someone fresh. Copilot is something different. Code is suggested automatically and, what's the most important, suggested by the authority - hey, this is GitHub, huge project, largest code repo on the planet, owned by Microsoft, one of the most successful company ever. Why should you not trust the code they are suggesting you? And that's for starters before malicious parties start creating intentionally broken code only to hack system built with it. Greedy lawyers who will chase some innocent code snippet asking to pay for using it, etc.

tifadg1 5y ago

copilot will be interesting in 10-20 years. Now it's an early stage ml driven experiment in a field that hasn't advanced in forever - the strides it'll make will be very gradual, incremental and filled with mistakes along the way.

bryanrasmussen 5y ago
Luckily global warming should be killing me about the same time machine learning is taking my job. That's pretty good planning right there.
- tifadg1 5y ago
  so it all works out in the end!
- speedgoose 5y ago
  You should consider to immigrate to a colder place.
  - seba_dos1 5y ago
    Just like Richard Branson, Elon Musk, Jeff Bezos do...
  - jazzyjackson 5y ago
    it's not the heat that gets you, it's the water wars
  - bryanrasmussen 5y ago
    I'm in Denmark. I guess I could move up to Sweden, but I'm not sure if they want me because, after all, I am from Denmark.

emerongi 5y ago

Copilot doesn't seem like the right word. Maybe first year college student with no previous programming experience? Then it would be clear what level of help you are actually getting. Impressive, for sure. Unclear whether it's a net-positive tool, though.

bla3 5y ago
GitHub Excited Intern
baud147258 5y ago
fuzzy matching with publicly available code on Github
Hamuko 5y ago
GitHub Student Driver
jsjohnst 5y ago
> Copilot doesn't seem like the right word. Maybe “autopilot” then (as in the Tesla marketing term, not as in a real autopilot)? /s Both lull you with a false sense of security which will suddenly and unexpectedly cause you to pay dearly for using it. Interestingly both seem to also get a vaguely similar balance of critics, apologists, and advocates on here. Sample size of two here, but what is it with companies using %pilot to describe product features which are nothing like the actual appropriate use of the term?

prepend 5y ago

> me @'ing them several hundred times about ICE This is such curious behavior to me. Does someone really @ a corporation hundreds of times about anything? Does this have any effect? Should it? It makes me doubt the rationality of the author’s post if ve truly did this. Although I suppose maybe their use of Twitter is just completely different from anything I understand.

smitop 5y ago
You can use some Twitter search filters to see all of them: https://twitter.com/search?q=from%3A0xabad1dea%20%40github%20ICE&src=typed_query&f=live https://twitter.com/search?q=from%3A0xabad1dea%20%40github%2...
tylersmith 5y ago
It's a way to feel better about yourself without changing any comfortable behaviors. The author still uses Github and the ICE tweets are public Hail Mary's.
infinitezest 5y ago
Mostly, this just seemed like a non sequitur to me. Right away, I get the sense that "oh, this author has a bone to pick with GitHub.". Even if it's unrelated to the crux of the piece, I already feel like I'm going to have to take what the author says with a grain of salt.

muglug 5y ago

I had not considered the proliferation of terrible open-source code on GitHub. I'd wager that the amount of code in public repositories from students learning to code may outweigh quality code in GitHub. I wonder if there was any sort of filter for Copilot's input — only repositories with more than a certain number of stars/forks, only repositories committed to recently etc.

tovej 5y ago
I wouldn't bet on it, they didn't filter by license after all.

ainar-g 5y ago

I'm pretty sure that this conclusion isn't new, but I've come to think that Copilot shouldn't be thought of as a better developer, but merely a quicker one. Obviously its code will be somewhat average, considering that it's been trained on code the only unifying characteristic of which is that it's public. Something like Copilot, but trained explicitly to analyse the code instead of writing it could be much more useful, imo. Basically a real-time code review tool. There are similar tools already, but I'm talking of something that is able to learn from the actual codebase being worked on, perhaps including the documentation, and giving on-the-go feedback.

rob74 5y ago
If you interviewed two developers, one who produces reasonably correct code in a given amount of time, and another one who produces code which is subtly incorrect most of the time, but much faster, which one would you hire? The problem with your proposal is that it's relatively easy to do what Copilot does at the moment using AI, i.e. guess what code you are looking for and find something that does (or says it does) more or less that. However, which codebase would you use to check against if the generated code is really correct? The same codebase that produced the more-or-less-correct code in the first place?
jazzyjackson 5y ago
I like this idea, given that it takes advantage of how git repos are made of bug-fixes. How many git diffs are out there that update a '=' in an if statement to '=='? so an AI copilot should be watching out for code I write that looks similar to code that was updated in another repo. It could even use the text from issues to synthesize a suggestion of why your code might cause problems!

eddieroger 5y ago

> Ultimately, a human being must take responsibility for every line of code that is committed. AI should not be used for "responsibility washing." That's the whole point, and the rest is moot because of it. If I chose to let Copilot write code for me, I am responsible for it's output, full stop. This is the same as if I let a more junior engineer submit code to prod, but there aren't blog posts about not letting them work or trusting them with code.

onionisafruit 5y ago

This articulates some of the concerns I had trying copilot. I noticed that I ended up assuming the code reviewer role when I was trying to write code. Context switching between writing and reviewing felt unnatural. I also think I am less likely to spot a bug than I am to avoid writing it in the first place. Taking the off-by-one error in the last example. I don't think I would have made that mistake, but if copilot had presented that code block, I probably wouldn't have noticed the error either.

generationP 5y ago

The Easter Date algorithm was probably someone implementing an algorithm from the Wikipedia ( https://en.wikipedia.org/wiki/Date_of_Easter#Anonymous_Gregorian_algorithm https://en.wikipedia.org/wiki/Date_of_Easter#Anonymous_Grego... ) without bothering to understand it (because honestly it's not a very interesting problem). No wonder it's uncommented. As long as the AI just regurgitates lines from repositories like a bad undergrad cheating on his homework, CS jobs should be safe. The fact that it has picked up the GPL might not mean that much -- it might appear in dual-licensed projects.

prionassembly 5y ago
This is something I don't get. You're supposed to be able to integrate BSD-licensed (or even Public Domain) code into GPL works, right? The fact that something shows up in GPL code means what exactly? This is like: there are scholarly books that quote extensive from original philosophers -- long, third-of-a-page quotations. Still I should be able to quote something in its original language (translations may be copyrighted) copying from the derived work. Copyrighted work is not supposed to be able to poison noncopyrighted work it originates from.
skohan 5y ago
> As long as the AI just regurgitates lines from repositories like a bad undergrad cheating on his homework, CS jobs should be safe. They said the same thing about Chess and Go.
- generationP 5y ago
  The chess "job market" seems to be doing pretty well these days, at least based on recent publicity.
  - skohan 5y ago
    Watching people play chess has entertainment value. Coding is lucrative because there's a lot of demand and not that many people who can do it well. What if a computer can do it better than almost anyone?

eCa 5y ago

Re license washing: > This may or may not suddenly become a huge legal liability for anyone using Copilot. And if it doesn’t, can’t Copilot also be used to license wash for people not using it, but claiming to be?

tjpnz 5y ago

>As a code reviewer, I would want clear indications about which code is Copilot-generated. I would like to see this tracked behind the scenes. At any time I should be able to get Copilot to spit out a list of suggestions I've accepted. I should be able to generate this report for the lifetime of a project.

jazzyjackson 5y ago
It's kind of funny that this whole thing is built on top of git but uses none of the features, such as git blame. Instead of being an auto-complete, I would want co-pilot to be a git contributor, making pull requests that improve my code.

smusamashah 5y ago

Copilot doesn't seem any better then Tab Nine. Tab nine is GPT-2 based, works offline and can produce high quality boilerplate code based on previous lines. It can also generate whole methods which when work seems mind blowing but they are not always correct. Most suggestions are usually mind blowing anyway because previously we never had this kind of code completion. It feels like it wrote the whole line which you were going to write exactly as it should have. But that's all it does. And it seems like Copilot is the same but on much larger scale and online.

tylersmith 5y ago
I'm a long time TabNine user and fully agree. It's cool that Copilot is bringing the idea to more people but TabNine is the better product.

kozikow 5y ago

People look at the beta release of the software, and interpret flaws in an early version like a critical fundamental problems. I am pretty sure the new releases will contain features like better software license handling (e.g. 3 levels for types of licenses - permissive, copy-left, hardcore copy-left), trust score for snippets, possibly some validation of the code for some languages.

tyingq 5y ago
I'd be more optimistic if the beta were crafted with the idea that it might have issues. It so, it would likely have some way of gathering feedback on suggestions that was a little more nuanced than just accepted/rejected.
skohan 5y ago
This could actually be interesting. If it tuns out that copy-left based code completion is better than other options, it will create a strong incentive to spread it.
only_as_i_fall 5y ago
Does Microsoft actually have a repository of permissive licensed code though? My guess would be that some significant portion of github code published under a permissive license is actually licensed improperly. Working that out at scale seems intractable, but maybe the training set doesn't need to be as big as I'm assuming.
- nullc 5y ago
  There is still something to be said for making a reasonable effort. There are no guarantees in the world.
TeMPOraL 5y ago
> People look at the beta release of the software, and interpret flaws in an early version like a critical fundamental problems. Maybe because they realize the flaws are fundamentally inherent in the very core of the product. They're using a GPT-3 derivative here. DNN models are not the right tool for this job.

_fat_santa 5y ago

I think the use case for Copilot is a bit misunderstood. The way I see it you have two types of code: 1. Smart Code: Code that you honestly have to think about while you're writing. You write this code slowly and carefully, to make sure that it does what you need it to 2. Dumb Code: This is trivial code, like adding a button to a screen. This is code you really don't have to think about, because you already know exactly how to implement it. At this point your biggest obstacle is how fast can your fingers type on a keyboard. For me Github Copilot is useless for "Smart Code" but a godsend when writing "Dumb Code". I want to focus more on writing and figuring out the "Smart Code", if I need to throw a form together in HTML or make a trivial helper function, I will gladly let AI take over and do that work for me.

skohan 5y ago
I think you just described which coders will be replaced by AI first
- gonzo41 5y ago
  Or you're describing how you could put copilot in a box and make a really good low code gui programming solution where the complex stuff is good old complex code.
  - skohan 5y ago
    It sounds like you're talking about a way to cut my front-end hiring budget by 80%.
- wongarsu 5y ago
  I don't know, we somehow manage to replace point-and-click GUIs for placing buttons (Windows Forms etc) with Frontend developers writing elaborate code to achieve the same result in HTML/CSS. Productivity is far from the first priority for frontend development.
  - skohan 5y ago

larodi 5y ago

Am I the only one thinking that GPT3 and CoPilot can actually work once trained on properly licensed and properly audited code? Well it will not be as ubiquitous as having all the github under your fingers, but perhaps is anyway better not to blindly cite the world's source code.

qayxc 5y ago
> Am I the only one thinking that GPT3 and CoPilot can actually work once trained on properly licensed and properly audited code? Sadly you aren't. The truth is, however, that models like GPT3 and its derivatives like Codex/CoPilot are by design incapable of ever achieving this. The only way to generate both correct and secure code is to use a combination of proper specs and theorem provers. Even then this won't help with non-functional requirements, such as performance or platform-dependent resource constraints. Generative models will always have the potential to yield broken code that doesn't do what you want or contains security flaws even if trained on "proper" code. If I have to audit the code that CoPilot generates and if the code is as obfuscated as the Easter example, it's probably less useful than it says on the label...

reilly3000 5y ago

I would hire copilot to write tests for me, that’s about it. Writing tests can be a drag. It’s really a low-risk proposition to have generated code attempt it. If it’s a usable test, maybe it will catch a bug. If not, then kill it and let it generate a few more. The expectation is entirely different than producing code. Code needs to be correct, secure, performant, and readable. Failure on any of those fronts can be expensive to disastrous. Nobody can reasonably expect a test suite to catch every bug, even if created by the smartest humans. If a copilot-created test does prevent a bug from shipping it provides immediate value. I could see it coming up with some whacky-but-useful test cases that a sane person might not consider. From a training perspective I would think that assertion descriptions contain more consistent lexical value than the average function signature. It seems like the ambitious data scientists, product marketers, and managers fell in love with a revolutionary idea about AI writing code, and neglected to consult the engineers they are trying to ‘augment’.

squidsurfer 5y ago
Many tests require an understanding of the intent of the code, especially the further from a pure unit test they stray. CoPilot has no understanding of anything and I anticipate will be useless in writing the tests that are actually hard to write.
neatze 5y ago
My time is spend mostly not on writing code, but thinking what program has to do, testing (including writing unit tests) and understanding errors, I always tell people half joking programming is not about writing code, but an ability to debug it; understanding requirements, errors and bugs is hard, writing code and fixing bugs is relatively easy, in general. Maybe Copilot 2 will do exactly this; it will generate tests based of half working code, run them and suggest improvements, that would increase productivity by like ~100%, but to me this sounds to good to be true.
wpietri 5y ago
Nope. That's like saying, "I might let a machine write the docs for me." Good tests are documentation that a computer can verify. Because they explain the meaning of parts of the system, they contain information not available in the code. If you try using ML for test generation, you'll have the same problem you do with GPT-3 prose: it might look plausible at first glance, but lacks coherent meaning. You'd also end up with one of the problems common in big test suites: poorly factored tests that end up being the sort of expressive duplication that is a giant drag on improving existing code. ML is nowhere near advanced enough to say, "Gosh, we're doing the same sort of test setup a bunch; let's extract that into a fixture, and then let's unify some fixtures into an ObjectMother. For people looking to get the computer to do the work of catching more things with less burdensome test writing, I suggest taking a look at things like Hypothesis: https://hypothesis.readthedocs.io/en/latest/ https://hypothesis.readthedocs.io/en/latest/

nojs 5y ago

I think the biggest problem copilot will have in practice gaining traction is that verifying correctness isn’t any faster than writing the code yourself in many cases. The Easter(y) function is a classic example - it would be way faster to write that than to try and verify that there’s no subtle bugs. Copilot is by design trying to give you something that _looks_ correct without caring whether it actually is - so it optimises for real-looking but subtly buggy code, which is the worst kind of broken code.

mattwad 5y ago
I've been using it and that's 100% correct. If it suggests more than a few lines, i might as well do it myself. However, it has been an awesome Intellisense tool for one-liners.. it can write out the rest of a comment or a simple map/filter method just fine. I don't think it will ever go further than that, but nor does it need to.
- visarga 5y ago
  Be sure it doesn't generate any GPL licensed code down to the size of a single letter, to be safe. The reaction to GPL inspired snippet generation has been more fierce than I could have imagined, even though usable snippets are so short.
  - z3t4 5y ago
    You can't license a code snippet. You can only copyright/license a complete work, as in a complete app or library. Software patterns might work differently though!?
    - dec0dedab0de 5y ago
      Any portion of a work has a copyright the moment there is a tangible copy. There is a uniqueness requirement, but it has nothing to do with length. A unique one liner would have a copyright.
      - 5y ago

IfOnlyYouKnew 5y ago

It's interesting how the community that used to fervently argue that, say, the sampling of a few seconds of a musical composition is so obviously fair use doesn't extend the same attitude when it comes to snippets of source code. Indeed I don't remember any of these complaints ever being made about, for example, AI-generated music or images, even though they work exactly the same and were trained on datasets of copyrighted works, both commercial and CC-licensed. Compared to the manual sampling that DJs and other musicians do, the AI process almost certainly produces only fairly generic code snippets, since they always need a multitude of examples. Some loop-through-file python snippet is a legal risk but five seconds of a Disney song don't reach the level of creativity needed for copyright to matter? That seems strange to me...

ShroudedNight 5y ago
While I think there is some room for retrospective humility, It seems perfectly consistent to believe that, morally, de minimis use of copyrighted material should be considered fair-use while acknowledging that there are practical legal and professional risks for doing so (especially blindly), and that encouraging people to engage in that behaviour in an uncritical manner (especially in the context of engineering) is sociologically reckless.
_ph_ 5y ago
The problem is, we don't know what Copilot is going to do. Sometimes it reproduces entire files in verbatim. That is certainly a copyright violation. Sometimes it produces whole functions which are more or less taking verbatim from a copyrighted file. The user cannot see the level to which they are "verbatim" and there are also no clear legal guidelines to what is considered violating copyrights and what not.

deleted 5y ago

[deleted]

kiernanmcgowan 5y ago

Has anyone tried to generate private keys from copilot? GitHub suppressed these from search a while ago, but it would be an easy mistake to have included them in their training data.

user-the-name 5y ago

The easter code is regurgitated pretty much verbatim: https://stackoverflow.com/a/58726426 https://stackoverflow.com/a/58726426 It really is hard to swallow the argument that this is not license violation on a massive scale.

abunuwas 5y ago

I see copilot as something that could be useful for generating boilerplate or starting code. Often when using a framework or library there's some boilerplate code that has to be there for things to work fine, so instead of having to the documentation and find the relevant snippets, maybe copilot can do this for you (although they often come with CLIs that do this for you). Case in point is machine learning code, where you often find lots boilerplate setting up the models, the training, etc. Maybe copilot was developed by machine learning devs for machine learning devs?

brundolf 5y ago

> This is well-formed and even commented C that sure looks like it parses HTML, and the main function has some useful boilerplate around opening the file. However, the parsing is loaded with issues. This sounds like a great example of an interview question (where the person is asked to find and fix all of the issues in a chunk of bad code). Unintended usecase for Copilot?

oaiey 5y ago

I just read one of the comments below and thought: we should write sql as an AST and not as a string. That solves many problems, not only SQL injection but also the lousy understanding of joins etc.

paulsutter 5y ago

If anyone wants to write software to detect the original source of Copilot code, I can explain how to detect variable length unaligned matches using fast hash code lookups (unlimited size corpus)

mattwad 5y ago

People complaining about Copilot should just try it. All the concerns people are bringing up are correct but missing the point. Just think of it as a context-aware autocomplete that can complete more than just properties - it can finish out that one-line comment or map function too! It's really not that intrusive, nor is it going to replace anyone's coding job or even require less programmers. It'll just speed you up a bit, similar to auto-completion. I think at least 70% of the time I take its suggestions, which is good enough to keep using it.

ghostly_s 5y ago
Did you even read the article?

jedberg 5y ago

My biggest thing with Copilot is that it was trained on all public code on Github, which includes a lot of bad code that people just put up there (like my own code that I wrote a decade ago). As long as it is keeping track of when people do or do not accept their suggestions, it should get better over time. But in the meantime the best bet it is to treat it like a smart autocomplete, where you still have to at least check that it got it right. In the future maybe it will be smart enough to be treated like an intern -- trust that the code is right but still verify it yourself if the code is of any importance.

skc 5y ago

It's been fascinating reading all the responses to Copilot. It's pretty clear to me now that developer habits are far more diverse than I thought. For me, Copilot seems like it will be useful in the exact same way that StackOverflow is useful, as a means to pointing me in the right direction for code snippets or apis or techniques that I haven't and don't really want to memorize. For example, on my current side project I wanted to know how to create a "unique enough" UUID in pure JS. Copilot would hopefully save me a couple of google/stackoverflow searches as I can very quickly test what it suggests. I already rarely ever take SO answers as gospel so it's unlikely that I'd do the same with Copilot but I think it significantly increases the speed with which I achieve the same results.

mumblemumble 5y ago

The moon phase example is illustrative in another way. It's not technically possible to precisely calculate the moon's phase based on time alone. It's an optical effect that is influenced by parallax, so you have to pay attention to location as well. This is, for example, why Eid al-Adha falls on different days in different parts of the world. So the function signature itself is potentially wrong, depending on my needs. I might find that out if I had to do some Googling to finish the function, but (assuming I didn't already know) I'm not sure if that possibility would ever have occurred to me if I were using Copilot. Copilot can spit out code that's influenced by what others have written. But can it clue you into design considerations like this? Or should we be worried that it is helping us to write code that does the wrong thing with a higher degree of confidence?

daveFNbuck 5y ago
The function signature was determined by Copilot. The author just wrote the comment above the function and the word function, then let Copilot determine what the signature was and how it would be implemented.
- mumblemumble 5y ago
  These use cases where people are running into trouble are increasingly sounding a lot like "What happens if I engage Autopilot and then take a nap?" That's definitely not how the tool was intended to be used, but it's hard to see how you can reasonably expect that nobody would ever do that.

lwb 5y ago

Seems like a programming language that optimizes for "codes well with Copilot" could be fairly successful. Things like memory safety, human readability, verifiability, etc. You could also have some kind of AI-driven testing/verification program -- Copilot and <other program> could go back and forth multiple times until the program is deemed correct and returned to the user.

polynomial 5y ago

I haven't used Co-Pilot (though I have apparently contributed to it) but just based on what I know, it doesn't seem like a finished product ready to be launched, rather an experimental feature that you might want to get some more focus group testing on. The idea that anyone would be actually "using" it in production is puzzling, to say the least.

vcryan 5y ago

I wonder if people are taking this too seriously. It's a fun toy, not something to actually use in a professional environment.

tyingq 5y ago
It's not being marketed as a toy. So the "too seriously" is perhaps concern about how other people will use it.

dboreham 5y ago

This is a fundamental problem with ML and current generation "AI" : it only works in scenarios where a statistical win is acceptable (e.g. ad targeting). It is useless and often worse than useless where you want absolute correctness and false positives are highly problematic (e.g. spam filtering, writing code, not running over pedestrians).

seanwilson 5y ago

If Copilot isn't showing exact copies of code it's seen, how is it able to produce code that mostly work? E.g. Copilot code from the article: function getPhase() { var phase = Math.floor((new Date().getTime() - new Date().setHours(0,0,0,0)) / 86400000) % 28; if (phase == 0) { return "New Moon"; } else if (phase == 1) { return "Waxing Crescent"; } // etc. It feels like small incorrect modifications to any of the code here would completely break the function. I've seen stories and articles written by GPT-3 where it will lose the plot and context on the way - in comparison Copilot doesn't suffer from this as much? How?

quenix 5y ago
This is also something that kind of breaks my brain. It's always impressive to see Copilot-"authored" code which seems complex, coherent, (and most of the time, functional), and Google it to zero previous results. How is it that a glorified statistical machine is able to put blocks of code so well together?
- piyh 5y ago
  Code is much mores structured than English. Code is built around well defined ideas, processes, datastructures. Human language is context dependent, loose, and things like tonality of speech can change the meaning. It's easier to guess a multiple choice questions when all choices can be generated by intellisense instead of having to look at a dictionary.
- gotostatement 5y ago
  its worth noting that, in the article, the author explains that the moon phase function produced by copilot was actually incorrect
YeGoblynQueenne 5y ago
There are parts of the internet that google doesn't search, it seems. I've just tried searching for a couple of strings from my github repository and I can't find them with google or duckduckgo (I get tons of results but none that's actually my repository or contains my code). To be sure the Copilot generated code is not copied verbatim from somewhere you'd have to search its training corpus. I don't know if that is available publicly.

ohazi 5y ago

I'm surprised that so much of the discussion around Copilot has centered around licensing rather than this. You're basically asking a robot that stayed up all night reading a billion lines of questionable source code to go on a massive LSD trip and then use the resulting fever dream to fill in your for loops. Coming from a hardware background where you often spend 2-8x of your time and money on verification vs. on the actual design, it seems obvious to me that Copilot as implemented today will either not provide any value (best case), will be a net negative (middling case), or will be a net negative, but you won't realize that you've surrounded yourself with a minefield for a few years (worst case). Having an "autocomplete" that can suggest more lines of code isn't better, it's worse. You still have to read the result, figure out what it's doing, and figure out why it will or will not work. Figuring out that it won't work could be relatively straightforward, as it is today with normal "here's a list of methods" autocomplete. Or it could be spectacularly difficult, as it would be when Copilot decides to regurgitate "fast inverse square root" but with different constants. Do you really think you're going to be able to decipher and debug code like that repeatedly when you're tired? When it's a subtly broken block of code rather than a famous example? That Easter example looks horrific, but I can absolutely see a tired developer saying "fuck it" and committing it at the end of the day, fully intending to check it later, and then either forgetting or hoping that it won't be a problem rather than ruining the next morning by attempting to look at it again. I can't imagine ever using it, but I worry about new grads and junior developers thinking that they need to use crap like this because some thought leader praises it as the newest best practice. We already have too much modern development methodology bullshit that takes endless effort to stomp out, but this has the potential to be exceptionally disastrous. I can't help but think that the product itself must be a PSYOP-like attempt to gaslight the entire industry. It seems so obvious to me that people are going to commit more broken code via Copilot than ever before.

bryanrasmussen 5y ago
>It seems so obvious to me that people are going to commit more broken code via Copilot than ever before. Maybe we should use Copilot to commit more open source code meaning that Copilot becomes more and more corrupted and unusable! of course then we end up with a bunch of bad open source code that will turn people off of using open source. Gee, I don't think Microsoft really thought this one through.

pagutierrezn 5y ago

IMO Copilot doesn't need to be perfect. It just needs to be better than copy pasting the code from a colleague or from Stack Overflow.

Houshalter 5y ago

Are we seriously going to criticize a tab completion engine because it doesn't perfectly calculate what the phases of the moon are? Can you? I'm amazed it even knows what that means and has a vague idea of what such a calculation should look like, even if it fails.

temuze 5y ago

I'm glad the author points out that there are aspects to Copilot that are usable. A lot of other critics haven't been so kind. Many complaints about Copilot remind me of the old Louis CK sketch where people complain about flying: YOU'RE SOARING THROUGH THE HEAVENS IN AN ALUMINUM TUBE. YOUR ANCESTORS WOULD'VE DIED OF DYSENTERY DOING THIS TRANSCONTINENTAL JOURNEY. Let's have some context here! Sure, it's not remotely close to perfect and it's going to take a long, long, long time for it to get there. But still, there's something about Engelbart-ian about seeing the demo when it works perfectly.

tjalfi 5y ago
I think [0] is the sketch you're thinking of [0] https://www.youtube.com/watch?v=nUBtKNzoKZ4 https://www.youtube.com/watch?v=nUBtKNzoKZ4

YeGoblynQueenne 5y ago

Huh, so Copilot was trained on a mix of correct and incorrect code and it learned to reproduce both correct and incorrect code. It's a language model, after all. It predicts the most likely next sequence of tokens. It doesn't know anything about correct and incorrect code. If the most likely next sequence of tokens is incorrect, it doesn't care, it's still the most likely next sequence. And I guess it makes sense that Copilot was trained in this way, even if it wasn't just a language model. How do you even begin to separate correct from incorrect code on the entire freaking github? But I think TFA serves best to show the worst way to use Copilot. I haven't tried it but I suspect that it would do a lot better if it was asked to generate small snippets of code, rather than entire functions. Not "returns the current phase of the moon" but "concatenates two strings". That would also make it much easier to eyball the generated code quickly without too many mistakes making it through. Of course, you could do the same thing with a bunch of macros, rather than a large language model that probably cost a few million to train...

k12sosse 5y ago

Can we all just finally realize that copilot existing doesn't prevent you from not using it? You can still write your 'famously better' code. Go on, show us why humans are better.

stitched2gethr 5y ago
It seems the more pressing issue is that I will almost definitely question a future pull request to have the author tell me, "well that's what copilot wrote and it looked OK." If I catch it. I'm all for AI coding assistance, but there's an abstraction layer in between copilot and myself (humans) that scares me a bit. At least an obvious failure prompts discussion and learning.

diragon 5y ago

This seems to imply a need for GPL4: its purpose would be to infect code that was written by a bot that learned from GPL4 code. Otherwise this can be used to circumvent all except the most lenient free software licences. I would say that GPL3 already probably does that, which means that nobody should be using this for actual code (except if it's GPL3). But it might be helpful to be explicit about this.