12 ms·

Quantifying GitHub Copilot’s impact on developer productivity and happiness

115 points by idan 4y ago

NoraCodes 4y ago
I'll tell you right now, having my GPL-licensed code stolen has a pretty negative impact on my happiness.
- JamesBarney 4y ago
  What snippets were stolen from your code and how has it affected your happiness?
- bastardoperator 4y ago
  Maybe you should switch to the Unlicense. I'm so much happier not giving a damn or punishing my users with pettiness.
  - NoraCodes 4y ago
    Licenses don't matter when Microsoft decides that AI training is fair use.
    - bastardoperator 4y ago
      I'll take "I didn't read the GitHub TOS" for 500 Alex.
      - throwaway1851 4y ago
        Well not everyone is in contractual privity with GitHub — like the poster a few up whose code was included in someone else’s repo. GitHub’s TOS have no effect in that circumstance.
- elcomet 4y ago
  What if I read your code? Now I have memory of it in my brain and it will influence my next projects. Is it stealing?
  - xmcqdpt2 4y ago
    Actually it could be! If you read GPL code and remember it well, then write a near exact copy of it into your proprietary software project, that's a violation of the GPL.
- sergnio 4y ago
  "The GPL is a free software license, and therefore it permits people to use and even redistribute the software without being required to pay anyone a fee for doing so." How is GHC breaking this license?
  - blowski 4y ago
    Say I drew a lovely picture and hung it outside my house for passers by to look at. I say “sure you can take a copy, no problem”, they then sell a billion prints, and don’t credit me at all. Legally, not stealing. Ethically? I’d call that stealing.
    - JohnHaugeland 4y ago
      Legally that actually is stealing. They must have documented reproduction rights. Misunderstanding a verbal comment will not fly in a court.
    - tducasse 4y ago
      Except that's not exactly what they are doing, is it? Following on your example, it'd be like selling the advice of someone who has seen a lot of these pictures passing by windows, with a very good memory and understanding of pictures. Github Copilot is not spitting out code verbatim, it's learning what code is, what shape it usually has, and trying to retrofit your own code onto the shape it thinks code should have. It's not like it's doing a query like `SELECT * FROM COPIED_CODE WHERE CODE STARTS_WITH "def my_func("`.
  - NoraCodes 4y ago
    GPL requires that derivative works - which, I argue, Copilot is - also be distributed under the GPL. Microsoft disagrees with my interpretation, and wins by default because I can't afford to sue them.
  - mtlynch 4y ago
    The GPL has requirements, and Copilot fails to adhere to those requirements. Here's an important one: >You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice
- baby 4y ago
  "wants to participate in and use open source, but is not happy when others use their code"
  - jacooper 4y ago
    *Use their code without complying with the license
    - nojito 4y ago
      Copilot is transformative work.
      - throwaway1851 4y ago
        Objection, legal conclusion.
    - baby 4y ago
      I couldn't care less about whatever restrictive license people choose and decide to call open source
      - jacooper 4y ago
        Well then dont use their code you have to legally follow the codes license.
- tducasse 4y ago
  I have no opinion on whether there's infrigement or not, as I am not a lawyer, but I found the argument that the terms of use specify that you allow them to analyze your code pretty convincing: > We need the legal right to do things like host Your Content, publish it, and share it. You grant us and our legal successors the right to store, archive, parse, and display Your Content, and make incidental copies, as necessary to provide the Service, including improving the Service over time. This license includes the right to do things like copy it to our database and make backups; show it to you and other users; parse it into a search index or otherwise analyze it on our servers; share it with other users; and perform it, in case Your Content is something like music or video.
  - paulryanrogers 4y ago
    Copilot seems to go further than all that. It can suggest verbatim, nontrivial, copyrighted work with no means of attribution. And the sources are so broad no human alive could say with certainty any given output is not infringing, unless it's so trivial it cannot be copyrighted.
    - tducasse 4y ago
      I've always had trouble understanding intellectual property when it comes to code, because if I read open source code, remember everything, and write my own version (even if slightly different) 10 years later, I am not 100% sure whether it's copyright infrigement or not. I see it as something akin to a painter studying someone else's work, then reproducing some of the techniques invented by the original artist... except that their techniques didn't have a license I guess? I think I am not equipped to fully understand the legal boundaries between inspiration and theft, and I think most programmers are in the same situation.
      - paulryanrogers 4y ago
        Painter analogy falls down when you consider Copilot can output verbatim chunks. So more like a photo of a painting or a stroke-for-stroke copy, even if portions changed or only parts taken. Now if changes are so significant it becomes impossible to recognize the reference then that could be legit. Though based on how Copilot works I don't think that can be assumed, or even proven. When I studied art the teachers usually taught us to begin with our own original photograph, ideally without trademarked items, for reference. And even in literature or journalism one must quote sources, even if paraphrased. I was taught to put down any inspiring work and only begin my own work after taking a break, to reduce the likelihood of unintentionally copying the original.
- asdff 4y ago
  I wonder if you could just ruin these systems by having some bots constantly upload bugged code into github
  - jrockway 4y ago
    I think my biggest worry about AI is that we hit a big wall right about now because training data is now contaminated with AI outputs. That's why OpenAI puts watermarks on images, I'm guessing, but Stable Diffusion doesn't, and the free one is probably going to get a lot more usage than the paid-for one. But, maybe not, because humans liked the outputs enough to share them and put them on the Internet, so maybe they don't have much of an effect quite yet.
    - asdff 4y ago
      I think the training limits other applications like self driving cars a lot too. Training is frequently done in contrived scenarios. Now, imagine if GM or Ford put sensors and telemetry for future self driving car use on their models decades ago. You'd have practically every road mapped with ten thousand observations a piece by now. It wasn't for lack of forsight to not collect these data, it was really for lack of profit. Sensors and other data collecting systems are not new technology. Neither is machine learning, its also decades old. The only thing different these days is that the compute required to run these models on these data has gotten cheaper, but you could have been collecting data the whole time with anticipation of cheap compute to run these models in the future. Then you'd have a leg up on everyone in the business with your massive dataset of decades of real world, not test tracks and other such situations utilizing an employed driver in all likelihood.
chank 4y ago
I ended up disabling it in the beta. It ended up getting in the way more often than not.
- AzzieElbab 4y ago
  Out of curiosity, what programming language you use, and what type of code you write?
  - mcluck 4y ago
    Not OP but I did the same thing. I build web apps using some combination of TypeScript, C#, F#, HTML, and CSS (or CSS-style languages). I spent about as much time correcting what it wrote as if I had written it myself. As someone else mentioned, it also interfered with proper inference given by the type system. Any time I wrote a more complicated algorithm (I work with a lot of graphs) I couldn't trust what it gave me. It's far easier to just gloss over some incorrect detail when the "shape" of the code is correct.
  - nprateem 4y ago
    I turned it off after it kept outputting go that looked feasible but wasn't valid because of the libraries I was or wasn't using. I only turn it on if I'm doing something mundane that it might be able to handle. The fact it overrides intellij's autocompletion was too annoying
- cesarvarela 4y ago
  Same. Felt like pairing with a drunk programmer.
  - verdverm 4y ago
    Did it ever repeat "developers!" loudly?
    - lopatin 4y ago
      I always figured those guys were on something else other than booze.
jeffbee 4y ago
"and timed how long it took them to write an HTTP server in JavaScript" OK ... interesting choice. This could mean almost anything. I can "write a web server" in a python 1-liner in about 2 seconds (ctrl-r SimpleHttpServer, enter).
- mgkimsal 4y ago
  There was a test suite to work against, so some standard notion of things that needed to be in place to be 'working'.
- idan 4y ago
  We'll be sharing some more about how we conducted this in an upcoming post! But no, the task was actually to write a little engine to process a limited subset of HTTP requests and return a valid response or relevant error.
  - verdverm 4y ago
    Would you pit it against my non-ai solution for the same problem? As a baseline to measure against? It would seem prudent to compare against other tools trying to alleviate the same problem, rather than just humans writing the code by hand https://github.com/hofstadter-io/demos/tree/main/full-stack-app https://github.com/hofstadter-io/demos/tree/main/full-stack-... I've since shortened that demo to be even faster for a user to complete
- blibble 4y ago
  it's a profoundly stupid task to measure to measure developer productivity too the hard bit of being a developer is deciding in precise, pedantic, utterly exact detail on what needs building to solve the business problem not bashing out the code in the editor now if they had given out a vague problem that ended up with the dev eventually producing a webserver in Javascript, then fine but then the difference copilot makes would be a much smaller percentage of the development effort I swear we're going back to the age of "lines of code written per hour", just now with arTiFiCiAL InTellIgEncE
  - 4y ago
adamddev1 4y ago
I mostly enjoyed my Copilot trial but one thing that really annoyed me was that it would often make suggestions for strings/keys that would be decent guesses by the AI but would be obvious type errors in TypeScript. Then it seemed a bit tricky to fight through the incorrect suggestion and get the old TypeScript Intellisense which was of course correct and what I wanted. Other suggestions by Copilot were awesome but when there was an obvious job for TypeScript it would be nice if Copilot would step back. For example I would often type: if (myTypedVar === "" <-- Back off Copilot, let TypeScript Intellisense give me the correct suggestions. Or at least run the Copilot suggestions through TS first and make sure it doesn't suggest type errors.
- matthewmacleod 4y ago
  Indeed this is one of the more annoying features of the integrations I’ve seen. Running through the type-checker is an interesting idea - even if you didn’t reject suggestions outright (which seems like it could frequently reject suggestions you actually wanted) it may be a useful weighting function. Similarly, running through eslint (or whatever linting system is appropriate) to avoid introducing obvious fixable formatting errors, like incorrect quotes, would be wonderful.
- capableweb 4y ago
  Sounds like it was your Copilot integration with whatever editor you used that was bad, not Copilot itself. The neovim integration worked very well, at least when I used it with Clojure and associated tooling. Could move seamlessly between what my Clojure tooling provides me with, and what Copilot did.
abdusco 4y ago
I liked using it. Though, most of the time, it was limited, in the sense that it could only offer completions using the code before the cursor as the context. I found it most helpful in setting up boilerplate in when writing tests. It could offer testing some edge cases before I could think of them.
wilg 4y ago
I use Copilot constantly and it's one of my favorite developer tools ever.
- davidkuennen 4y ago
  Same here. It's such a great tool and productivity boost. Don't want to miss it anymore.
avg_dev 4y ago
I read the article and I have some skepticism. I think my skepticism is well-founded but it may well be the case that a machine will one day do my job. I don't believe that time is at hand, though. First off, I don't see a link to the "HTTP server in JavaScript" task. It's really hard for me to place much faith in their conclusions when it's not even clear what the problem definition was. Second, I believe that a lot of more senior developers and development managers who take secure development practices somewhat seriously will not be able or willing to use Copilot in any sort of proprietary setting. Here is a quote from the Copilot FAQ: > [...] The GitHub Copilot extension sends your comments and code to the GitHub Copilot service, and it relies on context, as described in Privacy below - i.e., file content both in the file you are editing, as well as neighboring or related files within a project. It may also collect the URLs of repositories or file paths to identify relevant context. The comments and code along with context are then used by OpenAI Codex to synthesize and suggest individual lines and whole functions. - from https://github.com/features/copilot/#faq https://github.com/features/copilot/#faq - see "How does GitHub Copilot work?" I believe this makes it simply a nonstarter in a lot of environments. I am wondering if there are a number of places that have restrictions on sharing their code with a third-party but don't know or don't care and so end up using Copilot anyway. I believe that short-sighted thinking like this is more prevalent in shops that have low-quality code, and I believe that the higher-quality the code, the less likely someone is to use Copilot, simply for the "I can't share my code, even if I use the most restrictive no-telemetry settings" reason. Give me a self-hosted Copilot, and I may try it out in anger. Finally, I based some of my thinking on a recent Reddit /r/programming discussion of Copilot: https://old.reddit.com/r/programming/comments/wsnend/since_github_copilot_free_preview_ends_soon_what/ https://old.reddit.com/r/programming/comments/wsnend/since_g... After reading those posts, and internalizing them with my own view of coding, I believe Copilot is not ready for my personal use. Again: licensing considerations aside (if you actually can feel comfortable putting them aside, see NoraCodes comment in this HN thread e.g.), it is simply a non-starter for anything proprietary in nature. I am also of the mind that any code that is of necessity very tedious to write is in dire need of real attention, most likely in the form of tests and quite possibly refactoring to reduce the boilerplate if at all possible. I believe in the value of linters and automated code analysis tools and in continuous integration that runs after every commit. Give me a self-hosted Copilot, and we'll have a real chance to see how it works out - until then it's not going to be a boon to programmers.
_____s 4y ago
I really love Copilot. I had a trial, let it expire because I thought I wouldn't really need it, and then ended up paying for it. It's apparently got even better. For me, at least, it gets most things very right and takes a lot of mundanity out of day to day development!
- xwolfi 4y ago
  I work in Finance, if I type a certain client name out in public, I can be fired for it. Can copilot work without the internet and within only a corporate network ?
  - avg_dev 4y ago
    Not presently per their FAQ. > [...] The GitHub Copilot extension sends your comments and code to the GitHub Copilot service, and it relies on context, as described in Privacy below - i.e., file content both in the file you are editing, as well as neighboring or related files within a project. It may also collect the URLs of repositories or file paths to identify relevant context. The comments and code along with context are then used by OpenAI Codex to synthesize and suggest individual lines and whole functions. - from https://github.com/features/copilot/#faq https://github.com/features/copilot/#faq - see "How does GitHub Copilot work?"
  - skneko 4y ago
    You might be interested in IntelliCode from Visual Studio. It's an AI-guided autocompletion tool, like Copilot, although dumber and less powerful. It can run in local mode [0], looking at your existing codebase (opened solution) to feed future suggestions. [0]: https://docs.microsoft.com/en-us/visualstudio/intellicode/overview#telemetry https://docs.microsoft.com/en-us/visualstudio/intellicode/ov...
  - speedgoose 4y ago
    No, you can already use offline alternatives like tabnine but it's not as good. But I guess better opensource models and tools will be available eventually.
valenterry 4y ago
How's Copilot with really good statically typed languages like Haskell? I haven't tried it out because most comments say it doens't even work well with typescript yet...
- speedgoose 4y ago
  I use it almost daily with typescript and it works well. It will guess wrong some types once in a while but it's still a very nice help.
- siraben 4y ago
  It is not even remotely usable with languages like Haskell, the code it generates almost never typechecks, IME. I think the primary reasons being lack of training data (there is far more Python code on GitHub than the entirety of Hackage) and that Copilot is essentially a language model that does not consider AST and type information at all.
gamegoblin 4y ago
The place where Copilot shines most for me is boilerplate like: up = foo.north() right = and then it will correctly suggest up = foo.north() right = foo.east() down = foo.south() left = foo.west() This requires a kind of linguistic understanding that is beyond basic intellisense. If you are just wanting it for basic intellisense autocomplete, it's not much better than JetBrains stuff. But for things that require linguistic comprehension, it's incredibly useful. For example, if I have a `Point` object with `x` and `y` fields, and I make a function called `lowest_point`, it knows I want to the point with the minimum `y` value. Because it knows the name `y` is most typically used for vertical direction, and the word `lowest` also implies vertical direction. This is kind of a linguistic/semantic knowledge that is not found in just the type system.
- onemiketwelve 4y ago
  Especially boilerplate in a language or framework you don't usually use. It's such a time saver.
  - jlarocco 4y ago
    I've never used Copilot, but are you worried it will only spit out: up = foo.north() right = foo.east() down = foo.south() And you won't notice?
    - antihero 4y ago
      Not really no
      - richbell 4y ago
        Why not? From what I've seen GTP3 works exceptionally well until it doesn't, which can be easily missed.
        chrisseaton 4y ago
        But I read the code, and I run tests.
j-sizz 4y ago
Wish the auth was better
noizz 4y ago
I spent most of the trial hating Copilot and the way takes over my Tab key in Jetbrains software. It got in the way a bit too much. Every time I tried to use emmet shortcuts, live template triggers or anything muscle memory related, it would bring something else, so I had to undo, press Esc a couple of times and then continue with whatever I actually wanted. However, every now and then, there came a typed object or something mundane and Copilot exceled at it, so I decided that I want to find a way for it to coexist in a way that's as easy to use as pressing Tab to expand it. Lo and behold - I found an unused key below Tab (AKA caps lock), remapped it and now it doesn't get in the way and I can use Copilot's help only when I want.
- mccorrinall 4y ago
  In pycharm where indentation is key and you want to bring the current line into the current scope, it will just autofill the copilot suggestion which you then have to remove again. So annoying.
DethNinja 4y ago
There is a real need for offline co-pilot alternative. Privacy concerns due to copilot makes it unusable for companies with closed source code. I’m hoping that eventually there will be a stable diffusion level solution in this field.
- asdff 4y ago
  For some companies, maybe they have a large enough codebase of boilerplate functions that they can train their own models on. Oracle's own private co pilot instance or something like that, just pumping out more and more corporate code.
  - telotortium 4y ago
    Google does: https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html https://ai.googleblog.com/2022/07/ml-enhanced-code-completio...
jacobsenscott 4y ago
This is as credible as Ford doing a survey and finding out people got way more done with Ford trucks. I found copilot to be nothing but a hinderance and additional cognitive load. Not only do you need to think about the code you want, you need to review the (usually wrong) code copilot produces and either reject it or edit it to make it right. This is a lot more mental overhead than just writing the code.
- epolanski 4y ago
  I donk't like copilot. I don't like sending MS my code, I don't trust it to not violate copyright, and from what I used it, it feels like using dall-e trying to phrase what I expect. Moreover writing boilerplate is rarely my daily activity, when it is, it's faster to copy paste from a similar file. On the other hand, few exceptional devs I know, the kind of rockstar 10x engineer, insanely productive and very smart about product and business, not just great coders, people that were leads in faang 15 years ago already and are responsible for high impact quality sooftware, speak very highly of it and how better it makes them, which crushes my doubts about copilot.
  - jacobsenscott 4y ago
    As a 10x engineer with 24 years of experience, I stand by my opinion that it is meh at best.
    - DoesntMatter22 4y ago
      Are you actually saying you are a 10x developer? Like you are way better than average? Because that pretty much kills your credibility right there. "Oh I'm right because I'm so great". If you are indeed a 10x developer I'd love to see some of the great code you've written.
      - ipaddr 4y ago
        10x developers are not modest people
siraben 4y ago
I'm doubtful that the results from the experimental process can be used to draw such generalizing conclusions about the utility of Copilot. The task (implementing a HTTP web server in JS) is already something that would be over-represented in public codebases, this is like asking participants to implement two sum or a sudoku solver, of course the language model would have essentially memorized these types of problems.
habitue 4y ago
Any thread on copilot always seems to get a lot of comments about how it makes dumb mistakes. I guess I'm used to Intellisense etc making dumb suggestions all the time? I don't dwell on it for a second, either the computer makes a good suggestion that was what I was about to do and I accept the suggestion, or it doesn't and I don't accept it. I'm certainly not going to sit there and stew over how dumb it was to suggest such a thing, it's a damn computer! Cognitive load is massively decreased, I don't have to push much on the working memory stack to do small utility functions, they just write themselves now (especially if I write the type signature first). I can spend a lot more of my time at the middle level of abstraction, and keep moving.
- saurik 4y ago
  Earlier in my career--back when I still relied on Intellisense--I never really thought of it as a "suggestion" that could be "wrong"?... it is a tab completion mechanism, similar to how you can tab complete filenames at the shell, and that is extremely useful if you don't exactly remember the name of the function/file you are looking for--hell: I use tab completion in bash as a replacement for ls in most cases!--but it isn't something where the thing it generates somehow looks right but is actually a security vulnerability.
  - habitue 4y ago
    Yeah, I mean if you don't remember if it's `strlen`, `strnlen` when autocompleting, you can introduce a security vulnerability with intellisense just "tab completing" This is contrived, but in general (programmer + fuzzy memory + computer suggestion) means there's a possibility for it to be wrong, and you have to rely on the type system and reviewing it carefully to make sure there wasn't a mistake.
fezfight 4y ago
It should be free since it was trained using free software.
ktzar 4y ago
To me it's an added level of attention required, having to be extra careful (as I'd be when reviewing a junior's code) with any code suggestion copilot suggests... even if it's wrong 10% of the time, I need to be on the look for lines that would just cause a bug down the line. Ended up disabling it.
isatty 4y ago
I've not used it, and probably never will, but why do people want non deterministic snippets again? Seems like popularizing snippets wouldve gone further but then you miss out on the buzzwords.
- johnfn 4y ago
  Huh? Maybe you should try it out and see. It's nothing like snippets.
baby 4y ago
I don't understand all the negative comments, my experience with Copilot is just insane, it really is autocomplete on steroids. It's hard to fathom that some people have a bad experience with it.
xs83 4y ago
I wasn't that bothered about it until I used it. The biggest thing for me is when working with Terraform and some really badly documented parts of it and really not being able to work out the syntax required, the suggestions really helped expedite what would otherwise be a lot of iteration. What would be great would be for an enterprise pricing capability, we would love to purchase this on behalf of our employees via a single interface.
Stamp01 4y ago
mhils 4y ago
Anecdata: Copilot has been net-negative for me when writing Python, but the Rust completions are fantastic.
deleted 4y ago
[deleted]
wokwokwok 4y ago
This seems like something obvious, but I didn't really appreciate it until I was actually using copilot day to day. When you're using it on a code base that it's familiar with, it'll often suggestion quality of life suggestions that you may not know about. For example, using unity, did you know Mathf.RoundToInt Was a thing? I didn't, until I was using a normal way of doing it (eg. `(int) Math.Round(v);` and copilot popped this suggestion up. I had similar experiences using opencv in python. Maybe the suggestions aren't always perfect, but what it does do, is show you how other people have written code using the same engine/plugin/library you're using; and when you're learning a new API, that's extremely valuable. The alternative (finding github projects using library X, browsing through the code) lets you do the same thing too, but its much much slower to pickup the same 'tips' about an API that way. Specifically with regard to "give people a new task they haven't done before in language X with/without copilot", this kind of 'tip' for using an unfamiliar API seems high plausible for getting developers up to speed. I wish it had support for doing this in a way that was 'favour suggestions like existing code base' to help on-board new developers. That would be really really useful.
abrax3141 4y ago
Maybe I’m weird, but I kindof like programming. Even my own “boiler plate”. And good coding practices combined with well factored code design and planning (even in the micro-moment to moment sense of design and planning) usually avoids most “boiler plate”. (“Boiler plate” is usually a misnomer in modern well-designed languages. If the pl or ps makes you write something out, it usually is bcs you have the opportunity/obligation to think about it and usually to tweak it. If it was actually always the same, it would have good defaults and wouldn’t be there at all.)
IceDane 4y ago
ITT: A bunch of insecure clowns that somehow manage to feel personally threatened by a productivity tool. It's hilarious. Get over yourselves. Copilot is just an intelligent, context-aware search engine that is plugged directly into your code editing workflow. It's nothing more. It's the sci-fi version of having a hardcoded `site:stackoverflow.com` google search box right next to your editor. It's easily the nicest thing I've installed for productivity in a while, and it regularly amazes me what it can pick up from context. I can switch between code written in two different languages that is related, and copilot will often pick up enough context to give me correct suggestions in one language based on the code I just wrote in the other.
antihero 4y ago
Has anyone considered creating an extension like co-pilot that's trained on your own reddit/hn/twitter comments? Basically like an autocomplete for your own self.

115 points by idan 4y ago

NoraCodes 4y ago

I'll tell you right now, having my GPL-licensed code stolen has a pretty negative impact on my happiness.

JamesBarney 4y ago
What snippets were stolen from your code and how has it affected your happiness?
bastardoperator 4y ago
Maybe you should switch to the Unlicense. I'm so much happier not giving a damn or punishing my users with pettiness.
- NoraCodes 4y ago
  Licenses don't matter when Microsoft decides that AI training is fair use.
  - bastardoperator 4y ago
    I'll take "I didn't read the GitHub TOS" for 500 Alex.
    - throwaway1851 4y ago
      Well not everyone is in contractual privity with GitHub — like the poster a few up whose code was included in someone else’s repo. GitHub’s TOS have no effect in that circumstance.
elcomet 4y ago
What if I read your code? Now I have memory of it in my brain and it will influence my next projects. Is it stealing?
- xmcqdpt2 4y ago
  Actually it could be! If you read GPL code and remember it well, then write a near exact copy of it into your proprietary software project, that's a violation of the GPL.
sergnio 4y ago
"The GPL is a free software license, and therefore it permits people to use and even redistribute the software without being required to pay anyone a fee for doing so." How is GHC breaking this license?
- blowski 4y ago
  Say I drew a lovely picture and hung it outside my house for passers by to look at. I say “sure you can take a copy, no problem”, they then sell a billion prints, and don’t credit me at all. Legally, not stealing. Ethically? I’d call that stealing.
  - JohnHaugeland 4y ago
    Legally that actually is stealing. They must have documented reproduction rights. Misunderstanding a verbal comment will not fly in a court.
  - tducasse 4y ago
    Except that's not exactly what they are doing, is it? Following on your example, it'd be like selling the advice of someone who has seen a lot of these pictures passing by windows, with a very good memory and understanding of pictures. Github Copilot is not spitting out code verbatim, it's learning what code is, what shape it usually has, and trying to retrofit your own code onto the shape it thinks code should have. It's not like it's doing a query like `SELECT * FROM COPIED_CODE WHERE CODE STARTS_WITH "def my_func("`.
- NoraCodes 4y ago
  GPL requires that derivative works - which, I argue, Copilot is - also be distributed under the GPL. Microsoft disagrees with my interpretation, and wins by default because I can't afford to sue them.
- mtlynch 4y ago
  The GPL has requirements, and Copilot fails to adhere to those requirements. Here's an important one: >You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice
baby 4y ago
"wants to participate in and use open source, but is not happy when others use their code"
- jacooper 4y ago
  *Use their code without complying with the license
  - nojito 4y ago
    Copilot is transformative work.
    - throwaway1851 4y ago
      Objection, legal conclusion.
  - baby 4y ago
    I couldn't care less about whatever restrictive license people choose and decide to call open source
    - jacooper 4y ago
      Well then dont use their code you have to legally follow the codes license.
tducasse 4y ago
I have no opinion on whether there's infrigement or not, as I am not a lawyer, but I found the argument that the terms of use specify that you allow them to analyze your code pretty convincing: > We need the legal right to do things like host Your Content, publish it, and share it. You grant us and our legal successors the right to store, archive, parse, and display Your Content, and make incidental copies, as necessary to provide the Service, including improving the Service over time. This license includes the right to do things like copy it to our database and make backups; show it to you and other users; parse it into a search index or otherwise analyze it on our servers; share it with other users; and perform it, in case Your Content is something like music or video.
- paulryanrogers 4y ago
  Copilot seems to go further than all that. It can suggest verbatim, nontrivial, copyrighted work with no means of attribution. And the sources are so broad no human alive could say with certainty any given output is not infringing, unless it's so trivial it cannot be copyrighted.
  - tducasse 4y ago
    I've always had trouble understanding intellectual property when it comes to code, because if I read open source code, remember everything, and write my own version (even if slightly different) 10 years later, I am not 100% sure whether it's copyright infrigement or not. I see it as something akin to a painter studying someone else's work, then reproducing some of the techniques invented by the original artist... except that their techniques didn't have a license I guess? I think I am not equipped to fully understand the legal boundaries between inspiration and theft, and I think most programmers are in the same situation.
    - paulryanrogers 4y ago
      Painter analogy falls down when you consider Copilot can output verbatim chunks. So more like a photo of a painting or a stroke-for-stroke copy, even if portions changed or only parts taken. Now if changes are so significant it becomes impossible to recognize the reference then that could be legit. Though based on how Copilot works I don't think that can be assumed, or even proven. When I studied art the teachers usually taught us to begin with our own original photograph, ideally without trademarked items, for reference. And even in literature or journalism one must quote sources, even if paraphrased. I was taught to put down any inspiring work and only begin my own work after taking a break, to reduce the likelihood of unintentionally copying the original.
asdff 4y ago
I wonder if you could just ruin these systems by having some bots constantly upload bugged code into github
- jrockway 4y ago
  I think my biggest worry about AI is that we hit a big wall right about now because training data is now contaminated with AI outputs. That's why OpenAI puts watermarks on images, I'm guessing, but Stable Diffusion doesn't, and the free one is probably going to get a lot more usage than the paid-for one. But, maybe not, because humans liked the outputs enough to share them and put them on the Internet, so maybe they don't have much of an effect quite yet.
  - asdff 4y ago
    I think the training limits other applications like self driving cars a lot too. Training is frequently done in contrived scenarios. Now, imagine if GM or Ford put sensors and telemetry for future self driving car use on their models decades ago. You'd have practically every road mapped with ten thousand observations a piece by now. It wasn't for lack of forsight to not collect these data, it was really for lack of profit. Sensors and other data collecting systems are not new technology. Neither is machine learning, its also decades old. The only thing different these days is that the compute required to run these models on these data has gotten cheaper, but you could have been collecting data the whole time with anticipation of cheap compute to run these models in the future. Then you'd have a leg up on everyone in the business with your massive dataset of decades of real world, not test tracks and other such situations utilizing an employed driver in all likelihood.

chank 4y ago

I ended up disabling it in the beta. It ended up getting in the way more often than not.

AzzieElbab 4y ago
Out of curiosity, what programming language you use, and what type of code you write?
- mcluck 4y ago
  Not OP but I did the same thing. I build web apps using some combination of TypeScript, C#, F#, HTML, and CSS (or CSS-style languages). I spent about as much time correcting what it wrote as if I had written it myself. As someone else mentioned, it also interfered with proper inference given by the type system. Any time I wrote a more complicated algorithm (I work with a lot of graphs) I couldn't trust what it gave me. It's far easier to just gloss over some incorrect detail when the "shape" of the code is correct.
- nprateem 4y ago
  I turned it off after it kept outputting go that looked feasible but wasn't valid because of the libraries I was or wasn't using. I only turn it on if I'm doing something mundane that it might be able to handle. The fact it overrides intellij's autocompletion was too annoying
cesarvarela 4y ago
Same. Felt like pairing with a drunk programmer.
- verdverm 4y ago
  Did it ever repeat "developers!" loudly?
  - lopatin 4y ago
    I always figured those guys were on something else other than booze.

jeffbee 4y ago

"and timed how long it took them to write an HTTP server in JavaScript" OK ... interesting choice. This could mean almost anything. I can "write a web server" in a python 1-liner in about 2 seconds (ctrl-r SimpleHttpServer, enter).

mgkimsal 4y ago
There was a test suite to work against, so some standard notion of things that needed to be in place to be 'working'.
idan 4y ago
We'll be sharing some more about how we conducted this in an upcoming post! But no, the task was actually to write a little engine to process a limited subset of HTTP requests and return a valid response or relevant error.
- verdverm 4y ago
  Would you pit it against my non-ai solution for the same problem? As a baseline to measure against? It would seem prudent to compare against other tools trying to alleviate the same problem, rather than just humans writing the code by hand https://github.com/hofstadter-io/demos/tree/main/full-stack-app https://github.com/hofstadter-io/demos/tree/main/full-stack-... I've since shortened that demo to be even faster for a user to complete
blibble 4y ago
it's a profoundly stupid task to measure to measure developer productivity too the hard bit of being a developer is deciding in precise, pedantic, utterly exact detail on what needs building to solve the business problem not bashing out the code in the editor now if they had given out a vague problem that ended up with the dev eventually producing a webserver in Javascript, then fine but then the difference copilot makes would be a much smaller percentage of the development effort I swear we're going back to the age of "lines of code written per hour", just now with arTiFiCiAL InTellIgEncE
- 4y ago

adamddev1 4y ago

I mostly enjoyed my Copilot trial but one thing that really annoyed me was that it would often make suggestions for strings/keys that would be decent guesses by the AI but would be obvious type errors in TypeScript. Then it seemed a bit tricky to fight through the incorrect suggestion and get the old TypeScript Intellisense which was of course correct and what I wanted. Other suggestions by Copilot were awesome but when there was an obvious job for TypeScript it would be nice if Copilot would step back. For example I would often type: if (myTypedVar === "" <-- Back off Copilot, let TypeScript Intellisense give me the correct suggestions. Or at least run the Copilot suggestions through TS first and make sure it doesn't suggest type errors.

matthewmacleod 4y ago
Indeed this is one of the more annoying features of the integrations I’ve seen. Running through the type-checker is an interesting idea - even if you didn’t reject suggestions outright (which seems like it could frequently reject suggestions you actually wanted) it may be a useful weighting function. Similarly, running through eslint (or whatever linting system is appropriate) to avoid introducing obvious fixable formatting errors, like incorrect quotes, would be wonderful.
capableweb 4y ago
Sounds like it was your Copilot integration with whatever editor you used that was bad, not Copilot itself. The neovim integration worked very well, at least when I used it with Clojure and associated tooling. Could move seamlessly between what my Clojure tooling provides me with, and what Copilot did.

abdusco 4y ago

I liked using it. Though, most of the time, it was limited, in the sense that it could only offer completions using the code before the cursor as the context. I found it most helpful in setting up boilerplate in when writing tests. It could offer testing some edge cases before I could think of them.

wilg 4y ago

I use Copilot constantly and it's one of my favorite developer tools ever.

davidkuennen 4y ago
Same here. It's such a great tool and productivity boost. Don't want to miss it anymore.

avg_dev 4y ago

I read the article and I have some skepticism. I think my skepticism is well-founded but it may well be the case that a machine will one day do my job. I don't believe that time is at hand, though. First off, I don't see a link to the "HTTP server in JavaScript" task. It's really hard for me to place much faith in their conclusions when it's not even clear what the problem definition was. Second, I believe that a lot of more senior developers and development managers who take secure development practices somewhat seriously will not be able or willing to use Copilot in any sort of proprietary setting. Here is a quote from the Copilot FAQ: > [...] The GitHub Copilot extension sends your comments and code to the GitHub Copilot service, and it relies on context, as described in Privacy below - i.e., file content both in the file you are editing, as well as neighboring or related files within a project. It may also collect the URLs of repositories or file paths to identify relevant context. The comments and code along with context are then used by OpenAI Codex to synthesize and suggest individual lines and whole functions. - from https://github.com/features/copilot/#faq https://github.com/features/copilot/#faq - see "How does GitHub Copilot work?" I believe this makes it simply a nonstarter in a lot of environments. I am wondering if there are a number of places that have restrictions on sharing their code with a third-party but don't know or don't care and so end up using Copilot anyway. I believe that short-sighted thinking like this is more prevalent in shops that have low-quality code, and I believe that the higher-quality the code, the less likely someone is to use Copilot, simply for the "I can't share my code, even if I use the most restrictive no-telemetry settings" reason. Give me a self-hosted Copilot, and I may try it out in anger. Finally, I based some of my thinking on a recent Reddit /r/programming discussion of Copilot: https://old.reddit.com/r/programming/comments/wsnend/since_github_copilot_free_preview_ends_soon_what/ https://old.reddit.com/r/programming/comments/wsnend/since_g... After reading those posts, and internalizing them with my own view of coding, I believe Copilot is not ready for my personal use. Again: licensing considerations aside (if you actually can feel comfortable putting them aside, see NoraCodes comment in this HN thread e.g.), it is simply a non-starter for anything proprietary in nature. I am also of the mind that any code that is of necessity very tedious to write is in dire need of real attention, most likely in the form of tests and quite possibly refactoring to reduce the boilerplate if at all possible. I believe in the value of linters and automated code analysis tools and in continuous integration that runs after every commit. Give me a self-hosted Copilot, and we'll have a real chance to see how it works out - until then it's not going to be a boon to programmers.

_____s 4y ago

I really love Copilot. I had a trial, let it expire because I thought I wouldn't really need it, and then ended up paying for it. It's apparently got even better. For me, at least, it gets most things very right and takes a lot of mundanity out of day to day development!

xwolfi 4y ago
I work in Finance, if I type a certain client name out in public, I can be fired for it. Can copilot work without the internet and within only a corporate network ?
- avg_dev 4y ago
  Not presently per their FAQ. > [...] The GitHub Copilot extension sends your comments and code to the GitHub Copilot service, and it relies on context, as described in Privacy below - i.e., file content both in the file you are editing, as well as neighboring or related files within a project. It may also collect the URLs of repositories or file paths to identify relevant context. The comments and code along with context are then used by OpenAI Codex to synthesize and suggest individual lines and whole functions. - from https://github.com/features/copilot/#faq https://github.com/features/copilot/#faq - see "How does GitHub Copilot work?"
- skneko 4y ago
  You might be interested in IntelliCode from Visual Studio. It's an AI-guided autocompletion tool, like Copilot, although dumber and less powerful. It can run in local mode [0], looking at your existing codebase (opened solution) to feed future suggestions. [0]: https://docs.microsoft.com/en-us/visualstudio/intellicode/overview#telemetry https://docs.microsoft.com/en-us/visualstudio/intellicode/ov...
- speedgoose 4y ago
  No, you can already use offline alternatives like tabnine but it's not as good. But I guess better opensource models and tools will be available eventually.

valenterry 4y ago

How's Copilot with really good statically typed languages like Haskell? I haven't tried it out because most comments say it doens't even work well with typescript yet...

speedgoose 4y ago
I use it almost daily with typescript and it works well. It will guess wrong some types once in a while but it's still a very nice help.
siraben 4y ago
It is not even remotely usable with languages like Haskell, the code it generates almost never typechecks, IME. I think the primary reasons being lack of training data (there is far more Python code on GitHub than the entirety of Hackage) and that Copilot is essentially a language model that does not consider AST and type information at all.

gamegoblin 4y ago

The place where Copilot shines most for me is boilerplate like: up = foo.north() right = and then it will correctly suggest up = foo.north() right = foo.east() down = foo.south() left = foo.west() This requires a kind of linguistic understanding that is beyond basic intellisense. If you are just wanting it for basic intellisense autocomplete, it's not much better than JetBrains stuff. But for things that require linguistic comprehension, it's incredibly useful. For example, if I have a `Point` object with `x` and `y` fields, and I make a function called `lowest_point`, it knows I want to the point with the minimum `y` value. Because it knows the name `y` is most typically used for vertical direction, and the word `lowest` also implies vertical direction. This is kind of a linguistic/semantic knowledge that is not found in just the type system.

onemiketwelve 4y ago
Especially boilerplate in a language or framework you don't usually use. It's such a time saver.
- jlarocco 4y ago
  I've never used Copilot, but are you worried it will only spit out: up = foo.north() right = foo.east() down = foo.south() And you won't notice?
  - antihero 4y ago
    Not really no
    - richbell 4y ago
      Why not? From what I've seen GTP3 works exceptionally well until it doesn't, which can be easily missed.
      - chrisseaton 4y ago
        But I read the code, and I run tests.

j-sizz 4y ago

Wish the auth was better

noizz 4y ago

I spent most of the trial hating Copilot and the way takes over my Tab key in Jetbrains software. It got in the way a bit too much. Every time I tried to use emmet shortcuts, live template triggers or anything muscle memory related, it would bring something else, so I had to undo, press Esc a couple of times and then continue with whatever I actually wanted. However, every now and then, there came a typed object or something mundane and Copilot exceled at it, so I decided that I want to find a way for it to coexist in a way that's as easy to use as pressing Tab to expand it. Lo and behold - I found an unused key below Tab (AKA caps lock), remapped it and now it doesn't get in the way and I can use Copilot's help only when I want.

mccorrinall 4y ago
In pycharm where indentation is key and you want to bring the current line into the current scope, it will just autofill the copilot suggestion which you then have to remove again. So annoying.

DethNinja 4y ago

There is a real need for offline co-pilot alternative. Privacy concerns due to copilot makes it unusable for companies with closed source code. I’m hoping that eventually there will be a stable diffusion level solution in this field.

asdff 4y ago
For some companies, maybe they have a large enough codebase of boilerplate functions that they can train their own models on. Oracle's own private co pilot instance or something like that, just pumping out more and more corporate code.
- telotortium 4y ago
  Google does: https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html https://ai.googleblog.com/2022/07/ml-enhanced-code-completio...

jacobsenscott 4y ago

This is as credible as Ford doing a survey and finding out people got way more done with Ford trucks. I found copilot to be nothing but a hinderance and additional cognitive load. Not only do you need to think about the code you want, you need to review the (usually wrong) code copilot produces and either reject it or edit it to make it right. This is a lot more mental overhead than just writing the code.

epolanski 4y ago
I donk't like copilot. I don't like sending MS my code, I don't trust it to not violate copyright, and from what I used it, it feels like using dall-e trying to phrase what I expect. Moreover writing boilerplate is rarely my daily activity, when it is, it's faster to copy paste from a similar file. On the other hand, few exceptional devs I know, the kind of rockstar 10x engineer, insanely productive and very smart about product and business, not just great coders, people that were leads in faang 15 years ago already and are responsible for high impact quality sooftware, speak very highly of it and how better it makes them, which crushes my doubts about copilot.
- jacobsenscott 4y ago
  As a 10x engineer with 24 years of experience, I stand by my opinion that it is meh at best.
  - DoesntMatter22 4y ago
    Are you actually saying you are a 10x developer? Like you are way better than average? Because that pretty much kills your credibility right there. "Oh I'm right because I'm so great". If you are indeed a 10x developer I'd love to see some of the great code you've written.
    - ipaddr 4y ago
      10x developers are not modest people

siraben 4y ago

I'm doubtful that the results from the experimental process can be used to draw such generalizing conclusions about the utility of Copilot. The task (implementing a HTTP web server in JS) is already something that would be over-represented in public codebases, this is like asking participants to implement two sum or a sudoku solver, of course the language model would have essentially memorized these types of problems.

habitue 4y ago

Any thread on copilot always seems to get a lot of comments about how it makes dumb mistakes. I guess I'm used to Intellisense etc making dumb suggestions all the time? I don't dwell on it for a second, either the computer makes a good suggestion that was what I was about to do and I accept the suggestion, or it doesn't and I don't accept it. I'm certainly not going to sit there and stew over how dumb it was to suggest such a thing, it's a damn computer! Cognitive load is massively decreased, I don't have to push much on the working memory stack to do small utility functions, they just write themselves now (especially if I write the type signature first). I can spend a lot more of my time at the middle level of abstraction, and keep moving.

saurik 4y ago
Earlier in my career--back when I still relied on Intellisense--I never really thought of it as a "suggestion" that could be "wrong"?... it is a tab completion mechanism, similar to how you can tab complete filenames at the shell, and that is extremely useful if you don't exactly remember the name of the function/file you are looking for--hell: I use tab completion in bash as a replacement for ls in most cases!--but it isn't something where the thing it generates somehow looks right but is actually a security vulnerability.
- habitue 4y ago
  Yeah, I mean if you don't remember if it's `strlen`, `strnlen` when autocompleting, you can introduce a security vulnerability with intellisense just "tab completing" This is contrived, but in general (programmer + fuzzy memory + computer suggestion) means there's a possibility for it to be wrong, and you have to rely on the type system and reviewing it carefully to make sure there wasn't a mistake.

fezfight 4y ago

It should be free since it was trained using free software.

ktzar 4y ago

To me it's an added level of attention required, having to be extra careful (as I'd be when reviewing a junior's code) with any code suggestion copilot suggests... even if it's wrong 10% of the time, I need to be on the look for lines that would just cause a bug down the line. Ended up disabling it.

isatty 4y ago

I've not used it, and probably never will, but why do people want non deterministic snippets again? Seems like popularizing snippets wouldve gone further but then you miss out on the buzzwords.

johnfn 4y ago
Huh? Maybe you should try it out and see. It's nothing like snippets.

baby 4y ago

I don't understand all the negative comments, my experience with Copilot is just insane, it really is autocomplete on steroids. It's hard to fathom that some people have a bad experience with it.

xs83 4y ago

I wasn't that bothered about it until I used it. The biggest thing for me is when working with Terraform and some really badly documented parts of it and really not being able to work out the syntax required, the suggestions really helped expedite what would otherwise be a lot of iteration. What would be great would be for an enterprise pricing capability, we would love to purchase this on behalf of our employees via a single interface.

Stamp01 4y ago

mhils 4y ago

Anecdata: Copilot has been net-negative for me when writing Python, but the Rust completions are fantastic.

deleted 4y ago

[deleted]

wokwokwok 4y ago

This seems like something obvious, but I didn't really appreciate it until I was actually using copilot day to day. When you're using it on a code base that it's familiar with, it'll often suggestion quality of life suggestions that you may not know about. For example, using unity, did you know Mathf.RoundToInt Was a thing? I didn't, until I was using a normal way of doing it (eg. `(int) Math.Round(v);` and copilot popped this suggestion up. I had similar experiences using opencv in python. Maybe the suggestions aren't always perfect, but what it does do, is show you how other people have written code using the same engine/plugin/library you're using; and when you're learning a new API, that's extremely valuable. The alternative (finding github projects using library X, browsing through the code) lets you do the same thing too, but its much much slower to pickup the same 'tips' about an API that way. Specifically with regard to "give people a new task they haven't done before in language X with/without copilot", this kind of 'tip' for using an unfamiliar API seems high plausible for getting developers up to speed. I wish it had support for doing this in a way that was 'favour suggestions like existing code base' to help on-board new developers. That would be really really useful.

abrax3141 4y ago

Maybe I’m weird, but I kindof like programming. Even my own “boiler plate”. And good coding practices combined with well factored code design and planning (even in the micro-moment to moment sense of design and planning) usually avoids most “boiler plate”. (“Boiler plate” is usually a misnomer in modern well-designed languages. If the pl or ps makes you write something out, it usually is bcs you have the opportunity/obligation to think about it and usually to tweak it. If it was actually always the same, it would have good defaults and wouldn’t be there at all.)

IceDane 4y ago

ITT: A bunch of insecure clowns that somehow manage to feel personally threatened by a productivity tool. It's hilarious. Get over yourselves. Copilot is just an intelligent, context-aware search engine that is plugged directly into your code editing workflow. It's nothing more. It's the sci-fi version of having a hardcoded `site:stackoverflow.com` google search box right next to your editor. It's easily the nicest thing I've installed for productivity in a while, and it regularly amazes me what it can pick up from context. I can switch between code written in two different languages that is related, and copilot will often pick up enough context to give me correct suggestions in one language based on the code I just wrote in the other.

antihero 4y ago

Has anyone considered creating an extension like co-pilot that's trained on your own reddit/hn/twitter comments? Basically like an autocomplete for your own self.