33 ms·

Copilot regurgitating Quake code, including sweary comments

• influx 5y ago
  I get why marketing calls machine learning “AI”. I don’t get why engineers would think this is. Dumb.
  • konfusinomicon 5y ago
    it's the marketing magic bullet. each person shot is entranced by its promises, and given unlimited ammo to spread its lies. few possess armor capable of stopping them
  • SEMW 5y ago
    > I don’t get why engineers would think this is. This claim that "AI" only means artificial general / human-equivalent intelligence completely ignores the long history of how that term has been used, by computer science researchers, for the last 70-odd years, to include everything from Shannon's maze-solving algorithms, to Prolog-y systems, to simple reinforcement learning, and so on. https://web.archive.org/web/20070103222615/http://www.engagingexperience.com/2006/07/ai50_ai_past_pr.html https://web.archive.org/web/20070103222615/http://www.engagi... It's true that there has been linguistic drift in the direction of the definition getting narrower (to the point where it's a joke that some people use 'AI' to mean whatever computers can't do _yet_). And you can have reasons to prefer your own very-narrow definition. But claiming that your own definition is the only valid one to the point that anyone using a wider definition (one that has a long etymological history, and which remains in widespread use) are "dumb" is... not how language works.
    • influx 5y ago
      It hasn't been AI the entire time. It's borderline fraud, tbh.
  • squeaky-clean 5y ago
    I still consider anything with more than 3 if-statements to be AI. We just need more sensible expectations about what AI can do haha.
  • rvz 5y ago
    Exactly. The clever OpenAI marketing hype squad on HN and Twitter know that they are re-selling a snake oil contraption. This 'thing' completely needs assistance from a human since it is producing insecure code, code that is also copyrighted and most of the times garbage from other sources, which is again totally dangerous. Just look at this [0] Do a simple 'typo' in the signature and the whole implementation is wrong. I have to say that OpenAI, GitHub and Microsoft are very clever in selling this scam to engineers who use the code produced by this contraption as 'safe to use' in their projects; especially since GPT-3 still cannot explain why it is generating the code its generating, or if the code is under a license that is non-commercial or under a restrictive licence. No thanks and most certainly no deal. [0] https://twitter.com/leifg/status/1411083360756146177 https://twitter.com/leifg/status/1411083360756146177
• rasz 5y ago
  "0.1% of the time" indeed
• thinkingemote 5y ago
  From the GPLv2 licensed code: https://github.com/id-Software/Quake-III-Arena/blob/master/code/game/q_math.c#L552 https://github.com/id-Software/Quake-III-Arena/blob/master/c... copilot repeats it word for word almost, including comments, and adds an MIT like license up the top
  • Thomashuet 5y ago
    Actually the indentation of the first comment and the lack of preprocessor show it's not copied from this code directly but from Wikipedia (https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overview_of_the_code https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overv...) So It could be that the Quake source code is not part of the training set but the Wikipedia version is.
    • SamBam 5y ago
      While I strongly doubt they would use Wikipedia as a training set, has anyone done a search of GitHub code to see if other projects have copied-and-pasted that function from Wikipedia into their more-permissive codebases?
      • ajayyy 5y ago
        It is probably based off GPT-3 with a layer on top trained for programming specifically, like what is done with AI dungeon.
        • an_opabinia 5y ago
          Wait until people on the toxic orange site find out what has happened to AI Dungeon.
          • SamBam 5y ago
            I'm out of the loop.
• rozularen 5y ago
  Guess someone had to try it
• louthy 5y ago
  This is utterly damning. I have already instructed my team that Copilot can never be used for our projects. Compromising the product because of unknowable license demands isn't acceptable in the professional world of software engineering. But if we put the licensing to one side for a moment... 1/ Everything I've seen it generate so far is 'imperative hell'. It is practically a 'boilerplate generator'. That might be useful for pet projects, smaller code bases, or even unit-test writing. But large swathes of application code looking like the examples I've seen so far is hard to manage. 2/ The boilerplate is what bothers me the most (as someone who believes in the declarative approach to software engineering). The future for programming and programming languages should be an attempt to step up to a higher level of abstraction, that has been historically the way we step up to higher levels of productivity. As applications get larger and code-bases grow significantly we need abstraction, not more boilerplate. 3/ As someone who develops a functional framework for C# [1], I could see Copilot essentially side-lining my ideas and my approach to writing code in C#. Not just style, but choice of types, etc. I wonder if the fall out of what is Copilot's 'one true way' of generating code was ever considered? It appears to force a style that is at odds with many who are looking for more robust code. At worst it will homogenise code "people who wrote that, also wrote this" - stifling innovation and iterative improvements in the industry. 4/ Writing code is easy. Reading and understanding code written by another developer is hard. Will we spend most of our time as code-reviewers going forwards? Usually, you can ask the author what their intentions were, or why they think their approach is the correct one. Copilot (as far as I can tell) can't justify its decisions. So, beyond the simple boilerplate generation, will this destroy the art of programming? I can imagine many juniors using this as a crutch, and potentially never understanding the 'why'. I'm not against productivity tools per se; it's certainly a neat trick, and a very impressive feat of engineering in its own right. I am however dubious that this really adds value to professional code-bases, and actively may decrease code quality over time. Then there's the grey area of licensing, which I feel has been totally brushed to one side. [1] https://github.com/louthy/language-ext https://github.com/louthy/language-ext
  • ethangk 5y ago
    This is a little off topic, but your framework looks really interesting! How come you opted for building a functional framework in C#, vs using F#? I couldn’t see anything in the README about what was specifically frustrating about F#? I ask because we’re looking at introducing it at my company.
• stefan_ 5y ago
  Even includes the commented out code. Clearly Copilot has gained a deep understanding of code and is not simply the slowest way to make a terrible, opaque search engine ever!
  • aj3 5y ago
    Does GitHub Copilot write perfect code? No. GitHub Copilot tries to understand your intent and to generate the best code it can, but the code it suggests may not always work, or even make sense. While we are working hard to make GitHub Copilot better, code suggested by GitHub Copilot should be carefully tested, reviewed, and vetted, like any other code. As the developer, you are always in charge. https://copilot.github.com/ https://copilot.github.com/ EDIT: the text above is a direct quote from the Copilot website
    • danparsonson 5y ago
      > ...may not always work, or even make sense... Naively, as someone who just heard of this - that sounds worse than useless. If you can't trust its output and have to verify every line it produces and that the combination of those lines does what you wanted, surely it's quicker just to write the code yourself?
      • aj3 5y ago
        Then write the code yourself. It's not like you're forced to use this demo.
        • danparsonson 5y ago
          Well, you're right. I was somehow expecting there might be a silver lining I'd missed but perhaps not.
        • cjaybo 5y ago
          Not exactly a confidence-inspiring reply from someone who just identified themselves as representing the project here!
• danuker 5y ago
  They could train on solely MIT-licensed code, and dump ALL the copyright notices of code used for training into a file. Problem solved.
  • rebolek 5y ago
    Have you read MIT license? It explicitly says: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
  • Uehreka 5y ago
    Plenty of people probably copy-paste GPL code with the comments and stick MIT on it. This kind of thing violates the GPL, but I’m pretty sure (IANAL) that such code is “fruit of the poison tree”, and if you then copy it, you too can be held responsible. Sure, you might not get caught, but it’s a rough situation if you do.
• bob1029 5y ago
  This is pretty clearly just a search engine with more parameters. I thought there was something more going on with copilot, but the fact that it is regurgitating arbitrary code comments tells me that there is zero semantic analysis going on with the actual code being pulled in.
  • thegeomaster 5y ago
    It is decidedly not "just a search engine with more parameters." Language models are just prone to repeating training examples verbatim when they have a strong signal with the prompt. Arguably, in this case, it is the most correct continuation.
  • saynay 5y ago
    It's more that the model is so large it is capable of memorizing a lot. This can be seen in other language models like GPT-3 as well. Comments, I suspect, will be more likely to be memorized since the model would be trained to make syntactically correct outputs, and a comment will always be syntactically correct. That would mean there is nothing to 'punish' bad comments.
    • username90 5y ago
      The model in this case is just a lossy compression of github, and you search that.
  • josefx 5y ago
    They openly claim it is an AI. What about the state of AI currently in use made you think that there was any intelligence behind it?
    • philistine 5y ago
      When do words lose their meaning? There's the word intelligence in the thing, and yet, no intelligence in the concept itself as known today.
• sydthrowaway 5y ago
  What causes this in a net? I’m guessing the RNN gets in a catastrophic state..
  • otabdeveloper4 5y ago
    > catastrophic state No, overfitting is the normal state for neural nets.
  • captainmuon 5y ago
    I would say overfitting - the net doesn't "understand" the code in any meaningful sense. It just finds fitting examples and jumbles them a bit. Understanding would mean to have an internal representation related to the intention of the user, the expected behavior, and say the AST of the code. My pessimistic interpretation of this and many other recent AI applications is that it is a "better markov chain".
    • LeanderK 5y ago
      a markov chain can have an internal representation related to the intention of the user. I guess this example just got copied a lot and is therefore included multiple times in the training data, forcing the network to memorise it. Neural networks always memorise things that appear too frequent. Memorized Artifacts in an otherwise working neural network is usually seen as a "bug" (since the training allowed the network to cheat), not as a proof that the network didn't generalise.
  • salawat 5y ago
    Neural nets aren't magic. You actually need quite a bit of complexity and modeling of interrelated problem spaces to get anything more than a childlike naivete or trauma savant-like mastery of one particular area with crippling deficiencies elsewhere.
  • michaelt 5y ago
    This is the network working as designed. I mean, if you wrote an autocomplete system for written english and asked it to complete the sentence "O Romeo, Romeo" what would you expect to happen? You'd expect it to complete to "O Romeo, Romeo, wherefore art thou Romeo?" - a very famous quote. How else could you produce the single right output for that unique input, other than memorising and regurgitating?
• klohto 5y ago
  I’m really dumbfounded by the Copilot team decision to not exclude GPL licensed code. Why was this direction chosen? Is the inclusion of GPL really worth the risk and potential Google v. Oracle lawsuit? I’d like to know the reasoning.
  • throwaway287391 5y ago
    Isn't it entirely possible that they did exclude GPL licensed code, but somebody somewhere has violated copyright and copy-pasted that snippet into non-GPL-licensed code that they trained on? They could try to trace every single code snippet they train on to its "true source" and use the license for that, but that's not very well-defined, and is a lot harder, and it's never going to be 100%.
    • ttt0 5y ago
      Nope. They did include GPL code. > Once, GitHub Copilot suggested starting an empty file with something it had even seen more than a whopping 700,000 different times during training -- that was the GNU General Public License. https://docs.github.com/en/github/copilot/research-recitation#github-copilot-quotes-when-it-lacks-specific-context https://docs.github.com/en/github/copilot/research-recitatio...
      • pessimizer 5y ago
        Looks like Copilot is smart enough to understand its own licensing situation. It should continue to suggest this for any empty file.
    • another-dave 5y ago
      Which raises another question: ideally Copilot wouldn't be trained on "somebody somwhere", but is that happening? To use the old trope — if the majority of programmers can't implement Fizzbuzz, but they do have a Github profile, are they being included too? Hopefully there's some quality bar for the training set, i.e. some subset of "good" code (e.g. release candidate tags from fairly established OSS tools/frameworks in different languages) rather than any old code on the internet.
• bencollier49 5y ago
  This does make me wonder if this is susceptible to the same form of trolling as that MS AI got. Commit a load of grossly offensive material to multiple repos, and wait for Copilot to start parroting it. I think they're going to need some human moderation.
  • NullPrefix 5y ago
    Coding with Adolf?
    • heavyset_go 5y ago
      Jojo Rabbit except Adolf is in the cloud and not in a kid's imagination.
      • mhh__ 5y ago
        YC submission when?
  • lawl 5y ago
    Way better. It's susceptible to copyright trolling. Put up repos with snippets for things people might commonly write. Preferably use javascript so you can easily "prove" it. Write a crawler that crawls and parses JS files to search for matching stuff in the AST. Now go full patent troll, eh, i mean copyright troll.
    • handrous 5y ago
      1) Write a project heavily using Copilot (hell, automate it and write thousands of them, why not?) 2) AGPL all that code. 3) Search for large chunks of code very similar to yours, but written after yours, licensed more liberally than AGPL. Ideally in libraries used by major companies. 4) Point the offenders to your repos and offer a "convenient" paid dual-license to make the offenders' code legal for closed-source use, so they don't have to open source their entire product. 5) Profit?
• 0-_-0 5y ago
  This is a very famous function [0] and likely appears multiple times in the training set (Google gives 40 hits for GitHub), which makes it more likely to be memorized by the network. [0]: https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overview_of_the_code https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overv...
  • 0-_-0 5y ago
    It's worth keeping in mind that what a neural network like this (just like GPT3) is doing is generating the most probable continuation based on the training dataset. Not the best continuation (whatever that means), simply the most likely one. If the training dataset has mostly bad code, the most likely continuation is likely to be bad as well. I think this is still valuable, you just have to think before accepting a suggestion (just like you have to think before writing code from scratch or copying something from Stack Overflow).
    • abecedarius 5y ago
      > the most probable continuation based on the training dataset This is not wrong, but it's easy to misread it as implying little more than a glorified Markov model. If it's like https://www.gwern.net/GPT-3 https://www.gwern.net/GPT-3 then it's already significantly cleverer, and so you should expect to sometimes get the kind of less-blatant derivation that companies aim to avoid using a cleanroom process or otherwise forbidding engineers from reading particular sources.
    • dematz 5y ago
      I have no idea how this or GPT3 works or how to evaluate them, but couldn't you argue that it's working as it should? You tell copilot to write a fast inverse square root, it gives you the super famous fast inverse square root. It'd be weird and bad if this didn't happen. As far as licenses go, idk. Presumably it could delete associated comments and change variable names or otherwise obscure where it's taking code from. Maybe this part is shady.
      • 0-_-0 5y ago
• LeanderK 5y ago
  I think the problem might be in the training data. Famous code examples are probably copied a lot and therefore appear multiple times in the training data, prompting the neural network to memorise it completely.
  • avian 5y ago
    Famous code examples are also much more likely to be noticed. For all I know, the thing might be spewing random GPL'd code from the long tail of GitHub all the time and nobody notices because it was written by some random guy and not John Carmack.
    • LeanderK 5y ago
      Well, it's sure speculation on my part what the root cause is, but i think OpenAI is already trying to ensure the network generalises. It's just common behaviour for neural network to memorise frequent samples, so I think my guess is quite realistic. I don't think OpenAI would not notice large-scale memorisation in their model. But as long as they don't publish more details it's just guesswork. Just keep in mind that it's a statistical tool. You can't really formally prove that it won't memorise, but I think with enough work you can get it unlikely enough that it won't matter. It's their first iteration.
      • visarga 5y ago
        Hash 10-grams and make a bloom filter. It will not generate more than 10 GPLed tokens from a source.
    • mkl 5y ago
      Carmack didn't write this: https://en.wikipedia.org/wiki/Fast_inverse_square_root https://en.wikipedia.org/wiki/Fast_inverse_square_root
  • cblconfederate 5y ago
    Is this function used verbatim in multiple projects? I know it's famous but how often does one use an approximation of inverse sqrt instead of the readily available cpu call in the past 20 years
• cblconfederate 5y ago
  Clearly, swearing is the only right way to write that function
• kklisura 5y ago
  Phew! Are jobs are safe!
  • unknownOrigin 5y ago
    snickers
  • CookieMon 5y ago
    though our companies will one day be competing with product manufacturers in China who get to use it to its fullest
• tyingq 5y ago
  They have 4 hand picked examples on their homepage: https://copilot.github.com/ https://copilot.github.com/ One has the issue with form encoding: https://news.ycombinator.com/item?id=27697884 https://news.ycombinator.com/item?id=27697884 The python example is using floats for currency, in an expense tracking context. The golang one uses a word ("value") for a field name that's been a reserved word since SQL-1999. It will work in popular open source SQL databases, but I believe it would bomb in some servers if not delimited...which it is not. The ruby one isn't outright terrible, but shows a very Americanized way to do street addresses that would probably become a problem later. And these are the hand picked examples. This product seems like it needs some more thought. Maybe a way to comment, flag, or otherwise call out bad output?
  • slver 5y ago
    > And these are the hand picked examples. This product seems like it needs some more thought. Everyone's self-preservation instincts kicking in to attack Copilot is kinda amusing to watch. Copilot is not supposed to produce excellent code. It's not even supposed to produce final code, period. It produces suggestions to speed you up, and it's on you to weed out stupid shit, which is INEVITABLE. As a side note, Excel also uses floats for currency, so best practice and real world have a huge gap in-between as usual.
    • mkr-hn 5y ago
      Have you met programmers? Even those who care about quality are often under a lot of pressure to produce. Things slip through. Before, it was verbatim copies from Stack Overflow. Now it'll be using Copilot code as-is.
      • slver 5y ago
        So, nothing new, is your point?
        • mkr-hn 5y ago
          Then why are you complaining? Unless something is new that warrants you getting mad about people getting mad at technology.
• ChrisMarshallNY 5y ago
  I've always assumed that we would eventually have a low-code, or no-code junior dev replacement, and was wondering if this was it. GH and MS actually have [Ed. had?] some cred for this kind of thing. Nope. Game over. Play again?
  • jgilias 5y ago
    Copying GPLed code as your own and passing it under an MIT license is not too far fetched of a thing for a junior dev to do. Jokes aside, to have a proper junior dev replacement you need something that is able to learn and grow to eventually become a senior dev, an architect, or a CTO. That is the most important value of a junior dev. Not the ability to produce subpar code.
    • ChrisMarshallNY 5y ago
      Depends on who you ask. I think a lot of modern software development shops, these days, exist only to make their founder[s] as rich as possible, as quickly as possible. If they are willing to commit their entire future to a lowest-bid outsourcing shop, then I don’t think they are too concerned about playing the long game. Also, the software development industry, as an aggregate, has established a pervasive culture, based around developers staying at companies for 18-month stints. I don’t think many companies feel it’s to their advantage to incubate people who will bail out, as soon as they feel they have greener pastures, elsewhere.
  • marcosdumay 5y ago
    Most low-code and no-code platforms go for junior dev empowerment, and senior dev replacement. This one also seems to be aimed at empowering juniors, but looks like it missed the senior replacement by miles.
    • deleted 5y ago
      [deleted]
• gok 5y ago
  Quake and GitHub are both owned by Microsoft now, perhaps we can assume this is relicense?
  • Jyaif 5y ago
    Wow, Quake is owned by Microsoft. This is mind blowing, and a little sad.
    • josefx 5y ago
      It belongs to id software -> ZeniMax -> Xbox Game Studios -> Microsoft.
  • johndough 5y ago
    Is it possible that Copilot just put Quake's source code into the public domain? From the Copilot FAQ: Who owns the code GitHub Copilot helps me write? GitHub Copilot is a tool, like a compiler or a pen. The suggestions GitHub Copilot generates, and the code you write with its help, belong to you, and you are responsible for it. We recommend that you carefully test, review, and vet the code, as you would with any code you write yourself. Copilot can probably recite most of Quake's source code and according to the FAQ, the output of Copilot belongs to the user. I think a point where this argumentation might fail is that Quake's source code does not belong to Github directly, but instead both Github and Quake belong to Microsoft. However, I am not a lawyer, so I might be wrong.
• tyingq 5y ago
  Just the MP4, since it's hard to read in the smaller size: https://video.twimg.com/tweet_video/E5R5lsfXoAQDRkE.mp4 https://video.twimg.com/tweet_video/E5R5lsfXoAQDRkE.mp4
• deleted 5y ago
  [deleted]
• celeritascelery 5y ago
  From the Copilot FAQ: > The technical preview includes filters to block offensive words And somehow their filters missed f*k? That doesn’t give a lot of confidence in their ability filter more nuanced text. Or maybe it only filters truly terrible offensive words like “master”.
  • minimaxir 5y ago
    In my testing of Copilot, the content filters only work on input, not output. Attempting to generate text from code containing "genocide" just has Copilot refuse to run. But you can still coerce Copilot to return offensive output given certain innocuous prompts.
    • Closi 5y ago
      Ahh, so it's the most pointless interpretation of the phrase "filters to block offensive words", where it is stopping the user from causing offense to the AI rather than the other way around.
      • raffraffraff 5y ago
        Easily offended AI is exactly what the world needs
        • GenerocUsername 5y ago
          We have too many easily offended NPC's as is.
          • Jordrok 5y ago
            Yeah. If only we could cure all those delusional righties fantisizing about being the protagonist in their own personal RPG. eyeroll
      • derefr 5y ago
        I believe the concept is to stop users from prompting the AI to generate offensive stuff specifically, and then publishing the so-generated stream of offensive stuff as negative PR for GitHub, in the same way the generated stream of offensive stuff coming from Microsoft’s AI was a big PR disaster.
• dgellow 5y ago
  Another fascinating one, an "About me" page generated by copilot links to a real person's Github and twittter accounts! https://twitter.com/kylpeacock/status/1410749018183933952 https://twitter.com/kylpeacock/status/1410749018183933952
  • bencollier49 5y ago
    That's bonkers. And the beauty of it is that now someone could realistically do a GDPR Erasure request on the Neural Net. I do hope that they're able to reverse data out.
    • qayxc 5y ago
      Since the information is encoded in model weights, I doubt that erasure is even possible. Only post-retrieval filtering would be an option. It only goes to show that intransparent black-box models have no place in the industry. The networks leak information left and right, because it's way too easy to just crawl the web and throw terabytes of unfiltered data at the training process.
      • bencollier49 5y ago
        > Only post-retrieval filtering would be an option. And illegal, if the original information remains. I assume that there must be a process for altering the training data set and rerunning the entire thing.
        • gmueckl 5y ago
          The problem is that the information is in an opaque encoding that nobody can reverse engineer today. So it's impossible to prove that a certain subset of data has been removed from the model. Say, you have a model that repeats certain PII when prompted in a way that I figure out. I show you the prompt, you retrain the model to give a different, non-offensive answer. But now I go and alter the prompt and the same PII reappears. What now?
        • computerex 5y ago
• Thomashuet 5y ago
  It seems like a very sensible answer from copilot since the prompt includes "Q_" which makes it obvious that the programmer is specifically looking for the Quake version of this function. To me it doesn't show that copilot will regurgitate existing code when I don't want it to, just that if I ask it to copy some famous existing code for me it will oblige.
  • the_mitsuhiko 5y ago
    I was able to trigger it without the Q prompt before. It just made the a nicer looking gif that way. I got it to produce more GPL code too, that one is just not entertaining.
  • cjaybo 5y ago
    Apparently you haven't seen many of the demos that people are showing off? Because saying that this only occurs when the author is explicitly asking for copied code is blatantly false.
    • Thomashuet 5y ago
      No I haven't. If you think the other demos are more interesting please link to them. I'm just saying that this demo is biased and that we can't draw any conclusion from it. Actually the author has just confessed optimizing it for entertainment in a sister comment. That doesn't mean that the claim is false but it doesn't show that it is true either.
      • the_mitsuhiko 5y ago
        I think you misunderstood my comment. The same code gets generated if you call the function `float fast_rsqrt` or `float fast_isqrt` for instance. I intentionally wanted it to be looking like `Q_rsqrt` so that people pick up on it quicker.
        • heavyset_go 5y ago
          Thanks for making the video in the OP. Do you have more examples like this that I can share with those who don't use Twitter, like a repo or blog post?
• littlestymaar 5y ago
  Copilot was only trained on public code right? Because that would be an absolute PR disaster if some proprietary code was leaked that way.
  • ezekg 5y ago
    I'm curious if you can find certain keywords that may leak private code...
    • imnitishng 5y ago
      Waiting for the access to test this out
• antpls 5y ago
  One has to admit, Copilot raises many questions regarding global code quality, reviewing processes and copyright. It's a marketing success.
  • qayxc 5y ago
    Honestly, I see this exact issue as the main accomplishment of Copilot. It shows that the black-box machines are to be considered harmful and are incompatible with the current intellectual property and privacy frameworks. This issue goes way beyond just code - imagine GPT-like systems being used in medical diagnosis and results can suddenly depend on the date of the CT-scan or the name of patient, because the black-box simply regurgitates training data...
• comodore_ 5y ago
  funny, the youtube algo blessed me with an in dept video (~1y old) about this quake function yesterday.
• ronenlh 5y ago
  There are a lot of good points made against copilot. But I’m optimistic in that it will improve with time. At worst it’s an efficient code copy-pasting tool, but at best it could be the next level of abstraction.
• GhostVII 5y ago
  Sounds like we need another tool called "Auditor" that scans your code to see if it violates copyright laws.
• anyonecancode 5y ago
  I think copilot is solving the wrong problem. A future of programming where we're higher up the abstraction tree is absolutely something I want to see. I am taking advantage of that right now -- I'm a decently good programmer, in the sense that I can write useful, robust, reliable software, but I'm pretty high up the stack, working in languages like Java or even higher up the stack that free me from worrying about the fine details of memory allocation or the particular architecture of the hardware my code is running on. Copilot is NOT a shift up the abstraction tree. Over the last few years, though, I've realized the the concept of typing is. Typed programming is becoming more popular and prominent beyond just traditional "typed" languages -- see TypeScript in JS land, Sorbet in Ruby, type hinting in Python, etc. This is where I can see the future of programming being realized. An expressive type system lets you encode valid data and even valid logic so that the "building blocks" of your program are now bigger and more abstract and reliable. Declarative "parse don't validate"[1] is where we're eventually headed, IMO. An AI that can help us to both _create_ new, useful types, and then help us _choose_ the best type, would be super helpful. I believe that's beyond the current abilities of AI, but can imagine that in the future. And that would be amazing, as it would then truly be moving us up the abstraction tree in the same way that, for instance, garbage collection has done. [1] https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/ https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...
  • shadowgovt 5y ago
    A taller abstraction tree makes tradeoffs of specialization: the deeper the abstractions, the more one has to understand when the abstractions break or when one chooses to use them in novel ways. This is something I'm interested in regarding this approach... When it works as intended, it's basically shortening the loop in the dev's brain from idea to code-on-screen without adding an abstraction layer that someone has to understand in the future to interpret the code. The result is lower density, so it might take longer to read... Except what we know about linguistics suggests there's a balance between density and redundancy for interpreting information (i.e. the bottleneck may not be consuming characters, but fitting the consumed data into a usable mental model). I think the jury's out on whether something like this or the approach of dozens of DSLs and problem-domain-shifting abstractions will ultimately result in either more robust or more quickly-written code. But on the topic of types, I'm right there with you, and I think a copilot for a dense type forest (i.e. something that sees you writing a {name: string; address: string} struct and says "Do you want to use MailerInfo here?") would be pretty snazzy.
• hi41 5y ago
  I saw the gif on Twitter. Sorry, I am not able to understand what is going on. Is copilot a character in the Quake game?
  • dpassens 5y ago
    Copilot seems to be an AI tool to generate code for you[0]. In the gif, it's copying code from Quake, which is GPLv2 or later. If copying GPLed code wasn't bad enough, it then adds a MIT-like license header. [0] https://copilot.github.com/ https://copilot.github.com/
• dredmorbius 5y ago
  Backstory? WTF is "Copilot"?
  • gregsadetsky 5y ago
    It's a new product launched by GitHub in association with OpenAI https://news.ycombinator.com/item?id=27676266 https://news.ycombinator.com/item?id=27676266
  • loloquwowndueo 5y ago
    “Your AI pair programmer” - auto-completes entire functions while you’re coding. https://copilot.github.com/ https://copilot.github.com/
    • dredmorbius 5y ago
      Thanks.
• deleted 5y ago
  [deleted]
• woah 5y ago
  It looks like the author of the linked tweet intended for it to reproduce the Quake code, by using the exact same function name and comment. Whatever the merits of CoPilot, in this case the human intended to write the quake function into their file, and put the wrong license on it.
• toss1 5y ago
  License Laundering Like Money Laundering for cash equivalents, or Trust-Washing for disinformation, but for hijacking software IP. It might not be the intended use case, but that winds up being the practical result. (on a related note, it would make me want to fun GPT-* output through plagiarism filters, but maybe they already do that before outputting?)
• fencepost 5y ago
  So what I'm reading here is that "Tay for code" is maybe going to need to be rethought and perhaps trained differently?
• Tarucho 5y ago
  Is Copilot aimed at programmers or at non-technical hiring managers? I mean, it goes right away with the devaluing narrative of programming that is going around from the last couple of years. To the "anyone can code" narrative we are adding "more so, if they have AI assisted Copilot"
• swiley 5y ago
  So I guess we can just copy around copyrighted source now? Great! Now we can share all the proprietary driver and DSP code from Qualcomm.
  • supernintendo 5y ago
    I wonder when someone will try to use the “it came from Copilot” defense to get away with stealing copyrighted code.
• axiosgunnar 5y ago
  I hate to be the one that says this but I think it‘s true: "So you are an SWE and you take a break from work to go to Hackernews to complain that Github's Copilot, which is an AI-based solution meant to help SWEs, is utter shit and completely unusuable. And then you go back to writing AI-based solutions for some other profession. Which is totally not shit or anything.“ Can anybody put this more elegantly?
  • SamBam 5y ago
    "'I never thought leopards would eat MY face,' sobs woman who voted for the Leopards Eating People's Faces Party."
  • skinkestek 5y ago
    You mean like the insanely annoying AIs that replaced Google search? The idiotic one that files Javascript books under "Law" in Amazon or the insulting one who runs Ad Sense and thinks my wife isn't good enough and I am stupid enough to leave her for some mail order bride?
    • donkeybeer 5y ago
      Javascript books under "Law" is hilarious
      • drdaeman 5y ago
        I'm in for JavaScript Penal Code. Make that unwarranted type-coercing operator use punishable by law.
        • axiosgunnar 5y ago
          Instead of prison you go to callback hell
        • deleted 5y ago
          [deleted]
      • 5y ago
• saiojd 5y ago
  Based on all the negative comments so far, and based on this website's aptitude at predicting the viability of a product, it really seems like Copilot is bound to be a success.
  • sydthrowaway 5y ago
    We're gonna Dropbox this thing all the way to the top!
  • DantesKite 5y ago
    Right? Even given all its initial problems, I don't see a world where people completely avoid using it.
  • stusmall 5y ago
    No wireless. Less space than a nomad. Lame.
    • saiojd 5y ago
      Yeah. I get why people's initial reaction is to dislike it tbh. Honestly I doubt the utility will be huge for experts, mostly likely it will just alleviate having to remember certain how a certain language implements a specific concept.
• ethbr0 5y ago
  The irony is that we're whinging about a tool that generates code that will be difficult to understand in the future... ... and the example is mathematically- and floating-point-spec obtuse enough that it was incomprehensible at the time it was written. (As evidenced by id comments)
• mkl95 5y ago
  Copilot is one of the worst ideas that have made it to production in recent years. I predict it will be quite successful considering Microsoft's track record.
• villgax 5y ago
  That salesforce engineer is the average developer when prompting to write an About me section lol
• dempsey 5y ago
  I’ve always wondered this about the realistic photo generators. How do we know they’re generating new faces and not just regurgitating ingested faces?
• hawski 5y ago
  Copilot may do more to move open source projects out of GitHub than the message that Microsoft is the buyer. Now you can host the code on GitHub to get your license violated, or DMCA-ed in a long run, when your code will become a part of some big proprietary project. At least it makes me think about my choice for code hosting more then whatever happened before.
• unknownOrigin 5y ago
  I'm honestly kinda amazed this as upvoted here as it is. Typically anything ML-related is upvoted to the top positions and any dissent harshly ridiculed. Anyways... it appears those who thought about this as if it was a glorified code search engine were close to being right.
  • qayxc 5y ago
    I still don't think it's just a glorified code search engine. Context-sensitive data retrieval is undoubtedly a part of it, though and the question is how big and relevant is that part and what are the consequences? To me the biggest issue is that it's impossible to tell whether the suggestions are verbatim reproductions of training material and thus problematic. It goes to show that this tool and basically every tool relying on the same or similar technology must now be assumed to do this and thus any code suggestion must be regarded plagiarism until proven otherwise. As a consequence such tools are now off-limits for commercial or open source development...
• ocdtrekkie 5y ago
  Probably an excellent reminder that both Google and Microsoft decided to use your private emails for a training set to create Smart Reply behavior that can "write emails for you", and they swore up and down there's no way that could ever leak private information. We need legislation banning companies from ingesting data into AI training sets without explicit permission.
  • nebuke 5y ago
    Makes me wonder if github are using private repos in their training data.
    • ocdtrekkie 5y ago
      GitHub clearly stated they only used publicly available repos in this project. However, as many people are rightfully pointing out, those projects might still be either closed source or copylefted, and if Copilot regurgitates chunks of those projects, people who use it may be subject to infringement lawsuits in the future.
• mattowen_uk 5y ago
  With apologies to Martin Niemöller[1]: First the automation came for the farmers, and I did not speak out — Because I was not a farmer. Then the automation came for the factory workers, and I did not speak out — Because I was not a factory worker. Then the automation came for the accountants, and I did not speak out — Because I was not a accountant. Then the automation came for me (a programmer) — and there was no one left to speak for me. --- [1] https://en.wikipedia.org/wiki/First_they_came_ https://en.wikipedia.org/wiki/First_they_came_...
  • flatiron 5y ago
    Honestly I’ve automated a large chunk of my day job. The trick is keeping it secret!
    • mattowen_uk 5y ago
      Well... wait until all the programmer salaries crash to minimum wage because Management believe that "CoPilot does most of the work anyway".
      • Rooster61 5y ago
        Then wait for them to realize how brittle the code is when nobody is considering the context into which this code is being foisted. They'll TRIPLE our salaries! :D
• crescentfresh 5y ago
  Direct url to the "gif" in that twitter post: https://video.twimg.com/tweet_video/E5R5lsfXoAQDRkE.mp4 https://video.twimg.com/tweet_video/E5R5lsfXoAQDRkE.mp4 I could not figure out how to show it larger on the twitter UI. I don't have a twitter account so that may be the problem.
• guhayun 5y ago
  Copilot NEEDs to be trained on licensed code,so that it doesn't produce them
• sdevonoes 5y ago
  Not a problem. Just don't use Copilot :)
• LeicaLatte 5y ago
  Curious if Microsoft is training Co-Pilot on my private repositories.
• yepthatsreality 5y ago
  Co-pilot is just lowest common denominator solutions with flashy tabbing.
• gumby 5y ago
  stack overflow at its automated finest. Or should we call it the Tesla of software?
• king_magic 5y ago
  Yep, Copilot is insanely poorly thought out. Astonishing they'd release something as half-baked as this.
• FlyingSnake 5y ago
  The rate at which these bots implode imply something about the whole AI/ML zeitgeist.
• maliker 5y ago
  Copilot transitions programmers from writing code to reading auto-generated code. And the feeling is that reading code is 10x harder than reading it? Seems like a rich source of problems. (However, I'm still definitely going to try this out once I get off the waitlist.)
• amelius 5y ago
  Co-pilot fixes the wrong problem. It should be a tool capable of one-shot learning. I.e., I'm in the middle of a refactoring operation and have to do lots of repetitive work; the tool should help me by understanding what I'm trying to do after I give it 1 example.
• deleted 5y ago
  [deleted]
• rgbrenner 5y ago
  So this makes it official... this post[0] and the comments on the announcement[1] concerned about licensing issues were absolutely correct... and this product has the possibility of getting you sued if you use it. Unfortunately for GitHub, there's no turning back the clocks. Even if they fix this, everyone that uses it has been put on notice that it copies code verbatim and enables copyright infringement. Worse, there's no way to know if the segment it's writing for you is copyrighted... and no way for you to comply with license requirements. Nice proof of concept... but who's going to touch this product now? It's a legal ticking time bomb. 0. https://news.ycombinator.com/item?id=27687450 https://news.ycombinator.com/item?id=27687450 1. https://news.ycombinator.com/item?id=27676266 https://news.ycombinator.com/item?id=27676266
  • eganist 5y ago
    Adding to this: I run product security for a large enterprise, and I've already gotten the ball rolling on prohibiting copilot for all the reasons above. It's too big a risk. I'd be shocked if GitHub could remedy the negative impressions minted in the last day or so. Even with other compensating controls around open source management, this flies right under the radar with a c130's worth of adverse consequences.
    • fragmede 5y ago
      Do you also block stack overflow and give guidance to never copy code from that website or elsewhere on the Internet? I'm legitimately curious - my org internally officially denounces the copying of stack overflow snippets. Thankfully for my role it's moot as I mostly work with an internal non-public language, for better or worse, and I have no idea how well that's followed elsewhere in the wider company.
      • mediaman 5y ago
        Who really copies stack overflow snippets verbatim? It's usually just easier to refer to it for help figuring out the right structure and then adapt it for your own needs. Usually it needs customization for your own application anyway (variables, class instances, etc).
• yk 5y ago
  So, what would happen if I train a neural network to recreate Disney movies?
  • avaldes 5y ago
    Isn't already?
• oiu45hunegn 5y ago
  This reminds me of an issue that came up when I was working with a intelligence agency, training machine translation. If you think about language in general, individual words aren't very sensitive. The word for bomb in any language is public knowledge. But when you start getting to jargony phrases, some might be unique to an organization. And if you're training your MT on translated documents surreptitiously intercepted from West Nordistan's nuclear program, and make your MT model public, the West Nordistanis might notice - "hey, this accurately translates our non-public documents that contain rather novel phrases ... I think someone's been listening to us!"
• KoftaBob 5y ago
  Seems like this is less of an "AI that intelligently generates code based on context given" and more of a "google search autocomplete for code".
• mouzogu 5y ago
  I can't even get intellisense to work correctly half the time.
• PaulHoule 5y ago
  This kind of thing devalues the work that programmers do. The head example they show is using a sentiment analysis API which is about the most useless use of technology there is.
• omgwtfbbq 5y ago
  The uproar over Copilot is kind of hilarious. Maybe it's SWE's realizing that they might not be as irreplaceable as they seem but its an awful lot of salty comments. If anything I think Copilot is a really cool PoC and shows just how close we are getting to automating large portions of the code writing process which we should all welcome as more cycles can be spent on architecture and system design.
• _tom_ 5y ago
  I’m reminded of the old saying: The best person to have on your team is a productive, high-quality coder. The worst is a productive, low-quality coder. Copilot looks like it would give us more of the latter.
• timdaub 5y ago
  I felt the need to write an article about this whole situation too: "Built on Stolen Data": https://rugpullindex.com/blog#BuiltonStolenData https://rugpullindex.com/blog#BuiltonStolenData
• greyfox 5y ago
  Very interesting that this was posted as I literally JUST watched an even MORE interesting youtube upload about this very bit of code just last weekend. Here's the very fun video if anyone wants to take a look: https://www.youtube.com/watch?v=p8u_k2LIZyo https://www.youtube.com/watch?v=p8u_k2LIZyo
• marcodiego 5y ago
  Now, consider quake is GPL'ed. Any proprietary software using such code will have to bow to the license terms.
• nightowl_games 5y ago
  Almost feels like a developer cultural thing to hate on something like this. If you dont like it, dont use it. If you dont want your team using it, become senior and then set the rules. Kinda seems like maybe there's some level of insecurity at play here in the criticism. Like a "I coulda came up with that but its a bad idea" type of hater philosophy.
• deleted 5y ago
  [deleted]
• rpmisms 5y ago
  I like tabnine. It's an autocomplete tool and doesn't pretend to be anything more.
• meling 5y ago
  What I would love even more than copilot helping me write code is a copilot to write my tests for the code I write.
• fourseventy 5y ago
  Ahh yes the infamous "evil floating point bit level hacking" code
• nadsat2021 5y ago
  "sweary comments" When I hear phrases like that, I worry more about human intelligence. "I'm a little tea pot, short and stout," said social media. I watched Kubrik's "A Clockwork Orange" again this week, after a certain amount of fearful anticipation. When Alex said "eggy weggie", it clicked. It's like Burgess time-traveled to 2021 to document our modern infantilization and antisocialization. He forgot to include the Internet, loss of humor, and emerging AI, but I guess he was overwhelmed by the enormity of the "baddiwad". Later on, droogs.
• abeppu 5y ago
  I may be over-reading, but I think this kind of example not only demonstrates the pragmatic legal issues, but also the fundamental weaknesses of a solely text-oriented approach to suggesting code. It doesn't really seem to have a representation of the problem being solved, or the relationship between things it generates and such a goal. This is not surprising in a tool which claims to work at least a little for almost all languages (i.e. which isn't built around any firm concept of the language's semantics). I'd be much more excited by (and less unnerved by) a tool which brought program synthesis into our IDEs, with at least a partial description of intended behavior, especially if searching within larger program spaces could be improved with ML. E.g. here's an academic tool from last year which I would love to see productionized. https://www.youtube.com/watch?v=QF9KtSwtiQQ https://www.youtube.com/watch?v=QF9KtSwtiQQ
  • whimsicalism 5y ago
    > fundamental weaknesses of a solely text-oriented approach to suggesting code. I don't think it is clear that such "fundamental weaknesses" exist. A text-based approach can get you incredibly far.
    • abeppu 5y ago
      I mean, the cases where it tries to assign copyright to another person in a different year highlights that context other than the other text in the file is semantically extremely important, and not considered by this approach. Merely generating text which looks appropriate to the model given surrounding text is ... misguided? If you think about it, program synthesis is one of the few problems in which the system can have a perfectly faithful model dynamics of the problem domain. It can run any candidate it generates. It can examine the program graph. It can look at what parts of the environment were changed. To leave all that on the table in favor of blurting out text that seems to go with other text is like the toddler who knows that "five" comes after "four", but who cannot yet point to the pile of four candies. You gotta know the referents, not just the symbols. No one wants a half-broken Chinese Room.
      • whimsicalism 5y ago
• grawprog 5y ago
  I'm not surprised to be honest. I've played around with AI dungeon, which also uses GPT-3. It regularly reproduces content directly from its training material, including even comments attached to the stories they trained the ai on.
• sktrdie 5y ago
  I'm going to go against the flow here and say that worrying about this is similar to worrying about the license we give to snippets of code we copy-paste from other licensed code. The reality is that we never attribute the original source because we copy-paste it, change it up a bit, and make it our own. Literally everybody does this. I still care about licensing and proper attribution but the reality is that a snippet of code is not something so easy to attribute. Should we attribute all kinds of ideas, even the very small ones? How quickly is an idea copied, altered & reused? Can we attributes all the thoughts humans have?
• ezoe 5y ago
  Similar story. He tried to write Quine in Ruby, ended up conjuring up a copyright claim comment and fake licensing term. https://twitter.com/mametter/status/1410459840309125121 https://twitter.com/mametter/status/1410459840309125121
• deleted 5y ago
  [deleted]
• throwaway_egbs 5y ago
  Welp, guess I'll be taking all my code off of GitHub now, lest it be copied verbatim while ignoring my licenses. (I'm no John Carmack, but still.)
• throwaway_egbs 5y ago
  This reply from @AzureDevOps is bizarre: "We understand. However, the way to report this issues related to Windows 11 is through our Windows Insider even from another device. Thanks in advance." I think I'm gonna give "AI" a few more years. https://twitter.com/AzureDevOps/status/1411018079849619458 https://twitter.com/AzureDevOps/status/1411018079849619458
  • krick 5y ago
    Wow. To think of it, nothing in this HN thread, including your link, is truly new and unexpected, but in this context it felt somehow more dystopian than ever. Talking about machines pretending to be humans doing stupid stuff, getting automated responses from machines pretending to be humans, that also are the same kind of stupid stuff... Almost feels like drowning.
    • eightails 5y ago
      We understand. However, the way to report this issues related to im̄̽̚m͚͠i͙̬͈̟̹̳ͨ͆̀ͅn̲͚̻ͩ̐͒ͩ̊è̹̱͖̼̰n̘̯ͥ̿̌͛͌t̳̖̣̻̯̱ͥ̅̿̇͜ ̥̻̺͒ͣ͒͠A͔͔͓ͨÌ̖̲̆͒̐̍ͅ ̝̙̼̤͖͍̆̀ͪdͤͨ͑̈҉̭̖y̤͔̮͚̞̺ͬͦͦ̎ͮ͐́s̤͓̲͓̖̪̊t͎̰̤̩̞̞͇͐̎͂̉̆̚o̱̣̰͇̟̻͎̿͒̋̎p̫̰̮̌͐ͧ͗̔̀ͣi̫̱̩̠̫͔͒̉ͤa̶ͧͦͭͩ is through our Citizen Satisfaction Department, even from another Autonomous Azure® Sub-district. Thanks in advance.
    • heartbreak 5y ago
      The user that account replied to was having a conversation with Microsoft Support about Windows 11, and they replied to the wrong tweet thread with the wrong account. https://twitter.com/lnplum/status/1410599036311130113?s=21 https://twitter.com/lnplum/status/1410599036311130113?s=21
• coolspot 5y ago
  Time to write a GPL-licensed Win32 and Win64 -compatible OS with the help of CoPilot...
• nadsat2021 5y ago
  I don't understand why this is an issue. And now I have a tool to do my coding, as well as my social media contributions: "'Sweary comments' is not grammatically correct English." -> GPT-2 -> "Sweary comments" is not grammatically correct English. The "sweary" meaning is used as slang by those in charge, especially the ones who get a free ride through the media, for the sake of a quick buck, or the sake of getting a better job with a promotion. The word is commonly used in a social context and comes from an actual "sweary" comment. The statement or phrase, "You're too shy!" is derived from this phrase: "I wish he were as outgoing as you." Am I right?
• Osiris 5y ago
  I assumed it was trained on source code that was explicitly licensed with a permissive license. Are they training it using private unlicensed repos also?
• GenerocUsername 5y ago
  A closed beta with only a few previews out in the wild has bugs. Unbelievable. I cannot believe GitHub would do this.
• ianhorn 5y ago
  Remember that unlicensed content is still licensed. Generally the implicit license of a work is less permissive. So however you feel about this using GPL code, the situation is even worse for most ML models that have no license at all to use their training data.
• wly_cdgr 5y ago
  Welp, there go our licensing models
• astatine 5y ago
  I expect it will lead to greater compliance than today, in the near future even if not today. Making tens of thousands of engineers understand the nuances of various licenses is infinitely more difficult than making copilot start taking better care of licensing requirements.
• sivakon 5y ago
  Copilot will most likely be useful as a Stackoverflow indexer or a better GitHub search. Developer speed can become faster because this eliminates some time spent on GitHub and SO. I don’t think it’s anymore useful than that.
• Loeffeldude 5y ago
  I struggle to comprehend how anyone thought copilot is a good idea.law wise and concept wise.