12 ms·
The Good and the Limitations of Github Copilot
- make3 5y agoControlling the generation for code quality will be extremely hard. The only thing I see is that they could filter their dataset so that some bad proxy of code quality is taken into account, something like the number of stars (which is clearly a terrible metric, tell me if you think of something else). The idea would maybe start by training on all of the subset of Github it is ethical and legal to train on, and then filter down to higher code quality towards the end. Controlling for the time at which the code is emitted would be easier. Something like, retrieving similar contexts, and guiding the model to be more similar to the recent code if there is similar recent code that exists. I'm not sure exactly of how this would be done, but I can see it working.
- nicetryguy 5y agoIs === really necessary for Javascript? Type checking seems like overkill?
- kylemuir 5y agoDepends how much you care about the equality and how strict. by default == will do type coercion so you can get weird results where things like [] == 0 are true. More equality fun in JS can be seen here: https://dorey.github.io/JavaScript-Equality-Table/ https://dorey.github.io/JavaScript-Equality-Table/
- Arainach 5y agoYou should confirm that the inputs are what you expect them to be. There's almost no time when you shouldn't be checking the type.
- tlrobinson 5y agoIt’s a well known rule in JavaScript to default to “===“ for everything except checking for null or undefined, in which case “== null” is acceptable. https://eslint.org/docs/rules/eqeqeq#smart https://eslint.org/docs/rules/eqeqeq#smart Even better is to just use TypeScript.
- BigJono 5y agoChecking for 'null or undefined' is an antipattern anyway, so there's no real valid use of '=='.
- flippinburgers 5y agoDoes typescript really protect one from javascript though? I imagine it is fine. If it was just myself developing I would be fine, but the insufferable junior devs that learned "=== or die" rear their ugly heads.
- numlock86 5y agoIn best practice you should always use === unless you know exactly why you wouldn't for a specific case.
- pdpi 5y agoa == !a There’s several values of a where that yields true (of the top of my head, a = '0' is one of them). Yes, you do need ===.
- chx 5y agoIf someone doesn't understand this then here's why. You need to learn the Six Falsey Things In JavaScript. Everything else when cast to boolean will be true. These six things are What and Why and When And How And Where and Who. Wait, no, that's a totally different list of six... anyways false undefined null NaN 0 "" (empty string) That's it. Since '0' is neither when the negate operator casts it to boolean it'll become true. And then negating it becomes false. When doing a comparison between '0' and false, the standard says https://262.ecma-international.org/5.1/#sec-11.9.3 https://262.ecma-international.org/5.1/#sec-11.9.3 > If Type(y) is Boolean, return the result of the comparison x == ToNumber(y). ToNumber(false) is of course 0. So you are running '0' == 0 which is visibly true...
- pdpi 5y ago> So you are running '0' == 0 which is visibly true... Crucially, there's the extra step of converting '0' to a number.
- megous 5y agoIt's not, it's just one of the tools in the toolbox.
- chovybizzass 5y agoIf they are using my code anywhere I expect to be compensated.
- ronsor 5y agoIf you read the license you put on your project (and/or the GitHub Terms of Service), you wouldn't expect that.
- Gaelan 5y agoIf it’s anything but public domain, there’s probably at least a requirement for credit.
- alkonaut 5y agoI accept google's and Facebooks terms, and I still don't expect to be tracked across the internet to pay for the services they give me. Basically: my expectation is that companies aren't behaving as badly as their terms allow them to.
- nextaccountic 5y agoBeing credited for your work is a form of compensation. Nearly all open source licenses require proper attribution. Github is just violating the open source licenses of nearly all people that posted their code to Github itself, which is a huge breach of trust.
- capableweb 5y agoHuge leap of logic to assume what license another user publishes their code under. A charitable reading here of chovybizzass's comment leads me to the understanding that they are publishing their code under a license that requires compensation if used, this code can be published to GitHub as well. Would it be stupid? Probably. But even more stupid would be to produce MIT-licensed code and then complain about not being compensated.
- na85 5y agoThe new model pioneered by AirBNB and Uber is to break laws under the guise of "growth hacking" and then change the laws later once you're entrenched. You won't be compensated. Corporations write the laws.
- chrismorgan 5y ago> Can help you with Email Validation and API Calls It generates a nastily complex regular expression that is hopelessly wrong. Visible at https://www.youtube.com/watch?v=9Pw-Roo_duE&t=404 https://www.youtube.com/watch?v=9Pw-Roo_duE&t=404, here transcribed: /^([\w-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([\w-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$/ For the local part, it requires [\w-\.]+, which excludes many valid characters like everyone’s favourite, +. For the domain part, it tries to allow IPv4 addresses as well as normal domain labels (not IPv6 addresses, though), but it ends up tangling it up in a way that a human never would, allowing things like [12.34.56.com], [987.654.321.000, example.com] and example.123], while disallowing things like example.studio (the last label only allowing 2–4 letters) and IDN TLDs (which start with xn-- and must allow hyphen and numbers, not just [a-zA-Z]). The author makes no comment on how hideously bad it is, which makes me suspect he didn’t notice, which… yeah, shows the problems of the whole thing.
- visarga 5y agoCopilot acts like a search engine, you search, you find, then you judge. It was never the case with search engines that you could just copy some code you found without verifying it. Also, it has the same copyright problems as if you used Google to find the code.
- objclxt 5y ago> Copilot acts like a search engine, you search, you find, then you judge. OK, but that’s not how GitHub position it: “Your AI pair programmer” “Skip the docs and searching for examples” They literally say it’s not a search engine!
- tsimionescu 5y ago> Also, it has the same copyright problems as if you used Google to find the code. No, it has the same copyright problems as if you Google and instead of getting links to sites that host code and licenses, you get just the code.
- 5y ago
- minimaxir 5y agoA fun issue I keep hitting with Github Copilot in Python is that it's a coin flip whether it will give me a Python 3-style print statement or a Python 2.7 style print statement.
- hrithwikbha 5y agoSame lol . Most of the times I get Python 2 print statements
- mdaniel 5y agoGiven that no one should be _actively_ writing 2.7 code, and that GH is trying to monetize this, I would think they could retrain the main model that excludes 2.7 code, and then allow people who need 2.7 to "pay for supporting 2.7" just like I'm guessing folks stuck on old platforms always do You raise a fascinating point about older platforms of other languages, too; Java has a super backward compat story, but woe be unto the coder who tries to name a variable "enum" nowadays
- kzrdude 5y agoThis seems to be another side of the problem that it gives code that doesn't actually compile for the target language. They should have a check for that, that should be the baseline. For me, it sounds like it's closer to dumb copy paste than a smart code generator. AlphaZero wouldn't play a chess move that was against the rules.
- jononor 5y agoMost instances of that should be fixable with some post-processing in Copilot. They could maybe even run py2to3 on it?
- mkl 5y agoI think preprocessing, upgrade the input data before training.
- addandsubtract 5y ago
- wildpeaks 5y agoOne problem is the lack of context: unlike Stackoverflow, you don't get additional infos from other users who tried the same code. The only feedback Copilot receives is whether you keep it or not, you can't tell it a few days later that it wasn't a good fit after all (whereas you can comment on a Stackoverflow answer). In its current form, it amplifies bias whereas code needs accuracy.
- alephnan 5y agoThat seems like an additional feature. Maybe append some comments to caveat the code snippet
- alephnan 5y agoSo does querying Google or Stackoverflow. Looking for “inspiration” should guide you, but you should always be skeptical and not let your brain fall out
- qayxc 5y agoThe danger is that CoPilot becomes "AutoPilot", because as the author points out, you get used to it way too quickly. Searching the web still implies extra effort on your side and (hopefully) some scepticism towards the code you find (SO comments, etc.)
- nxpnsv 5y agoI wonder how much worse or better simply automating a stackexchange search and pasting the results is...
- forgotpwd16 5y agoSearching SO and converting the highest rated answer's code that doesn't syntax error to a Python module is about 150 LoC[0]. [0]: https://github.com/drathier/stack-overflow-import https://github.com/drathier/stack-overflow-import
- mrweasel 5y agoSomeone did a Stackoverflow auto-complete for JavaScript: https://emilschutte.com/stackoverflow-autocomplete/ https://emilschutte.com/stackoverflow-autocomplete/
- deleted 5y ago[deleted]
- FractalHQ 5y agoCopilot is going to improve incredibly fast with millions of devs providing ReCaptcha style training data around the clock.
- nprateem 5y agoA week in and I'm already bored of Copilot submissions... It's not perfect. What do you expect?
- TenToedTony 5y agoThat it's perfect.
- qayxc 5y agoThere are two types of products: the ones that people complain about and the ones that nobody uses or cares about; based on a famous Bjarne Stroustrup quote. The fact that people post about it, point out flaws, leave GH out of spite, etc. only goes to show how impactful a system like this can be. It's a big deal, which is why people talk about it.
- deleted 5y ago[deleted]
- ratww 5y agoNot being perfect would be acceptable, but at this point it looks like something with the potential of setting back software quality and security [1], with the added bonus of also breaking copyright/licensing. [1] I understand that everyone should review code before committing, but even if me and my team do it, there's no way all the proprietary and open-source software I use has teams doing the same. That's why I worry for my security.
- progx 5y agoOur daily GithHub Copilot message give us today.
- L33tCrown 5y agoA copilot for commit messages will be as useful as this.
- bellyfullofbac 5y agoSkimmed through this "review", nothing wowing me. > Hey Stephen, If you are reading this, please follow me on hashnode . God, that's obnoxious...
- thrwyoilarticle 5y agoThat's what we call a 'joke'
- PierceJoy 5y agoWhat’s with the editorialized title based on a single sentence in the article?
- corobo 5y agoHonestly I just ditched the idea of Copilot the moment it spewed out an entire file of copied code. Nice work on making a markov bot with extra steps GitHub. Please, do take my money..
- maccard 5y agoIt never did that. It got the license wrong so it wasn't an exact copy...
- corobo 5y agoUntil a court decides what it does and doesn't do I'm staying clear Here's it outputting Quake code, including handy comments it came up with for each line and even an entire line of commented out code. Maybe it decided it was a good choice to comment it out but still include it I guess Being word for word from the original is just a weird coincidence too I truly wanted it to be as good as it was sold to us too, but it isn't https://twitter.com/mitsuhiko/status/1410886329924194309 https://twitter.com/mitsuhiko/status/1410886329924194309 HN discussion at https://news.ycombinator.com/item?id=27710287 https://news.ycombinator.com/item?id=27710287 Additionally here's it somehow requesting and using API keys for use in your code https://twitter.com/passcod/status/1410822834272694275 https://twitter.com/passcod/status/1410822834272694275
- maccard 5y agoIt didn't copy the entire file for quake - it got the license wrong (which is even worse!) That what I said in my original comment!
- deleted 5y ago[deleted]
- corobo 5y agoAh I see. I think my perception of your point was altered by you having been downvoted by someone making the text grey (can't read text = bad comment). Sorry about that. I'll try to avoid letting downvotes bias me in future
- peterkelly 5y agoGitHub Copilot is going to end up being the Google Glass of developer tools.
- smnscu 5y agoIt could've been a fantastic April Fool's release. I like to imagine that was the original intent and a manager went "eh, release it anyway".
- thinkingemote 5y agoGitHub should release a ML powered copyright plagarism app to spot when code has been copied and suggest to the user the license they need to use.
- lysecret 5y agoYou know they are hitting something interesting if all of hackernews gets mad haha
- majewsky 5y ago> Developers with experience can definitely handle this out but what if a newbie directly starts with the help of AI, he will spend more time on stack overflow than writing actual code. (oh wait that's how most of the developers are lmao) Spontaneous Ask HN: How much truth is there in this "devs be copy-pasting from SO all day" trope? Personally, I have used SO quite heavily in its early years, circa 2010-2014, including posting my own questions and sometimes posting answers to others' questions. But now, I don't use it actively anymore. Sure, when I search for a concrete question and SO happens to be in the search results, it's sometimes a valuable resource. But it's not the go-to for programming questions that it once was for me. I'm honestly not sure if that's indicative of my own growth as a developer, or caused by outside factors. I have a vague feeling that developer documentation in general got better in the last decade, at least for the technologies that I'm using... but then again it could be also a sign of personal growth that I'm more comfortable with the upstream documentation. Finding answers for webdev questions on MDN is another sort of game than finding answers for webdev questions on SO, after all. What do you all think?
- nirui 5y agoBased on my own experience, I remembered back in the day when I first started programming, in PHP, I copied many code from other commenters on PHP.net a lot (their online manual allows others to post code underneath the main docs). So I expect some new programmers to do the same but with StackOverflow. Of course, I stopped doing that once I got hold on the real knowledge. In fact, now I look back the code that I copied from the others, it's just like watching a horror movie, and I rather rewrite the whole thing in my own term. I guess the fairer statement to put in is "some programmer copy those code to 'get started'".
- ok2938 5y agoI'm all for moving forward, but Copilot just seems to be a bad idea, amplified. What I want is careful, thoughtful, knowledgeable people, who have learned and honed their skills over years in various areas and can come up with creative, maintainable solutions to complex problems. This is not something you autocomplete. If it would be, we could autocomplete 80% of all jobs tomorrow. I don't want code monkeys on steroids. But maybe I'm to far off Silicon Valley.
- banana_giraffe 5y agoSo far the most WTF thing I've gotten out of it is: import base64 test = base64.b64decode("""SSdtIGtpbGxpbmcgeW91ciBicmFpbiBsaWtlIGEgcG9pc29ub3VzIG11c2hyb29t""".encode()) print(test) # b"I'm killing your brain like a poisonous mushroom" And the most odd thing: # The base URL for all API requests base_url = 'https://api.gdax.com/' # The base URL for all non-API requests (e.g. static content) base_url_static = 'https://static.gdax.com/' Which are URLs that haven't been a thing for 2 years, I think and I can't find any code in github that uses them still.