9 ms·
They have 4 hand picked examples on their homepage: https://copilot.github.com/ https://copilot.github.com/ One has the issue with form encoding: https://news.
by tyingq 5y ago
They have 4 hand picked examples on their homepage: https://copilot.github.com/ https://copilot.github.com/
One has the issue with form encoding: https://news.ycombinator.com/item?id=27697884 https://news.ycombinator.com/item?id=27697884
The python example is using floats for currency, in an expense tracking context.
The golang one uses a word ("value") for a field name that's been a reserved word since SQL-1999. It will work in popular open source SQL databases, but I believe it would bomb in some servers if not delimited...which it is not.
The ruby one isn't outright terrible, but shows a very Americanized way to do street addresses that would probably become a problem later.
And these are the hand picked examples. This product seems like it needs some more thought. Maybe a way to comment, flag, or otherwise call out bad output?
- slver 5y ago> And these are the hand picked examples. This product seems like it needs some more thought. Everyone's self-preservation instincts kicking in to attack Copilot is kinda amusing to watch. Copilot is not supposed to produce excellent code. It's not even supposed to produce final code, period. It produces suggestions to speed you up, and it's on you to weed out stupid shit, which is INEVITABLE. As a side note, Excel also uses floats for currency, so best practice and real world have a huge gap in-between as usual.
- mkr-hn 5y agoHave you met programmers? Even those who care about quality are often under a lot of pressure to produce. Things slip through. Before, it was verbatim copies from Stack Overflow. Now it'll be using Copilot code as-is.
- slver 5y agoSo, nothing new, is your point?
- mkr-hn 5y agoThen why are you complaining? Unless something is new that warrants you getting mad about people getting mad at technology.
- saiojd 5y agoNot the parent, but people really like to get riled up on the same topics, over and over again, which quickly monopolizes and derails all conversion. Facebook bad, UIs suck, etc. We can now add to the list, "AI will never reduce demand for software engineering".
- slver 5y agoWell, "never" is a long time. Copilot is definitely no replacement for anything except copying from Stack Overflow for juniors. But in the long run, AI is us basically us creating our own replacement. As a species. We don't realize it yet. It'll be really funny in retrospective. Too bad I probably won't be alive to see it.
- volta83 5y agoSo how do you know if the code that Copilot regurgitates is almost a 1:1 verbatim copy of some GPL'ed code or not ? Because if you don't realize this, you might be introducing GPL'ed code into your propiertary code base, and that might end up forcing you to distribute all of the other code in that code base as GPL'ed code as well. Like, I get that Copilot is really cool, and that software engineers like to use the latest and bestest, but even if the code produced by Copilot is "functionally" correct, it might still be a catastrophic error to use it in your code base due to licenses. This issue looks solvable. Train 2 copilots, one using only BSD-like licensed software, and one using also GPL'ed code, and let users choose, and/or warn when the snippet has been "heavily inspired" by GPL'ed code. Or maybe just train an adversarial neural network to detect GPL'ed code, and use it to warn on snippets, or...
- slver 5y agoIt's very easy: don't use copilot code verbatim, and you won't have GPL code verbatim.
- volta83 5y ago> It's very easy: don't use copilot Fixed that for you. Verbatim isn't the problem / solution. If you take a GPL'ed library and rename all symbols and variables, the output is still a GPL'ed library. Just seeing the output of GPL'ed code spitted by copilot and writing different code "inspired" by it can result in GPL'ed code. That's why "clean room"s exist. Copilot is going to make for a very interesting to follow law case, because probably until somebody sues, and courts decide, nobody will have a definitive answer of whether it is safe to use or not.
- throw_2021-07 5y agoStack Overflow content is licensed under CC-BY-SA. Terms [1]: * Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. * ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. In over a decade of software engineering, I've seen many reuses of Stack Overflow content, occasionally with links to underlying answers. All Stack Overflow content use I've seen would clearly fail the legal terms set out by the license. I suspect Copilot usage will similarly fail a stringent interpretation of underlying licenses, and will similarly face essentially no enforcement. [1] https://creativecommons.org/licenses/by-sa/4.0/ https://creativecommons.org/licenses/by-sa/4.0/
- pydry 5y agoIt's true I probably wouldnt have laughed quite as loudly if there werent a chorus of smug economists telling us that tools like this are gonna put me out of a job.
- slver 5y agoBusiness types hate dealing with programmers, that's a fact. And these claims of "we'll replace programmers" happen with certain precise regularity. Ruby on Rails was advertised as so simple, startup founders who can't program were making their entire products in it in a few days, with zero experience. As if.
- astrange 5y agoEconomists don't believe this. It's non-economists who do. Economists know that it's not possible to run out of jobs because demand is infinite.
- j-pb 5y agoIf I want random garbage in my codebase that I have to fix anyways I might as well hire a underpaid intern/junior. It's easier to write correct code than to fix buggy code. For the former you have to understand the problem, for the latter you have to understand the problem, and a slightly off interpretation of it.
- saiojd 5y agoI agree with you. This is basically similar to autocomplete on cellphone keyboard (useful because typing is hard on cellphone), but for programming (useful because what we type tends to involve more memorization than prose).
- tyingq 5y ago>As a side note, Excel also uses floats for currency It's still problematic, but the defaults and handling there avoid some issues. So, for example: Excel: =1.03-.42 produces 0.61, by default, even if you expand out the digits very far. Python: 1.03-.42 produces 0.6100000000000001, by default.
- slver 5y agoExcel rounds doubles to 15 digits for display and comparison. The exact precision of doubles is something like 15.6 digits, those remaining 0.6 digits causing some of those examples floating (heh) around.
- okl 5y agoThat depends https://randomascii.wordpress.com/2012/03/08/float-precisionfrom-zero-to-100-digits-2/ https://randomascii.wordpress.com/2012/03/08/float-precision...
- slver 5y agoA lot of these edge cases are about theoretical concerns like "how many digits we need in decimal to represent an exact IEEE binary float". In practice a double is 15.6 digits precise, which Excel rounds to 15 to eliminate some weirdness. In their documentation they do cite their number type as 15 digit precision type. Ergo that's the semantic they've settled on.
- tyingq 5y ago"self-preservation" My suggestion was a way to comment or flag, not to kill the product. These were particularly notable to me because someone hand-picked these 4 to be the front page examples of what a good product it was.
- Supermancho 5y ago> Everyone's self-preservation instincts kicking in to attack Copilot is kinda amusing to watch Nobody is threatened by this, assuredly. As with IDEs giving us autocomplete, duplication detection, etc this can only be helpful. There is an infinite amount of code to write for the foreseeable future, so it would be great if copilot had more utility.
- gentleman11 5y ago> The python example is using floats for currency. Dumb question, but what is the proper way to handle currency? Custom number objects? Strings for any number of decimal places?
- stickfigure 5y agoCreate a Money class, or use one off the shelf. It should store the currency and the amount. There are a few popular ways of storing amounts (integer cents, fixed decimal) but it should not be exposed outside the Money class. There's plenty of good advice in this subthread for how to represent currency inside your Money abstraction, but whatever you do, keep it hidden. If you pass around numbers as currency values you will be in for a world of pain as your application grows.
- ronnier 5y agoIn its base unit. So cents in USD. Which can be an int64 Or if your language has something specific built in, use that.
- umanwizard 5y agoNot necessarily. It depends on the application.
- ainar-g 5y ago> Or if your language has something specific built in, use that. Unless your language is PostgreSQL's dialect of SQL, apparently. https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use...
- pilif 5y agoIt has the same issue that the other suggestion of your parent comment had: it can’t deal with fractions of cents, which is an issue you will most likely run into before you will into floating point rounding issues.
- 5y ago
- rjknight 5y ago> This product seems like it needs some more thought. Maybe a way to comment, flag, or otherwise call out bad output? Wait for your colleagues to use it, fix the bad code in the pull request, and wait for copilot to learn from the new training data you just provided!
- more_corn 5y agoThis is actually a good idea that is missing from nearly every machine learning product. How do you back propagate lessons from user interaction into future training of the model? It can be done, I can't think of a place I've seen it done though.
- adriancr 5y agoIt would be viewed as IP theft by most companies to upload private code to this for use by others
- dgb23 5y agoIt would have to be in the same range of what is suggested, small patches and opt in. If snippets are a legal problem, then Copilot is problematic by default, since it suggests code that may or may not be sourced from free software.
- adriancr 5y agoEven free software snippets have clauses like GPL or attribution. Putting GPL code in proprietary codebase would cause a company massive headaches... So I agree copilot is problematic by default, liability to lawsuits for employers and forced open sourcing, liability to IP lawsuits as well which will end up on employees shoulders.
- TeMPOraL 5y agoIt's tricky, because once you start accepting user feedback, you need to moderate it, or else someone will poison your model for fun and profit.
- mempko 5y agoThese are great examples. I wrote about how this will propagate all sorts of bugs. But my argument was that it's good enough developers may get complacent and not review the auto complete closely enough. But maybe I'm wrong! Maybe it's not that good yet.
- ssss11 5y ago“Maybe a way to comment, flag, or otherwise call out bad output?” A copilot for copilot? :)
- gameswithgo 5y agopost process with some language aware heuristics maybe
- xyzzy_plugh 5y ago> The golang one uses a word ("value") for a field name that's been a reserved word since SQL-1999. It will work in popular open source SQL databases, but I believe it would bomb in some servers if not delimited...which it is not. In their defense they created the table with this column before invoking the autocomplete, so they sort of reap what the sow here. It could at least auto-quote the column names to remove the ambiguity, but it's not a compiler, is it.
- bierjunge 5y agoThe Golang example would not even compile, because `sql` is not imported.
- IncRnd 5y agoThat's for the best. We don't want products that pretend to write code for us, while copying other's code without attribution and that may not even work.
- shadowgovt 5y agoNow that they have an AI that can be trained to replicate code, it looks like the next step is training it to replicate good code. That will be non-trivial, since step one is identifying good code and they may not have much big data signal to draw from for that. We know you can't use StackOverflow upvotes. However, they should have enough signal to identify what snippets of code have been most frequently copy-pasted from one project to another. Question is whether that serves as a good proxy for good code identification.
- foobiekr 5y agoCopilot appears to be “give more efficiency leverage to the worst kind of coder.”
- ticviking 5y agoI'm not really sure that type of tool could really be anything else. How would a model become aware of all of the various edge cases that depend on which SQL database you use or differences in language versions over time?
- gmadsen 5y agoa large data set covering exactly what you just mentioned?
- sbr464 5y agoCan it submit pull requests to itself with if/else boolean logic/hacks?
- TeMPOraL 5y ago> I'm not really sure that type of tool could really be anything else. It can't be, because they've chosen to use a deep learning approach. That makes it a dead end right from the start. > How would a model become aware of all of the various edge cases that depend on which SQL database you use or differences in language versions over time? A lot of things that we call "edge cases" are only a problem for humans. They're not "edge cases" from the point of view of the grammar / semantics of programming languages and libraries. The way a hypothetical, better Copilot could work, is by having directly encoded grammars and semantics metadata corresponding to popular languages and tools. It could generate code in principled and introspectable way, by having a model of the computation it wants to express and encoding it in a target language. Of course, such hypothetical Copilot is a harder task - someone would have to come up with a structure for explicitly representing understanding of the abstract computation the user wants to happen, and then translate user input into that structure. That's a lot of drudgery, and from my vague understanding of the "classical" AI space, there might be a bunch of unsolved problems on the way. Real Copilot uses DNNs, because they let you ignore all that - you just keep shoving code at it, until the black-box model starts to give you mostly correct answers. The hard work is done automagically. It makes sense for some tasks, less for others - and I think code generation is one of those things where black-box DNNs are a bad idea.
- stevelosh 5y agoThe golang one also silently drops rows.Err() on the floor. https://golang.org/pkg/database/sql/#Rows https://golang.org/pkg/database/sql/#Rows
- amelius 5y agoWhat are the statistics of Copilot based on a validation set? How often does it get code right? I want to see hard statistics, not 4 hand-picked examples.
- mnky9800n 5y agoThat's what I thought when I first started working in text generation too. It's highly annoying people pitch their successful models with hand picked examples. It's literally the opposite of STATISTICAL learning imo.
- rubatuga 5y agoYeah. And like how would you even devise a metric? Like compile it down to assembly and see if it’s similar logic?
- amelius 5y agoWell, this is the question which the producers of the tool should answer. You can't just release a ML tool onto the public if you haven't validated it first.
- jacurtis 5y ago> The ruby one isn't outright terrible, but shows a very Americanized way to do street addresses that would probably become a problem later. As someone who has been coding up address storage and validation for the past week in my current job, that one really made me laugh. Mostly because it tries to simplify all the stuff I have been analyzing and mulling over for a week into a single auto-complete. Spoiler: The Github Copilot's solution simply won't work. It would barely work for Americanized addresses, but even then not be ideal. Of course trying to internationalize it, this thing isn't even close. I get what Copilot is trying to do. But at the same time I don't get it. Because from my experience, typing code is the fastest part of my job. I don't really have a problem typing. I spend most of my time thinking about the problem, how to solve it, and considering ramifications of my decisions before ever putting code in the IDE. So Copilot comes around and it autocompletes code for me. But I still have to read what it suggested, making edits to it, and consider if this is solving the problem appropriately. I'm still doing everything I used to do, except it saved me from typing out a block of code initially. I still have to most likely rebuild, edit, or change the function somewhat. So it just saves me from typing that first pass. Well that's the easy part of the job. I have never had a manager come to me and ask why a project is taking so long where I could answer "it just takes so long to type out the code, i wish I had a copilot that could type it for me". That's why we call it software engineering and not coding. Coding is easy. Software engineering is hard. Github Copilot helps with coding, but doesn't help with Software Engineering.
- ph0rque 5y ago> I spend most of my time thinking about the problem, how to solve it... A few years ago, I got a small but painful cut on my fingertip. I thought I would have a hard time on the job as a dev. To my surprise, I realized I spend 90-95% of my time thinking, and only 5-10% of the time typing. It turned out to be almost a non-issue.
- lwhi 5y ago>Because from my experience, typing code is the fastest part of my job. I don't really have a problem typing. I spend most of my time thinking about the problem, how to solve it, and considering ramifications of my decisions before ever putting code in the ID So very true. [1] Understanding the problem > [2] thinking about all possible solutions > [3] working out which solution fits best > [4] working out which implementations are possible > [5] working out the most suitable implementation ... and finally, [6] implementing via code.
- eulers_secret 5y agoI haven't seen anyone mention this issue for some reason, but in fetch_tweets.py: fetch_tweets_from_user(user_name): ... tweets = api.user_timeline(screen_name=user, count=200, include_rts=False) 'user' isn't defined, should be user_name, right? Side note, 'copilot' is a decent name for this (though copilots are usually very competent, moreso than this right now). You must check the suggestions carefully. Maybe it'll make folks better at code review, lol.
- TeMPOraL 5y ago- The Go one (averaging) is non-idiomatic, and has a nasty bug in it: https://news.ycombinator.com/item?id=27698287 https://news.ycombinator.com/item?id=27698287 - The JavaScript one (memoization) is a bad implementation, it doesn't handle some argument types you'd expect it to handle: https://news.ycombinator.com/item?id=27698125 https://news.ycombinator.com/item?id=27698125 You can tell a lot about what to expect, if there are so many bugs in the very examples used to market this product.