4 ms·

Copilot appears to be “give more efficiency leverage to the worst kind of coder.”

ticviking 5y ago
I'm not really sure that type of tool could really be anything else. How would a model become aware of all of the various edge cases that depend on which SQL database you use or differences in language versions over time?
- gmadsen 5y ago
  a large data set covering exactly what you just mentioned?
- sbr464 5y ago
  Can it submit pull requests to itself with if/else boolean logic/hacks?
- TeMPOraL 5y ago
  > I'm not really sure that type of tool could really be anything else. It can't be, because they've chosen to use a deep learning approach. That makes it a dead end right from the start. > How would a model become aware of all of the various edge cases that depend on which SQL database you use or differences in language versions over time? A lot of things that we call "edge cases" are only a problem for humans. They're not "edge cases" from the point of view of the grammar / semantics of programming languages and libraries. The way a hypothetical, better Copilot could work, is by having directly encoded grammars and semantics metadata corresponding to popular languages and tools. It could generate code in principled and introspectable way, by having a model of the computation it wants to express and encoding it in a target language. Of course, such hypothetical Copilot is a harder task - someone would have to come up with a structure for explicitly representing understanding of the abstract computation the user wants to happen, and then translate user input into that structure. That's a lot of drudgery, and from my vague understanding of the "classical" AI space, there might be a bunch of unsolved problems on the way. Real Copilot uses DNNs, because they let you ignore all that - you just keep shoving code at it, until the black-box model starts to give you mostly correct answers. The hard work is done automagically. It makes sense for some tasks, less for others - and I think code generation is one of those things where black-box DNNs are a bad idea.
  - jhgb 5y ago
    > The way a hypothetical, better Copilot could work, is by having directly encoded grammars and semantics metadata corresponding to popular languages and tools. It could generate code in principled and introspectable way, by having a model of the computation it wants to express and encoding it in a target language. But that sounds like too much work, let's just throw a lot of data into an NN and see what comes out! /s > and introspectable Which most importantly means "debuggable", I assume. From what I get there doesn't seem to be any way to ad-hoc fix an NN's output.
codyb 5y ago
Hmm... I mean, these all seem like mistakes I could make and I don't think I'm the "worst kind of coder". The currency one I learned a while back, but it's not like I intuited using integers by default. Value being a reserved keyword, I'm not sure I'd know that and I do Postgres work as part of my myriad duties at the startup I work at. Maybe I'd make that mistake in a migration, maybe I have already. In a way, is it much different then what we do now as engineers? I'm hard pressed to call it much of an engineering discipline considering most teams I work on barely do design reviews before they launch in to writing code, documentation and meeting minutes are generally an afterthought, and the code review process while decent isn't perfect either and often times relies on arcane knowledge derived over months and years of wrangling with particular <framework, project, technology>. It's pretty neat, presumably it'll learn as people correct it, and it'll get better over time. I mean it's not even version one. I get the concerns, but I think they're a bit overblown, and this'll be really useful for people who want to learn how to code. Sure they'll run into some bugs, but, I mean, they were going to do that anyways.
- voakbasda 5y ago
  Is this any worse? Maybe not. Is it better? Absolutely not. This kind of tool will only further entrench the production of mediocre, bug-ridden code that plagues the world. As implemented, this will not be a solution; it is a express lane in the race to the bottom.
  - pasquinelli 5y ago
    it is a race to the bottom, and people are trying to win. any skilled trade is being turned into an unskilled job. it might suck, the results might suck, but it's more profitable, and that's what matters.
majormajor 5y ago
It takes what should be your method of last resort - copypaste - and makes it the first thing you try. All the steps in between - looking at the docstring for the function you're calling, googling for more general information, looking at and deciding not to use not-applicable or poorly-written SO answers - get pushed aside. So instead of you having to convince yourself "yes, it's safe to copy-paste these lines from SO, they actually fit my problem" you're presented with magic and I think the burden for rejecting it is going to be higher once it's in your editor than when you're just reading it on a SO post or Github snippet. Even for a newcomer looking to learn, working on simple stuff that it has great completions for, it seems like it will sabotage your long-term growth, since it takes all the why and the reasoning out of it. Autocomplete for a function name isn't that relevant to gaining a deeper understanding. Knowing why a certain block of code is passed in in a certain style, or needs to be written at all? Probably that is.
- majormajor 5y ago
  Thinking about it more: there's a very small subset of problems that I think this is actually great for. And I do run into this somewhat often: relatively new libraries or frameworks that don't really care about thorough documentation so they only show you a few happy path snippets and nothing about how to do something more interesting, so you have to bridge the gap between "this one line in the doc obviously doesn't work with me, but I'd like to figure it out without reading all their source code from scratch..." - getting more example snippets barfed up onto my screen from other people who've figured it out before could be a sort of replacement for the library writers having provided documentation in the first place. But ... this is a somewhat insane way to work around a problem of shitty code documentation, and is still insufficient in a couple ways: * some poor bastard is going to have to be the first person to figure out how to do something, so that copilot itself can know * any non-code nuances around "oh, if you do that, your memory usage is going to explode" or "oh, by the way, if you do that, make sure you don't do your own threading" will still fail to be communicated.
groby_b 5y ago
I've called it "The Fully Mechanized Stack Overflow Brigade" before, and everything that comes to light supports that assessment. On the upside, think of the consultancy fees you can charge to clean up those messes.
heroHACK17 5y ago
This is my thought as well. I get the "make productive engineers even more productive" angle, but productive engineers' bottleneck isn't coding. Sure, coding up a boilerplate Go web server is tedious, but I have done it so many times that it takes me two seconds now. On the flip side, coding can be the bottleneck for the worst kind of coder. When I first started coding, coding was hard simply because I had very little reps and was just learning to understand how to code common solutions, data structures, libraries, etc. Fast forward a few years and, if I were still struggling to understand these concepts, Copilot is a lifeline.
- hamandcheese 5y ago
  I’m gonna have to disagree - coding can and does take significant amounts of time even when I know exactly what problem I am solving. I admit that at many organizations there are so many other factors and bottlenecks, but it’s not uncommon that I find myself 8+ hours deep into a coding task that I had expected would be much shorter. On the other hand, usually that’s due to refactoring or otherwise not being satisfied with the quality of my initial solution, so copilot probably wouldn’t help…
captn3m0 5y ago
I find it is reducing my research time by providing a decent starting solution space. Especially for boring stuff where you just need to google the signature of some standard library function.

I'm not really sure that type of tool could really be anything else. How would a model become aware of all of the various edge cases that depend on which SQL database you use or differences in language versions over time?

gmadsen 5y ago
a large data set covering exactly what you just mentioned?
sbr464 5y ago
Can it submit pull requests to itself with if/else boolean logic/hacks?
TeMPOraL 5y ago
> I'm not really sure that type of tool could really be anything else. It can't be, because they've chosen to use a deep learning approach. That makes it a dead end right from the start. > How would a model become aware of all of the various edge cases that depend on which SQL database you use or differences in language versions over time? A lot of things that we call "edge cases" are only a problem for humans. They're not "edge cases" from the point of view of the grammar / semantics of programming languages and libraries. The way a hypothetical, better Copilot could work, is by having directly encoded grammars and semantics metadata corresponding to popular languages and tools. It could generate code in principled and introspectable way, by having a model of the computation it wants to express and encoding it in a target language. Of course, such hypothetical Copilot is a harder task - someone would have to come up with a structure for explicitly representing understanding of the abstract computation the user wants to happen, and then translate user input into that structure. That's a lot of drudgery, and from my vague understanding of the "classical" AI space, there might be a bunch of unsolved problems on the way. Real Copilot uses DNNs, because they let you ignore all that - you just keep shoving code at it, until the black-box model starts to give you mostly correct answers. The hard work is done automagically. It makes sense for some tasks, less for others - and I think code generation is one of those things where black-box DNNs are a bad idea.
- jhgb 5y ago
  > The way a hypothetical, better Copilot could work, is by having directly encoded grammars and semantics metadata corresponding to popular languages and tools. It could generate code in principled and introspectable way, by having a model of the computation it wants to express and encoding it in a target language. But that sounds like too much work, let's just throw a lot of data into an NN and see what comes out! /s > and introspectable Which most importantly means "debuggable", I assume. From what I get there doesn't seem to be any way to ad-hoc fix an NN's output.

codyb 5y ago

Hmm... I mean, these all seem like mistakes I could make and I don't think I'm the "worst kind of coder". The currency one I learned a while back, but it's not like I intuited using integers by default. Value being a reserved keyword, I'm not sure I'd know that and I do Postgres work as part of my myriad duties at the startup I work at. Maybe I'd make that mistake in a migration, maybe I have already. In a way, is it much different then what we do now as engineers? I'm hard pressed to call it much of an engineering discipline considering most teams I work on barely do design reviews before they launch in to writing code, documentation and meeting minutes are generally an afterthought, and the code review process while decent isn't perfect either and often times relies on arcane knowledge derived over months and years of wrangling with particular <framework, project, technology>. It's pretty neat, presumably it'll learn as people correct it, and it'll get better over time. I mean it's not even version one. I get the concerns, but I think they're a bit overblown, and this'll be really useful for people who want to learn how to code. Sure they'll run into some bugs, but, I mean, they were going to do that anyways.

voakbasda 5y ago
Is this any worse? Maybe not. Is it better? Absolutely not. This kind of tool will only further entrench the production of mediocre, bug-ridden code that plagues the world. As implemented, this will not be a solution; it is a express lane in the race to the bottom.
- pasquinelli 5y ago
  it is a race to the bottom, and people are trying to win. any skilled trade is being turned into an unskilled job. it might suck, the results might suck, but it's more profitable, and that's what matters.

majormajor 5y ago

It takes what should be your method of last resort - copypaste - and makes it the first thing you try. All the steps in between - looking at the docstring for the function you're calling, googling for more general information, looking at and deciding not to use not-applicable or poorly-written SO answers - get pushed aside. So instead of you having to convince yourself "yes, it's safe to copy-paste these lines from SO, they actually fit my problem" you're presented with magic and I think the burden for rejecting it is going to be higher once it's in your editor than when you're just reading it on a SO post or Github snippet. Even for a newcomer looking to learn, working on simple stuff that it has great completions for, it seems like it will sabotage your long-term growth, since it takes all the why and the reasoning out of it. Autocomplete for a function name isn't that relevant to gaining a deeper understanding. Knowing why a certain block of code is passed in in a certain style, or needs to be written at all? Probably that is.

majormajor 5y ago
Thinking about it more: there's a very small subset of problems that I think this is actually great for. And I do run into this somewhat often: relatively new libraries or frameworks that don't really care about thorough documentation so they only show you a few happy path snippets and nothing about how to do something more interesting, so you have to bridge the gap between "this one line in the doc obviously doesn't work with me, but I'd like to figure it out without reading all their source code from scratch..." - getting more example snippets barfed up onto my screen from other people who've figured it out before could be a sort of replacement for the library writers having provided documentation in the first place. But ... this is a somewhat insane way to work around a problem of shitty code documentation, and is still insufficient in a couple ways: * some poor bastard is going to have to be the first person to figure out how to do something, so that copilot itself can know * any non-code nuances around "oh, if you do that, your memory usage is going to explode" or "oh, by the way, if you do that, make sure you don't do your own threading" will still fail to be communicated.

groby_b 5y ago

I've called it "The Fully Mechanized Stack Overflow Brigade" before, and everything that comes to light supports that assessment. On the upside, think of the consultancy fees you can charge to clean up those messes.

heroHACK17 5y ago

This is my thought as well. I get the "make productive engineers even more productive" angle, but productive engineers' bottleneck isn't coding. Sure, coding up a boilerplate Go web server is tedious, but I have done it so many times that it takes me two seconds now. On the flip side, coding can be the bottleneck for the worst kind of coder. When I first started coding, coding was hard simply because I had very little reps and was just learning to understand how to code common solutions, data structures, libraries, etc. Fast forward a few years and, if I were still struggling to understand these concepts, Copilot is a lifeline.

hamandcheese 5y ago
I’m gonna have to disagree - coding can and does take significant amounts of time even when I know exactly what problem I am solving. I admit that at many organizations there are so many other factors and bottlenecks, but it’s not uncommon that I find myself 8+ hours deep into a coding task that I had expected would be much shorter. On the other hand, usually that’s due to refactoring or otherwise not being satisfied with the quality of my initial solution, so copilot probably wouldn’t help…

captn3m0 5y ago

I find it is reducing my research time by providing a decent starting solution space. Especially for boring stuff where you just need to google the signature of some standard library function.