9 ms·

I was pretty sceptical of Copilot when it was announced, but after having used it for a while I think it's more that it's been kind of sold as something that it

I was pretty sceptical of Copilot when it was announced, but after having used it for a while I think it's more that it's been kind of sold as something that it's not. What it does offer for me, in practice, is basically "clever autocomplete". Say I wrote a little bit of code: bounds.max.x = bounds.min.x + width Then copilot will suggest that the next line is: bounds.max.y = bounds.min.y + height That's reasonably smart, and it adapts pretty well to whatever code you are currently writing. It makes writing those little bits of slightly repetitive code less annoying, and I like that – it feels like a useful productivity boost in the same way as regular "dumb" autocomplete is. However I'd say too much of the initial coverage was along the lines of "you can just write the function name and it will fill in the body!". That will work if you want to write `isEven` or `capitalise` or something, which again is quite nice. But I have found that basically any other prompts product a function that either does something completely wrong, or introduces a subtle and non-obvious bug. And this actually feels kind of dangerous – I'd absolutely caught myself a few times from just going "it'll probably be fine" and accepting whatever Copilot has spewed out. I'll remain sceptical of it as a system that sucks up and regurgitates vast lakes of open-source code and sends my every keystroke to Microsoft, among other things. But it's definitely something I'd pay for a local version of.

kannanvijayan 5y ago
I agree with your last point - with an added dash of being wary of getting dependent on sophisticated tools that have one vendor. I'm not skeptical of the idea at all, however. I see it as the rough analogy of mathematicians using proof assistants. This is a coding assistant. It's generally true that the vast, vast majority of code one writes is boilerplate or otherwise rote. The core logic and control flow that defines the meaningful bits of any complex program are nested in the middle of tons of error handling, logging, data munging, incidental book-keeping, and other very prosaic tasks. It will not just be handy to have a friendly intelligence assist in these tasks. It doesn't require much "intelligence" (of the sort required for figuring out what the program should do in the first place and how to architect it). Once the technology gets firmed up a bit, the productivity multiplier will be palpable and impossible to ignore. Programmers using these tools will complete tasks faster than ones that don't, and the market will shift. Further extensions of this technology lead to automatic generation of unit tests, automatic refactoring, etc. These all exist as very specialized tools that are explicitly coded, but it's clear to see that adding a bit of ML-driven intuition to them would greatly increase their scope and effectiveness.
ianai 5y ago
Thank you, I was wondering what the real use was like.
kromem 5y ago
That's where it's at...today. We're only a year into this thing existing. I'd be VERY surprised if the data from all the interactions with it don't result in a product people care less and less about Microsoft harvesting their data from because of the value produced in exchange. By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition.
- abel_ 5y ago
  As someone else already mentioned, the scaling laws paint a different story empirically: we haven't hit diminishing returns at all, and there's no end in sight. But more anecdotally, the first applied neural network paper in 1989 by LeCunn has pretty much the same format as the GPT paper: a large neural network trained on a large dataset (all relative to the era). https://karpathy.github.io/2022/03/14/lecun1989/ https://karpathy.github.io/2022/03/14/lecun1989/ It really just seems that there are a certain number of flops you need before certain capabilities can emerge.
- baryphonic 5y ago
  > By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition. Is there a single example in AI, or even technology as a whole, where simply continuing to apply one technique has led to compounding growth? Even Moore's Law, according to none other than Jim Keller[1], is more a consequence of thousands of individual innovations that are each quite distinct from others but that build on each other to create this compounding growth we see. There is no similar curve for AI. In this case, GPT-3 (released in 2020) uses the same architecture as GPT-2 (released in 2019), expanded to have ~100x more parameters. It's not hard to see that compounding this process will rapidly hit diminishing returns quickly in terms of time, power consumption, cost of hardware, etc. Honestly, if Google, Amazon and Microsoft didn't see increased computational cost as a financial benefit for their cloud services, people might be willing to admit that GPT-3 is a diminishing return itself: for 100x parameters, is GPT-3 over 100x better than GPT-2? It seems that the big quantum leaps in AI come from new architectures applied in just the right way. CNNs and now transformers (based on multi-head attention) are the ways we've found to scale this thing, but those seem to come around every 25 years or so. Even the classes of problems they solve seem to change discretely and then approach some asymptote. Copilot will probably improve, but I doubt we will see much compounding. My best guess is that Copilot will steadily improve "arithmetically" as users react to its suggestions, or even that it will just change sporadically in response to this. [0]https://youtu.be/Nb2tebYAaOA?t=1975 https://youtu.be/Nb2tebYAaOA?t=1975
karmicthreat 5y ago
Yea, this is exactly where I am at with Copilot. So github copilot is pretty neat I entered this code: result_dataFrame['yearOverYearMin'] = year_dataFrame.groupby('partId')['unitCost'].min() result_dataFrame['yearOverYearMax'] = year_dataFrame.groupby('partId')['unitCost'].max() Then when I put in result_dataFrame['yearOverYearChange'] Copilot gave me result_dataFrame['yearOverYearChange'] = result_dataFrame['yearOverYearMax'] - result_dataFrame['yearOverYearMin'] So it's a fancy context aware autocomplete. And big productivity booster on code entry.
- verhovsky 5y ago
  I've had Copilot teach/tell me about things I didn't know I needed to be considering. I was writing a parser for URL query strings (?these=things&key=value) and at one point Copilot suggested return [decodeURIComponent(key), decodeURIComponent(value)]; and until I saw that I didn't even know that percent encoded text in URLs is something I needed to be thinking about.
- hackernewds 5y ago
  Absolutely cannot wait for this to come to SQL. Or is there a way already to adopt this?
- mordymoop 5y ago
  It is also a lifesaver for complicated fiddly array index code. One of the most cognitively costly thing for me, decades into programming, is still anything related to array partitioning, or remapping indices. Copilot pretty quickly just gets what you’re trying to do and auto fills a correct line of code. It’s basically never wrong about this sort of thing. And even in the rare case where it’s not correct (usually because it’s mistaken about what I’m trying to do), it’s much faster for me to read the line and figure out what it does than it is to mentally slog through the problem and write the line from scratch.
- csee 5y ago
  Is there a way to fast track being able to try out Copilot? I put my email down a few days after it was announced and haven't heard anything.
abecedarius 5y ago
I wonder if the usefulness varies with your approach to comments. (Haven't used Copilot yet because it didn't have Emacs support.)
Zerverus 5y ago
It’s also great at translation. Actually really great down down to interpolated strings containing variables. Take English strings.en.json, copy to strings.th.json, open side by side, delete English text and watch copilot fill in the thai translation
- 2muchcoffeeman 5y ago
  Are the translations any good though?
udbhavs 5y ago
This could be useful for languages like Go where simple repetition and loops are preferred over clever language features like object or array destructuring that can add cognitive or performance overhead.
sireat 5y ago
I too was pretty skeptical of Copilot when I started using it last summer. However it is shockingly good for filling out Python snippets - ie smarter autocomplete when teaching. Popular libraries like Pandas, Beautiful Soup, Flask are perfect for this. About 80% time it will fill out the code exactly they way I would want. About 10% time it will be something you want to correct or nudge. Then about 10% of time it will be a howler or anti-pattern. Then you simply explain it to students why it is not so great to say insert something at a beginning of a Python list. Edit: Copilot is also great for filling out comments when teaching # it can generate Captain Obvious what comments and also some nice why type comments as well
teaearlgraycold 5y ago
It's also really good at writing test cases. Once you've written a couple test cases manually it can take the "it should" string and auto-complete the body of the test with very high accuracy.
alisonkisk 5y ago
siddboots 5y ago
I largely agree with this. Occasionally it suggests a few lines that are more or less what I want, but rarely more than that. One use case that I reach for relatively often is writing a comment to describe the one line of code I want when I can’t remember the specifics of the API I’m working with. E.g. # reset numpy random seed, and copilot pretty reliably gives me what I want.

kannanvijayan 5y ago

I agree with your last point - with an added dash of being wary of getting dependent on sophisticated tools that have one vendor. I'm not skeptical of the idea at all, however. I see it as the rough analogy of mathematicians using proof assistants. This is a coding assistant. It's generally true that the vast, vast majority of code one writes is boilerplate or otherwise rote. The core logic and control flow that defines the meaningful bits of any complex program are nested in the middle of tons of error handling, logging, data munging, incidental book-keeping, and other very prosaic tasks. It will not just be handy to have a friendly intelligence assist in these tasks. It doesn't require much "intelligence" (of the sort required for figuring out what the program should do in the first place and how to architect it). Once the technology gets firmed up a bit, the productivity multiplier will be palpable and impossible to ignore. Programmers using these tools will complete tasks faster than ones that don't, and the market will shift. Further extensions of this technology lead to automatic generation of unit tests, automatic refactoring, etc. These all exist as very specialized tools that are explicitly coded, but it's clear to see that adding a bit of ML-driven intuition to them would greatly increase their scope and effectiveness.

ianai 5y ago

Thank you, I was wondering what the real use was like.

kromem 5y ago

That's where it's at...today. We're only a year into this thing existing. I'd be VERY surprised if the data from all the interactions with it don't result in a product people care less and less about Microsoft harvesting their data from because of the value produced in exchange. By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition.

abel_ 5y ago
As someone else already mentioned, the scaling laws paint a different story empirically: we haven't hit diminishing returns at all, and there's no end in sight. But more anecdotally, the first applied neural network paper in 1989 by LeCunn has pretty much the same format as the GPT paper: a large neural network trained on a large dataset (all relative to the era). https://karpathy.github.io/2022/03/14/lecun1989/ https://karpathy.github.io/2022/03/14/lecun1989/ It really just seems that there are a certain number of flops you need before certain capabilities can emerge.
baryphonic 5y ago
> By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition. Is there a single example in AI, or even technology as a whole, where simply continuing to apply one technique has led to compounding growth? Even Moore's Law, according to none other than Jim Keller[1], is more a consequence of thousands of individual innovations that are each quite distinct from others but that build on each other to create this compounding growth we see. There is no similar curve for AI. In this case, GPT-3 (released in 2020) uses the same architecture as GPT-2 (released in 2019), expanded to have ~100x more parameters. It's not hard to see that compounding this process will rapidly hit diminishing returns quickly in terms of time, power consumption, cost of hardware, etc. Honestly, if Google, Amazon and Microsoft didn't see increased computational cost as a financial benefit for their cloud services, people might be willing to admit that GPT-3 is a diminishing return itself: for 100x parameters, is GPT-3 over 100x better than GPT-2? It seems that the big quantum leaps in AI come from new architectures applied in just the right way. CNNs and now transformers (based on multi-head attention) are the ways we've found to scale this thing, but those seem to come around every 25 years or so. Even the classes of problems they solve seem to change discretely and then approach some asymptote. Copilot will probably improve, but I doubt we will see much compounding. My best guess is that Copilot will steadily improve "arithmetically" as users react to its suggestions, or even that it will just change sporadically in response to this. [0]https://youtu.be/Nb2tebYAaOA?t=1975 https://youtu.be/Nb2tebYAaOA?t=1975

karmicthreat 5y ago

Yea, this is exactly where I am at with Copilot. So github copilot is pretty neat I entered this code: result_dataFrame['yearOverYearMin'] = year_dataFrame.groupby('partId')['unitCost'].min() result_dataFrame['yearOverYearMax'] = year_dataFrame.groupby('partId')['unitCost'].max() Then when I put in result_dataFrame['yearOverYearChange'] Copilot gave me result_dataFrame['yearOverYearChange'] = result_dataFrame['yearOverYearMax'] - result_dataFrame['yearOverYearMin'] So it's a fancy context aware autocomplete. And big productivity booster on code entry.

verhovsky 5y ago
I've had Copilot teach/tell me about things I didn't know I needed to be considering. I was writing a parser for URL query strings (?these=things&key=value) and at one point Copilot suggested return [decodeURIComponent(key), decodeURIComponent(value)]; and until I saw that I didn't even know that percent encoded text in URLs is something I needed to be thinking about.
hackernewds 5y ago
Absolutely cannot wait for this to come to SQL. Or is there a way already to adopt this?
mordymoop 5y ago
It is also a lifesaver for complicated fiddly array index code. One of the most cognitively costly thing for me, decades into programming, is still anything related to array partitioning, or remapping indices. Copilot pretty quickly just gets what you’re trying to do and auto fills a correct line of code. It’s basically never wrong about this sort of thing. And even in the rare case where it’s not correct (usually because it’s mistaken about what I’m trying to do), it’s much faster for me to read the line and figure out what it does than it is to mentally slog through the problem and write the line from scratch.
csee 5y ago
Is there a way to fast track being able to try out Copilot? I put my email down a few days after it was announced and haven't heard anything.

abecedarius 5y ago

I wonder if the usefulness varies with your approach to comments. (Haven't used Copilot yet because it didn't have Emacs support.)

Zerverus 5y ago

It’s also great at translation. Actually really great down down to interpolated strings containing variables. Take English strings.en.json, copy to strings.th.json, open side by side, delete English text and watch copilot fill in the thai translation

2muchcoffeeman 5y ago
Are the translations any good though?

udbhavs 5y ago

This could be useful for languages like Go where simple repetition and loops are preferred over clever language features like object or array destructuring that can add cognitive or performance overhead.

sireat 5y ago

I too was pretty skeptical of Copilot when I started using it last summer. However it is shockingly good for filling out Python snippets - ie smarter autocomplete when teaching. Popular libraries like Pandas, Beautiful Soup, Flask are perfect for this. About 80% time it will fill out the code exactly they way I would want. About 10% time it will be something you want to correct or nudge. Then about 10% of time it will be a howler or anti-pattern. Then you simply explain it to students why it is not so great to say insert something at a beginning of a Python list. Edit: Copilot is also great for filling out comments when teaching # it can generate Captain Obvious what comments and also some nice why type comments as well

teaearlgraycold 5y ago

It's also really good at writing test cases. Once you've written a couple test cases manually it can take the "it should" string and auto-complete the body of the test with very high accuracy.

alisonkisk 5y ago

siddboots 5y ago

I largely agree with this. Occasionally it suggests a few lines that are more or less what I want, but rarely more than that. One use case that I reach for relatively often is writing a comment to describe the one line of code I want when I can’t remember the specifics of the API I’m working with. E.g. # reset numpy random seed, and copilot pretty reliably gives me what I want.