15 ms·

When you see people write articles about copilot like this, it’s clear they haven’t understood how it works, specifically that it is not a “one shot” generator

When you see people write articles about copilot like this, it’s clear they haven’t understood how it works, specifically that it is not a “one shot” generator like chatgpt, and uses other files in your project as part of the prompt [1]. If you’re getting inconsistent results, it’s most likely you’re doing experiments in a single project, and opening a “new file” expecting it to be a “clean slate” like a “new chat” in chat gpt. …but codex doesn’t work that way. If you find it “suddenly” figures out how to do something, particularly something you’ve done before, it’s most likely your solution is being fed in as part of the prompt, not that you’ve figured out some magical prompting keywords. Copilot just isn’t that good at code gen. Personally I’ve found that codex is very predictable in terms of performance and consistency, but, broadly speaking, an order of magnitude less capable than chatgpt. [1] - https://thakkarparth007.github.io/copilot-explorer/posts/copilot-internals.html#secret-sauce-1-prompt-engineering https://thakkarparth007.github.io/copilot-explorer/posts/cop...

SOLAR_FIELDS 3y ago
I’ve been using copilot for a few months now and I wouldn’t consider it as anything more than really good autocomplete. I don’t expect it to write code for me like ChatGPT does or really just do anything other than save me the hassle of typing out repetitive lines in testing and config files. And for that it’s great! Absolutely worth the $20 USD a month or whatever I pay for it It’s wild how much the narrative has changed around copilot since the release of ChatGPT. When copilot first came out there was so much breathless hemming and hawing over how it was so problematic for violating licenses. Turns out, it’s not really good enough to even be used in such a way that it could do that
- still_grokking 3y ago
  Oh, it think it's good enough to violate licenses. Because there is not much to it given how draconian copyright gets exercised in the western world. Copy paste a function signature… Copyright issue… Ask for example Google about that. (I mean the company, not necessary the search engine.) The class actions against MS & Friends regarding Copilot are still rolling: https://githubcopilotlitigation.com/case-updates.html https://githubcopilotlitigation.com/case-updates.html
  - naikrovek 3y ago
    Google didn't get sued because of a single method signature, they got sued because they copied the function signatures of an entire language's standard library completely. that's a very different thing than copying a method signature here or there, and the suit was about the collection as a whole and would not have made it into a courtroom if it were just a few method signatures.
spenczar5 3y ago
Agreed completely. In fact, one can start to write code in a copilot-friendly manner to take advantage of this. A nit, though: codex is the model itself, and doesn’t do anything to slurp up nearby files. That work is entirely done client-side, and is part of the “UI” of the tool. Code is heuristically scanned for relevant snippets using an 11-dimensional logistic regression model which is bundled into the copilot source code and executes locally, and those snippets are included automatically as preamble comments in the prompt. Thakkar Parth’s work showed this and you can further explore it by running a MITM proxy to capture the network activity and look at the prompts directly.

SOLAR_FIELDS 3y ago

I’ve been using copilot for a few months now and I wouldn’t consider it as anything more than really good autocomplete. I don’t expect it to write code for me like ChatGPT does or really just do anything other than save me the hassle of typing out repetitive lines in testing and config files. And for that it’s great! Absolutely worth the $20 USD a month or whatever I pay for it It’s wild how much the narrative has changed around copilot since the release of ChatGPT. When copilot first came out there was so much breathless hemming and hawing over how it was so problematic for violating licenses. Turns out, it’s not really good enough to even be used in such a way that it could do that

still_grokking 3y ago
Oh, it think it's good enough to violate licenses. Because there is not much to it given how draconian copyright gets exercised in the western world. Copy paste a function signature… Copyright issue… Ask for example Google about that. (I mean the company, not necessary the search engine.) The class actions against MS & Friends regarding Copilot are still rolling: https://githubcopilotlitigation.com/case-updates.html https://githubcopilotlitigation.com/case-updates.html
- naikrovek 3y ago
  Google didn't get sued because of a single method signature, they got sued because they copied the function signatures of an entire language's standard library completely. that's a very different thing than copying a method signature here or there, and the suit was about the collection as a whole and would not have made it into a courtroom if it were just a few method signatures.

spenczar5 3y ago

Agreed completely. In fact, one can start to write code in a copilot-friendly manner to take advantage of this. A nit, though: codex is the model itself, and doesn’t do anything to slurp up nearby files. That work is entirely done client-side, and is part of the “UI” of the tool. Code is heuristically scanned for relevant snippets using an 11-dimensional logistic regression model which is bundled into the copilot source code and executes locally, and those snippets are included automatically as preamble comments in the prompt. Thakkar Parth’s work showed this and you can further explore it by running a MITM proxy to capture the network activity and look at the prompts directly.