4 ms·

The moment I can train Copilot on my codebase and improve its predictions, I’d gladly pay more than $20. Even now, I find that I can’t live without it simply as

The moment I can train Copilot on my codebase and improve its predictions, I’d gladly pay more than $20. Even now, I find that I can’t live without it simply as a slightly more intelligent auto-complete.

reustle 3y ago
Check out https://cursor.so https://cursor.so - It does this pretty well without requiring much setup.
bloopernova 3y ago
Yeah, I really, really want to be able to configure copilot to explicitly include directories to scan in addition to its built-in corpus of knowledge. That would increase its worth to me a lot.
- visarga 3y ago
  That's what I expected Copilot 2.0 to be, what a bummer.
grogenaut 3y ago
I've used some prototypes that tried it. One of the things it does it regurgitate the old patterns you want to stop using and not the new stuff you do. Basically it's a copy pasta tech debt generator.
- js8 3y ago
  That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders.
  - visarga 3y ago
    They are great information meat grinders, the best. /s Prompting allows you to direct the model into a different way than the training data. If it were not so, LLMs would never solve problems that were not explicitly in their training set.
  - criley2 3y ago
    >That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders. This is an issue with your prompting not the models. If you tell someone "Do this thing" they'll just do it, LLM's too. And how it is done will probably be terrible. If you ask someone "come up with 5 ways to accomplish this, compare and contrast between them, list pros and cons, think through best practices, maintainability, readability, security, performance and cost. Come to a final decision and then do it" their response will be different. So too will the LLM's. You can even have the LLM present its opinions and recommendation but pause and wait for your choice on the go-ahead.
    - js8 3y ago
      Fair enough, let's put aside that in Copilot, prompting is automatic. Even if you do prompt it correctly, and it responded in the abstract with all these pros and cons, you cannot be reasonably sure that the code it also provided actually follows these best practices. It will just anytime mindlessly wander from "the best experts on the Internet are saying this" territory to "I just made this up" territory. Compare that with googling a human-written stack overflow answer on the topic, there usually is some good soul who pointed out the inconsistency, if there is one. Look, I like grinded meat. But the fact is, without a detailed analysis, it's hard to tell what's actually in it.
      - criley2 3y ago
        >Even if you do prompt it correctly, and it responded in the abstract with all these pros and cons, you cannot be reasonably sure that the code it also provided actually follows these best practices. It will just anytime mindlessly wander from "the best experts on the Internet are saying this" territory to "I just made this up" territory. And? As a Sr. Engineer, this is exactly how all the slop my Jr's and Mids send me looks. Copy pasted, Stack Overflow, and when I read through Jr code my mind is boggled. I want to shake them "did you even read your own code. do you even know what your code is doing? Why did you do this?" Forget style, best practices, etc, In my opinion, LLM's produce code that is as good or better than most Jr engineers and as a Sr it is my responsibility to audit, review and test all code. As a Sr level engineer I spend 90% of my time judging/fixing/improving others code, and less than 10% of my time writing my own. LLM is just another source, and unlike the Jr, I can quickly ask it why it did what it did or to refine it. You ask the Jr to iterate on the project and you won't hear from them until they mention a blocker at tomorrow's stand-up (or you just pair it out and spend 2 hours teaching them, while the LLM turns it around in 15 seconds. There is value in teaching of course, but we build quickly, too).
  - ben_w 3y ago
    For me, "intelligence"[0] is a real-valued vector rather than a boolean. If it was a boolean, the limits you describe would certainly exclude some of the human programmers whose nonsense I have had to fix. [0] and "generality", though "AGI when?" is a separate topic
- falcor84 3y ago
  I've actually had the opposite happen to me several times - of trying to get Copilot to help me fix something I wrote that isn't quite working, and instead of giving me an alternative solution, it just returns my bad code.
lvncelot 3y ago
I would love for Copilot to receive lsp information together with the raw code, I think that could already provide it with enough context, especially in strongly typed codebases.
benterix 3y ago
I have mixed feeling on this. On the one hand, sometimes it's able to guess correctly and save me a couple of seconds, up to half a minute I think. On the other hand, very often it's completely wrong, so I needed to constantly verify the suggestions and I found out it interrupted my flow and in the end I'd code faster without it.
- visarga 3y ago
  Maybe you just need shorter predictions, like one single line or 3-4 words
_fat_santa 3y ago
I tried Copilot in the early days but stopped using it pretty quickly as it just wasn't that useful in an already established codebase. Often times I'm not writing completely new code but rather implementing a pattern that I've previously defined somewhere else. In this case Copilot just completely falls over because it doesn't know about that existing code. My other gripe is language specific but in VSCode it breaks Typescript hints, often times I'm writing something like `MyCustomObject.` and expect Typescript to help me autofill the field, this breaks with Copilot as it always tries to suggest some other code. The second issue I could work around but combined with the fact that it doesn't have context of my current codebase, my impression was that it would be a fantastic tool for someone going to school for CS but practically useless in a professional codebase.
- stephendause 3y ago
  It's very interesting to me that you say that because it hasn't been my experience. I am working in a ~20k-50k SLOC codebase with a few common patterns, and it seems to have picked up on most of them. It's not always right, but even when it isn't, I can write a comment above an empty line to nudge it in the right direction.
- simonw 3y ago
  Copilot pulls in similar code from your current project as part of its prompt. If you understand how it works you can sometimes lay out your code in a way that makes it more likely to include the relevant examples to get the effect that you want. There is no guidance or documentation for this at all! I've been running a hack to help me see the prompts in order to better understand what's going on: https://gist.github.com/simonw/30fec0ac20244b4b921532b63b8b04e5 https://gist.github.com/simonw/30fec0ac20244b4b921532b63b8b0...
- GrinningFool 3y ago
  Could it be the language and/or the editor's integration? In neovim w/ a medium-sized go project, I"m finding that it's really good at providing meaningful and at least directionally-accurate contextual suggestions. It seems that the plugin is providing a good bit of context, but maybe the vscode one doesn't?
esalman 3y ago
It's amazing and all, but can be so subtly wrong. I let it autocomplete a function for generating time zone offset. It sneaked a minus sign in there so central US offset would be +0600 instead of -0600. My manager caught it during code review. We both are proponents of copilot but it was still embarrassing for me.
- gwern 3y ago
  Sounds like something a few tests would have caught. I've found my GPT-4 coding use to be much more satisfactory when I have it generate a few dozen test cases after finishing (either my code or its code), and then auto-repair based on any errors.

reustle 3y ago

Check out https://cursor.so https://cursor.so - It does this pretty well without requiring much setup.

bloopernova 3y ago

Yeah, I really, really want to be able to configure copilot to explicitly include directories to scan in addition to its built-in corpus of knowledge. That would increase its worth to me a lot.

visarga 3y ago
That's what I expected Copilot 2.0 to be, what a bummer.

grogenaut 3y ago

I've used some prototypes that tried it. One of the things it does it regurgitate the old patterns you want to stop using and not the new stuff you do. Basically it's a copy pasta tech debt generator.

js8 3y ago
That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders.
- visarga 3y ago
  They are great information meat grinders, the best. /s Prompting allows you to direct the model into a different way than the training data. If it were not so, LLMs would never solve problems that were not explicitly in their training set.
- criley2 3y ago
  >That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders. This is an issue with your prompting not the models. If you tell someone "Do this thing" they'll just do it, LLM's too. And how it is done will probably be terrible. If you ask someone "come up with 5 ways to accomplish this, compare and contrast between them, list pros and cons, think through best practices, maintainability, readability, security, performance and cost. Come to a final decision and then do it" their response will be different. So too will the LLM's. You can even have the LLM present its opinions and recommendation but pause and wait for your choice on the go-ahead.
  - js8 3y ago
    Fair enough, let's put aside that in Copilot, prompting is automatic. Even if you do prompt it correctly, and it responded in the abstract with all these pros and cons, you cannot be reasonably sure that the code it also provided actually follows these best practices. It will just anytime mindlessly wander from "the best experts on the Internet are saying this" territory to "I just made this up" territory. Compare that with googling a human-written stack overflow answer on the topic, there usually is some good soul who pointed out the inconsistency, if there is one. Look, I like grinded meat. But the fact is, without a detailed analysis, it's hard to tell what's actually in it.
    - criley2 3y ago
      >Even if you do prompt it correctly, and it responded in the abstract with all these pros and cons, you cannot be reasonably sure that the code it also provided actually follows these best practices. It will just anytime mindlessly wander from "the best experts on the Internet are saying this" territory to "I just made this up" territory. And? As a Sr. Engineer, this is exactly how all the slop my Jr's and Mids send me looks. Copy pasted, Stack Overflow, and when I read through Jr code my mind is boggled. I want to shake them "did you even read your own code. do you even know what your code is doing? Why did you do this?" Forget style, best practices, etc, In my opinion, LLM's produce code that is as good or better than most Jr engineers and as a Sr it is my responsibility to audit, review and test all code. As a Sr level engineer I spend 90% of my time judging/fixing/improving others code, and less than 10% of my time writing my own. LLM is just another source, and unlike the Jr, I can quickly ask it why it did what it did or to refine it. You ask the Jr to iterate on the project and you won't hear from them until they mention a blocker at tomorrow's stand-up (or you just pair it out and spend 2 hours teaching them, while the LLM turns it around in 15 seconds. There is value in teaching of course, but we build quickly, too).
- ben_w 3y ago
  For me, "intelligence"[0] is a real-valued vector rather than a boolean. If it was a boolean, the limits you describe would certainly exclude some of the human programmers whose nonsense I have had to fix. [0] and "generality", though "AGI when?" is a separate topic
falcor84 3y ago
I've actually had the opposite happen to me several times - of trying to get Copilot to help me fix something I wrote that isn't quite working, and instead of giving me an alternative solution, it just returns my bad code.

lvncelot 3y ago

I would love for Copilot to receive lsp information together with the raw code, I think that could already provide it with enough context, especially in strongly typed codebases.

benterix 3y ago

I have mixed feeling on this. On the one hand, sometimes it's able to guess correctly and save me a couple of seconds, up to half a minute I think. On the other hand, very often it's completely wrong, so I needed to constantly verify the suggestions and I found out it interrupted my flow and in the end I'd code faster without it.

visarga 3y ago
Maybe you just need shorter predictions, like one single line or 3-4 words

_fat_santa 3y ago

I tried Copilot in the early days but stopped using it pretty quickly as it just wasn't that useful in an already established codebase. Often times I'm not writing completely new code but rather implementing a pattern that I've previously defined somewhere else. In this case Copilot just completely falls over because it doesn't know about that existing code. My other gripe is language specific but in VSCode it breaks Typescript hints, often times I'm writing something like `MyCustomObject.` and expect Typescript to help me autofill the field, this breaks with Copilot as it always tries to suggest some other code. The second issue I could work around but combined with the fact that it doesn't have context of my current codebase, my impression was that it would be a fantastic tool for someone going to school for CS but practically useless in a professional codebase.

stephendause 3y ago
It's very interesting to me that you say that because it hasn't been my experience. I am working in a ~20k-50k SLOC codebase with a few common patterns, and it seems to have picked up on most of them. It's not always right, but even when it isn't, I can write a comment above an empty line to nudge it in the right direction.
simonw 3y ago
Copilot pulls in similar code from your current project as part of its prompt. If you understand how it works you can sometimes lay out your code in a way that makes it more likely to include the relevant examples to get the effect that you want. There is no guidance or documentation for this at all! I've been running a hack to help me see the prompts in order to better understand what's going on: https://gist.github.com/simonw/30fec0ac20244b4b921532b63b8b04e5 https://gist.github.com/simonw/30fec0ac20244b4b921532b63b8b0...
GrinningFool 3y ago
Could it be the language and/or the editor's integration? In neovim w/ a medium-sized go project, I"m finding that it's really good at providing meaningful and at least directionally-accurate contextual suggestions. It seems that the plugin is providing a good bit of context, but maybe the vscode one doesn't?

esalman 3y ago

It's amazing and all, but can be so subtly wrong. I let it autocomplete a function for generating time zone offset. It sneaked a minus sign in there so central US offset would be +0600 instead of -0600. My manager caught it during code review. We both are proponents of copilot but it was still embarrassing for me.

gwern 3y ago
Sounds like something a few tests would have caught. I've found my GPT-4 coding use to be much more satisfactory when I have it generate a few dozen test cases after finishing (either my code or its code), and then auto-repair based on any errors.