3 ms·

From my experience, Github Copilot is useful for its seamless autocompletes. But Github Copilot chat, a new feature, noticeably lags behind GPT-4. I get far b

by jdlyga 2y ago

From my experience, Github Copilot is useful for its seamless autocompletes. But Github Copilot chat, a new feature, noticeably lags behind GPT-4. I get far better performance copying and pasting code into ChatGPT (GPT-4) than using Github Copilot. Both definitely help productivity, especially when generating code using an unfamiliar framework or library. It's great for generating skeleton code for small programs using new libraries, but it in its current form can only get you around 80% of the way there. You will spend hours fighting with the prompt to try and get the remaining 20% complete when it's much easier to just finish it yourself.

simonw 2y ago
Weird, because Copilot chat /is/ GPT-4 as far as I know. Or maybe they run it on 3.5?
- heethjain21 2y ago
  It runs both. Had seen this on the discussions on Github. https://github.com/community/community/discussions/58059#discussioncomment-7900899 https://github.com/community/community/discussions/58059#dis...
  - simonw 2y ago
    That original note about it using both 4 and 3.5 was from this comment on November 2nd: https://github.com/microsoft/vscode-copilot-release/issues/6#issuecomment-1791319670 https://github.com/microsoft/vscode-copilot-release/issues/6... GPT-4 Turbo - the much cheaper version of 4 - was announced on November 6th. So there's a good change they've switched to that at some point in the last 6 months.
- anonzzzies 2y ago
  They mix it to save on compute.
fendy3002 2y ago
And for those who say that it doesn't speed up development, then maybe they're right, because the main benefit isn't at speed. It reduce the devs mental / cognitive load and pass it to the ai instead, we just need to check and review the result. If it doesn't monumentally increase the speed, it helps devs to stay productive longer
- frizlab 2y ago
  Reviewing code is famously harder than writing it. I don’t see how it’s a gain to have to review more code than to write.
  - KaoruAoiShiho 2y ago
    It's shorter, you're working directly with the code base you're already thinking about, and it's plugged into it rather than having to start a new mental task when you're looking at a pull request.
    - Seb-C 2y ago
      When I write code, I usually get a complete understanding of the problem and constraints. I have a picture in my mind of all the edge cases and branching that can happen, which most of the time allows me to write correct code without bugs. When I review code, I am completely unable to get this mental picture, and I usually miss a lot of issues that I would otherwise have avoided. Which is why I don't see the point of Copilot.
      - KaoruAoiShiho 2y ago
        When you review pull requests you're unable to get the mental picture. But it's different with copilot because it fits in your mental picture, you can compartmentalize what you're working on, and the copilot output can come in the form of small pieces that are easily understood and slot in the prepared places where you need it. I agree 100% that reviewing human code is a lot of work but not with copilot IMO because of the immediacy of it all.
bmitc 2y ago
This has been exactly my experience. It is great at autocomplete and boilerplate stuff. It's very hit and miss for StackOverflow type of stuff. And it's usually a complete miss when you ask it something where it needs to understand even basic things of what you're actually trying to do. It can also get very confused and circular when it hits those edges as well. I've found myself fighting with it, and so part of my use of it has been learning when to let it go.
- collyw 2y ago
  I find ChatGPT 4 is the best place to ask a question, then often use it's answer to find more in docs or stack overflow. It's basically a better search engine for certain coding tasks. I wasn't overly impressed with ChatGPT 3.5 as it gave me wrong answers enough of the time that it cancelled out the amount of time it saved me.
collyw 2y ago
That's been my experience as well. Doesn't help me that much in Django where I am pretty familiar. It helps me loads in the frontend, where I am less familiar.

by jdlyga 2y ago

simonw 2y ago

Weird, because Copilot chat /is/ GPT-4 as far as I know. Or maybe they run it on 3.5?

heethjain21 2y ago
It runs both. Had seen this on the discussions on Github. https://github.com/community/community/discussions/58059#discussioncomment-7900899 https://github.com/community/community/discussions/58059#dis...
- simonw 2y ago
  That original note about it using both 4 and 3.5 was from this comment on November 2nd: https://github.com/microsoft/vscode-copilot-release/issues/6#issuecomment-1791319670 https://github.com/microsoft/vscode-copilot-release/issues/6... GPT-4 Turbo - the much cheaper version of 4 - was announced on November 6th. So there's a good change they've switched to that at some point in the last 6 months.
anonzzzies 2y ago
They mix it to save on compute.

fendy3002 2y ago

And for those who say that it doesn't speed up development, then maybe they're right, because the main benefit isn't at speed. It reduce the devs mental / cognitive load and pass it to the ai instead, we just need to check and review the result. If it doesn't monumentally increase the speed, it helps devs to stay productive longer

frizlab 2y ago
Reviewing code is famously harder than writing it. I don’t see how it’s a gain to have to review more code than to write.
- KaoruAoiShiho 2y ago
  It's shorter, you're working directly with the code base you're already thinking about, and it's plugged into it rather than having to start a new mental task when you're looking at a pull request.
  - Seb-C 2y ago
    When I write code, I usually get a complete understanding of the problem and constraints. I have a picture in my mind of all the edge cases and branching that can happen, which most of the time allows me to write correct code without bugs. When I review code, I am completely unable to get this mental picture, and I usually miss a lot of issues that I would otherwise have avoided. Which is why I don't see the point of Copilot.
    - KaoruAoiShiho 2y ago
      When you review pull requests you're unable to get the mental picture. But it's different with copilot because it fits in your mental picture, you can compartmentalize what you're working on, and the copilot output can come in the form of small pieces that are easily understood and slot in the prepared places where you need it. I agree 100% that reviewing human code is a lot of work but not with copilot IMO because of the immediacy of it all.

bmitc 2y ago

This has been exactly my experience. It is great at autocomplete and boilerplate stuff. It's very hit and miss for StackOverflow type of stuff. And it's usually a complete miss when you ask it something where it needs to understand even basic things of what you're actually trying to do. It can also get very confused and circular when it hits those edges as well. I've found myself fighting with it, and so part of my use of it has been learning when to let it go.

collyw 2y ago
I find ChatGPT 4 is the best place to ask a question, then often use it's answer to find more in docs or stack overflow. It's basically a better search engine for certain coding tasks. I wasn't overly impressed with ChatGPT 3.5 as it gave me wrong answers enough of the time that it cancelled out the amount of time it saved me.

collyw 2y ago

That's been my experience as well. Doesn't help me that much in Django where I am pretty familiar. It helps me loads in the frontend, where I am less familiar.