6 ms·

Has anyone used Gemini Code Assist? I'm curious how it compares with Github Copilot and Cursor.

spotlmnop 2y ago
God, it sucks
spiralk 2y ago
The Aider leaderboards seem like a good practical test of coding usefulness: https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/. I haven't tried Cursor personally but I am finding Aider with Sonnet more useful that Github Copilot and its nice to be able to pick any model API. Eventually even a local model may be viable. This new Gemini model does not rank very high unfortunately.
- kendallchuang 2y ago
  Thanks for the link. That's unfortunate, though perhaps the benchmarks will be updated after this latest Gemini release. Cursor with Sonnet is great, I'll have to give Aider a try as well.
  - spiralk 2y ago
    It is updated actually, gemini-1.5-pro-002 is this new model.
    - kendallchuang 2y ago
      That was fast, I missed it!
  - KaoruAoiShiho 2y ago
    It's on the leaderboard, it's tied with qwen 2.5 72b and far below SOTA of o1, claude sonnet, and deepseek. (also below very old models like gpt-4-0314 lol)
dudus 2y ago
I use it and find it very helpful. Never tried cursor or copilot though
- therein 2y ago
  I tried Cursor the other day. It was actually pretty cool. My thought was, I'll open this open source project and use it to grok my way around the codebase. It was very helpful. After that I accidentally pasted an API secret into the document; had to consider it compromised and re-issued the credential.
therein 2y ago
I know you aren't necessarily talking about in-editor code assist but something about in-editor AI cloud code assist makes me super uncomfortable. It makes sense I need to be careful not to commit secrets to public repositories but now I have to avoid not only saving credentials into a file but even to paste them by accident into my editor?
mil22 2y ago
I have used Github Copilot extensively within VS Code for several months. The autocomplete - fast and often surprisingly accurate - is very useful. My only complaint is when writing comments, I find the completions distracting to my thought process. I tried Gemini Code Assist and it was so bad by comparison that I turned it off within literally minutes. Too slow and inaccurate. I also tried Codestral via the Continue extension and found it also to be slower and less useful than Copilot. So I still haven't found anything better for completion than Copilot. I find long completions, e.g. writing complete functions, less useful in general, and get the most benefit from short, fast, accurate completions that save me typing, without trying to go too far in terms of predicting what I'm going to write next. Fast is the key - I'm a 185 wpm on Monkeytype, so the completion had better be super low latency otherwise I'll already have typed what I want by the time the suggestion appears. Copilot wins on the speed front by far. I've also tried pretty much everything out there for writing algorithms and doing larger code refactorings, and answering questions, and find myself using Continue with Claude Sonnet, or just Sonnet or o1-preview via their native web interfaces, most of the time.
- imp0cat 2y ago
  Have you tried Gitlab Duo and if so, what are your thoughts on that?
  - mil22 2y ago
    Not yet, hadn't heard of it. Thanks for the suggestion.
- kendallchuang 2y ago
  I see, perhaps with Gemini because the model is larger it takes longer to generate the completions. I would expect with a larger model it would perform better on larger codebases. It sounds like for you, it's faster to work on a smaller model with shorter more accurate completions rather than letting the model guess what you're trying to write.
danmaz74 2y ago
I tried it briefly and didn't like it. On the other hand, I found Gemini pro better than sonnet or 4o at some more complex coding tasks (using continue.dev)

God, it sucks

The Aider leaderboards seem like a good practical test of coding usefulness: https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/. I haven't tried Cursor personally but I am finding Aider with Sonnet more useful that Github Copilot and its nice to be able to pick any model API. Eventually even a local model may be viable. This new Gemini model does not rank very high unfortunately.

kendallchuang 2y ago
Thanks for the link. That's unfortunate, though perhaps the benchmarks will be updated after this latest Gemini release. Cursor with Sonnet is great, I'll have to give Aider a try as well.
- spiralk 2y ago
  It is updated actually, gemini-1.5-pro-002 is this new model.
  - kendallchuang 2y ago
    That was fast, I missed it!
- KaoruAoiShiho 2y ago
  It's on the leaderboard, it's tied with qwen 2.5 72b and far below SOTA of o1, claude sonnet, and deepseek. (also below very old models like gpt-4-0314 lol)

dudus 2y ago

I use it and find it very helpful. Never tried cursor or copilot though

therein 2y ago
I tried Cursor the other day. It was actually pretty cool. My thought was, I'll open this open source project and use it to grok my way around the codebase. It was very helpful. After that I accidentally pasted an API secret into the document; had to consider it compromised and re-issued the credential.

therein 2y ago

I know you aren't necessarily talking about in-editor code assist but something about in-editor AI cloud code assist makes me super uncomfortable. It makes sense I need to be careful not to commit secrets to public repositories but now I have to avoid not only saving credentials into a file but even to paste them by accident into my editor?

mil22 2y ago

I have used Github Copilot extensively within VS Code for several months. The autocomplete - fast and often surprisingly accurate - is very useful. My only complaint is when writing comments, I find the completions distracting to my thought process. I tried Gemini Code Assist and it was so bad by comparison that I turned it off within literally minutes. Too slow and inaccurate. I also tried Codestral via the Continue extension and found it also to be slower and less useful than Copilot. So I still haven't found anything better for completion than Copilot. I find long completions, e.g. writing complete functions, less useful in general, and get the most benefit from short, fast, accurate completions that save me typing, without trying to go too far in terms of predicting what I'm going to write next. Fast is the key - I'm a 185 wpm on Monkeytype, so the completion had better be super low latency otherwise I'll already have typed what I want by the time the suggestion appears. Copilot wins on the speed front by far. I've also tried pretty much everything out there for writing algorithms and doing larger code refactorings, and answering questions, and find myself using Continue with Claude Sonnet, or just Sonnet or o1-preview via their native web interfaces, most of the time.

imp0cat 2y ago
Have you tried Gitlab Duo and if so, what are your thoughts on that?
- mil22 2y ago
  Not yet, hadn't heard of it. Thanks for the suggestion.
kendallchuang 2y ago
I see, perhaps with Gemini because the model is larger it takes longer to generate the completions. I would expect with a larger model it would perform better on larger codebases. It sounds like for you, it's faster to work on a smaller model with shorter more accurate completions rather than letting the model guess what you're trying to write.

danmaz74 2y ago

I tried it briefly and didn't like it. On the other hand, I found Gemini pro better than sonnet or 4o at some more complex coding tasks (using continue.dev)