21 ms·
Ask HN: What is the, currently, best Programming LLM (copilot) subscription?
Primarily interested in Go, Python, Swift, Android (Java) and Web stuff. So, relatively mainstream languages, nothing too esoteric.
What have people found to be the best copilot paid subscription, at the moment?
- vector_rotcev 3y agoThe AI Assistant in JetBrains is the best I've encountered. In Pycharm, specifically. I've tried using my OpenAI subscription directly to use GPT4, but copy-pasting between the browser and the lack of context (that GPT has of my code) is limiting. I've tried Copilot, but it just produces large amounts of junk code, and it gets irritating. JetBrain's AI Assistant is built into the IDE I use, it melds well with all the pre-existing coding assistance it offers, the conversation interface is useful and can answer questions on specific parts of code (I habitually highlight whatever I'm interested in and ask it about that), and it's good when I ask it to write tests.
- codelikeawolf 3y agoJinx!
- barbazoo 3y agoTIL that there's also Copilot plugins for Jetbrain IDEs
- Filligree 3y agoYou also get them for Vim, Emacs, <insert editor here> just about.
- switchbak 3y agoI don't think it gets much love. I'm using it, and ... well I guess I get to stay inside my IDE, but other than that it just gets in the way. Then again, I haven't given it much of a fare shake perhaps. I do find AI Assistant is a lot more compelling.
- infecto 3y agoI need to try it out. I have been using a mixture of the Copilot plugin for autocomplete and then a third party package that uses an API key for OpenAI. I enjoy the third party package because I can just use 3.5/4 directly with my own system prompt instead of the garbage interface for Copilot or ChatGPT directly. Edit: Just tried it out again, honestly pretty disappointing. It works and good they released it, Github has been ignoring the Jetbrains experience. Their implementation has bloated UI and I don't like the way they handle prewritten prompts. I really like this implementation. https://github.com/didalgolab/chatgpt-intellij-plugin https://github.com/didalgolab/chatgpt-intellij-plugin
- codelikeawolf 3y agoIf you use JetBrains products, I've been pretty happy with their AI assistant. I feel like it's less disruptive than GitHub Copilot. If I ask it to write some code, it does a pretty decent job, otherwise it stays out of my way.
- SOLAR_FIELDS 3y agoOpenAI API key with GPT4 plus aider[1]: https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider For reference against the actual product called “Copilot”, I would say this is actually useful, vs Copilot which I would use words like “glorified autocomplete” and “slightly better intellisense” to describe. Only really good for saving you some rote typing periodically. The primary limit with “aider” is the GPT4 context window, so really it’s about learning to work within that as its only learning curve. I have been curious about sourcegraph’s “Cody” product if anyone here has tried it
- beastman82 3y agoI love aider but it feels like a giant black box that consumes copious credits. The output is amazing though, so I keep using it. But I feel like I have no insight or control. Maybe I should read the docs more carefully.
- SOLAR_FIELDS 3y agoI use mine in earnest with the full context window filled prompting probably 20-30x a week and have not gotten a monthly bill over $5 USD in a long while. When I first started using it when GPT4 access was limited last year and the tokens were expensive it was a lot more. I think my bills were around $200 USD per month at that point
- harlanlewis 3y agore Cody - I found it very similar to Codium, Copilot, et al. Impressive when it works, but consistently has trouble identifying the right context to inject or using it effectively. At its best when writing code that does not require a lot of local context - speeding up standard operations that might need a lot of boilerplate or using publicly documented + stable APIs are where they shine. re getting some keys - for most tasks I have not yet found a better tool than finding a chat interface to your liking, setting a fairly generic “you’re an expert programmer” system prompt with output instructions tuned to your needs, and manually adding relevant context (copy/paste) to your messages when relevant. I can’t wait for big content windows and RAG methods to improve enough to replace all that with one of assistants, but it’s just not there yet.
- burkaman 3y agoI don't think it's the best assistant by ability, but if you're like me and you bought a lifetime Tabnine subscription for $29 when it debuted on HN in 2018 (https://news.ycombinator.com/item?id=18393364 https://news.ycombinator.com/item?id=18393364), maybe reinstall it and try it out. I've been using it and I would say it is better than nothing.
- mediaman 3y agoThe creator of tabnine also just released supermaven. I’ve liked it so far because it’s fast and holds a lot more of my project in context (300k tokens) so it’s better at using data structures or functions I’ve defined elsewhere.
- lolc 3y agoWhat I like most about Supermaven is how it's trying to complete the diff, not the file. It knows the field I just added in another file. And it works towards a commit. I still have to figure out which files to open, and I hope there will be solutions to that.
- meeech 3y agoI've been pretty happy with Codeium free offering. it does good context completions for me, and based on context, gives me the next bits I'm about to type. I used cody at work, and the best feature for that I found was chore work - I could highlight a struct (for example) and give it instructions how to transform it and it did a good job there.
- mypalmike 3y agoIn addition to paid subscriptions, you might want to consider running an LLM locally. One of a number of projects enabling this approach is "continue" - https://github.com/continuedev/continue https://github.com/continuedev/continue
- geepytee 3y agoThe issue with local IMO is that you'll going to have to compromise on quality. And I really don't want to waste my time talking with inferior models when better models are widely available.
- SOLAR_FIELDS 3y agoYeah, at the time of writing this comment it feels like if you want to easily convince me that your local LLM is good enough, just do a side by side comparison of prompts with your LLM vs GPT4. It doesn’t even have to be better. It just has to be kind of in the same ballpark
- fzzzy 3y agoI'd like to try this, do you have some prompts you like to use for evaluation?
- singularity2001 3y agopeople who think that their local LLM's are in the same ballpark as GPT or Claude are delusional
- 3abiton 3y agoHaven't found any good local model yet that rivals or come close to gpt-4
- fzzzy 3y agoHave you tried miqu q5
- dimal 3y agoI’m pretty happy with Continue, which is a free VS Code extension. You can use an OpenAI API key or local models. I don’t think it does autocomplete, just pair programming, but I’ve found AI autocomplete to be more of a distraction than anything, so it’s good for me.
- bpc777 3y agoThe beta version includes autocomplete: https://continue.dev/docs/walkthroughs/tab-autocomplete https://continue.dev/docs/walkthroughs/tab-autocomplete
- shakabrah 3y agoCursor is really great. It will do codebase search as well as actually apply the code it suggest from the chat interface. Some other neat features like checking your staged changes for bugs.
- pps 3y agoCopilot++ in Cursor is a beast, it's my favorite feature of this app.
- singularity2001 3y agoOne thing that's really annoying is that it doesn't offer suggestions or fixes for (compile) errors which some other plug-ins do
- thekevan 3y agoThis isn't completely answering the question you asked but https://www.blackbox.ai/ https://www.blackbox.ai/ was my favorite after testing out some free ones. I know I liked it better than TabNine, CodeGPT and Codeium in VS Code. I'm pretty sure I tested a few others from the top of list of extensions when you search the extension marketplace, but I forgot which ones. Blackbox just seems more intuitive in use and had better answers. They don't have many followers on Twitter, I'm surprised more people aren't talking about it.
- windowshopping 3y agoThe giant typos in "assistant" throughout the page give this such an unprofessional feel.
- cheema33 3y agoI have been a big user of ChatGPT-4 ever since it came out. I have used in various forms. Through the chatbot, through GitHub Copilot etc. For the past year, I have looked high and low to see an alternative that performed better. Nothing came close. Until a few days ago with Anthropic released Claude 3. For software development work, it is very very impressive. I've been running GPT-4 and Claude 3 Opus side by side for a few days. And I have now decided to cancel all AI subscriptions except Claude 3. For software development work, no other model comes close.
- Filligree 3y agoHow do you get access to Claude 3? It doesn’t seem to be available yet.
- simpaticoder 3y agoWas an early adopter of Github Copilot but I quickly turned it off because of the noise. I really like AI for "gap filling" (for example, I used AI extensively to help me write my bash deployment scripts for simpatico.io) and porting tasks, both of which lend themselves to copy/paste into a browser.
- geepytee 3y agoWhen you say noise do you mean the autocomplete just spamming you all the time, like for example in the middle of writing a comment?
- simpaticoder 3y agoYes, it would often offer unwanted and/or incorrect suggestions that I found distracting. It did not justify the 10% of useful suggestions it made. My usage pattern turns out to be quite copy/paste friendly - it leaves my editor quiet but still gives me the ability to answer "knowledge gap" type questions.
- AlexanderNull 3y agoNeither Github Copilot nor GPT4 are worth your time. At best they partially guess the name of a function you're thinking about typing, at worst they give you almost a correct answer. I've been shocked by how close those models will get to almost understanding what you're attempting to do, while still fundamentally getting it wrong. Last month, after a while of realizing I was spending more time correcting suggestions than I was saving I stopped using them and will need to see some major improvements before I can feel comfortable using them again.
- switchbak 3y agoI see a whole lot of potential in these tools, and in some domains they are starting to deliver on some of the promise. But by and large I agree with this statement - they're actually costing me time because I have to do the research to see where they went wrong. I'm better off learning it properly and idiomatically from scratch right now.
- willsmith72 3y agoI don't get this at all. What kind of code are you writing that you have to literally go and research what it spat out? In my experience 90% of code is 90% the same as another piece of code in the same repo, with small differences, and copilot will make you fly writing that code. If you can't read the output code, does it mean the rest of your codebase is similarly unreadable? The complexity in a codebase or a system is usually from different parts integrating or an overall architecture, but that's totally different to an individual function
- onel 3y agoThis can happen when you are working with new codebases or APIs. For example, recently I tried to build this small gnome extension [0] but I had 0 experience with the API. So I tried chat gpt. Even though the structure of the code in the file was ok, it called some APIs that did not exist, it created a new var `this._menu` for the dropdown that was not needed (this.menu already exists) and in the end I still had to go through gnome extensions docs to figure out how to do it right. Overall I don't regret using it but the experience wasn't magical, as I guess we all want it. [0]: https://github.com/onel/keyboard-cat-defense https://github.com/onel/keyboard-cat-defense
- drewnick 3y agoPhind released a new model I think about a week or two back and I added their VS code plug-in. I really like it because it makes using GPT 4 or their proprietary model much easier than copying and pasting between a browser and VS code. It is a paid subscription, but it includes 500 messages per day, which is way better than the time window on ChatGPT for me.
- IanCal 3y agowww.phind.com is what I've found to be the best generally, using searches. Their recent 70B model seems really good.
- thiodrio 3y ago> Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context (phind.com) https://news.ycombinator.com/item?id=38088538 https://news.ycombinator.com/item?id=38088538
- thiodrio 3y agoConsider Phind Model: > Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context (https://phind.com https://phind.com) https://news.ycombinator.com/item?id=38088538 https://news.ycombinator.com/item?id=38088538
- oksurewhynot 3y agoSupermaven slaps
- brunooliv 3y agoIs Claude 3 on a pay per use case or a fixed monthly fee? I’ve used it a bit with the free 5USD they have and it’s looking so so much better than gpt 4… the problem is I have the feeling it’s also extremely more expensive
- geepytee 3y agoSoftware dev is one of those careers where top performers earn a lot of money, so IMO a tool that generates higher quality code is a no brainer, even if it's 'expensive'. Similarly to a Bloomberg terminal if you are in IB (those things are $25k/mo). That being said if you want to keep playing with Claude 3 Opus, we have it available on our free tier at https://double.bot https://double.bot
- fragmede 3y agoin my limited testing, greptile, posted here recently, was really good for understanding an existing codebase, Reich is a large part of real-world programming.
- geepytee 3y agoI was asking myself the same question not so long ago because Github Copilot's bugs kept bothering me and this is what got us to go into YC and build https://double.bot https://double.bot If what you are looking for a high quality UX + access to the most capable models (today that'd be the GPT-4 killer, Claude 3 Opus), then I think you'll like what we've built. Double ha similar features to all the other Copilots, code autocomplete, chat, etc but we've put particularly care around getting the small details right (even the smallest thing like making sure our autocomplete closes the brackets appropriately). If you're coming from Github Copilot you should check out this side-by-side comparison: https://docs.double.bot/copilot https://docs.double.bot/copilot
- mike986 3y agoA lot of us are already paying for API access from openai/azure/claude Can double's plugin accept bring your own key like cursor.sh ? Would love to use an extension with my favorite editor instead of adapting a whole separate editor like cursor.sh Thanks
- westoncb 3y agoI'm using supermaven these days. Big enough context to understand my full codebase well (I am often working on relatively small projects though), and it's super fast. They trained a custom model, I believe on edit sequences. These copilots are way more useful when their knowledge context is cross-file; anything restricted to current active file or making me do work to select some file subset I'm gonna pass on.
- srameshc 3y agoDuet AI from Google. I have initially tried Copilot and it was sometimes slow in responding. Duet I feel is hyper fast and it sometimes suggest elegant solutions, I don't know if it has to do with any data trained on Google's own code base, I felt this particulary with Go code.
- bernaferrari 3y agoFor me, personally, Copilot is the best because you are typing and it is just suggesting. Very often I type something, wait 2 seconds for its answer (knowing it is going to get it right 80% of the time) and press tab. Sure, it misses a ton, but I see it as a great autocorrect that is always there trying to help. Now, copilot chat and everything else are very bad. I don't like them. If you are "intentional" I usually just use ChatGPT-4 directly. I've had a super pleasant experience with Cursor, too. You can just open Cursor and say "write readme for me", it will scan your project and right a decent readme. It is also eeeeextremely goooood at "I need to know where in the codebase is doing this" and it finds. But for day to day I still prefer chatgpt 4. So I would say: - If you want a nice autocorrect, copilot - If you want something that knows your project and tries to help, cursor - If you just want something smart, chatgpt with gpt-4. I would say copilot lets me be 15% faster daily, and gpt-4 about 30% faster (when I need it), but I need it less often - maybe twice per day.
- geepytee 3y agoHave you ran into issues with Copilot where it does things like: - Doesn't close brackets / adds too many closing brackets - Spams completions while you're writing comments - Doesn't auto import the required libraries after accepting a suggestion - No multi cursor mode - Refuses to name variables - Refuses to trigger in the middle of a line - Lower quality code than ChatGPT (probably because they use a smaller model) Asking because I was also a heavy Github Copilot user but after 1-2 years of constant subpar UX and seemingly no effort from the Copilot team to fix it, I just went ahead and built my own extension that fixes all of the above: https://double.bot https://double.bot Also it wasn't on your list but you mentioned GPT-4 is smart. You should try the new Claude 3 Opus, we implemented it on Monday and have been getting super positive feedback from users!
- henry2023 3y agoI’d love to always know how much code I’m sending to a third party when using a tool like this. What about autocompleting on sensitive files like .env or so.
- drdaeman 3y agoChatGPT 4 with manual explanations and code snippets seem to work best so far. IDE-integrated tools have the potential, but are really immature yet. In particular, I haven't found any tools that can do either of the following: - Comprehend a large codebase. All assistants seem to have a pretty narrow scope for the context, like the current file or few of its neighbors. E.g. they can write some code that already exists somewhere as a ready-to-use utility function, if this function isn't mentioned anywhere near to the location I'm working on. - Automate anything. IDE integration is extremely minimal in everything I've seen so far. A copilot can spew a new version of the code piece that can be placed with a click, but that's the extent of it. It otherwise can't replace a human in the loop, even if the work is simple, trivial and highly repetitive. E.g. doing a regex-powered "replace all" still cannot be replaced with a natural language query. - Do any repetitive work without making mistakes or hallucinating something. E.g. a copilot can try to generate an unit test for me and the overall structure can be quite decent, but I cannot ever be sure it won't miss or invent something if it involves creating some complex object and checking all its fields after some transformation.
- adt 3y agoHave a look at StarCoder 2. And here's my db of 300+ LLMs: https://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/
- geepytee 3y agoSo basically your db says Claude 3 Opus is the best coding LLM, correct?
- dankwizard 3y agoChatGPT 4 but I'm bias (Paid to use)
- geepytee 3y agoHow do I get paid to use LLMs? That's the dream.
- callamdelaney 3y agoI think he means he paid to use chatgpt..
- ado__dev 3y agoI'm biased, but Sourcegraph Cody has changed the way I write code. https://sourcegraph.com/cody https://sourcegraph.com/cody And we just shipped support for Claude 3 (Opus and Sonnet) as well as using any local LLM via Ollama for completions and chat.
- adl 3y agoDoes the Claude 3 support only apply to the VSCode plugin right now? I'm using the Jetbrains plugin and I don't see an option to use Claude 3.
- cyberbiosecure 3y agofor that answer of greatest validity you should e. g. check out "HumanEval" LLM benchmark on HuggingFace website. That is one of the best objective source of info on this issue. Currently Claude 3 is the best, far superior to other models. (85% correct code tasks done. ChatGPT 4 has 65% aprox. This is a giant difference (35% errors vs 15% , so Claude 3 is x2.5 better in terms of code quality losses)
- __rito__ 3y agoEven the free ChatGPT-3.5 beats paid GitHub Copilot in code quality and problem solving. I used Copilot for a while, and it's no more than very smart autofill and good at generating boilerplate. ChatGPT actually solves (some, minor) problems. I kept using Copilot because of comfort, and it having the context. But I stopped using it 2-3 months ago, and haven't looked back.