11 ms·

Claude 3 beats GPT-4 on Aider's code editing benchmark

202 points by goranmoomin 3y ago

submeta 3y ago
I use ChatGPT4 ever since it was released and it changed my computing life. - I started using Claude 3 when it was released. I realized that it’s answers were slightly different, sometimes better. Especially when I feed it long text. At some point I was so intrigued by its answers I became a paid subscriber. Now after a few weeks its my goto tool for programming. It beats both VS Code‘s Copilot (also a subscriber) and ChatGPT4 in coding, and aftentimes in text generation. But I still use both. Happy to have more options.
- shmoogy 3y ago
  I don't need another $20 subscription - are you using the api? I wouldn't mind spending 5-8 a month through the api for opus..
  - submeta 3y ago
    I use both the API and the web interface as that’s more convenient. I hardly use Google anymore these days. These tools are well worth every penny for me. They save me endless hours of research/ parsing the results and bringing it in a format I can use.
    - londons_explore 3y ago
      If you're a coder earning $150/hour, a tool that saves 'countless hours' is worth a lot more than $20
      - burkaman 3y ago
        To your employer maybe, but not to you. You get paid either way.
        lexapro 3y ago
        Enhancing your productivity can lead to an improvement in job performance, which in turn, may significantly increase your job security.
      - k4rli 3y ago
        Yes but it's still nice to spend <$10 per month on per token APIs (and get faster responses) than $25 for chatgpt UI (that has horrid captchas besides being much slower than api). Also it's nice to store the data locally with Librechat.
      - ricardobeat 3y ago
        If you’re sitting all day, it’s worth investing $1000 on a chair. If you’re looking at the monitor all day, it’s worth the $3000 for a Studio display. If you’re programming all day, it’s worth getting the top end M3 Max for $5k. A mechanical keyboard, ergonomic mouse, overhead light. It’s worth paying a lot more than $20 for your text editor, your terminal app, window manager, your music subscription, your calendar, your email, todo app, notion, figma, miro.. Most of these make sense in isolation, but if you apply this thinking to everything it quickly adds up. I’m not keen on spending $3k-4k per year on software alone. Even if it provided massive productivity gains (hasn’t for me) my pay would not increase accordingly.
        rsanek 3y ago
        it's just a question of roi. if those investments increase your earnings over their useful lifetime more than they cost, then yeah, they're worth it. of course, you need to be an owner that can get the benefits of the increased productivity. if you're salaried, may not make sense.
        submeta 3y ago
        There are things that I have solved with this in a fraction of time that I otherwise wouldn’t have persued. Side projects I had in my mind. Or migration projects (old data) with lots of edge cases where I‘d have spent weeks to finish, I did in two days. Or apps I needed written in GTK as an interface for data or a workflow. Or semi-legal emails that I‘d have reached a lawyer for. There‘s a ton of things I can do with these tools in fractions of time. Or that I can do at all. This gives me superpowers. And it’s worth dropping a restaurant visit to pay for this.
        bredren 3y ago
        > it’s worth the $3000 for a Studio display It’s worth $6000 for the XDR Pro Display
        llm_trw 3y ago
        > it’s worth getting the top end M3 Max for $5k May I suggest learning enough Linux to remove the impedance mismatch between your dev and prod machines?
      - swader999 3y ago
        It's like trying to out dig a backhoe with a shovel. Well not yet but...
  - asterix_pano 3y ago
    You can use a service such as nano-gpt.com where you pay for every single query only. I rarely spend more than $20 a month and I can choose whichever model I want.
- nkotov 3y ago
  Similar experience - my only issue is hitting the rate limitation with Opus.
rubymamis 3y ago
From my experience, Claude 3 Opus is subpar compared to GPT-4 and Mistral Large. Mistral Large is competitive against GPT-4 in coding tasks.
- maelito 3y ago
  How do you use Mistral in your coding tasks ? Is it connected to your IDE ? Does it only take an API key to use it ?
  - rubymamis 3y ago
    Nothing fancy, I’m using the chat interface. The only thing connected to my IDE is Copilot (which seems subpar to all of the above?).
    - simonw 3y ago
      As far as I can tell Copilot autocomplete is still using gpt-3.5-turbo. Copilot Chat uses GPT-4.
- rubymamis 3y ago
  Okay, my results keep changing, it's hard to tell. Just tried some coding tasks that GPT-4 couldn't complete compared to Claude 3 Opus. It's a very close competition right now.
jessep 3y ago
Two things interest me about Claude being better than GPT-4: 1) We are all breathless that it is better. But a year has passed since GPT4. It’s like we’re excited that someone beat Usain Bolt’s 100 meter time from when he was 7. Impressive, but … he’s twenty now, has been training like a maniac, and we’ll see what happens when he runs his next race. 2) It’s shown AI chat products have no switching costs right now. I now use mostly Claude and pay them money. Each chat is a universe that starts from scratch, so … very easy to switch. Curious if deeper integrations with my data, or real chat memory, will change that.
- tempusalaria 3y ago
  The current version of GPT-4 is 3 months old not 1 year old. Anthropic are legitimately ahead on performance for cost right now. But their API latency I don’t think matches OpenAI We’ll see what GPT4.5 looks like in the next 6 months.
  - lynx23 3y ago
    Did you mean two or four months? Because 3 months is somewhere in december, and there were no updates around that time.
- londons_explore 3y ago
  They're comparing against gpt-4-0125-preview, which was released at the end of January 2024. So they really are beating the market leader for this test.
  - humansareok1 3y ago
    Model Updates != New Models. GPT5 will be substantially better than even the latest GPT4 update.
    - nicce 3y ago
      What matters here is that what I can use today. I can either use Claude 3 or GPT 4. If the Claude is better, it is best on the market. Let’s see what the story is tomorrow.
chmod775 3y ago
My gut feeling is that the softcap for coding skills of current LLMs is somewhere here. There is only so far you can go if your model is a directed graph that cannot think/symbolically execute while it has to write code by emitting it incrementally top to bottom. A different architecture is needed to go further.
- simonw 3y ago
  We already know the trick to get massively improved coding performance from an existing model: set up a mechanism for it to execute the code that it writes and then react to any errors or test failures that occur. ChatGPT Code Interpreter has been demonstrating the effectiveness of this trick for over a year now. Anthropic are working on their own version. I got Code Interpreter to write, compile, debug and fix C code the other day, despite it pretending it can only work with Python: https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/ https://simonwillison.net/2024/Mar/23/building-c-extensions-...
  - febed 3y ago
    Did you have any success in trying automate the whole development/debug loop similar what the auto coder tools are trying to achieve?
- unraveller 3y ago
  Code interpreter seem to have found a path to perfection, i don't care how bad the first response is (if there have to be >1 turns) as long as we can sync up quickly from any misunderstanding, mine or theirs. Here they kind of made the most easily nudgeable code LLM and won the benchmarks that way. https://opencodeinterpreter.github.io https://opencodeinterpreter.github.io
kleiba 3y ago
> Unfortunately, Claude.ai is only available in certain regions right now. We're working hard to expand to other regions soon. Yay, DRM has arrived in AI land.
- h4x0rr 3y ago
  Which doesn't even work. It just shows a notification it's not available in germany, yet just works anyways
  - jss326 3y ago
    I can’t sign up from the EU
- Zambyte 3y ago
  There is a difference between DRM and just not providing a service to a region.
  - kleiba 3y ago
    What is that difference in your opinion?
    - Zambyte 3y ago
      DRM is software that stops media or other software from working on your machine. It could be based in individual authorization, license key, or it could be something like a geo check. Providing or not providing a service based on a region could be done with DRM (as I mentioned at the end of the previous example), or I could block IPs in that region, or I could require information to use the service that people in different regions may not have (ie SSN), or I could simply ask the user where they are and not service them if they are in certain regions. Claude is not running on end users computers. If it were and they were restricting use within certain regions, that would be done with DRM.
- CapsAdmin 3y ago
  I can't sign up from Norway. I can choose a different country (I'm actually in Vietnam) but it refuses my Norwegian credit card.
jewba 3y ago
A bit off topic. But I never understood these country staggered releases that some companies, like Anthropic, does. The country list seems completely random. https://www.anthropic.com/claude-ai-locations https://www.anthropic.com/claude-ai-locations
- tzekid 3y ago
  Those are just the countries where you can release an AI product without worrying that you'll get sued into oblivion by the state. See e.g. EU AI regulation
- deleted 3y ago
  [deleted]
- sireat 3y ago
  There is one common factor: No EU countries. Similarly I can not use Gemini AI Studio from within EU. https://ai.google.dev/available_regions https://ai.google.dev/available_regions - Think data collection policies.
lettergram 3y ago
> The Claude models refused to perform a number of coding tasks and returned the error “Output blocked by content filtering policy”. They refused to code up the beer song program, which makes some sort of superficial sense. But they also refused to work in some larger open source code bases, for unclear reasons. haha this is almost exactly why I wont use Claude models for any task. I can't risk something being blocked with a customer facing application.
lucumo 3y ago
It's interesting that GPT is available in the EU, but Claude isn't. Both are by American companies (despite the French name), but OpenAI clearly has found ways of working within the EU regulations. Why is it more difficult for Anthropic?
- Chabsff 3y ago
  According to them: https://support.anthropic.com/en/articles/7996881-what-is-your-approach-to-gdpr-or-related-issues https://support.anthropic.com/en/articles/7996881-what-is-yo... The very first part of the answer to "How do you approach GDPR" is: > "We approach data privacy and security holistically, [...]" Which reads to me as a polite way to say: We don't want to be GDPR-compliant.
- danpalmer 3y ago
  I think a lot of regulatory issues right now with AI are not that the technology can't be used, but that it takes time to get everything in place for it to be allowed to be used, or to have the compliance procedures in place so it's unlikely you'll be sued for it. Given how fast this space is moving, it's understandable that these companies are opening it up in different countries as soon as they can.
- worldsayshi 3y ago
  It looks like API access is enabled in EU? I can at least use it here(1) and I'm in the EU. 1 - https://console.anthropic.com https://console.anthropic.com
  - sixhobbits 3y ago
    I could use the free trial / starter credits from EU, but the moment I tried to upgrade it said it wasn't available in my region.
    - k4rli 3y ago
      I'm also still on trial credits in EU, but I read on HN that using VPN once for the topup is enough for it to allow continuing to use it. Yet to try but might work
deleted 3y ago
[deleted]
_andrei_ 3y ago
The prompt also matters a lot, this is just a review of GPT vs Claude models for the Aider prompts. In my experience, with some fairly extensive prompts, Opus eats GPT-4's lunch, with the quality difference being huge.
sva_ 3y ago
Claude 3 Opus also surpassed GPT4 on the LMSYS Chatbot Arena Leaderboard[0] just today. I have tried it a bit through Chatbot Arena[1], and it really seems qualitatively significantly better, especially on technical engineering/mathematical tasks. Hope the chat interface becomes available in EU soon. 0. https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... 1. https://chat.lmsys.org/ https://chat.lmsys.org/
mattkevan 3y ago
I've used GPT-4 for programming since it came out and it's massively improved my productivity, despite being frustrating at times. It quickly forgets details and starts hallucinating, so I have to constantly remind it by pasting in code. After a few hours it gets so confused I have to start a new chat to reset things. I've been using Claude pretty intensively over the last week and it's so much better than GPT. The larger context window (200k tokens vs ~16k) means that it can hold almost the entire codebase in memory and is much less likely to forget things. The low request quota even for paid users is a pain though.
- bredren 3y ago
  Which languages are you primarily writing in? Do you mean you are throttled by it more often? (~“You must wait X mins to make more queries”)
- varispeed 3y ago
  It is still remarkably better than working with junior developers. I can ask it to code something based on specification and most of the time it will do better job than if I send the same task to a human. It often makes some small mistakes, but little nudge here and there corrects them, whereas with human you have to spend a lot of time explaining why something is wrong and why this and that way would rectify it. The difference is probably because GPT has access to a vast array of knowledge a human can't reasonably compete with.
  - aiisjustanif 3y ago
    > It is still remarkably better than working with junior developers. Definitely not all junior developers. I have yet to see it do well at handling code migrations, updating APIs, writing end to end tests, and front end code changes with existing UX specifications to name a few things.
    - greenavocado 3y ago
      Wow, you have good junior developers.
ClearAndPresent 3y ago
Sadly, for us, comparisons are moot, as what was previously a rejection of our phone number for verification has become an instant account suspension when once again attempting to sign up. We understand this has been a recent issue and Anthropic's community team have stepped in to resolve issues occasionally(1), but it's not an endearing first experience. So we'll continue to give money to OpenAI, and increasingly MistralAI, for our services, and make do. (1) https://old.reddit.com/r/ClaudeAI/comments/1b7yz9w/i_just_registered_an_account_and_got_banned/ https://old.reddit.com/r/ClaudeAI/comments/1b7yz9w/i_just_re...
ionwake 3y ago
So is there anyway to include "context" documents like you can with GPT 4? What I mean is add some documents that a referenced in every new chat instantiation Im really surprised I havent seen that. Its vital for me. I think in GPT 4 is called custom instructions. Or am I being obtuse and not seeing the setting?
- master-lincoln 3y ago
  That is just some UI code that enables to do this. Has nothing to do with the model
  - ionwake 3y ago
    Nice one, now can you answer my question? EDIT> Not trying to be harsh mate
- cosmojg 3y ago
  Just copy and paste the contents into context?
  - ionwake 3y ago
    Yeah I could also walk instead of taking the bus, its a simple UI update and I hav e no idea why they havent done it.
deleted 3y ago
[deleted]
sireat 3y ago
From the first look it looks like Claude 3 performed as well as the rest of the models. Did not see any massive improvement. I tried a simple prompt driven web devel approach using OpenRouter to test GPT4-Turbo, Claude 3 Opus and Mistral Large, along with GPT3.5 and some other models. I prompted about 6 small content sections each with different requirements (headline, main text, motto, download links, footer). Each model was able to provide reasonable HTML and CSS. However, ALL models started losing context dropping elements and were unable to finish the page with full content. I had to prompt the missing content again. Surely there would be enough context window for a few lines of text? Disclaimer: I've been using Copilot for almost 3 years now but mostly for Python.
ipopoff 3y ago
Claude is seriously impressive
danw1979 3y ago
Claude Opus has really struggled to do code review of a basic Terraform repo for me. I’ve spent longer on “Prompt Engineering” work than it would have done to knock up a deterministic HCL-parser to check code structure.
ilaksh 3y ago
does anyone know how to run aider with claude 3 opus? aider --model anthropic/claude-3-opus ValueError: No known tokenizer for model: anthropic/claude-3-opus
- anotherpaulg 3y ago
  The linked article shows how to use aider with Claude via OpenRouter.
okasaki 3y ago
I subscribed due to the HN and reddit hype but I don't get it. Claude still easily makes stuff up, like below. For reference both ChatGPT4 and Mistral (free) got it right. I'll probably cancel after one month because this doesn't seem useful compared to ChatGPT. User: does the maps.google.com site use one of the big javascript frameworks? Claude Opus: Yes, the Google Maps website (maps.google.com) uses a JavaScript framework developed by Google called Angular. Angular is a popular open-source web application framework maintained by Google. It allows developers to build complex, single-page applications with features like two-way data binding, dependency injection, and a powerful template syntax. Google Maps relies heavily on JavaScript and Angular to provide its interactive mapping features, real-time updates, and smooth user experience. Angular enables Google Maps to efficiently render map tiles, handle user interactions, and dynamically update the map content without refreshing the entire page. It's worth noting that Google Maps also utilizes various other technologies and libraries alongside Angular, such as HTML, CSS, and the Google Maps JavaScript API, to create the full functionality of the mapping platform.
ilaksh 3y ago
I will have to update aider and get Claude set up with it. Have been having too many problems with the diffs from gpt-4-turbo in aider. Actually made my own cli tool for Opus but if aider supports Opus then will switch because my program doesn't do much. Ideally there is an option to go with whole file edits even in Opus.
aurareturn 3y ago
Great to see competition and excited to see what GPT5 will do.
poszlem 3y ago
Given that Gemini, Claude, and ChatGPT are all relatively similar in sophistication, my primary criterion for selecting one is based on its responsiveness to my requests versus its tendency to educate me on the "potential harm" of my inquiries. Claude falls somewhere between Gemini and ChatGPT but is notably less advanced than ChatGPT in providing direct answers to my queries. It is really castrated, though obviously less than Gemini. For example, when I asked Claude to rephrase the above statement it responded with: "I apologize, but I don't feel comfortable rephrasing your statement as written, as it makes some claims I disagree with. While I respect that you may have a preference for AI assistants that are less cautious about potentially harmful content, I don't believe that providing requested information without any regard for potential harms should be the main metric for sophistication or usefulness." ChatGPT just did it. Another example was me asking Claude to rephrase some text I wrote about MacOS being more closed than Windows and it schooled me about how it "enforces negative stereotypes about operating systems" (WTF). You are the machine here, I tell you what to do, not the other way around.
- HarHarVeryFunny 3y ago
  It seems that people in general prefer Claude-3 Opus over GPT-4 https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... > You are the machine here, I tell you what to do, not the other way around. Your example seems fairly innocuous, but what if this was, for example, someone trying to subvert the next election, asking one of these systems to rephrase their propaganda/psychological-manipulation statement to be less obvious, or better take advantage of human psychology, etc? For all Claude knows, you might indeed be a propagandist, trying to use it to craft a better stated message to push an ultimately dangerous point of view.
  - wastewastewaste 3y ago
    So what? Someone else can do the same to do the opposite
    - HarHarVeryFunny 3y ago
deleted 3y ago
[deleted]
mlcrypto 3y ago
Anyone else think of themselves as a GPT? When I write my brain just predicts good words that come next based on my experiences. No doubt in my mind AGI is coming.
- randomsolutions 3y ago
  Hopefully the idea you are trying to convey has slightly more coherence than predicting individual words.
- SketchySeaBeast 3y ago
  No, because when I write I'm trying to convey a concept. The words are a means to the end, they aren't the end. I'm taking your written words, transforming them into ideas in my head, processing those ideas and then producing my own ideas and outputting words to try and match those ideas. I know that my ideas and my words aren't the same thing because I'm always at least slightly unhappy with how I'm expressing myself, which tells me there is a separation and dissonance between the two.
- Workaccount2 3y ago
  I don't think humans are LLM's, but I think prompting as a concept has strong overlap with general human interaction.
- deleted 3y ago
  [deleted]
- pilotneko 3y ago
  Not really, that sounds like the kind of person who starts a sentence without knowing where it'll end up :-) Kidding aside, I think this a version of differences in visual imagery between individuals.
- inference-lord 3y ago
  No, maybe save the low value comments for some other forum.
  - 3y ago
crazal 3y ago
Claude 3 is a very clear improvment on GPT-4 but where GPT-4 does have the edge is that it doesn't rate limit you as quickly or as harshly... I find myself running out of Claude prompts very quickly. Not because I'm asking questions best suited to a smaller model but when attempting to debug a prompt or hard problem it will quickly run out of requests if you get stuck in a corner.
- sokz 3y ago
  I am curious how perplexity handles the rate limiting. I use it a lot during the course of the day and find no rate limiting while Claude 3 Opus set as the default model.
- CuriouslyC 3y ago
  Claude 3 is a clear improvement in stylish writing, because it hasn't been turbo-aligned to produce "helpful short form article answers." Coding wise, it depends on the language and a lot of other factors, but I don't think it's a clear winner over GPT4. I've noticed that Claude likes to really ham up its writing though, and you have to actively prompt it to be less hammy. GPT4's writing is less hammy, but sounds vaguely like marketing material even when it's clearly not supposed to be.
- jug 3y ago
  Yes, and they basically conclude that OpenAI might be a better choice for you despite Claude 3 Opus technically performing better. > While Opus got the highest score, it was only a few points higher than the GPT-4 Turbo results. Given the extra costs of Opus and the slower response times, it remains to be seen which is the most practical model for daily coding use. > ... snip ... > Claude 3 Opus and Sonnet are both slower and more expensive than OpenAI’s models. You can get almost the same coding skill faster and cheaper with OpenAI’s models. It's an interesting time of AI. Is this the first sign in a launched commercial product hitting diminishing returns given current LLM design? I'm going to be very interested in seeing where OpenAI is headed next, and "GPT-5" performance. Also, given these indicators, the real news here might not be that Opus just barely has an edge on GPT-4 at a high cost, but what's going on at the lower/cheaper end where both Sonnet and Haiku now beats some current versions of GPT-4 on LMSys Chatbot Arena. https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Considering that Sonnet is offered for free on claude.ai, ChatGPT 3.5 in particular now looks hopelessly behind.
deleted 3y ago
[deleted]
stavros 3y ago
I've never heard of OpenRouter before, and it looks cool, but I don't see any pricing anywhere, and it's a bit opaque on even how to use the models. Has anyone used it?
- creesch 3y ago
  Pricing depends on the model you use. You can get a quick overview with prices in their documentation: https://openrouter.ai/docs#models https://openrouter.ai/docs#models Or browse: https://openrouter.ai/models https://openrouter.ai/models for a few more details. As far as usage goes, the documentation I linked goes into it more deeply. But to quickly get started, and assuming you are familiar with the openAI API, you basically can use openAI api calls by changing the baseurl. https://openrouter.ai/docs#quick-start https://openrouter.ai/docs#quick-start edit: I don't remember if this was turned on or off by default. But in your settings (https://openrouter.ai/account#default-model https://openrouter.ai/account#default-model) there is an analytics setting: > Allow logging of inputs & outputs and get a 1% discount on all LLMs I have turned this off. Having said that, they effectively route the calls to the LLM providers and these might have their own policies on data. Having said all that, it is a nice way to play around with various models.
  - stavros 3y ago
    Thank you, that pricing table is very very hidden (I even visited that section and missed it). It looks like they don't charge anything extra if you use GPT-4, for example, which is great, as I can point my code to them to use GPT-4 and then just switch models out as necessary.
    - creesch 3y ago
      > It looks like they don't charge anything extra if you use GPT-4 Yup, which makes it great to experiment without extra cost. It also should give you pause before you implement this in production, as it very much raises the question of what their income model is. Generally speaking, there are a few indicators that make me a bit hesitant to see this as anything more than a way to do some experimenting. Then again, a lot of that is true for making anything long term with the current AI offering.
buremba 3y ago
It's the time for OpenAI to set a new bar with GPT-5 then. For reference, it's been more than a year since they released GPT-4 and Sam was indicating that the leap from GPT-4 to GPT-5 is going to be similar to how people felt GPT-3 -> GPT-4 here: https://www.youtube.com/watch?v=jvqFAi7vkBc&t=5274s https://www.youtube.com/watch?v=jvqFAi7vkBc&t=5274s
- Workaccount2 3y ago
  I listened to the entire poscast, and Sam gives away basically nothing more than anyone could easily assume to be true. The only not-dry bits were: New non-chatgpt products in the pipeline coming soon. GPT5 maybe this year. Smaller increment releases as opposed to big milestone is probably better.
icholy 3y ago
I've been using claude 3 with semgrep and the results are impressive https://github.com/icholy/semgrepx https://github.com/icholy/semgrepx
creesch 3y ago
I feel like the key points are at the bottom. > Claude 3 Opus and Sonnet are both slower and more expensive than OpenAI’s models. You can get almost the same coding skill faster and cheaper with OpenAI’s models. GPT-4 already isn't cheap. Certainly for code tasks I have seen cheaper models also be very capable, I wonder how those stack up here. > Claude 3 has a 2X larger context window than the latest GPT-4 Turbo, which may be an advantage when working with larger code bases. No comment here > The Claude models refused to perform a number of coding tasks and returned the error “Output blocked by content filtering policy”. They refused to code up the beer song program, which makes some sort of superficial sense. But they also refused to work in some larger open source code bases, for unclear reasons. Depending on how often this occurs, this basically can be a dealbreaker entirely. > The Claude APIs seem somewhat unstable, returning HTTP 5xx errors of various sorts. Aider automatically recovers from these errors with exponential backoff retries, but it’s a sign that Anthropic made be struggling under surging demand. Considering they are using openrouter I'd say it might as well be related to that. Certainly if Anthropic offers Claude in a different format and openrouter is doing conversions.
- danielbln 3y ago
  > Claude 3 has a 2X larger context window than the latest GPT-4 Turbo I feel this is selling claude-3 short. Not only is the context window double the size that of GPT-4's, recall over long contexts is significantly better.
Smaug123 3y ago
This baffles me. For the past few days I've been using Opus and ChatGPT through their chat interfaces side by side, and I don't think I've got a single correct answer out of Opus yet (except the one where I asked "can you tell me this, or is the given context insufficient" and it correctly replied "insufficient"). I'm getting fed up of "I apologize for the confusion" followed by nonsense. ChatGPT at least waffles on without trying to answer, if it can't answer, which is much better than getting a fake answer.
- simonw 3y ago
  A lot of this comes down to personal prompting style. My own style is such that I consistently get slightly better results (at least for coding questions) from Opus compared to GPT-4.
- Donald 3y ago
  Do you have any examples of your prompts? I've found Opus to be vastly superior to ChatGPT for intricate tasks.
  - Smaug123 3y ago
    Sure. --- In the following, Opus bombed hard by ignoring the "when" component, replying with "MemoryStream"; where ChatGPT (I think correctly) said "no": > In C#, is there some kind of class in the standard library which implements Stream but which lets me precisely control when and what the Read call returns? --- In the following, Opus bombed hard by inventing `Task.WaitUntilCanceled`, which simply doesn't exist; ChatGPT said "no", which actually isn't true (I could `.ContinueWith` to set a `TaskCancelationSource`, or there's probably a way to do it with an await in a try-catch and a subsequent check for the task's status) but does at least immediately make me think about how to do it rather than going through a loop of trying a wrong answer. > In C#, can I wait for a Task to become cancelled? --- In the following exchange, Opus and ChatGPT both bombed (the correct answer turns out to be "this is undefined behaviour under the POSIX standard, and .NET guarantees nothing under those conditions"), but Opus got into a terrible mess whereas ChatGPT did not: > In .NET, what happens when you read from stdin from a process which has its stdin closed? For example, when it was started with { ./bin/Debug/net7.0/app; } <&- (both engines reply "the call immediately returns with EOF" or similar) > I am observing instead the call to Console.Read() hangs. Riddle me that! ChatGPT replies with basically "I can't explain this" and gives a list of common I/O problems related to file handles; Opus replies with word salad and recommends checking whether stdin has been redirected (which is simply a bad answer: that check has all the false positives in the world). --- > In Neovim, how might I be able to detect whether the user has opened Neovim by invoking Ctrl+X Ctrl+E from the terminal? Normally I have CHADtree open automatically in Neovim, but when the user has just invoked $EDITOR to edit a command line, I don't want that. Claude invents `if v:progname != '-e'`; ChatGPT (I think correctly) says "you can't do that, try setting env vars in your shell to detect this condition instead"
nirav72 3y ago
As a non API users - I also found the chat interface to be slow on the response. While it did a much better job than ChatGPT with 4 on many coding related interactions I had -Claude 3 pro for the $20/month also has a daily rate limit. I signed up for it for a month. But probably will cancel and stick with Chapgpt subscription for now.

202 points by goranmoomin 3y ago

I use ChatGPT4 ever since it was released and it changed my computing life. - I started using Claude 3 when it was released. I realized that it’s answers were slightly different, sometimes better. Especially when I feed it long text. At some point I was so intrigued by its answers I became a paid subscriber. Now after a few weeks its my goto tool for programming. It beats both VS Code‘s Copilot (also a subscriber) and ChatGPT4 in coding, and aftentimes in text generation. But I still use both. Happy to have more options.

shmoogy 3y ago
I don't need another $20 subscription - are you using the api? I wouldn't mind spending 5-8 a month through the api for opus..
- submeta 3y ago
  I use both the API and the web interface as that’s more convenient. I hardly use Google anymore these days. These tools are well worth every penny for me. They save me endless hours of research/ parsing the results and bringing it in a format I can use.
  - londons_explore 3y ago
    If you're a coder earning $150/hour, a tool that saves 'countless hours' is worth a lot more than $20
    - burkaman 3y ago
      To your employer maybe, but not to you. You get paid either way.
      - lexapro 3y ago
        Enhancing your productivity can lead to an improvement in job performance, which in turn, may significantly increase your job security.
    - k4rli 3y ago
      Yes but it's still nice to spend <$10 per month on per token APIs (and get faster responses) than $25 for chatgpt UI (that has horrid captchas besides being much slower than api). Also it's nice to store the data locally with Librechat.
    - ricardobeat 3y ago
      If you’re sitting all day, it’s worth investing $1000 on a chair. If you’re looking at the monitor all day, it’s worth the $3000 for a Studio display. If you’re programming all day, it’s worth getting the top end M3 Max for $5k. A mechanical keyboard, ergonomic mouse, overhead light. It’s worth paying a lot more than $20 for your text editor, your terminal app, window manager, your music subscription, your calendar, your email, todo app, notion, figma, miro.. Most of these make sense in isolation, but if you apply this thinking to everything it quickly adds up. I’m not keen on spending $3k-4k per year on software alone. Even if it provided massive productivity gains (hasn’t for me) my pay would not increase accordingly.
      - rsanek 3y ago
        it's just a question of roi. if those investments increase your earnings over their useful lifetime more than they cost, then yeah, they're worth it. of course, you need to be an owner that can get the benefits of the increased productivity. if you're salaried, may not make sense.
      - submeta 3y ago
        There are things that I have solved with this in a fraction of time that I otherwise wouldn’t have persued. Side projects I had in my mind. Or migration projects (old data) with lots of edge cases where I‘d have spent weeks to finish, I did in two days. Or apps I needed written in GTK as an interface for data or a workflow. Or semi-legal emails that I‘d have reached a lawyer for. There‘s a ton of things I can do with these tools in fractions of time. Or that I can do at all. This gives me superpowers. And it’s worth dropping a restaurant visit to pay for this.
      - bredren 3y ago
        > it’s worth the $3000 for a Studio display It’s worth $6000 for the XDR Pro Display
      - llm_trw 3y ago
        > it’s worth getting the top end M3 Max for $5k May I suggest learning enough Linux to remove the impedance mismatch between your dev and prod machines?
    - swader999 3y ago
      It's like trying to out dig a backhoe with a shovel. Well not yet but...
- asterix_pano 3y ago
  You can use a service such as nano-gpt.com where you pay for every single query only. I rarely spend more than $20 a month and I can choose whichever model I want.
nkotov 3y ago
Similar experience - my only issue is hitting the rate limitation with Opus.

rubymamis 3y ago

From my experience, Claude 3 Opus is subpar compared to GPT-4 and Mistral Large. Mistral Large is competitive against GPT-4 in coding tasks.

maelito 3y ago
How do you use Mistral in your coding tasks ? Is it connected to your IDE ? Does it only take an API key to use it ?
- rubymamis 3y ago
  Nothing fancy, I’m using the chat interface. The only thing connected to my IDE is Copilot (which seems subpar to all of the above?).
  - simonw 3y ago
    As far as I can tell Copilot autocomplete is still using gpt-3.5-turbo. Copilot Chat uses GPT-4.
rubymamis 3y ago
Okay, my results keep changing, it's hard to tell. Just tried some coding tasks that GPT-4 couldn't complete compared to Claude 3 Opus. It's a very close competition right now.

jessep 3y ago

Two things interest me about Claude being better than GPT-4: 1) We are all breathless that it is better. But a year has passed since GPT4. It’s like we’re excited that someone beat Usain Bolt’s 100 meter time from when he was 7. Impressive, but … he’s twenty now, has been training like a maniac, and we’ll see what happens when he runs his next race. 2) It’s shown AI chat products have no switching costs right now. I now use mostly Claude and pay them money. Each chat is a universe that starts from scratch, so … very easy to switch. Curious if deeper integrations with my data, or real chat memory, will change that.

tempusalaria 3y ago
The current version of GPT-4 is 3 months old not 1 year old. Anthropic are legitimately ahead on performance for cost right now. But their API latency I don’t think matches OpenAI We’ll see what GPT4.5 looks like in the next 6 months.
- lynx23 3y ago
  Did you mean two or four months? Because 3 months is somewhere in december, and there were no updates around that time.
londons_explore 3y ago
They're comparing against gpt-4-0125-preview, which was released at the end of January 2024. So they really are beating the market leader for this test.
- humansareok1 3y ago
  Model Updates != New Models. GPT5 will be substantially better than even the latest GPT4 update.
  - nicce 3y ago
    What matters here is that what I can use today. I can either use Claude 3 or GPT 4. If the Claude is better, it is best on the market. Let’s see what the story is tomorrow.

chmod775 3y ago

My gut feeling is that the softcap for coding skills of current LLMs is somewhere here. There is only so far you can go if your model is a directed graph that cannot think/symbolically execute while it has to write code by emitting it incrementally top to bottom. A different architecture is needed to go further.

simonw 3y ago
We already know the trick to get massively improved coding performance from an existing model: set up a mechanism for it to execute the code that it writes and then react to any errors or test failures that occur. ChatGPT Code Interpreter has been demonstrating the effectiveness of this trick for over a year now. Anthropic are working on their own version. I got Code Interpreter to write, compile, debug and fix C code the other day, despite it pretending it can only work with Python: https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/ https://simonwillison.net/2024/Mar/23/building-c-extensions-...
- febed 3y ago
  Did you have any success in trying automate the whole development/debug loop similar what the auto coder tools are trying to achieve?
unraveller 3y ago
Code interpreter seem to have found a path to perfection, i don't care how bad the first response is (if there have to be >1 turns) as long as we can sync up quickly from any misunderstanding, mine or theirs. Here they kind of made the most easily nudgeable code LLM and won the benchmarks that way. https://opencodeinterpreter.github.io https://opencodeinterpreter.github.io

kleiba 3y ago

> Unfortunately, Claude.ai is only available in certain regions right now. We're working hard to expand to other regions soon. Yay, DRM has arrived in AI land.

h4x0rr 3y ago
Which doesn't even work. It just shows a notification it's not available in germany, yet just works anyways
- jss326 3y ago
  I can’t sign up from the EU
Zambyte 3y ago
There is a difference between DRM and just not providing a service to a region.
- kleiba 3y ago
  What is that difference in your opinion?
  - Zambyte 3y ago
    DRM is software that stops media or other software from working on your machine. It could be based in individual authorization, license key, or it could be something like a geo check. Providing or not providing a service based on a region could be done with DRM (as I mentioned at the end of the previous example), or I could block IPs in that region, or I could require information to use the service that people in different regions may not have (ie SSN), or I could simply ask the user where they are and not service them if they are in certain regions. Claude is not running on end users computers. If it were and they were restricting use within certain regions, that would be done with DRM.
CapsAdmin 3y ago
I can't sign up from Norway. I can choose a different country (I'm actually in Vietnam) but it refuses my Norwegian credit card.

jewba 3y ago

A bit off topic. But I never understood these country staggered releases that some companies, like Anthropic, does. The country list seems completely random. https://www.anthropic.com/claude-ai-locations https://www.anthropic.com/claude-ai-locations

tzekid 3y ago
Those are just the countries where you can release an AI product without worrying that you'll get sued into oblivion by the state. See e.g. EU AI regulation
deleted 3y ago
[deleted]
sireat 3y ago
There is one common factor: No EU countries. Similarly I can not use Gemini AI Studio from within EU. https://ai.google.dev/available_regions https://ai.google.dev/available_regions - Think data collection policies.

lettergram 3y ago

> The Claude models refused to perform a number of coding tasks and returned the error “Output blocked by content filtering policy”. They refused to code up the beer song program, which makes some sort of superficial sense. But they also refused to work in some larger open source code bases, for unclear reasons. haha this is almost exactly why I wont use Claude models for any task. I can't risk something being blocked with a customer facing application.

lucumo 3y ago

It's interesting that GPT is available in the EU, but Claude isn't. Both are by American companies (despite the French name), but OpenAI clearly has found ways of working within the EU regulations. Why is it more difficult for Anthropic?

Chabsff 3y ago
According to them: https://support.anthropic.com/en/articles/7996881-what-is-your-approach-to-gdpr-or-related-issues https://support.anthropic.com/en/articles/7996881-what-is-yo... The very first part of the answer to "How do you approach GDPR" is: > "We approach data privacy and security holistically, [...]" Which reads to me as a polite way to say: We don't want to be GDPR-compliant.
danpalmer 3y ago
I think a lot of regulatory issues right now with AI are not that the technology can't be used, but that it takes time to get everything in place for it to be allowed to be used, or to have the compliance procedures in place so it's unlikely you'll be sued for it. Given how fast this space is moving, it's understandable that these companies are opening it up in different countries as soon as they can.
worldsayshi 3y ago
It looks like API access is enabled in EU? I can at least use it here(1) and I'm in the EU. 1 - https://console.anthropic.com https://console.anthropic.com
- sixhobbits 3y ago
  I could use the free trial / starter credits from EU, but the moment I tried to upgrade it said it wasn't available in my region.
  - k4rli 3y ago
    I'm also still on trial credits in EU, but I read on HN that using VPN once for the topup is enough for it to allow continuing to use it. Yet to try but might work

deleted 3y ago

[deleted]

_andrei_ 3y ago

The prompt also matters a lot, this is just a review of GPT vs Claude models for the Aider prompts. In my experience, with some fairly extensive prompts, Opus eats GPT-4's lunch, with the quality difference being huge.

sva_ 3y ago

Claude 3 Opus also surpassed GPT4 on the LMSYS Chatbot Arena Leaderboard[0] just today. I have tried it a bit through Chatbot Arena[1], and it really seems qualitatively significantly better, especially on technical engineering/mathematical tasks. Hope the chat interface becomes available in EU soon. 0. https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... 1. https://chat.lmsys.org/ https://chat.lmsys.org/

mattkevan 3y ago

I've used GPT-4 for programming since it came out and it's massively improved my productivity, despite being frustrating at times. It quickly forgets details and starts hallucinating, so I have to constantly remind it by pasting in code. After a few hours it gets so confused I have to start a new chat to reset things. I've been using Claude pretty intensively over the last week and it's so much better than GPT. The larger context window (200k tokens vs ~16k) means that it can hold almost the entire codebase in memory and is much less likely to forget things. The low request quota even for paid users is a pain though.

bredren 3y ago
Which languages are you primarily writing in? Do you mean you are throttled by it more often? (~“You must wait X mins to make more queries”)
varispeed 3y ago
It is still remarkably better than working with junior developers. I can ask it to code something based on specification and most of the time it will do better job than if I send the same task to a human. It often makes some small mistakes, but little nudge here and there corrects them, whereas with human you have to spend a lot of time explaining why something is wrong and why this and that way would rectify it. The difference is probably because GPT has access to a vast array of knowledge a human can't reasonably compete with.
- aiisjustanif 3y ago
  > It is still remarkably better than working with junior developers. Definitely not all junior developers. I have yet to see it do well at handling code migrations, updating APIs, writing end to end tests, and front end code changes with existing UX specifications to name a few things.
  - greenavocado 3y ago
    Wow, you have good junior developers.

ClearAndPresent 3y ago

Sadly, for us, comparisons are moot, as what was previously a rejection of our phone number for verification has become an instant account suspension when once again attempting to sign up. We understand this has been a recent issue and Anthropic's community team have stepped in to resolve issues occasionally(1), but it's not an endearing first experience. So we'll continue to give money to OpenAI, and increasingly MistralAI, for our services, and make do. (1) https://old.reddit.com/r/ClaudeAI/comments/1b7yz9w/i_just_registered_an_account_and_got_banned/ https://old.reddit.com/r/ClaudeAI/comments/1b7yz9w/i_just_re...

ionwake 3y ago

So is there anyway to include "context" documents like you can with GPT 4? What I mean is add some documents that a referenced in every new chat instantiation Im really surprised I havent seen that. Its vital for me. I think in GPT 4 is called custom instructions. Or am I being obtuse and not seeing the setting?

master-lincoln 3y ago
That is just some UI code that enables to do this. Has nothing to do with the model
- ionwake 3y ago
  Nice one, now can you answer my question? EDIT> Not trying to be harsh mate
cosmojg 3y ago
Just copy and paste the contents into context?
- ionwake 3y ago
  Yeah I could also walk instead of taking the bus, its a simple UI update and I hav e no idea why they havent done it.

deleted 3y ago

[deleted]

sireat 3y ago

From the first look it looks like Claude 3 performed as well as the rest of the models. Did not see any massive improvement. I tried a simple prompt driven web devel approach using OpenRouter to test GPT4-Turbo, Claude 3 Opus and Mistral Large, along with GPT3.5 and some other models. I prompted about 6 small content sections each with different requirements (headline, main text, motto, download links, footer). Each model was able to provide reasonable HTML and CSS. However, ALL models started losing context dropping elements and were unable to finish the page with full content. I had to prompt the missing content again. Surely there would be enough context window for a few lines of text? Disclaimer: I've been using Copilot for almost 3 years now but mostly for Python.

ipopoff 3y ago

Claude is seriously impressive

danw1979 3y ago

Claude Opus has really struggled to do code review of a basic Terraform repo for me. I’ve spent longer on “Prompt Engineering” work than it would have done to knock up a deterministic HCL-parser to check code structure.

ilaksh 3y ago

does anyone know how to run aider with claude 3 opus? aider --model anthropic/claude-3-opus ValueError: No known tokenizer for model: anthropic/claude-3-opus

anotherpaulg 3y ago
The linked article shows how to use aider with Claude via OpenRouter.

okasaki 3y ago

I subscribed due to the HN and reddit hype but I don't get it. Claude still easily makes stuff up, like below. For reference both ChatGPT4 and Mistral (free) got it right. I'll probably cancel after one month because this doesn't seem useful compared to ChatGPT. User: does the maps.google.com site use one of the big javascript frameworks? Claude Opus: Yes, the Google Maps website (maps.google.com) uses a JavaScript framework developed by Google called Angular. Angular is a popular open-source web application framework maintained by Google. It allows developers to build complex, single-page applications with features like two-way data binding, dependency injection, and a powerful template syntax. Google Maps relies heavily on JavaScript and Angular to provide its interactive mapping features, real-time updates, and smooth user experience. Angular enables Google Maps to efficiently render map tiles, handle user interactions, and dynamically update the map content without refreshing the entire page. It's worth noting that Google Maps also utilizes various other technologies and libraries alongside Angular, such as HTML, CSS, and the Google Maps JavaScript API, to create the full functionality of the mapping platform.

ilaksh 3y ago

I will have to update aider and get Claude set up with it. Have been having too many problems with the diffs from gpt-4-turbo in aider. Actually made my own cli tool for Opus but if aider supports Opus then will switch because my program doesn't do much. Ideally there is an option to go with whole file edits even in Opus.

aurareturn 3y ago

Great to see competition and excited to see what GPT5 will do.

poszlem 3y ago

Given that Gemini, Claude, and ChatGPT are all relatively similar in sophistication, my primary criterion for selecting one is based on its responsiveness to my requests versus its tendency to educate me on the "potential harm" of my inquiries. Claude falls somewhere between Gemini and ChatGPT but is notably less advanced than ChatGPT in providing direct answers to my queries. It is really castrated, though obviously less than Gemini. For example, when I asked Claude to rephrase the above statement it responded with: "I apologize, but I don't feel comfortable rephrasing your statement as written, as it makes some claims I disagree with. While I respect that you may have a preference for AI assistants that are less cautious about potentially harmful content, I don't believe that providing requested information without any regard for potential harms should be the main metric for sophistication or usefulness." ChatGPT just did it. Another example was me asking Claude to rephrase some text I wrote about MacOS being more closed than Windows and it schooled me about how it "enforces negative stereotypes about operating systems" (WTF). You are the machine here, I tell you what to do, not the other way around.

HarHarVeryFunny 3y ago
It seems that people in general prefer Claude-3 Opus over GPT-4 https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... > You are the machine here, I tell you what to do, not the other way around. Your example seems fairly innocuous, but what if this was, for example, someone trying to subvert the next election, asking one of these systems to rephrase their propaganda/psychological-manipulation statement to be less obvious, or better take advantage of human psychology, etc? For all Claude knows, you might indeed be a propagandist, trying to use it to craft a better stated message to push an ultimately dangerous point of view.
- wastewastewaste 3y ago
  So what? Someone else can do the same to do the opposite
  - HarHarVeryFunny 3y ago

deleted 3y ago

[deleted]

mlcrypto 3y ago

Anyone else think of themselves as a GPT? When I write my brain just predicts good words that come next based on my experiences. No doubt in my mind AGI is coming.

randomsolutions 3y ago
Hopefully the idea you are trying to convey has slightly more coherence than predicting individual words.
SketchySeaBeast 3y ago
No, because when I write I'm trying to convey a concept. The words are a means to the end, they aren't the end. I'm taking your written words, transforming them into ideas in my head, processing those ideas and then producing my own ideas and outputting words to try and match those ideas. I know that my ideas and my words aren't the same thing because I'm always at least slightly unhappy with how I'm expressing myself, which tells me there is a separation and dissonance between the two.
Workaccount2 3y ago
I don't think humans are LLM's, but I think prompting as a concept has strong overlap with general human interaction.
deleted 3y ago
[deleted]
pilotneko 3y ago
Not really, that sounds like the kind of person who starts a sentence without knowing where it'll end up :-) Kidding aside, I think this a version of differences in visual imagery between individuals.
inference-lord 3y ago
No, maybe save the low value comments for some other forum.
- 3y ago

crazal 3y ago

Claude 3 is a very clear improvment on GPT-4 but where GPT-4 does have the edge is that it doesn't rate limit you as quickly or as harshly... I find myself running out of Claude prompts very quickly. Not because I'm asking questions best suited to a smaller model but when attempting to debug a prompt or hard problem it will quickly run out of requests if you get stuck in a corner.

sokz 3y ago
I am curious how perplexity handles the rate limiting. I use it a lot during the course of the day and find no rate limiting while Claude 3 Opus set as the default model.
CuriouslyC 3y ago
Claude 3 is a clear improvement in stylish writing, because it hasn't been turbo-aligned to produce "helpful short form article answers." Coding wise, it depends on the language and a lot of other factors, but I don't think it's a clear winner over GPT4. I've noticed that Claude likes to really ham up its writing though, and you have to actively prompt it to be less hammy. GPT4's writing is less hammy, but sounds vaguely like marketing material even when it's clearly not supposed to be.
jug 3y ago
Yes, and they basically conclude that OpenAI might be a better choice for you despite Claude 3 Opus technically performing better. > While Opus got the highest score, it was only a few points higher than the GPT-4 Turbo results. Given the extra costs of Opus and the slower response times, it remains to be seen which is the most practical model for daily coding use. > ... snip ... > Claude 3 Opus and Sonnet are both slower and more expensive than OpenAI’s models. You can get almost the same coding skill faster and cheaper with OpenAI’s models. It's an interesting time of AI. Is this the first sign in a launched commercial product hitting diminishing returns given current LLM design? I'm going to be very interested in seeing where OpenAI is headed next, and "GPT-5" performance. Also, given these indicators, the real news here might not be that Opus just barely has an edge on GPT-4 at a high cost, but what's going on at the lower/cheaper end where both Sonnet and Haiku now beats some current versions of GPT-4 on LMSys Chatbot Arena. https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Considering that Sonnet is offered for free on claude.ai, ChatGPT 3.5 in particular now looks hopelessly behind.

deleted 3y ago

[deleted]

stavros 3y ago

I've never heard of OpenRouter before, and it looks cool, but I don't see any pricing anywhere, and it's a bit opaque on even how to use the models. Has anyone used it?

creesch 3y ago
Pricing depends on the model you use. You can get a quick overview with prices in their documentation: https://openrouter.ai/docs#models https://openrouter.ai/docs#models Or browse: https://openrouter.ai/models https://openrouter.ai/models for a few more details. As far as usage goes, the documentation I linked goes into it more deeply. But to quickly get started, and assuming you are familiar with the openAI API, you basically can use openAI api calls by changing the baseurl. https://openrouter.ai/docs#quick-start https://openrouter.ai/docs#quick-start edit: I don't remember if this was turned on or off by default. But in your settings (https://openrouter.ai/account#default-model https://openrouter.ai/account#default-model) there is an analytics setting: > Allow logging of inputs & outputs and get a 1% discount on all LLMs I have turned this off. Having said that, they effectively route the calls to the LLM providers and these might have their own policies on data. Having said all that, it is a nice way to play around with various models.
- stavros 3y ago
  Thank you, that pricing table is very very hidden (I even visited that section and missed it). It looks like they don't charge anything extra if you use GPT-4, for example, which is great, as I can point my code to them to use GPT-4 and then just switch models out as necessary.
  - creesch 3y ago
    > It looks like they don't charge anything extra if you use GPT-4 Yup, which makes it great to experiment without extra cost. It also should give you pause before you implement this in production, as it very much raises the question of what their income model is. Generally speaking, there are a few indicators that make me a bit hesitant to see this as anything more than a way to do some experimenting. Then again, a lot of that is true for making anything long term with the current AI offering.

buremba 3y ago

It's the time for OpenAI to set a new bar with GPT-5 then. For reference, it's been more than a year since they released GPT-4 and Sam was indicating that the leap from GPT-4 to GPT-5 is going to be similar to how people felt GPT-3 -> GPT-4 here: https://www.youtube.com/watch?v=jvqFAi7vkBc&t=5274s https://www.youtube.com/watch?v=jvqFAi7vkBc&t=5274s

Workaccount2 3y ago
I listened to the entire poscast, and Sam gives away basically nothing more than anyone could easily assume to be true. The only not-dry bits were: New non-chatgpt products in the pipeline coming soon. GPT5 maybe this year. Smaller increment releases as opposed to big milestone is probably better.

icholy 3y ago

I've been using claude 3 with semgrep and the results are impressive https://github.com/icholy/semgrepx https://github.com/icholy/semgrepx

creesch 3y ago

I feel like the key points are at the bottom. > Claude 3 Opus and Sonnet are both slower and more expensive than OpenAI’s models. You can get almost the same coding skill faster and cheaper with OpenAI’s models. GPT-4 already isn't cheap. Certainly for code tasks I have seen cheaper models also be very capable, I wonder how those stack up here. > Claude 3 has a 2X larger context window than the latest GPT-4 Turbo, which may be an advantage when working with larger code bases. No comment here > The Claude models refused to perform a number of coding tasks and returned the error “Output blocked by content filtering policy”. They refused to code up the beer song program, which makes some sort of superficial sense. But they also refused to work in some larger open source code bases, for unclear reasons. Depending on how often this occurs, this basically can be a dealbreaker entirely. > The Claude APIs seem somewhat unstable, returning HTTP 5xx errors of various sorts. Aider automatically recovers from these errors with exponential backoff retries, but it’s a sign that Anthropic made be struggling under surging demand. Considering they are using openrouter I'd say it might as well be related to that. Certainly if Anthropic offers Claude in a different format and openrouter is doing conversions.

danielbln 3y ago
> Claude 3 has a 2X larger context window than the latest GPT-4 Turbo I feel this is selling claude-3 short. Not only is the context window double the size that of GPT-4's, recall over long contexts is significantly better.

Smaug123 3y ago

This baffles me. For the past few days I've been using Opus and ChatGPT through their chat interfaces side by side, and I don't think I've got a single correct answer out of Opus yet (except the one where I asked "can you tell me this, or is the given context insufficient" and it correctly replied "insufficient"). I'm getting fed up of "I apologize for the confusion" followed by nonsense. ChatGPT at least waffles on without trying to answer, if it can't answer, which is much better than getting a fake answer.

simonw 3y ago
A lot of this comes down to personal prompting style. My own style is such that I consistently get slightly better results (at least for coding questions) from Opus compared to GPT-4.
Donald 3y ago
Do you have any examples of your prompts? I've found Opus to be vastly superior to ChatGPT for intricate tasks.
- Smaug123 3y ago
  Sure. --- In the following, Opus bombed hard by ignoring the "when" component, replying with "MemoryStream"; where ChatGPT (I think correctly) said "no": > In C#, is there some kind of class in the standard library which implements Stream but which lets me precisely control when and what the Read call returns? --- In the following, Opus bombed hard by inventing `Task.WaitUntilCanceled`, which simply doesn't exist; ChatGPT said "no", which actually isn't true (I could `.ContinueWith` to set a `TaskCancelationSource`, or there's probably a way to do it with an await in a try-catch and a subsequent check for the task's status) but does at least immediately make me think about how to do it rather than going through a loop of trying a wrong answer. > In C#, can I wait for a Task to become cancelled? --- In the following exchange, Opus and ChatGPT both bombed (the correct answer turns out to be "this is undefined behaviour under the POSIX standard, and .NET guarantees nothing under those conditions"), but Opus got into a terrible mess whereas ChatGPT did not: > In .NET, what happens when you read from stdin from a process which has its stdin closed? For example, when it was started with { ./bin/Debug/net7.0/app; } <&- (both engines reply "the call immediately returns with EOF" or similar) > I am observing instead the call to Console.Read() hangs. Riddle me that! ChatGPT replies with basically "I can't explain this" and gives a list of common I/O problems related to file handles; Opus replies with word salad and recommends checking whether stdin has been redirected (which is simply a bad answer: that check has all the false positives in the world). --- > In Neovim, how might I be able to detect whether the user has opened Neovim by invoking Ctrl+X Ctrl+E from the terminal? Normally I have CHADtree open automatically in Neovim, but when the user has just invoked $EDITOR to edit a command line, I don't want that. Claude invents `if v:progname != '-e'`; ChatGPT (I think correctly) says "you can't do that, try setting env vars in your shell to detect this condition instead"

nirav72 3y ago

As a non API users - I also found the chat interface to be slow on the response. While it did a much better job than ChatGPT with 4 on many coding related interactions I had -Claude 3 pro for the $20/month also has a daily rate limit. I signed up for it for a month. But probably will cancel and stick with Chapgpt subscription for now.