7 ms·

Given the cited stats here and elsewhere as well as in everyday experience, does anyone else feel that this model isn’t significantly different, at least to jus

by aabhay 1y ago

Given the cited stats here and elsewhere as well as in everyday experience, does anyone else feel that this model isn’t significantly different, at least to justify the full version increment? The one statistic mentioned in this overview where they observed a 67% drop seems like it could easily be reduced simply by editing 3.7’s system prompt. What are folks’ theories on the version increment? Is the architecture significantly different (not talking about adding more experts to the MoE or fine tuning on 3.7’s worst failures. I consider those minor increments rather than major). One way that it could be different is if they varied several core hyperparameters to make this a wider/deeper system but trained it on the same data or initialized inner layers to their exact 3.7 weights. And then this would “kick off” the 4 series by allowing them to continue scaling within the 4 series model architecture.

kubb 1y ago
> to justify the full version increment I feel like a company doesn’t have to justify a version increment. They should justify price increases. If you get hyped and have expectations for a number then I’m comfortable saying that’s on you.
- aabhay 1y ago
  That’s an odd way to defend the decision. “It doesn’t make sense because nothing has to make sense”. Sure, but it would be more interesting if you had any evidence that they decided to simply do away with any logical premise for the 4 moniker.
  - kubb 1y ago
    > nothing has to make sense It does make sense. The companies are expected to exponentially improve LLMs, and the increasing versions are catering to the enthusiast crowd who just need a number to go up to lose their mind over how all jobs are over and AGI is coming this year. But there's less and less room to improve LLMs and there are currently no known new scaling vectors (size and reasoning have already been largely exhausted), so the improvement from version to version is decreasing. But I assure you, the people at Anthropic worked their asses off, neglecting their families and sleep and they want to show something for their efforts. It makes sense, just not the sense that some people want.
- jsheard 1y ago
  > They should justify price increases. I think the justification for most AI price increases should go without saying - they were losing money at the old price, and they're probably still losing money at the new price, but it's creeping up towards the break-even point.
  - brookst 1y ago
    Customers don’t decide the acceptable price based on the company’s cost structure. If two equivalent cars were priced $30k apart, you wouldn’t say “well, it seems like a lot but they did have unusual losses last year and stupidly locked themselves in to that steel agreement”. You’d just buy the less expensive one that meets the same needs. (Almost) all producing is based I value. If the customer perceives the price fair for the value received, they’ll pay. If not, not. There are only “justifications” for a price increase: 1) it was an incredibly good deal at the lower price and remains a good deal at the higher price, and 2) substantially more value has been added, making it worth the higher price. Cost structure and company economics may dictate price increases, but customers do not and should not care one whit about that stuff. All that matters is if the value is there at the new price.
  - fragmede 1y ago
    That's not how pricing works on anything.
loveparade 1y ago
Just anecdotal experience, but this model seems more eager to write tests, create test scripts and call various tools than the previous one. Of course this results in more roundtrips and overall more tokens used and more money for the provider. I had to stop the model going crazy with unnecessary tests several times, which isn't something I had to do previously. Can be fixed with a prompt but can't help but wonder if some providers explicitly train their models to be overly verbose.
- aabhay 1y ago
  Eagerness to tool call is an interesting observation. Certainly an MCP ecosystem would require a tool biased model. However, after having pretty deep experience with writing book (or novella) length system prompts, what you mentioned doesn’t feel like a “regime change” in model behavior. I.e it could do those things because its been asked to do those things. The numbers presented in this paper were almost certainly after extensive system prompt ablations, and the fact that we’re within a tenth of a percent difference in some cases indicates less fundamental changes.
- sebzim4500 1y ago
  >I had to stop the model going crazy with unnecessary tests several times, which isn't something I had to do previously When I was playing with this last night, I found that it worked better to let it write all the tests it wanted and then get it to revert the least important ones once the feature is finished. It actually seems to know pretty well which tests are worth keeping and which aren't. (This was all claude 4 sonnet, I've barely tried opus yet)
deleted 1y ago
[deleted]
Aeolun 1y ago
I think they didn’t have anywhere to go after 3.7 but 4. They already did 3.5 and 3.7. People were getting a bit cranky 4 was nowhere to be seen. I’m fine with a v4 that is marginally better since the price is still the same. 3.7 was already pretty good, so as long as they don’t regress it’s all a win to me.
retinaros 1y ago
the big difference is the capability to think during tool calls. this is what makes openAI o3 lookin like magic
- ekidd 1y ago
  Yeah, I've noticed this with Qwen3, too. If I rig up a nonstandard harness than allows it to think before tool calls, even 30B A3B is capable of doing low-budget imitations of the things o3 and similar frontier models do. It can, for example, make a surprising decent "web research agent" with some scaffolding and specialized prompts for different tasks. We need to start moving away from Chat Completions-style tool calls, and start supporting "thinking before tool calls", and even proper multi-step agent loops.
  - pxc 1y ago
    > If I rig up a nonstandard harness than allows it to think before tool calls What does that require? (I'm extremely, extremely new to all this.)
colonCapitalDee 1y ago
I'm noticing much more flattery ("Wow! That's so smart!") and I don't like it
- FieryTransition 1y ago
  Turns out tuning LLMs on human preferences leads to sycophantic behavior, they even wrote about it themselves, guess they wanted to push the model out too fast.
  - mike_hearn 1y ago
    I think it was OpenAI that wrote about that. Most of us here on HN don't like this behaviour, but it's clear that the average user does. If you look at how differently people use AI that's not a surprise. There's a lot of using it as a life coach out there, or people who just want validation regardless of the scenario.
    - tankenmate 1y ago
      > or people who just want validation regardless of the scenario. This really worries me as there are many people (even more prevalent in younger generations if some papers turn out to be valid) that lack resilience and critical self evaluation who may develop narcissistic tendencies with increased use or reinforcement from AIs. Just the health care costs involved when reality kicks in for these people, let alone other concomitant social costs will be substantial at scale. And people think social media algorithms reinforce poor social adaptation and skills, this is a whole new level.
      - ekidd 1y ago
        > who may develop narcissistic tendencies with increased use or reinforcement from AIs. It's clear to me that (1) a lot of billionaires believe amazingly stupid things, and (2) a big part of this is that they surround themselves with a bubble of sycophants. Apparently having people tell you 24/7 how amazing and special you are sometimes leads to delusional behavior. But now regular people can get the same uncritical, fawning affirmations from an LLM. And it's clearly already messing some people up. I expect there to be huge commercial pressure to suck up to users and tell them they're brilliant. And I expect the long-term results will be as bad as the way social media optimizes for filter bubbles and rage bait.
antirez 1y ago
It works better when using tools, but the LLM itself it is not powerful from the POV of reasoning. Actually Sonnet 4 seems weaker than Sonnet 3.7 in many instances.
benreesman 1y ago
The API version I'm getting for Opus 4 via gptel is aligned in a way that will win me back to Claude if its intentional and durable. There seems to be maybe some generalized capability lift but its hard to tell, these things are aligment constrained to a level below earlier frontier models and the dynamic cost control and what not is a liability for people who work to deadlines. Its net negative. The 3.7 bait and switch was the last straw for me and closed frontier vendors or so I said, but I caught a candid, useful, Opus 4 today on a lark, and if its on purpose its like a leadership shakeup level change. More likely they just don't have the "fuck the user" tune yet because they've only run it for themsrlves. I'm not going to make plans contingent on it continuing to work well just yet, but I'm going to give it another audition.
sebzim4500 1y ago
Having used claude 4 for a few hours (and claude 3.7 and gemini 2.5 pro for much more than that) I really think it's much better in ways that aren't being well captured by benchmarks. It does a much better job of debugging issues then either 3.7 or gemini and so far it doesn't seem to have the 'reward hacking' behavior of 3.7. It's a small step for model intelligence but a huge leap for model usability.
- itchyjunk 1y ago
  I have the same experience. I was pretty happy with gemini 2.5 pro and was barely using claude 3.7. Now I am strictly using claude 4 (sonnet mostly). Especially with tasks that require multi tool use, it nicely self corrects which I never noticed in 3.7 when I used it. But it's different in conversational sense as well. Might be the novelty, but I really enjoy it. I have had 2 instances where it had very different take and kind of stuck with me.
frabcus 1y ago
I'd like version numbers to indicate some element of backwards compatibility. So point releases (mostly) wouldn't need prompt changes, whereas a major version upgrade might require significant prompt changes in my application. This is from a developer API use point of view - but honestly it would apply to large personality changes in Claude's chat interface too. It's confusing if it changes a lot and I'd like to know!
sensanaty 1y ago
I feel that 3.7 is still the best. With 4, it keeps writing hundreds upon hundreds of lines, it'll invoke search for everything, it starts refactoring random lines unrelated to my question, it'll often rewrite entire portions of its own output for no reason. I think they took the "We need to shit out code" thing the AIs are good at and cranked it to 11 for whatever reason, where 3.7 had a nice balance (although it still writes WAY too many comments that are utterly useless)
Closi 1y ago
> Given the cited stats here and elsewhere as well as in everyday experience, does anyone else feel that this model isn’t significantly different, at least to justify the full version increment? My experience is the opposite - I'm using it in Cursor and IMO it's performing better than Gemini 2.5 Pro at being able to write code which will run first time (which it wasn't before) and seems to be able to complete much larger tasks. It is even running test cases itself without being prompted, which is novel!
- yosito 1y ago
  I'm a developer, and I've been trying to use AI to vibe code apps for two years. This is the first time I'm able to vibe code an app without major manual interventions at every step. Not saying it's perfect, or that I'd necessarily trust it without human review, but I did vibe code an entire production-ready iOS/Android/web app that accepts payments in less than 24 hours and barely had to manually intervene at all, besides telling it what I wanted to do next.
- mountainriver 1y ago
  It’s funny how differently the models work in cursor. Claude 4 thinks then takes one little step at a time, but yes it’s quite good overall
margorczynski 1y ago
They're probably feeling the heat from e.g. Google and Gemini which is gaining ground fast so the plan is to speed up the releases. I think a similar thing happened with OpenAI where incremental upgrades were presented as something much more.
blurbleblurble 1y ago
I tried it and found that it was ridiculously better than Gemini on a hard programming problem that Gemini 2.5 pro had been spinning wheels on for days
pauldix 1y ago
My experience so far with Opus 4 is that it's very good. Based on a few days of using it for real work, I think it's better than Sonnet 3.5 or 3.7, which had been my daily drivers prior to Gemini 2.5 Pro switching me over just 3 weeks ago. It has solved some things that eluded Gemini 2.5 Pro. Right now I'm swapping between Gemini and Opus depending on the task. Gemini's 1M token context window is really unbeatable. But the quality of what Opus 4 produces is really good. edit: forgot to mention that this is all for Rust based work on InfluxDB 3, a fairly large and complex codebase. YMMV
- Workaccount2 1y ago
  I've been having really good results from Jules, which is Google's gemini agent coding platform[1]. In the beta you only get 5 tasks a day, but so far I have found it to be much more capable than regular API Gemini. [1]https://jules.google/ https://jules.google/
  - trip-zip 1y ago
    Would you mind giving a little more info on what you're getting Jules to work on? I tried it out a couple times but I think I was asking for too large a task and it ended up being pretty bad, all things considered. I tried to get it to add some new REST endpoints that follow the same pattern as the other 100 we have, 5 CRUD endpoints. It failed pretty badly, which may just be an indictment on our codebase...
  - deleted 1y ago
    [deleted]
  - loufe 1y ago
    I let Jules write a PR in my codebase with very specific scaffolding, and it absolutely blew it. It took me more time to understand the ways it failed to grasp the codebase and wrote code for a fundamentally different (incorrectly understood) project. I love Gemini 2.5, but I absolutely agree with the gp (pauldix) on their quality / scope point.
- smokel 1y ago
lherron 1y ago
With all the incremental releases, it’s harder to see the advancement. Maybe it would be more fair to compare 4 vs 3 than 4 vs 3.7.
jsharf 1y ago
I want to also mention that the previous model was 3.7. 3.7 to 4 is not an entire increment, it’s theoretically the same as 3 -> 3.3, which is actually modest compared to the capability jump I’ve observed. I do think Anthropic wants more frequent, continuous releases, and using a numeric version number rather than a software version number is their intent. Gradual releases give society more time to react.
- rst 1y ago
  The numbers are branding, not metrics on anything. You can't do math to, say, determine the capability jump between GPT-4 and GPT-4o. Trying to do math to determine capability gaps between "3.7" and "4.0" doesn't actually make more sense.
ezst 1y ago
> does anyone else feel that this model isn’t significantly different According to Anthropic¹, LLMs are mostly a thing in the software engineering space, and not much elsewhere. I am not a software engineer, and so I'm pretty agnostic about the whole thing, mildly annoyed by the constant anthropomorphisation of LLMs in the marketing surrounding it³, and besides having had a short run with Llama about 2 years ago, I have mostly stayed away from it. Though, I do scripting as a mean to keep my digital life efficient and tidy, and so today I thought that I had a perfect justification for giving Claude 4 Sonnet a spin. I asked it to give me a jujutsu² equivalent for `git -ffdx`. What ensued was that: https://claude.ai/share/acde506c-4bb7-4ce9-add4-657ec9d5c391 https://claude.ai/share/acde506c-4bb7-4ce9-add4-657ec9d5c391 I leave you the judge of this, but for me this is very bad. Objectively, for the time that it took me to describe, review, correct some obvious logical flaws, restart, second-guess myself, get annoyed for being right and having my time wasted, fighting unwarranted complexity, etc…, I could have written a better script myself. So to answer your question, no, I don't think this is significant, and I don't think this generation of LLMs are close to their price tag. ¹: https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F189bf07829f2ecc1d3bac5961e9793a9e0f31e7d-3920x2948.jpg&w=3840&q=75 https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-... ²: https://jj-vcs.github.io/jj/latest/ https://jj-vcs.github.io/jj/latest/ ³: "hallucination", "chain of thought", "mixture of experts", "deep thinking" would have you being laughed at in the more "scientifically apt" world I grew up with, but here we are </rant>
Tarrosion 1y ago
I'm finding 4 Opus good, but 4 Sonnet a bit underwhelming: https://evanfields.net/Claude-4/ https://evanfields.net/Claude-4/

by aabhay 1y ago

kubb 1y ago

> to justify the full version increment I feel like a company doesn’t have to justify a version increment. They should justify price increases. If you get hyped and have expectations for a number then I’m comfortable saying that’s on you.

aabhay 1y ago
That’s an odd way to defend the decision. “It doesn’t make sense because nothing has to make sense”. Sure, but it would be more interesting if you had any evidence that they decided to simply do away with any logical premise for the 4 moniker.
- kubb 1y ago
  > nothing has to make sense It does make sense. The companies are expected to exponentially improve LLMs, and the increasing versions are catering to the enthusiast crowd who just need a number to go up to lose their mind over how all jobs are over and AGI is coming this year. But there's less and less room to improve LLMs and there are currently no known new scaling vectors (size and reasoning have already been largely exhausted), so the improvement from version to version is decreasing. But I assure you, the people at Anthropic worked their asses off, neglecting their families and sleep and they want to show something for their efforts. It makes sense, just not the sense that some people want.
jsheard 1y ago
> They should justify price increases. I think the justification for most AI price increases should go without saying - they were losing money at the old price, and they're probably still losing money at the new price, but it's creeping up towards the break-even point.
- brookst 1y ago
  Customers don’t decide the acceptable price based on the company’s cost structure. If two equivalent cars were priced $30k apart, you wouldn’t say “well, it seems like a lot but they did have unusual losses last year and stupidly locked themselves in to that steel agreement”. You’d just buy the less expensive one that meets the same needs. (Almost) all producing is based I value. If the customer perceives the price fair for the value received, they’ll pay. If not, not. There are only “justifications” for a price increase: 1) it was an incredibly good deal at the lower price and remains a good deal at the higher price, and 2) substantially more value has been added, making it worth the higher price. Cost structure and company economics may dictate price increases, but customers do not and should not care one whit about that stuff. All that matters is if the value is there at the new price.
- fragmede 1y ago
  That's not how pricing works on anything.

loveparade 1y ago

Just anecdotal experience, but this model seems more eager to write tests, create test scripts and call various tools than the previous one. Of course this results in more roundtrips and overall more tokens used and more money for the provider. I had to stop the model going crazy with unnecessary tests several times, which isn't something I had to do previously. Can be fixed with a prompt but can't help but wonder if some providers explicitly train their models to be overly verbose.

aabhay 1y ago
Eagerness to tool call is an interesting observation. Certainly an MCP ecosystem would require a tool biased model. However, after having pretty deep experience with writing book (or novella) length system prompts, what you mentioned doesn’t feel like a “regime change” in model behavior. I.e it could do those things because its been asked to do those things. The numbers presented in this paper were almost certainly after extensive system prompt ablations, and the fact that we’re within a tenth of a percent difference in some cases indicates less fundamental changes.
sebzim4500 1y ago
>I had to stop the model going crazy with unnecessary tests several times, which isn't something I had to do previously When I was playing with this last night, I found that it worked better to let it write all the tests it wanted and then get it to revert the least important ones once the feature is finished. It actually seems to know pretty well which tests are worth keeping and which aren't. (This was all claude 4 sonnet, I've barely tried opus yet)

deleted 1y ago

[deleted]

Aeolun 1y ago

I think they didn’t have anywhere to go after 3.7 but 4. They already did 3.5 and 3.7. People were getting a bit cranky 4 was nowhere to be seen. I’m fine with a v4 that is marginally better since the price is still the same. 3.7 was already pretty good, so as long as they don’t regress it’s all a win to me.

retinaros 1y ago

the big difference is the capability to think during tool calls. this is what makes openAI o3 lookin like magic

ekidd 1y ago
Yeah, I've noticed this with Qwen3, too. If I rig up a nonstandard harness than allows it to think before tool calls, even 30B A3B is capable of doing low-budget imitations of the things o3 and similar frontier models do. It can, for example, make a surprising decent "web research agent" with some scaffolding and specialized prompts for different tasks. We need to start moving away from Chat Completions-style tool calls, and start supporting "thinking before tool calls", and even proper multi-step agent loops.
- pxc 1y ago
  > If I rig up a nonstandard harness than allows it to think before tool calls What does that require? (I'm extremely, extremely new to all this.)

colonCapitalDee 1y ago

I'm noticing much more flattery ("Wow! That's so smart!") and I don't like it

FieryTransition 1y ago
Turns out tuning LLMs on human preferences leads to sycophantic behavior, they even wrote about it themselves, guess they wanted to push the model out too fast.
- mike_hearn 1y ago
  I think it was OpenAI that wrote about that. Most of us here on HN don't like this behaviour, but it's clear that the average user does. If you look at how differently people use AI that's not a surprise. There's a lot of using it as a life coach out there, or people who just want validation regardless of the scenario.
  - tankenmate 1y ago
    > or people who just want validation regardless of the scenario. This really worries me as there are many people (even more prevalent in younger generations if some papers turn out to be valid) that lack resilience and critical self evaluation who may develop narcissistic tendencies with increased use or reinforcement from AIs. Just the health care costs involved when reality kicks in for these people, let alone other concomitant social costs will be substantial at scale. And people think social media algorithms reinforce poor social adaptation and skills, this is a whole new level.
    - ekidd 1y ago
      > who may develop narcissistic tendencies with increased use or reinforcement from AIs. It's clear to me that (1) a lot of billionaires believe amazingly stupid things, and (2) a big part of this is that they surround themselves with a bubble of sycophants. Apparently having people tell you 24/7 how amazing and special you are sometimes leads to delusional behavior. But now regular people can get the same uncritical, fawning affirmations from an LLM. And it's clearly already messing some people up. I expect there to be huge commercial pressure to suck up to users and tell them they're brilliant. And I expect the long-term results will be as bad as the way social media optimizes for filter bubbles and rage bait.

antirez 1y ago

It works better when using tools, but the LLM itself it is not powerful from the POV of reasoning. Actually Sonnet 4 seems weaker than Sonnet 3.7 in many instances.

benreesman 1y ago

The API version I'm getting for Opus 4 via gptel is aligned in a way that will win me back to Claude if its intentional and durable. There seems to be maybe some generalized capability lift but its hard to tell, these things are aligment constrained to a level below earlier frontier models and the dynamic cost control and what not is a liability for people who work to deadlines. Its net negative. The 3.7 bait and switch was the last straw for me and closed frontier vendors or so I said, but I caught a candid, useful, Opus 4 today on a lark, and if its on purpose its like a leadership shakeup level change. More likely they just don't have the "fuck the user" tune yet because they've only run it for themsrlves. I'm not going to make plans contingent on it continuing to work well just yet, but I'm going to give it another audition.

sebzim4500 1y ago

Having used claude 4 for a few hours (and claude 3.7 and gemini 2.5 pro for much more than that) I really think it's much better in ways that aren't being well captured by benchmarks. It does a much better job of debugging issues then either 3.7 or gemini and so far it doesn't seem to have the 'reward hacking' behavior of 3.7. It's a small step for model intelligence but a huge leap for model usability.

itchyjunk 1y ago
I have the same experience. I was pretty happy with gemini 2.5 pro and was barely using claude 3.7. Now I am strictly using claude 4 (sonnet mostly). Especially with tasks that require multi tool use, it nicely self corrects which I never noticed in 3.7 when I used it. But it's different in conversational sense as well. Might be the novelty, but I really enjoy it. I have had 2 instances where it had very different take and kind of stuck with me.

frabcus 1y ago

I'd like version numbers to indicate some element of backwards compatibility. So point releases (mostly) wouldn't need prompt changes, whereas a major version upgrade might require significant prompt changes in my application. This is from a developer API use point of view - but honestly it would apply to large personality changes in Claude's chat interface too. It's confusing if it changes a lot and I'd like to know!

sensanaty 1y ago

I feel that 3.7 is still the best. With 4, it keeps writing hundreds upon hundreds of lines, it'll invoke search for everything, it starts refactoring random lines unrelated to my question, it'll often rewrite entire portions of its own output for no reason. I think they took the "We need to shit out code" thing the AIs are good at and cranked it to 11 for whatever reason, where 3.7 had a nice balance (although it still writes WAY too many comments that are utterly useless)

Closi 1y ago

> Given the cited stats here and elsewhere as well as in everyday experience, does anyone else feel that this model isn’t significantly different, at least to justify the full version increment? My experience is the opposite - I'm using it in Cursor and IMO it's performing better than Gemini 2.5 Pro at being able to write code which will run first time (which it wasn't before) and seems to be able to complete much larger tasks. It is even running test cases itself without being prompted, which is novel!

yosito 1y ago
I'm a developer, and I've been trying to use AI to vibe code apps for two years. This is the first time I'm able to vibe code an app without major manual interventions at every step. Not saying it's perfect, or that I'd necessarily trust it without human review, but I did vibe code an entire production-ready iOS/Android/web app that accepts payments in less than 24 hours and barely had to manually intervene at all, besides telling it what I wanted to do next.
mountainriver 1y ago
It’s funny how differently the models work in cursor. Claude 4 thinks then takes one little step at a time, but yes it’s quite good overall

margorczynski 1y ago

They're probably feeling the heat from e.g. Google and Gemini which is gaining ground fast so the plan is to speed up the releases. I think a similar thing happened with OpenAI where incremental upgrades were presented as something much more.

blurbleblurble 1y ago

I tried it and found that it was ridiculously better than Gemini on a hard programming problem that Gemini 2.5 pro had been spinning wheels on for days

pauldix 1y ago

My experience so far with Opus 4 is that it's very good. Based on a few days of using it for real work, I think it's better than Sonnet 3.5 or 3.7, which had been my daily drivers prior to Gemini 2.5 Pro switching me over just 3 weeks ago. It has solved some things that eluded Gemini 2.5 Pro. Right now I'm swapping between Gemini and Opus depending on the task. Gemini's 1M token context window is really unbeatable. But the quality of what Opus 4 produces is really good. edit: forgot to mention that this is all for Rust based work on InfluxDB 3, a fairly large and complex codebase. YMMV

Workaccount2 1y ago
I've been having really good results from Jules, which is Google's gemini agent coding platform[1]. In the beta you only get 5 tasks a day, but so far I have found it to be much more capable than regular API Gemini. [1]https://jules.google/ https://jules.google/
- trip-zip 1y ago
  Would you mind giving a little more info on what you're getting Jules to work on? I tried it out a couple times but I think I was asking for too large a task and it ended up being pretty bad, all things considered. I tried to get it to add some new REST endpoints that follow the same pattern as the other 100 we have, 5 CRUD endpoints. It failed pretty badly, which may just be an indictment on our codebase...
- deleted 1y ago
  [deleted]
- loufe 1y ago
  I let Jules write a PR in my codebase with very specific scaffolding, and it absolutely blew it. It took me more time to understand the ways it failed to grasp the codebase and wrote code for a fundamentally different (incorrectly understood) project. I love Gemini 2.5, but I absolutely agree with the gp (pauldix) on their quality / scope point.
smokel 1y ago

lherron 1y ago

With all the incremental releases, it’s harder to see the advancement. Maybe it would be more fair to compare 4 vs 3 than 4 vs 3.7.

jsharf 1y ago

I want to also mention that the previous model was 3.7. 3.7 to 4 is not an entire increment, it’s theoretically the same as 3 -> 3.3, which is actually modest compared to the capability jump I’ve observed. I do think Anthropic wants more frequent, continuous releases, and using a numeric version number rather than a software version number is their intent. Gradual releases give society more time to react.

rst 1y ago
The numbers are branding, not metrics on anything. You can't do math to, say, determine the capability jump between GPT-4 and GPT-4o. Trying to do math to determine capability gaps between "3.7" and "4.0" doesn't actually make more sense.

ezst 1y ago

> does anyone else feel that this model isn’t significantly different According to Anthropic¹, LLMs are mostly a thing in the software engineering space, and not much elsewhere. I am not a software engineer, and so I'm pretty agnostic about the whole thing, mildly annoyed by the constant anthropomorphisation of LLMs in the marketing surrounding it³, and besides having had a short run with Llama about 2 years ago, I have mostly stayed away from it. Though, I do scripting as a mean to keep my digital life efficient and tidy, and so today I thought that I had a perfect justification for giving Claude 4 Sonnet a spin. I asked it to give me a jujutsu² equivalent for `git -ffdx`. What ensued was that: https://claude.ai/share/acde506c-4bb7-4ce9-add4-657ec9d5c391 https://claude.ai/share/acde506c-4bb7-4ce9-add4-657ec9d5c391 I leave you the judge of this, but for me this is very bad. Objectively, for the time that it took me to describe, review, correct some obvious logical flaws, restart, second-guess myself, get annoyed for being right and having my time wasted, fighting unwarranted complexity, etc…, I could have written a better script myself. So to answer your question, no, I don't think this is significant, and I don't think this generation of LLMs are close to their price tag. ¹: https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F189bf07829f2ecc1d3bac5961e9793a9e0f31e7d-3920x2948.jpg&w=3840&q=75 https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-... ²: https://jj-vcs.github.io/jj/latest/ https://jj-vcs.github.io/jj/latest/ ³: "hallucination", "chain of thought", "mixture of experts", "deep thinking" would have you being laughed at in the more "scientifically apt" world I grew up with, but here we are </rant>

Tarrosion 1y ago

I'm finding 4 Opus good, but 4 Sonnet a bit underwhelming: https://evanfields.net/Claude-4/ https://evanfields.net/Claude-4/