5 ms·

I agree with your overall message - rapid growth appears to encourage competition and forces companies to put their best foot forward. However, unfortunately,

I agree with your overall message - rapid growth appears to encourage competition and forces companies to put their best foot forward. However, unfortunately, I cannot shower much praise on Claude 3.7. And if you (or anyone) asks why - 3.7 seems much better than 3.5, surely? - Then I’m moderately sure that you use Claude much more for coding than for any kind of conversation. In my opinion, even 3.5 Haiku (which is available for free during high loads) is better than 3.7 Sonnet. Here’s a simple test. Try asking 3.7 to intuitively explain anything technical - say, mass dominated vs spring dominated oscillations. I’m a mechanical engineer who studied this stuff and I could not understand 3.7’s analogies. I understand that coders are the largest single group of Claude’s users, but Claude went from being my most used app to being used only after both chatgpt and Gemini, something that I absolutely regret.

airstrike 1y ago
I too like 3.5 better than 3.7 and I use it pretty often. It's like 3.7 is better in 2 metrics but worse in 10 different ones
deleted 1y ago
[deleted]
joshstrange 1y ago
I use Claude mostly for coding/technical things and something about 3.7 does not feel like an upgrade. I haven't gone back to 3.5 (mostly started using Gemini Pro 2.5 instead). I haven't been able to use Claude research yet (it's not rolled out to the Pro tier) but o1 -> o3 deep research was a massive jump IMHO. It still isn't perfect but o1 would often give me trash results but o3 deep research actually starts to be useful. 3.5->3.7 (even with extended thinking) felt like a nothingburger.
mattlutze 1y ago
The expectation that one model be top marks for all things is, imo, asking too much.
tiberriver256 1y ago
3.7 did score higher in coding benchmarks but in practice 3.5 is much better at coding. 3.7 ignores instructions and does things you didn't ask it to do.
- UncleEntity 1y ago
  I think it just does that to eat up your token quota and get you to upgrade. Like, ask it a simple question and it comes up with a full repo, complete with a README and a Makefile, when all you wanted to know was how efficient a particular algorithm would be in the included code. Can't wait until the add research to the Pro plan because, you know, I have questions...
  - vineyardmike 1y ago
    > I think it just does that to eat up your token quota and get you to upgrade. If you pay for a subscription then they don’t have an incentive to use more tokens for the same answer. It’s definitely because feedback from people has “taught” it that more boilerplate is better. It’s the same reason ChatGPT is annoyingly complementary.
  - kruxigt 1y ago
    [dead]
- spaceman_2020 1y ago
  3.7 is too overactive I prefer Gemini 2.5 pro for all code now
  - conception 1y ago
    2.5 is my “okay Claude can’t get it” but first I check my “bank account” to see if I can afford it.
    - ralusek 1y ago
      Isn’t 2.5 pro significantly cheaper?
garrickvanburen 1y ago
My current hypothesis: the more familiar you are with a topic the worse the results from any LLM.
- jeswin 1y ago
  > My current hypothesis: the more familiar you are with a topic the worse the results from any LLM. That's not really true, since your prompts are also getting better. Better input leads to better output remains true, even with LLMs (when you see it as a tool).
  - franga2000 1y ago
    Being more familiar with the topic definitely doesn't always make your prompts better. For a lot of things it doesn't really change (explain X, compare X and Y...) - and this is what is being discussed it. For giving "building" instructions (like writing code) it helps a bit, but even if you know exactly what you want it to write, getting it to do that is pretty much trial and errror (too much detail makes it follow word-for-word and produce bad code, too little and it misses important parts or makes dumb mistakes).
  - jm547ster 1y ago
    The opposite may be true, the more effective the model the lazier the prompting as it can seemingly handle not being micromanaged as with earlier versions.
- mac-mc 1y ago
  He was saying that 3.5 is better than 3.7 on the same topic he knows well tho.
- user_7832 1y ago
  That is certainly the case in niche topics where published information is lacking, or needs common sense to synthesize proper outputs [1]. However in this specific example, I don't remember if it was chatgpt or gemini or 3.5 Haiku but the other(s) explained it well enough. I think I re-asked 3.5 Haiku at a later point of time, and to my complete non-surprise, it gave an answer that was quite decent. 1 - For example, the field of DIY audio - which was funnily enough the source of my question. I'm no speaker designer, but combining creativity with engineering basics/rules of thumb seems to be something LLms struggle with terribly. Ask them to design a speaker and they come up with the most vanilla, tired, textbook design - despite several existing market products that are already so much ahead/innovative. I'm confident that if you asked an LLM an identical question for which there is more discourse - eg make an interesting/innovative phone - you'd get relatively much better results.
fastball 1y ago
Seems clear to me that Claude 3.7 suffers from overfitting, probably due to Anthropic seeing that 3.5 was a smash hit in the LLM coding space and deciding their North star for 3.7 should be coding benchmarks (which, like all benchmarks, do not properly capture the process of real-world coding). If it was actually good they would've named it 4.0, the fact that they went from 3.5 to 3.7 (weird jump) speaks volumes imo.
- snewman 1y ago
  The numbering jump is because there was "Claude 3.5" and then "Claude 3.5 (new)" and they decided to retroactively stop the madness and rename the later to 3.6 (which is what everyone was calling it anyway).
  - kruxigt 1y ago
    [dead]
csomar 1y ago
Plateauing overall but apparently you can gain in certain directions while you lose on some. I've written an article a while back that current models are not that far from GPT-3.5: https://omarabid.com/gpt3-now https://omarabid.com/gpt3-now 3.7 is definitively better at coding but you feel it lost a bit of maneuverability at other domains. For someone who wants code generated, it doesn't matter but I've found myself using DeepSeek first and then getting code output by 3.7.

airstrike 1y ago

I too like 3.5 better than 3.7 and I use it pretty often. It's like 3.7 is better in 2 metrics but worse in 10 different ones

deleted 1y ago

[deleted]

joshstrange 1y ago

I use Claude mostly for coding/technical things and something about 3.7 does not feel like an upgrade. I haven't gone back to 3.5 (mostly started using Gemini Pro 2.5 instead). I haven't been able to use Claude research yet (it's not rolled out to the Pro tier) but o1 -> o3 deep research was a massive jump IMHO. It still isn't perfect but o1 would often give me trash results but o3 deep research actually starts to be useful. 3.5->3.7 (even with extended thinking) felt like a nothingburger.

mattlutze 1y ago

The expectation that one model be top marks for all things is, imo, asking too much.

tiberriver256 1y ago

3.7 did score higher in coding benchmarks but in practice 3.5 is much better at coding. 3.7 ignores instructions and does things you didn't ask it to do.

UncleEntity 1y ago
I think it just does that to eat up your token quota and get you to upgrade. Like, ask it a simple question and it comes up with a full repo, complete with a README and a Makefile, when all you wanted to know was how efficient a particular algorithm would be in the included code. Can't wait until the add research to the Pro plan because, you know, I have questions...
- vineyardmike 1y ago
  > I think it just does that to eat up your token quota and get you to upgrade. If you pay for a subscription then they don’t have an incentive to use more tokens for the same answer. It’s definitely because feedback from people has “taught” it that more boilerplate is better. It’s the same reason ChatGPT is annoyingly complementary.
- kruxigt 1y ago
  [dead]
spaceman_2020 1y ago
3.7 is too overactive I prefer Gemini 2.5 pro for all code now
- conception 1y ago
  2.5 is my “okay Claude can’t get it” but first I check my “bank account” to see if I can afford it.
  - ralusek 1y ago
    Isn’t 2.5 pro significantly cheaper?

garrickvanburen 1y ago

My current hypothesis: the more familiar you are with a topic the worse the results from any LLM.

jeswin 1y ago
> My current hypothesis: the more familiar you are with a topic the worse the results from any LLM. That's not really true, since your prompts are also getting better. Better input leads to better output remains true, even with LLMs (when you see it as a tool).
- franga2000 1y ago
  Being more familiar with the topic definitely doesn't always make your prompts better. For a lot of things it doesn't really change (explain X, compare X and Y...) - and this is what is being discussed it. For giving "building" instructions (like writing code) it helps a bit, but even if you know exactly what you want it to write, getting it to do that is pretty much trial and errror (too much detail makes it follow word-for-word and produce bad code, too little and it misses important parts or makes dumb mistakes).
- jm547ster 1y ago
  The opposite may be true, the more effective the model the lazier the prompting as it can seemingly handle not being micromanaged as with earlier versions.
mac-mc 1y ago
He was saying that 3.5 is better than 3.7 on the same topic he knows well tho.
user_7832 1y ago
That is certainly the case in niche topics where published information is lacking, or needs common sense to synthesize proper outputs [1]. However in this specific example, I don't remember if it was chatgpt or gemini or 3.5 Haiku but the other(s) explained it well enough. I think I re-asked 3.5 Haiku at a later point of time, and to my complete non-surprise, it gave an answer that was quite decent. 1 - For example, the field of DIY audio - which was funnily enough the source of my question. I'm no speaker designer, but combining creativity with engineering basics/rules of thumb seems to be something LLms struggle with terribly. Ask them to design a speaker and they come up with the most vanilla, tired, textbook design - despite several existing market products that are already so much ahead/innovative. I'm confident that if you asked an LLM an identical question for which there is more discourse - eg make an interesting/innovative phone - you'd get relatively much better results.

fastball 1y ago

Seems clear to me that Claude 3.7 suffers from overfitting, probably due to Anthropic seeing that 3.5 was a smash hit in the LLM coding space and deciding their North star for 3.7 should be coding benchmarks (which, like all benchmarks, do not properly capture the process of real-world coding). If it was actually good they would've named it 4.0, the fact that they went from 3.5 to 3.7 (weird jump) speaks volumes imo.

snewman 1y ago
The numbering jump is because there was "Claude 3.5" and then "Claude 3.5 (new)" and they decided to retroactively stop the madness and rename the later to 3.6 (which is what everyone was calling it anyway).
- kruxigt 1y ago
  [dead]

csomar 1y ago

Plateauing overall but apparently you can gain in certain directions while you lose on some. I've written an article a while back that current models are not that far from GPT-3.5: https://omarabid.com/gpt3-now https://omarabid.com/gpt3-now 3.7 is definitively better at coding but you feel it lost a bit of maneuverability at other domains. For someone who wants code generated, it doesn't matter but I've found myself using DeepSeek first and then getting code output by 3.7.