6 ms·

The leap frogging at this point is getting insane (in a good way, I guess?). The amount of time each state of the art feature gets before it's supplanted is a f

• user_7832 1y ago
  I agree with your overall message - rapid growth appears to encourage competition and forces companies to put their best foot forward. However, unfortunately, I cannot shower much praise on Claude 3.7. And if you (or anyone) asks why - 3.7 seems much better than 3.5, surely? - Then I’m moderately sure that you use Claude much more for coding than for any kind of conversation. In my opinion, even 3.5 Haiku (which is available for free during high loads) is better than 3.7 Sonnet. Here’s a simple test. Try asking 3.7 to intuitively explain anything technical - say, mass dominated vs spring dominated oscillations. I’m a mechanical engineer who studied this stuff and I could not understand 3.7’s analogies. I understand that coders are the largest single group of Claude’s users, but Claude went from being my most used app to being used only after both chatgpt and Gemini, something that I absolutely regret.
  • airstrike 1y ago
    I too like 3.5 better than 3.7 and I use it pretty often. It's like 3.7 is better in 2 metrics but worse in 10 different ones
  • deleted 1y ago
    [deleted]
  • joshstrange 1y ago
    I use Claude mostly for coding/technical things and something about 3.7 does not feel like an upgrade. I haven't gone back to 3.5 (mostly started using Gemini Pro 2.5 instead). I haven't been able to use Claude research yet (it's not rolled out to the Pro tier) but o1 -> o3 deep research was a massive jump IMHO. It still isn't perfect but o1 would often give me trash results but o3 deep research actually starts to be useful. 3.5->3.7 (even with extended thinking) felt like a nothingburger.
  • mattlutze 1y ago
    The expectation that one model be top marks for all things is, imo, asking too much.
  • tiberriver256 1y ago
    3.7 did score higher in coding benchmarks but in practice 3.5 is much better at coding. 3.7 ignores instructions and does things you didn't ask it to do.
    • UncleEntity 1y ago
      I think it just does that to eat up your token quota and get you to upgrade. Like, ask it a simple question and it comes up with a full repo, complete with a README and a Makefile, when all you wanted to know was how efficient a particular algorithm would be in the included code. Can't wait until the add research to the Pro plan because, you know, I have questions...
      • vineyardmike 1y ago
        > I think it just does that to eat up your token quota and get you to upgrade. If you pay for a subscription then they don’t have an incentive to use more tokens for the same answer. It’s definitely because feedback from people has “taught” it that more boilerplate is better. It’s the same reason ChatGPT is annoyingly complementary.
      • kruxigt 1y ago
        [dead]
    • spaceman_2020 1y ago
      3.7 is too overactive I prefer Gemini 2.5 pro for all code now
      • conception 1y ago
        2.5 is my “okay Claude can’t get it” but first I check my “bank account” to see if I can afford it.
        • ralusek 1y ago
          Isn’t 2.5 pro significantly cheaper?
  • garrickvanburen 1y ago
    My current hypothesis: the more familiar you are with a topic the worse the results from any LLM.
    • jeswin 1y ago
      > My current hypothesis: the more familiar you are with a topic the worse the results from any LLM. That's not really true, since your prompts are also getting better. Better input leads to better output remains true, even with LLMs (when you see it as a tool).
      • franga2000 1y ago
        Being more familiar with the topic definitely doesn't always make your prompts better. For a lot of things it doesn't really change (explain X, compare X and Y...) - and this is what is being discussed it. For giving "building" instructions (like writing code) it helps a bit, but even if you know exactly what you want it to write, getting it to do that is pretty much trial and errror (too much detail makes it follow word-for-word and produce bad code, too little and it misses important parts or makes dumb mistakes).
      • jm547ster 1y ago
        The opposite may be true, the more effective the model the lazier the prompting as it can seemingly handle not being micromanaged as with earlier versions.
    • mac-mc 1y ago
      He was saying that 3.5 is better than 3.7 on the same topic he knows well tho.
    • user_7832 1y ago
      That is certainly the case in niche topics where published information is lacking, or needs common sense to synthesize proper outputs [1]. However in this specific example, I don't remember if it was chatgpt or gemini or 3.5 Haiku but the other(s) explained it well enough. I think I re-asked 3.5 Haiku at a later point of time, and to my complete non-surprise, it gave an answer that was quite decent. 1 - For example, the field of DIY audio - which was funnily enough the source of my question. I'm no speaker designer, but combining creativity with engineering basics/rules of thumb seems to be something LLms struggle with terribly. Ask them to design a speaker and they come up with the most vanilla, tired, textbook design - despite several existing market products that are already so much ahead/innovative. I'm confident that if you asked an LLM an identical question for which there is more discourse - eg make an interesting/innovative phone - you'd get relatively much better results.
  • fastball 1y ago
    Seems clear to me that Claude 3.7 suffers from overfitting, probably due to Anthropic seeing that 3.5 was a smash hit in the LLM coding space and deciding their North star for 3.7 should be coding benchmarks (which, like all benchmarks, do not properly capture the process of real-world coding). If it was actually good they would've named it 4.0, the fact that they went from 3.5 to 3.7 (weird jump) speaks volumes imo.
    • snewman 1y ago
      The numbering jump is because there was "Claude 3.5" and then "Claude 3.5 (new)" and they decided to retroactively stop the madness and rename the later to 3.6 (which is what everyone was calling it anyway).
      • kruxigt 1y ago
        [dead]
  • csomar 1y ago
    Plateauing overall but apparently you can gain in certain directions while you lose on some. I've written an article a while back that current models are not that far from GPT-3.5: https://omarabid.com/gpt3-now https://omarabid.com/gpt3-now 3.7 is definitively better at coding but you feel it lost a bit of maneuverability at other domains. For someone who wants code generated, it doesn't matter but I've found myself using DeepSeek first and then getting code output by 3.7.
• ilrwbwrkhv 1y ago
  None of those reports are any good though. Maybe for shallow research, but I haven't found them deep. Can you share what kind of research you have been trying there where it has done a great job of actual deep research.
  • deleted 1y ago
    [deleted]
  • Balgair 1y ago
    I'm echoing this sentiment. Deep Research hasn't really been that good for me. Maybe I'm just using it wrong? Example: I want the precipitation in mm and monthly high and low temperature in C for the top 250 most populous cities in North America. To me, this prompt seems like a pretty anodyne and obvious task for Deep Research. It's long, tedious, but mostly coming from well structured data sources (wikipedia) across two languages at most. But when I put this in to any of the various models, I mostly get back ways to go and find that data myself. Like, I know how to look at Wikipedia, it's that I don't want to comb through 250 pages manually or try to write a script to handle all the HTML boxes. I want the LLM/model to do this days long tedious task for me.
    • 85392_school 1y ago
      The funny thing is that if your request only needed the top 100's temperature or the top 33's precipitation, it could just read "List of cities by average temperature" or "List of cities by average precipitation" and that would be it, but the top 250 requires reading 184x more pages. My perspective on this is that if Deep Research can't do something, you should do it yourself and put the results on the internet. It'll help other humans and AIs trying to do the same task.
      • Balgair 1y ago
        Yeah, that was intentional, well, somewhat. The project requires the full list of every known city in the western hemisphere and also Japan, Korea, and Taiwan. But that dataset is just maddeningly large, if it is possible at all. Like, I expect it to take me years, as I have to do a lot of translations. So, I figured that I'd be nice and just as for the top 250 for the various models. There's a lot more data that we're trying to get too and I'm hoping that I can get approval to post it as its a work thing.
• greymalik 1y ago
  Out of curiosity - can you give any examples of the programming questions you are using deep research on? I’m having a hard time thinking of how it would be helpful and could use the inspiration.
  • dimitri-vs 1y ago
    Easy, any research task that will take you 5 minutes to complete it's worth firing off a Deep Research request while you work on something else in parallel. I use it a lot when documentation is vague or outdated. When Gemini/o3 can't figure something out after 2 tries. When I am working with a service/API/framework/whatever that I am very unfamiliar with and I don't even know what to Google search.
    • jerpint 1y ago
      Have you tried using llms.txt when available? Very useful resource
  • emorning3 1y ago
    I often use Chrome to valid what I think I know. I recently asked Chrome to show me how to apply the Knuth-Bendix completion procedure to propositional logic, and I had already formed my own thoughts about how to proceed (I'm building a rewrite system that does automated reasoning). The response convinced me that I'm not a total idiot. I'm not an academic and I'm often wrong about theory so the validation is really useful to me.
    • scargrillo 1y ago
      That’s a perfect example of LLMs providing epistemic scaffolding — not just giving you answers, but helping you check your footing as you explore unfamiliar territory. Especially valuable when you’re reasoning through something structurally complex like rewrite systems or proof strategies. Sometimes just seeing your internal model reflected back (or gently corrected) is enough to keep you moving.
    • miki_oomiri 1y ago
• risyachka 1y ago
  What are you talking about It is literally stagnated for a year now All that changed is they connect more apis. And add a thinking loop with same model powering it This is the reason it seems fast - nothing really happens except easy things
  • tymscar 1y ago
    I totally agree with you, especially if you actually try using these models, not just looking at random hype posters on twitter or skewed benchmarks. That being said, isn’t it strange how the community has polar opposite views about this? Did anything like this ever happen before?
• itissid 1y ago
  I've been using it for pre scoping things I have no idea about and rapidly iterating by refeeding it a version with guard rails and conditions from previous chats. Like I wanted to scope how to build a home made TrueNAS Scale unit, it helped me with a avoiding pitfalls like knowing that I needed two GPUs minimum to run the OS and local llms, and speed up config for a CLI back up of my Dropbox locally(it told me to use the right filesystem format over ZFS to make Dropbox client work). It has researched how I can structure my web app for building payment system on the web(something I knew nothing about) to writing small tools to talk to my document collection and index them into collections in Anki in one day.
• wilg 1y ago
  o3 since it can web search while reasoning is a really useful lighter weight deep research
• spaceman_2020 1y ago
  Gemini 2.5 pro was the moment for me where I really thought “this is where true adoption happens” All those talks about AI replacing people seemed a little far fetched in 2024. But in 2025, I really think models are getting good enough
  • antupis 1y ago
    You still need "human in the loop" because with simple tasks or some tasks that have lots of training material, models can one-shot answer and are like super good. But if the domain grows too complex, there are some not-so-obvious dependencies, or stuff that is in bleeding edge. Models fail pretty badly. So you need someone to split those complex tasks to more simpler familiar steps.
• iLoveOncall 1y ago
  Calling some APIs is leap-frogging? You could do this with GPT-3, nothing has changed except it's branded under a new name and tries to establish a (flawed) standard. If there was truly any innovation still happening in OpenAI, Anthropic, etc., they would be working on models only, not on side features that someone could already develop over a weekend.
  • never_inline 1y ago
    Why would you love on-call though?
    • iLoveOncall 1y ago
      In my previous team most of our oncall requests came from bug reports by customers on various tools that we owned, so to be able to work on random tools that my team owned was a nice change of pace / scenery compared to working on the same thing for 3 months uninterrupted. Now I'm in a new team where 99% of our oncall tickets come from automated alarms and 80% of them are a subset of a few issues where the root-cause isn't easy to address but there is either nothing to actually do once investigated, or the fix is a one time process that is annoying to run, so the username isn't accurate anymore :) I still like the change of pace though, 0 worries about sprint tasks or anything else for a week every few months.
• apwell23 1y ago
  > DeepResearch which started to actually come up with useful results on more complex programming questions Is there a youtube video of ppl using this on complex open source projects like linux kernel or maybe something like pytorch. How come none of the oss pojects( atleast not the ones i follow) are progressing fast(er) from AI like 'deepresearch'