5 ms·
Using up to 32k thinking tokens, Sonnet 3.7 set SOTA with a 64.9% score. 65% Sonnet 3.7, 32k thinking 64% R1+Sonnet 3.5 62% o1 high 60% Sonnet 3.7, no
by anotherpaulg 2y ago
Using up to 32k thinking tokens, Sonnet 3.7 set SOTA with a 64.9% score.
65% Sonnet 3.7, 32k thinking
64% R1+Sonnet 3.5
62% o1 high
60% Sonnet 3.7, no thinking
60% o3-mini high
57% R1
52% Sonnet 3.5
- pclmulqdq 2y agoAlso for $36.83 compared to o1's $186.50
- pzo 2y agoBut also for $36.83 compared to DeepSeek R1 + claude-3-5 it's $13.29 and for latter "Percent using correct edit format" is 100% vs 97.8% for 3.7. edit: would be interesting to see how combo DeepSeek R1 + claude-3-7 performs.
- tw1984 2y agois there any public info on why such DeepSeek R1 + claude-3-5 combo worked better than using a single model?
- Ballas 2y agoFrom my experiments with the Deepseek Qwen-32b distill model, the Deepseek model did not follow the edit instructions - the format was wrong. I know the distill models are not at all the same as the full model, but that could provide a clue. Combine that information with the scores, then you have a reasonable hypothesis.
- re-thc 2y ago> I know the distill models are not at all the same as the full model It's far worse than that. It's not the model (Deepseek) at all. It's Qwen enhanced with Deepseek. So it's Qwen still.
- alienthrowaway 2y agoSonnet 3.5 is the best non-Chain-of-Thought code-authoring model. When paired with R1's CoT output, Sonnet 3.5 performs even better - outperforming vanilla R1 (and eveything else), which suggests Sonnet is better than R1 at utilizing R1's CoT. It's scenario where the result is greater than the sum of it's parts
- WiSaGaN 2y agoMy personal experience is that R1 is smarter than 3.5 sonnet, but 3.5 sonnet is a better coder. Thus it may be better to let R1 to tackle the problem, but let 3.5 sonnet to implement the solution.
- pythonaut_16 2y agoSpecialization of AI models is cool. Just like some people might be better planners and some are better at raw coding ability.
- VectorLock 2y agoHow does it stack up against Grok3? I've seen some discussion that Grok3 is good for coding.
- viraptor 2y agoIt isn't available over api yet, as far as I know. So it can't be really tested independently.
- VectorLock 2y agoThe comparisons I saw I think were manual, so it makes sense it can run a whole suite- these were just some basic prompts and showed the difference in how the produced output ran.
- pclmulqdq 2y agoPro tip: It's hard to trust Twitter for opinions on Grok. The thumb is very clearly on the scale. I have personally seen very few positive opinions of Grok outside of Twitter.
- _xtrimsky 2y agoI thought Grok 2 was pretty bad, but Grok 3 is actually quite good. I'm mostly impressed by the speed of answering. But Claude is still the king of code.
- VectorLock 2y agoI agree with you, and I hate to say this, but I saw them on LinkedIn. One purportedly used the same prompts to make a "pacman like" game and the results from Grok3 were at least better, assuming the post is on the up and up, better looking than o3-mini-high.
- mikae1 2y agoIt's clear that progress is incremental at this point. At the same time Anthropic and OpenAI are bleeding money. It's unclear to me how they'll shift to making money while providing almost no enhanced value.
- khafra 2y agoYudkowsky just mentioned that even if LLM progress stopped right here, right now, there are enough fundamental economic changes to provide us a really weird decade. Even with no moat, if the labs are in any way placed to capture a little of the value they've created, they could make high multiples of their investors' money.
- jonplackett 2y agoYep totally agree. It will also depend who captures the most eyeballs. ChatGPT is already my default first place to check something, where it was Google for the previous 20+ years.
- ssl-3 2y agoI use it for all kinds of unique things, but ChatGPT is the last place I look for facts.
- sarchertech 2y agoEyeballs aren’t enough though. Unlike Google ChatGPT is very expensive to run. It’s unlikely they can just slap ads on it like Google did.
- AJ007 2y agoInference costs will keep dropping. The stuff the average consumer does will be trivially cheap. More stuff will move on device. The edge capabilities of these models are already far beyond what the average person can use or comprehend. The point I wonder about is the sustainability of every query being 30+ requests. Site owners aren't ready to have 98% of their requests be non-monetizable bot traffic. However, sites that have something to sell are..
- vessenes 2y agoPaul, I saw in the notes that using claude with thinking mode requires yml config updates -- any pointers here? I was parsing some commits, and I couldn't tell if you only added architect support through openrouter. Thanks!
- anotherpaulg 2y agoHere are the current docs for changing the thinking token limits. https://aider.chat/docs/llms/anthropic.html#thinking-tokens https://aider.chat/docs/llms/anthropic.html#thinking-tokens I'll make this less clunky soon.
- vessenes 2y agoThanks. FWIW, it feels to me like this would be best as a global setting, not per-repo? Or, I guess it might be more aider-y to have sane defaults in the app and command line changes. Anyway, happily plugging away with the architect settings now!