7 ms·
I ran the same test I ran on Opus 4.6: feeding it my whole personal collection of ~900 poems which spans ~16 years It is a far cry from Opus 4.6. Opus 4.6 was
by jorl17 8mo ago
I ran the same test I ran on Opus 4.6: feeding it my whole personal collection of ~900 poems which spans ~16 years
It is a far cry from Opus 4.6.
Opus 4.6 was (is!) a giant leap, the largest since Gemini 2.5 pro. Didn't hallucinate anything and produced honestly mind-blowing analyses of the collection as a whole. It was a clear leap forward.
Sonnet 4.6 feels like an evolution of whatever the previous models were doing. It is marginally better in the sense that it seemed to make fewer mistakes or with a lower level of severity, but ultimately it made all the usual mistakes (making things up, saying it'll quote a poem and then quoting another, getting time periods mixed up, etc).
My initial experiments with coding leave the same feeling. It is better than previous similar models, but a long distance away from Opus 4.6. And I've really been spoiled by Opus.
- K0balt 8mo agoOpus 4.6 is outstanding for code, and for the little I have used it outside of that context, in everything else I have used it with. The productivity with code is at least 3x what I was getting with 5.2, and it can handle entire projects fairly responsibly. It doesn’t patronize the user, and it makes a very strong effort to capture and follow intentions. Unlike 5.2, I’ve never had to throw out a days work that it covertly screwed up taking shortcuts and just guessing.
- renmillar 8mo agoThat last part is a real one though, mine tried to debug a Dockerfile by poking around my local environment outside of Docker today.
- josephg 8mo agoI’ve had it make some pretty obvious mistakes. I have to hold back the impulse to “unstick” it manually. In my case, it’s been surprisingly good at eventually figuring out what it was doing wrong - though sometimes it burns a few minutes of tokens in the process.
- tiltowait 8mo agoClaude's willingness to poke outside of its present directory can definitely be a little worrying. Just the other day, it started trying to access my jails after I specifically told it not to.
- teaearlgraycold 8mo agoFor the moment it’s best practice to run it and all of your dev stuff in a VM.
- e1g 8mo agoOn a Mac, I use built-in sandboxing to jail Claude (and every other agent) to $CWD so it doesn’t read/write anything it shouldn’t, doesn’t leak env, etc. This is done by dynamically generating access policies and I open sourced this at https://agent-safehouse.dev https://agent-safehouse.dev
- nowahe 8mo agoBy any chance, do you know what Claude Code's sandbox feature uses under the hood and how that relates to your solution ? From what I remember it also uses the native MacOS sandbox framework, but I haven't looked too deep into it and don't trust it fully
- e1g 8mo agoClaude Code sandboxing uses the same basic OS primitive but grants read access to the entire filesystem and includes escape hatches (some commands bypass sandboxing). Also, I wanted something solid I can use to limit every agent (OpenCode, Pi, Auggie, etc).
- qalmakka 8mo agoOn Linux in a pinch you can use bubblewrap to hide and replace directories for a given process
- caspar 8mo ago
- linolevan 8mo agoOh! Poem guy is back, hey! I like seeing this analysis on new model releases, any chance you can aggregate your opinions in one place (instead of the hackernews comment sections for these model releases)?
- stingraycharles 8mo agoGiven than Sonnet is the cheaper “workhorse” alternative for Opus, isn’t this expected?
- slopinthebag 8mo agoHow do you evaluate the analyses?
- hypercube33 8mo agoOpus 4.6 has been awful for me and my team. It goes immediately off the rails and jumps to conclusions on wants and asks and just keeps chugging along forever and won't let anything stop it down whatever path it decides. 4.5 was awesome and is our still go-to model.
- 1broseidon 8mo agoI have found this to be true too and I thought I was the only one. Everyone is praising 4.6 and while it’s great at agentic and tool use, it does not follow instructions as cleanly as 4.5 - I also feel like 4.5 was just way more efficient too
- qalmakka 8mo agoI think that's because not everyone does the same job within the same stack and constraints. I'm yet to find an LLM that writes the kind of C++ I dabble with without having to manually tweak it myself (or that truly understands our codebase). Conversely, I find that LLMs are now excellent at python and orchestration tasks for instance. It's very situational
- 1broseidon 8mo ago100% - you are very right. 4.6 is amazing for orchestration. I even built some tools around agent to agent contracting. I use 4.6 as the brain and then handoff to a more rigid llm like GPT 5.2 or Opus 4.5
- majora2007 8mo agoThat's interesting, 4.6 is finally when AI started to become good in my eyes. I have a very strict plan phase, argue, plan then partial execute. I like it to do boilerplate then I do the hard stuff myself and have it do a once over at the end. Although I have had it try to debug something and just get stuck chugging tokens.
- jxmesth 8mo agoI'm curious how this would compare with codex 5.3. I've heard Codex actually is pretty good but Opus 4.6 has become synonymous with AI coding because all the big names praise it. I haven't compared them against each other though so can't really draw a conclusion.
- zarzavat 8mo agoThere are no universals. You have to try it on your particular codebase and see what works for you. For me, OpenAI is ahead in intelligence, and Anthropic is ahead in alignment. I use both but for different tasks. Given the pace of change, intuition is somewhat of a liability: what's true today may not be true tomorrow. You have to constantly keep an open mind and try new things. Listening to influencers is a waste of time.
- Valakas_ 8mo agoThanks for testing and sharing your results.
- cube2222 8mo agoThis seems to agree with my own previous tests of Sonnet vs Opus (not on this version). If I give them a task with a large list of constraints ("do this, don't do this, make sure of this"), like 20-40, Sonnet will forget half of it, while Opus correctly applies all directives. My intuition is this is just related to model size / its "working memory", and will likely neither be fixed by training Sonnet with Opus nor by steadily optimizing its agentic capabilities.
- versteegen 8mo agoI'd agree that this effect is probably mainly due to architectural parameters such as the number and dimensions of heads, and hidden dimension. But not so much the model size (number of parameters) or less training. Saw something about Sonnet 4.6 having had a greatly increased amount of RL training over 4.5.
- hesgyrxgh 8mo agoI'm curious if you tried the same prompt for chatgpt 5.2 Did it not give you a mind blowing analysis?