3 ms·

I wonder if this is one of the few times simonw's pelican was broken on the first try [1]: https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2

I wonder if this is one of the few times simonw's pelican was broken on the first try [1]: https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fraw.githubusercontent.com%2Fsimonw%2Fllm-anthropic%2F8272dfee5bdb65d5c88eef083da3ad885539b7df%2Flog.md https://tools.simonwillison.net/markdown-svg-renderer#url=ht... My experience with Opus 5 thus far haven't been that great either. It's been making mistake after mistake editing my coding plans that were being reviewed by GPT-6 Sol. [1] https://simonwillison.net/2026/Jul/24/introducing-claude-opus-5/ https://simonwillison.net/2026/Jul/24/introducing-claude-opu...

simonw 3mo ago
Yeah this was pretty surprising. For almost every model I've run the pelican against the first attempt was at least recognizable enough that I didn't feel like the model needed a second shot. It's always a roll of a dice, but it's surprising that the dice rolls so infrequently come up bad, yet Opus 5 rolled a bad pelican this one time. I suspect it's just a freak occurrence. I rolled a few more and they were all fine, I think Opus 5 just got unlucky.
- ValentineC 3mo ago
  What I think is more alarming is that Claude's advice for prompting Opus 5 says that Opus will verify its own work [1]: > Claude Opus 5 verifies its own work without being told to. If your prompt contains explicit verification instructions ("include a final verification step for any non-trivial task," "use a subagent to verify"), remove them: instructions like these cause over-verification on Claude Opus 5, and removing them reduces wasted tokens with no loss in quality. My own experience over the past ~20 hours hasn't been great either, with Opus producing sloppy mockups (e.g. buttons overflowing past cards) without doing any of the purported verification. [1] https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5#task-scope-and-over-verification https://platform.claude.com/docs/en/build-with-claude/prompt...
  - simonw 3mo ago
    The way I run the pelican benchmark prevents it from checking its own work. It gets one API call to return an SVG. If I ran the benchmark in Claude Code or a similar harness it could render the SVG as an image, look at what it created, then make tweaks to it.
    - ValentineC 3mo ago
      Ah, thanks for clarifying!

simonw 3mo ago

Yeah this was pretty surprising. For almost every model I've run the pelican against the first attempt was at least recognizable enough that I didn't feel like the model needed a second shot. It's always a roll of a dice, but it's surprising that the dice rolls so infrequently come up bad, yet Opus 5 rolled a bad pelican this one time. I suspect it's just a freak occurrence. I rolled a few more and they were all fine, I think Opus 5 just got unlucky.

ValentineC 3mo ago
What I think is more alarming is that Claude's advice for prompting Opus 5 says that Opus will verify its own work [1]: > Claude Opus 5 verifies its own work without being told to. If your prompt contains explicit verification instructions ("include a final verification step for any non-trivial task," "use a subagent to verify"), remove them: instructions like these cause over-verification on Claude Opus 5, and removing them reduces wasted tokens with no loss in quality. My own experience over the past ~20 hours hasn't been great either, with Opus producing sloppy mockups (e.g. buttons overflowing past cards) without doing any of the purported verification. [1] https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5#task-scope-and-over-verification https://platform.claude.com/docs/en/build-with-claude/prompt...
- simonw 3mo ago
  The way I run the pelican benchmark prevents it from checking its own work. It gets one API call to return an SVG. If I ran the benchmark in Claude Code or a similar harness it could render the SVG as an image, look at what it created, then make tweaks to it.
  - ValentineC 3mo ago
    Ah, thanks for clarifying!