3 ms·
Wow! It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Al
by HarHarVeryFunny 4mo ago
Wow!
It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Also great to see different harnesses being tested, with considerably different results.
Definitely a few surprises here:
1) GLM 5.2 using Pi performs identically in terms of pass rate (~87.5%) to Opus 4.8 high using Claude Code, but significantly cheaper ($1.25 per task vs $2)
2) Absolute best pass rate (90%) was from Opus 4.8 x-high using Pi, beating out Opus 4.8 using Claude Code
3) Pareto frontier performance from any of the models (Opus 4.8, GPT 5.5, GLM 2.5) was using Pi rather than native harnesses
Apparently Pi used 3x less context than Claude Code, and one takeaway is to use Pi regardless of what model you are using. The other takeaway is that in real-world performance GLM 5.2 is the equal of Opus 4.8 unless you run Opus 4.8 on x-high in which case you can eke out a 2.5% increase in pass rate at the expense of doubling your cost over GLM 5.2
- zaphar 4mo agoThe catch-22 here is that you still save money using Claude Code directly here if you are on one of their subscription plan due to how heavily they are subsidizing that. Using Pi means you are using the API which is both a more accurate pricing model but also a more expensive one.
- agentdev001 4mo agoObligatory yes, but only if you're subscription-based and not pay-per-token as enterprise users are.
- HarHarVeryFunny 4mo agoTrue, but OpenAI are OK with using their subscription plans with Pi, so GPT 5.6 with Pi may be a good combination. There seems to be a lot of good buzz about GPT 5.6 on Twitter - people (incl. OpenCode team) preferring it to Fable 5.
- ryt 4mo agoPi has a subscription flow for Claude Code. Are they goading people to get banned or did they figure out a way to work with the subscription?
- sgc 4mo agoIt seems like they are actually using the subscription providers' respective cli tools and managing context for them. In which case I believe it is not against the ToS any more than invoking codex cli from a custom python script would be? I personally cannot find an openai-hosted clarification, just on hn and x.
- ryt 4mo agoAwesome - that's what I do in my own harness for agy. Hopefully someone yells if they get banned.
- calumcl 4mo agoDatabricks wouldn't be using a subscription plan though due to the size of their engineering team which is what matters here. Free lunch is over for basically all providers' biz coding plans