Just played around with Opus. I'm starting to wonder if benchmarks are deviating from real world performance systematically - it doesn't seem actually better t

Just played around with Opus. I'm starting to wonder if benchmarks are deviating from real world performance systematically - it doesn't seem actually better than GPT-4, slightly worse if anything. Basic calculus/physics questions were worse off (it ignored my stating deceleration is proportional to velocity and just assumed constant). A traffic simulation I've been using (understanding traffic light and railroad safety and walking through the AI like a kid) is underperforming GPT-4's already poor results, forgetting previous concepts discussed earlier in the conversation about directions/etc. A test I conduct with understanding of primary light colors with in-context teaching is also performing worse. On coding, it slightly underperformed GPT-4 at the (surprisingly hard for AI) question of computing long term capital gains tax, given ordinary income, capital gains, and ltcg brackets. Took another step of me correcting it (neither model can do it right 0 shot)