2 ms·

Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).

• sothatsit 2mo ago
  If I need something smarter I use Fable. Medium works well and is quick. Opus 5 medium feels much better to me than Opus 4.8 medium.
  • dhorthy 2mo ago
    yeah someone will have to re-run this bench on various effort levels. unfortunately it is not cheap
• conception 2mo ago
  https://cognition.com/frontiercode https://cognition.com/frontiercode Quality vs cost - medium is the sweet (perhaps better too!) spot.
  • dhorthy 2mo ago
    my issue with frontier code is that it uses a model judge for quality whereas slop code bench forces a model to grapple with its own garbage code in order to receive a functionality reward
  • fastball 2mo ago
    That is just a single benchmark tho
• ValentineC 2mo ago
  Medium or low supposedly prevents Opus 5 from overthinking: https://xcancel.com/danshipper/status/2080700057892815114 https://xcancel.com/danshipper/status/2080700057892815114