3 ms·
Not sure if I'm reading this right, but the "success rate" table for OpenAI models shows Clojure near the bottom. And if I switch provider to Anthropic, success
by andai 5mo ago
Not sure if I'm reading this right, but the "success rate" table for OpenAI models shows Clojure near the bottom. And if I switch provider to Anthropic, success rate for most languages, including Clojure, goes up dramatically.
- gertlabs 5mo agoSuccess rate includes syntax/compilation failures as well as environment rule violations, and is almost entirely from one-shot code generations. Percentile shows how well the working submissions perform. In long horizon agentic coding evaluations, strong models fix the syntax and percentile and it becomes a direct comparison of which submissions per language performed the best on average. You can filter for that here: https://gertlabs.com/rankings?provider=openai&mode=agentic_coding https://gertlabs.com/rankings?provider=openai&mode=agentic_c...