3 ms·

VibeCodingBench: Benchmark Vibe Coding Models for Fun

2 points by jiayaoqijia 8mo ago

jiayaoqijia 8mo ago
VibeCodingBench: We benchmarked 15 AI coding models on what developers actually do Current benchmarks have an ecological validity crisis. Models score 70%+ on SWE-bench but struggle in production. Why? They optimize for bug fixes in Python repos—not the auth flows, API integrations, and CRUD dashboards that occupy 80% of real dev work. So we built VibeCodingBench: 180 tasks across SaaS features, glue code, AI integration, frontend, API integrations, and code evolution. Multi-dimensional scoring: Functional (40%) + Visual (20%) + Quality (20%) - Cost/Speed penalties. Security gate: Any OWASP Top 10 vuln = automatic 0. Top 5 Results (Jan 2026): Claude Opus 4.5 — 89.2% | $12.31 | 44s Claude Haiku 4.5 — 89.0% | $3.03 | 22s Grok 4 Fast — 88.8% | $0.21 | 70s 4⃣ OpenAI GPT-5.2 — 88.8% | $5.01 | 28s 5⃣ Qwen3 Max — 88.6% | $5.42 | 45s The real story? Cost varies 60x between similar performers. Grok 4 Fast matches GPT-5.2 at 1/25th the cost. Claude Haiku 4.5 delivers near-Opus quality for $3 total. Live dashboard: https://vibecoding.llmbench.xyz/ GitHub repo: https://github.com/alt-research/vibe-coding-benchmark-public Thesis: https://github.com/alt-research/vibe-coding-benchmark-public/blob/main/docs/THESIS.md The ultimate test isn't fixing a bug in scikit-learn. It's shipping a feature your users need—safely, efficiently—before the sprint ends. Open source. Contributions welcome.

2 points by jiayaoqijia 8mo ago

VibeCodingBench: We benchmarked 15 AI coding models on what developers actually do Current benchmarks have an ecological validity crisis. Models score 70%+ on SWE-bench but struggle in production. Why? They optimize for bug fixes in Python repos—not the auth flows, API integrations, and CRUD dashboards that occupy 80% of real dev work. So we built VibeCodingBench: 180 tasks across SaaS features, glue code, AI integration, frontend, API integrations, and code evolution. Multi-dimensional scoring: Functional (40%) + Visual (20%) + Quality (20%) - Cost/Speed penalties. Security gate: Any OWASP Top 10 vuln = automatic 0. Top 5 Results (Jan 2026): Claude Opus 4.5 — 89.2% | $12.31 | 44s Claude Haiku 4.5 — 89.0% | $3.03 | 22s Grok 4 Fast — 88.8% | $0.21 | 70s 4⃣ OpenAI GPT-5.2 — 88.8% | $5.01 | 28s 5⃣ Qwen3 Max — 88.6% | $5.42 | 45s The real story? Cost varies 60x between similar performers. Grok 4 Fast matches GPT-5.2 at 1/25th the cost. Claude Haiku 4.5 delivers near-Opus quality for $3 total. Live dashboard: https://vibecoding.llmbench.xyz/ GitHub repo: https://github.com/alt-research/vibe-coding-benchmark-public Thesis: https://github.com/alt-research/vibe-coding-benchmark-public/blob/main/docs/THESIS.md The ultimate test isn't fixing a bug in scikit-learn. It's shipping a feature your users need—safely, efficiently—before the sprint ends. Open source. Contributions welcome.