4 ms·

Parametric CAD Bench

• gnucleus_peggy 5mo ago
  [dead]
• mjzh 5mo ago
  interesting, per https://cadbench.ai/leaderboard https://cadbench.ai/leaderboard, gpt5.5 is the best, not the opus 4.7, why opus 4.7 is with mini-swe-agent, not claude code.
  • handcrafted 5mo ago
    GPT-5.5 and Opus 4.7 are comparable when using the same harness mini-swe-agent. GPT-5.5 demonstrates a significant performance delta only when integrated with the Codex module. We hypothesize that the superior performance of Opus 4.7 on mini-swe-agent relative to the more complex Claude Code harness stems from the tight feedback loop (edit-run-check), well suited for the CAD generation task.
    • bigskydog 5mo ago
      There are also a benchmark called BenchCAD that came out recently, which shows similiar results, Opus 4.7 seems to be the best. https://benchcad.github.io/BenchCAD_webpage/ https://benchcad.github.io/BenchCAD_webpage/
      • handcrafted 5mo ago
        We have a section "How we compare to CadBench and BenchCAD" in our bench report comparing our Parametric CAD Bench with CADBench and BenchCAD: https://cadbench.ai/news/parametric-cad-bench https://cadbench.ai/news/parametric-cad-bench
• little_cad 5mo ago
  I only see closed-source models on your leaderboard so far: https://cadbench.ai/leaderboard https://cadbench.ai/leaderboard It would be interesting to see how open-source models perform on CAD tasks.