4 ms·

swe-bench's bigger problems include (1) labs train on the test and (2) 50% of the tickets are from django; it's not a representative dataset even if all you car

swe-bench's bigger problems include (1) labs train on the test and (2) 50% of the tickets are from django; it's not a representative dataset even if all you care about is Python. I created a new benchmark from Java commits that are new in the past 6 months to add some variety: https://brokk.ai/power-ranking https://brokk.ai/power-ranking

lostmsu 1y ago
No GLM?
- jbellis 1y ago
  no, I'm pretty skeptical that it's better than qwen3 coder but if you have evidence that it could be, I'm down to test it
  - lostmsu 1y ago
    It has the same score on https://lmarena.ai/leaderboard/webdev https://lmarena.ai/leaderboard/webdev , but AFAIK Air version is much smaller.
    - jbellis 1y ago
      I've added results for GLM 4.5 and 4.5 Air.