4 ms·
Are these just leetcode exercises? What I would like to see is an independent benchmark based on real tasks in codebases of varying size.
by evantbyrne 1y ago
Are these just leetcode exercises? What I would like to see is an independent benchmark based on real tasks in codebases of varying size.
- KaoruAoiShiho 1y agoAider is not just leetcode exercises I think? livecodebench is leetcode exercises though.
- rafram 1y agoAider uses a dataset of 500 GitHub issues, so not LeetCode-style work.
- evantbyrne 1y agoIt says right on that linked page: > Aider’s polyglot benchmark tests LLMs on 225 challenging Exercism coding exercises across C++, Go, Java, JavaScript, Python, and Rust. I looked up Exercism and they appear to be story problems that you solve by coding on mostly/entirely blank slates, unless I'm missing something? That format would seem to explain why the models are reportedly performing so well, because they definitely aren't that reliable on mature codebases.