3 ms·
The barplot is wrong, the numbers are correct. Looks like they had a dummy plot and never updated it, only the numbers to prevent leaking? Screenshot of the bl
by croemer 1y ago
The barplot is wrong, the numbers are correct. Looks like they had a dummy plot and never updated it, only the numbers to prevent leaking?
Screenshot of the blog plot: https://imgur.com/a/HAxIIdC https://imgur.com/a/HAxIIdC
- hnuser123456 1y agoHaha, even with that, it says 4o does worse with 2 passes than with 1. Edit: Nevermind, just now the first one is SWE-bench and 2nd is aider.
- croemer 1y agoThose are different benchmarks
- hnuser123456 1y agoI see now on the website, the screenshot cut off the header for the first benchmark, looked like it was just comparing 1-pass and 2-pass.
- croemer 1y agoYes, sorry didn't fit everything on the screenshot.
- bufferoverflow 1y ago[dead]