34 ms·

Any time I see humaneval comparisons I feel the need to point out that humaneval is only 164 questions. 66.5 vs. 65.9% is a difference between 109 and 108 solut

Any time I see humaneval comparisons I feel the need to point out that humaneval is only 164 questions. 66.5 vs. 65.9% is a difference between 109 and 108 solutions, a single question. Still interesting work though

deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
moyix 3y ago
At least they used HumanEval+, which adds a bunch more test cases and fixes some errors in the original benchmark!