2 ms·
The website talks about a specific benchmark: > Python code generation. Held out dataset HumanEval-like, not leaked on the web What I find interesting here is
by eru 3y ago
The website talks about a specific benchmark:
> Python code generation. Held out dataset HumanEval-like, not leaked on the web
What I find interesting here is that for this particular benchmark _not_ publishing the benchmark is advertised as a feature (instead of as a sign of 'trust me, bro, we have a great benchmark'), and I can understand why. Still these are strange times we live in.