2 ms·
I'm really glad this got created, but the examples are... terrible? How is the "kernS: 'kern' referenced before assignment" problem an example of a good/verifi
by viraptor 2y ago
I'm really glad this got created, but the examples are... terrible?
How is the "kernS: 'kern' referenced before assignment" problem an example of a good/verified sample? The comment (if it is a comment?) is using // in python and I can't tell if it's an example of something failing, or a line from the error, or something else.
I would hope we can concentrate on benchmarking on full, well structured sentences before we go into "my first hasty SO question" territory. Which don't get me wrong, is useful if you want to write very short prompts, but... we're not great with basics yet and it can be always refined to a quality prompt by the LLM in a separate step. It feels really weird to me to include that in the benchmark.