3 ms·
> in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. and how do you know all these benchmarks not leaked? I think
by riku_iki 3y ago
> in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off.
and how do you know all these benchmarks not leaked? I think they all scrapped from web sites, the same as training data for LLM, so risk of contamination is extremely high.
The best way to measure this is through synthetic datasets, which generate new tasks every time and model can't memorize them during training. One example is BigBench has multiple such tasks, but researchers usually(always) not regenerating those datasets.