4 ms·
One interesting tidbit from the technical report: >HumanEval is an industry standard open-source evaluation benchmark (Chen et al., 2021), but we found control
by scarmig 3y ago
One interesting tidbit from the technical report:
>HumanEval is an industry standard open-source evaluation benchmark (Chen et al., 2021), but we
found controlling for accidental leakage on webpages and open-source code repositories to be a non-trivial task, even with conservative filtering heuristics. An analysis of the test data leakage of
Gemini 1.0 Ultra showed that continued pretraining on a dataset containing even a single epoch of
the test split for HumanEval boosted scores from 74.4% to 89.0%, highlighting the danger of data contamination. We found that this sharp increase persisted even when examples were embedded
in extraneous formats (e.g. JSON, HTML). We invite researchers assessing coding abilities of these
models head-to-head to always maintain a small set of truly held-out test functions that are written
in-house, thereby minimizing the risk of leakage. The Natural2Code benchmark, which we announced
and used in the evaluation of Gemini 1.0 series of models, was created to fill this gap. It follows the
exact same format of HumanEval but with a different set of prompts and tests.