4 ms·

You will want to look at HumanEval (https://github.com/abacaj/code-eval https://github.com/abacaj/code-eval) and Eval+ (https://github.com/my-other-github-accou

by lhl 3y ago

You will want to look at HumanEval (https://github.com/abacaj/code-eval https://github.com/abacaj/code-eval) and Eval+ (https://github.com/my-other-github-account/llm-humaneval-benchmarks https://github.com/my-other-github-account/llm-humaneval-ben...) results for coding. While Llama2 is an improvement over LLaMA v1, it's still nowhere near even the best open models (currently, sans test contamination, WizardCoder-15B, a StarCoder fine tune is at top). It's really not a competition atm though, ChatGPT-4 wipes the floor for coding atm.

riku_iki 3y ago
this all numbers can be missleading, and simply indicate that gpt have these tasks in training data, and another model doesn't.
- lhl 3y ago
  While there's some contamination, it's not like the community isn't aware of it. For example, here's this discussion: https://huggingface.co/sahil2801/replit-code-instruct-glaive/discussions/3 https://huggingface.co/sahil2801/replit-code-instruct-glaive... This was the HumanEval contamination one dev measured: ``` replit_glaive: 56.71% replit: 7.32% wizard: 4.88% ``` From the WizardCoder paper https://arxiv.org/pdf/2306.08568.pdf https://arxiv.org/pdf/2306.08568.pdf you can see that it hits SOTA (for open models) in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. For those interested in reading more about various considerations for coding models, I highly recommend reading the MSR phi-1 paper: https://arxiv.org/pdf/2306.11644.pdf https://arxiv.org/pdf/2306.11644.pdf Looking forward to if they ever publish code/model/dataset since it has extremely strong performance trained on a very small number of tokens very manageable 1.3B and 350M parameter models.
  - riku_iki 3y ago
    > in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. and how do you know all these benchmarks not leaked? I think they all scrapped from web sites, the same as training data for LLM, so risk of contamination is extremely high. The best way to measure this is through synthetic datasets, which generate new tasks every time and model can't memorize them during training. One example is BigBench has multiple such tasks, but researchers usually(always) not regenerating those datasets.

by lhl 3y ago

riku_iki 3y ago

this all numbers can be missleading, and simply indicate that gpt have these tasks in training data, and another model doesn't.

lhl 3y ago
While there's some contamination, it's not like the community isn't aware of it. For example, here's this discussion: https://huggingface.co/sahil2801/replit-code-instruct-glaive/discussions/3 https://huggingface.co/sahil2801/replit-code-instruct-glaive... This was the HumanEval contamination one dev measured: ``` replit_glaive: 56.71% replit: 7.32% wizard: 4.88% ``` From the WizardCoder paper https://arxiv.org/pdf/2306.08568.pdf https://arxiv.org/pdf/2306.08568.pdf you can see that it hits SOTA (for open models) in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. For those interested in reading more about various considerations for coding models, I highly recommend reading the MSR phi-1 paper: https://arxiv.org/pdf/2306.11644.pdf https://arxiv.org/pdf/2306.11644.pdf Looking forward to if they ever publish code/model/dataset since it has extremely strong performance trained on a very small number of tokens very manageable 1.3B and 350M parameter models.
- riku_iki 3y ago
  > in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. and how do you know all these benchmarks not leaked? I think they all scrapped from web sites, the same as training data for LLM, so risk of contamination is extremely high. The best way to measure this is through synthetic datasets, which generate new tasks every time and model can't memorize them during training. One example is BigBench has multiple such tasks, but researchers usually(always) not regenerating those datasets.