3 ms·
Those charts show pass@k metric (expectation at least k generated samples are correct out of n) on OpenAI and Octopack problem evals for code. WaveCoder: https
by ignoramous 3y ago
Those charts show pass@k metric (expectation at least k generated samples are correct out of n) on OpenAI and Octopack problem evals for code.
WaveCoder: https://arxiv.org/abs/2312.14187 https://arxiv.org/abs/2312.14187 (section 3.2)
Octopack: https://github.com/bigcode-project/octopack https://github.com/bigcode-project/octopack
- srikanth235 3y agoWhile testing internally, Mistral worked well. But these models are just starting points. Will add support for the models WaveCoder-Ultra-6.7B, WizardCoder-33B, Magiccoder-S-DS-6.7B, etc soon.