4 ms·
Did you use the same pass@1 generation method as in the code llama paper (greedy decoding)? I couldn't find this in the blog post.
by vikp 3y ago
Did you use the same pass@1 generation method as in the code llama paper (greedy decoding)? I couldn't find this in the blog post.
- rushingcreek 3y agoWe used sampling with temperature=0.1. Reproduction details can be found on the Huggingface model card: https://huggingface.co/Phind/Phind-CodeLlama-34B-v1 https://huggingface.co/Phind/Phind-CodeLlama-34B-v1
- vikp 3y agoGot it, thanks - and thanks for the model! I'd be interested in the results if anyone benchmarks without sampling. Edit: it could also be misleading to directly compare humaneval pass@1 against codellama without the same generation methodology. (possibly against GPT-4, also, but I don't know their methodology).