Got it, thanks - and thanks for the model! I'd be interested in the results if anyone benchmarks without sampling. Edit: it could also be misleading to direct

Got it, thanks - and thanks for the model! I'd be interested in the results if anyone benchmarks without sampling. Edit: it could also be misleading to directly compare humaneval pass@1 against codellama without the same generation methodology. (possibly against GPT-4, also, but I don't know their methodology).