Yeah, although looks like it currently has some issues with coqa: https://github.com/EleutherAI/lm-evaluation-harness/issues/238 https://github.com/EleutherAI/l

Yeah, although looks like it currently has some issues with coqa: https://github.com/EleutherAI/lm-evaluation-harness/issues/238 https://github.com/EleutherAI/lm-evaluation-harness/issues/2... There's also the bigscience fork, but I ran into even more problems (although I didn't try too hard) https://github.com/bigscience-workshop/lm-evaluation-harness https://github.com/bigscience-workshop/lm-evaluation-harness And there's https://github.com/EleutherAI/lm-eval2/ https://github.com/EleutherAI/lm-eval2/ (not sure if it's just starting over w/ a new repo or what?) but it has limited tests available