4 ms·
Results and dataset explorer here: https://huggingface.co/spaces/nuprl/verbal-reasoning-challenge https://huggingface.co/spaces/nuprl/verbal-reasoning-challen..
by enum 2y ago
Results and dataset explorer here: https://huggingface.co/spaces/nuprl/verbal-reasoning-challenge https://huggingface.co/spaces/nuprl/verbal-reasoning-challen...
- mh- 2y agoFor ID=3, it shows o1 getting it wrong, but it seems to have succeeded? It did add a space between Tinker and bell, but that is the canonical way of spelling the character apparently. (That just one caught my attention because I was curious what challenge o1-mini got correct that o1 did not.)
- enum 2y agoThanks, fixed. (Spaces rebuilding.) We have manually combed labelled-wrong answers and tweaked the predicates that check correctness. Sorry we missed this one.
- mh- 2y agoI thought maybe I was just missing something silly.. was hesitant to comment without additional coffee. And no problem, thanks for sharing this!
- ks2048 2y agoIs there some standard markup assumed for LLMs? I just looked at a couple and saw markdown-like text (which I expected), but also "$\boxed{foo}$". Looks like LaTeX, but is there some subset of LaTeX LLMs are assumed to know/use?