The sample answers for the horse race question are crazy. [0] Pretty much all the LLM really want to split 6 horses into two groups of three. Only LLAMA 3 make

The sample answers for the horse race question are crazy. [0] Pretty much all the LLM really want to split 6 horses into two groups of three. Only LLAMA 3 makes the justification that only 2 horses can be raced at a time, but then gets its modified question wrong by racing three horses. I personally would consider an answer that presumes some restriction to how the horses can be raced to be valid if it answers the restricted version correctly. [0]: https://arxiv.org/html/2405.19616v2#S9.SS2.SSS1 https://arxiv.org/html/2405.19616v2#S9.SS2.SSS1