nah, it's probably an okay-ish benchmark for humans, but it's just that, a filter to weed out those who can't learn hundreds of pages of legal trivia. the mode

nah, it's probably an okay-ish benchmark for humans, but it's just that, a filter to weed out those who can't learn hundreds of pages of legal trivia. the model is great at this, because the training set is full of this stuff. the questions are static and simple. (the actual text of the questions change, of course, but the format and the answers are from a fixed set. and the LLM doesn't need to generate text basically, just one token. A B C or D ... that said I'm curious how it's administered to the LLMs and how much that influences their performance.)