3 ms·

Human students also get to see those sample questions. Is the Bar exam a bad test? Maybe. But it's how we judge the merit of new lawyers.

• jfengel 4y ago
  Humans and LLMs solve the bar exam differently. A human can't memorize the complete set of solved examples, so they have to generate it at least somewhat from scratch. It's why we give the bar exam under controlled circumstances, so that they're not just plagiarizing from existing sources. And that's what you want to test them for: the ability to apply their knowledge to novel situations. LLMs don't have memory limitations, so they can produce equivalent answers using very different means. But that means that it's a poor proxy for those novel situations. The real challenge a lawyer faces is sitting down with a client and getting a stream of verbiage, some of which is relevant and some of which isn't. In that, it's much like a programmer's job. Extracting the actual requirements is usually harder than writing the code for them. As with programming, some days are easier than others. There's a middle ground of tasks that are harder than just filling in a form, and easier than inventing a new algorithm or legal theory. Perhaps it's the "no code" of law: good enough for users to solve a lot of problems on their own, and they need to call out to a professional for more things. The bar exam could potentially be a proxy for computers to prove that they're ready to take up that middle ground. But it hasn't been tested on that yet.