5 ms·

I wonder how long it takes AI to get good at law. Right now the verbal tasks it excels at are similar to the artistic ones: namely, solving problems with enormo

I wonder how long it takes AI to get good at law. Right now the verbal tasks it excels at are similar to the artistic ones: namely, solving problems with enormous solution spaces that are robust to small perturbations. That is, change a good picture of an angry tree man slightly and it's still probably a good picture of an angry tree man.

Dr4kn 3y ago
Depends what you would classify as good, but IBM Watson is already used in law firms [today](https://www.ibm.com/case-studies/legalmation https://www.ibm.com/case-studies/legalmation) LLMs iare most often best at helping humans do their tasks more effectively, not replacing them completely
qingcharles 3y ago
I've tried using a lot for writing motions. It can actually do a pretty decent job of writing motions, and it can come up with some arguments that you might not have thought of. You just have to ignore all its citations and look everything up yourself, otherwise this: https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/ https://www.reuters.com/legal/new-york-lawyers-sanctioned-us...
matheusmoreira 3y ago
Isn't ChatGPT getting progressively better scores on medical and law exams? It will probably pass the USMLE and the bar one day. If it doesn't already. It's gonna be interesting.
- Jeff_Brown 3y ago
  Yes and that's amazing -- but law exams resemble programming exams. In the wild, both labors require you to keep a mountain of project-specific context in your head, something that tests like the LSAT cannot evaluate.
- coffeefirst 3y ago
  Yes, but we should expect that, the answers are in its training data. The problem is passing tests are an okay proxy for competence in humans, but if you think of LLMs as a giant library search engine, the thing it is competent at is identifying and regurgitating compiled phrases from its records. Which is awesome. It can't be a doctor.
  - benreesman 3y ago
    Relatedly, a lot of the cap R fanfic crowd points at stuff pretty clearly lifted out of the Metamorphosis of Prime Intellect or whatever. And it’s dope that it can do that! But let’s keep our heads about what it is.

Dr4kn 3y ago

Depends what you would classify as good, but IBM Watson is already used in law firms [today](https://www.ibm.com/case-studies/legalmation https://www.ibm.com/case-studies/legalmation) LLMs iare most often best at helping humans do their tasks more effectively, not replacing them completely

qingcharles 3y ago

I've tried using a lot for writing motions. It can actually do a pretty decent job of writing motions, and it can come up with some arguments that you might not have thought of. You just have to ignore all its citations and look everything up yourself, otherwise this: https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/ https://www.reuters.com/legal/new-york-lawyers-sanctioned-us...

matheusmoreira 3y ago

Isn't ChatGPT getting progressively better scores on medical and law exams? It will probably pass the USMLE and the bar one day. If it doesn't already. It's gonna be interesting.

Jeff_Brown 3y ago
Yes and that's amazing -- but law exams resemble programming exams. In the wild, both labors require you to keep a mountain of project-specific context in your head, something that tests like the LSAT cannot evaluate.
coffeefirst 3y ago
Yes, but we should expect that, the answers are in its training data. The problem is passing tests are an okay proxy for competence in humans, but if you think of LLMs as a giant library search engine, the thing it is competent at is identifying and regurgitating compiled phrases from its records. Which is awesome. It can't be a doctor.
- benreesman 3y ago
  Relatedly, a lot of the cap R fanfic crowd points at stuff pretty clearly lifted out of the Metamorphosis of Prime Intellect or whatever. And it’s dope that it can do that! But let’s keep our heads about what it is.