2 ms·

LLMs are the first genuinely useful legal tech since the Internet. I'm pretty shocked, though, at the delta between how competent Claude is on code versus legal

LLMs are the first genuinely useful legal tech since the Internet. I'm pretty shocked, though, at the delta between how competent Claude is on code versus legal work. It's good for research and data organization, but terrible for drafting. I wonder if this is a structural problem with the lack of feedback loops. In law, there's no compiler to check for logical or continuity errors in your brief, and there's no unit tests to check for correctness or performance. Even without that, I think it'll be extremely valuable to clients to allow them to answer simple questions without a lawyer, figure out the lay of the land so they can supervise their counsel, etc.

throwaway20222 9d ago
I have found that it’s useful generally speaking to get the intent of contracts and red lines, but actual drafting I agree is where I lose all confidence. My guess is that the significance of the difference between using a word like “and“ or “or“ can be so meaningful that that level of nuance can often be lost. But I know nothing I’m not in the space, I just pay too much money for lawyers.
samtp 9d ago
> LLMs are the first genuinely useful legal tech since the Internet That is an incredible statement that could not be further from the truth. Large scale adoption of email, searchable document databases like Westlaw, LexisNexis, PACER, etc.. , OCR Software, electronic signatures, and tons more have had a much more defineably positive impact on the legal profession since the internet came about.
- rayiner 9d ago
  I think "email" and "westlaw" fairly count as "the Internet." LLMs might be bigger than either of those.
  - samtp 9d ago
    The internet came well before both of those. And none of the example I gave have lead to lawyers regularly being fined in court because of "hallucinations"
- alansaber 8d ago
  But emphasising "tech" advancements, that really only covers OCR.
qarl 9d ago
You should know - for coding they make terrible mistakes as well. But programmers have this concept of a "code review" where another person looks at the code to look for problems. We use this same technique with our LLMs. Most mistakes are caught by having a second LLM look at it. Doesn't even need to be a different model - just make sure it has a different context.
- flyinglizard 9d ago
  Programming has a long standing culture of accepting the code to be somewhat wrong, so we have various tests, linters, reviews and error handling. Also in programming there are many ways to do something and it's the end result that matters most. Not so in other knowledge work. There's no test harness for a contract and error is non-recoverable. Likewise in finance. There are specific ways of doing things and these ways are many times set in regulations. LLMs can assist all day, sure. But replacing the human, in highly regulated, zero tolerance for error environment?
  - qarl 9d ago
    I don't understand. I just provided advice about getting better output. Are you trying to reply to someone else?
    - flyinglizard 9d ago
      You made some points worthy of expansion: >> You should know - for coding they make terrible mistakes as well. >> But programmers have this concept of a "code review" where another person looks at the code to look for problems.
  - jll29 8d ago
    You could say the Salvatorian Clause in contracts is like exception handling: a "catch (all)": even if some clauses in this contract are illegal, the remaining contract stays in place. Logically, this actually doesn't make sense strictly speaking because the sentence creates a paradox: doesn't it make clear whether it includes itself or not, and each reading ends up in trouble. There is a "tradition" in law around the world to accept the only benign reading of such clauses, which I always found funny given that in all other ways lawyers adopt the most adversarial mindset imaginable.
cannonpalms 8d ago
Law is certainly a less verifiable domain
shim__ 8d ago
I reckon law and medicine are really premier use case for llms since those areas are all about having vast knowledge(knowing all about obsure cases in law or about an very rare disease the average doctor wouldn't have heard about)
- alansaber 8d ago
  I've heard this argument (basically, you can just have good legal skills and adapt your practice to more areas of law) but we're still pretty far from a layperson confidently navigating court (you could say the same about code).
alansaber 8d ago
That's exactly it. You basically can't set up deterministic regression tests, which makes development of (legal capabilities) even more abstract and messy.

throwaway20222 9d ago

I have found that it’s useful generally speaking to get the intent of contracts and red lines, but actual drafting I agree is where I lose all confidence. My guess is that the significance of the difference between using a word like “and“ or “or“ can be so meaningful that that level of nuance can often be lost. But I know nothing I’m not in the space, I just pay too much money for lawyers.

samtp 9d ago

> LLMs are the first genuinely useful legal tech since the Internet That is an incredible statement that could not be further from the truth. Large scale adoption of email, searchable document databases like Westlaw, LexisNexis, PACER, etc.. , OCR Software, electronic signatures, and tons more have had a much more defineably positive impact on the legal profession since the internet came about.

rayiner 9d ago
I think "email" and "westlaw" fairly count as "the Internet." LLMs might be bigger than either of those.
- samtp 9d ago
  The internet came well before both of those. And none of the example I gave have lead to lawyers regularly being fined in court because of "hallucinations"
alansaber 8d ago
But emphasising "tech" advancements, that really only covers OCR.

qarl 9d ago

You should know - for coding they make terrible mistakes as well. But programmers have this concept of a "code review" where another person looks at the code to look for problems. We use this same technique with our LLMs. Most mistakes are caught by having a second LLM look at it. Doesn't even need to be a different model - just make sure it has a different context.

flyinglizard 9d ago
Programming has a long standing culture of accepting the code to be somewhat wrong, so we have various tests, linters, reviews and error handling. Also in programming there are many ways to do something and it's the end result that matters most. Not so in other knowledge work. There's no test harness for a contract and error is non-recoverable. Likewise in finance. There are specific ways of doing things and these ways are many times set in regulations. LLMs can assist all day, sure. But replacing the human, in highly regulated, zero tolerance for error environment?
- qarl 9d ago
  I don't understand. I just provided advice about getting better output. Are you trying to reply to someone else?
  - flyinglizard 9d ago
    You made some points worthy of expansion: >> You should know - for coding they make terrible mistakes as well. >> But programmers have this concept of a "code review" where another person looks at the code to look for problems.
- jll29 8d ago
  You could say the Salvatorian Clause in contracts is like exception handling: a "catch (all)": even if some clauses in this contract are illegal, the remaining contract stays in place. Logically, this actually doesn't make sense strictly speaking because the sentence creates a paradox: doesn't it make clear whether it includes itself or not, and each reading ends up in trouble. There is a "tradition" in law around the world to accept the only benign reading of such clauses, which I always found funny given that in all other ways lawyers adopt the most adversarial mindset imaginable.

cannonpalms 8d ago

Law is certainly a less verifiable domain

shim__ 8d ago

I reckon law and medicine are really premier use case for llms since those areas are all about having vast knowledge(knowing all about obsure cases in law or about an very rare disease the average doctor wouldn't have heard about)

alansaber 8d ago
I've heard this argument (basically, you can just have good legal skills and adapt your practice to more areas of law) but we're still pretty far from a layperson confidently navigating court (you could say the same about code).

alansaber 8d ago

That's exactly it. You basically can't set up deterministic regression tests, which makes development of (legal capabilities) even more abstract and messy.