3 ms·

Any lawyers here who have used AI agents heavily for their work? From what I've heard, they're currently very good at searching, analyzing and drafting document

by keeda 17d ago

Any lawyers here who have used AI agents heavily for their work? From what I've heard, they're currently very good at searching, analyzing and drafting documents like contracts and patents, but some say they suck at interpreting the law.

amelius 17d ago
Probably because the AI was also trained on HN comments.
- john_strinlai 17d ago
  worse, it was trained on reddit law comments
  - alansaber 16d ago
    Could be worse, you should see the facebook law comments
qingcharles 17d ago
They are excellent, especially the latest models. That said, (a) I wouldn't feel safe filing something without a real lawyer looking at it; (b) it can't (easily? legally?) do oral arguments for you; and (c) a lot can happen in the hallways outside the courtroom to move a case forward that the AI can't easily do.
shicholas 17d ago
We do and it saves so much time. Of course human judgment is needed but it’s like cooking with someone else doing the mise en place.
droidjj 17d ago
I’m sure there are a wide variety of experiences out there, but here’s my perspective as a former biglaw associate and current solo litigator: I have had some success using frontier models from the last 6ish months, but only when I can break up my work into discrete and verifiable tasks. For example, I had ~15k pages of discovery I needed to dig through for a summary judgment motion. Instead of just asking Claude to find the best evidence, I asked it first to run a clean, high quality OCR pass (it was almost entirely PDFs). Then I had it generate embeddings and write some reusable python scripts to make keyword and semantic searching easy for agents. While I was writing the brief, I would routinely ask my agent (Claude Code) to use both keyword and semantic searching to find the best evidence supporting whatever assertion I was trying to make. I trusted it because there were traces I could follow. In other cases/situations, I’ve tried just giving a model access to all the docs and saying “write a brief arguing X,” but it’s always terrible at this. It writes briefs with lots of evocative jargon and rhetorical flourish, but a low signal-to-noise ratio. Again, I’m sure others’ experiences differ based on workflow, legal area, etc.
- deleted 17d ago
  [deleted]
- shimman 17d ago
  What are you using for semantic searching?
- Digory 17d ago
  Agreed. Six months ago, it was basically a gloried grammarly. But lately, I’ve been taking hints from the “company brain” models, where it develops a running model of the case, and assesses each new piece as it comes in and updates the file. I’ve also been using “Ralph Wiggum”-type models where you pass letter or contract drafts back and forth between agents with different goals (rules compliance, grammar, conciseness, ai slop detector, an opposing counsel critic, etc.). After a few rounds, it’s not perfect — but I start with a very good first draft in my hands.
- Otterly99 16d ago
  How did you mix up keyword and semantic search? Are they separate searches, or some sort of weighted search?
NoboruWataya 17d ago
In my experience it basically doesn't even try to interpret the law. It just summarises the publicly available law/guidance out there and, if there is a question about how to interpret some provision, might set out the arguments for each interpretation. It doesn't really take a position. It is pretty good at drafting though. (Legora)
- keeda 17d ago
  > It doesn't really take a position. Interesting! In other unrelated domains, models seem more willing to take a position. It may be nuanced, but they do tend to take a stance. I think it's good that it leaves the interpretation to humans, but I wonder if this is also some sort of a guardrail to minimize liability...
deleted 17d ago
[deleted]
deleted 16d ago
[deleted]

by keeda 17d ago

amelius 17d ago

Probably because the AI was also trained on HN comments.

john_strinlai 17d ago
worse, it was trained on reddit law comments
- alansaber 16d ago
  Could be worse, you should see the facebook law comments

qingcharles 17d ago

They are excellent, especially the latest models. That said, (a) I wouldn't feel safe filing something without a real lawyer looking at it; (b) it can't (easily? legally?) do oral arguments for you; and (c) a lot can happen in the hallways outside the courtroom to move a case forward that the AI can't easily do.

shicholas 17d ago

We do and it saves so much time. Of course human judgment is needed but it’s like cooking with someone else doing the mise en place.

droidjj 17d ago

I’m sure there are a wide variety of experiences out there, but here’s my perspective as a former biglaw associate and current solo litigator: I have had some success using frontier models from the last 6ish months, but only when I can break up my work into discrete and verifiable tasks. For example, I had ~15k pages of discovery I needed to dig through for a summary judgment motion. Instead of just asking Claude to find the best evidence, I asked it first to run a clean, high quality OCR pass (it was almost entirely PDFs). Then I had it generate embeddings and write some reusable python scripts to make keyword and semantic searching easy for agents. While I was writing the brief, I would routinely ask my agent (Claude Code) to use both keyword and semantic searching to find the best evidence supporting whatever assertion I was trying to make. I trusted it because there were traces I could follow. In other cases/situations, I’ve tried just giving a model access to all the docs and saying “write a brief arguing X,” but it’s always terrible at this. It writes briefs with lots of evocative jargon and rhetorical flourish, but a low signal-to-noise ratio. Again, I’m sure others’ experiences differ based on workflow, legal area, etc.

deleted 17d ago
[deleted]
shimman 17d ago
What are you using for semantic searching?
Digory 17d ago
Agreed. Six months ago, it was basically a gloried grammarly. But lately, I’ve been taking hints from the “company brain” models, where it develops a running model of the case, and assesses each new piece as it comes in and updates the file. I’ve also been using “Ralph Wiggum”-type models where you pass letter or contract drafts back and forth between agents with different goals (rules compliance, grammar, conciseness, ai slop detector, an opposing counsel critic, etc.). After a few rounds, it’s not perfect — but I start with a very good first draft in my hands.
Otterly99 16d ago
How did you mix up keyword and semantic search? Are they separate searches, or some sort of weighted search?

NoboruWataya 17d ago

In my experience it basically doesn't even try to interpret the law. It just summarises the publicly available law/guidance out there and, if there is a question about how to interpret some provision, might set out the arguments for each interpretation. It doesn't really take a position. It is pretty good at drafting though. (Legora)

keeda 17d ago
> It doesn't really take a position. Interesting! In other unrelated domains, models seem more willing to take a position. It may be nuanced, but they do tend to take a stance. I think it's good that it leaves the interpretation to humans, but I wonder if this is also some sort of a guardrail to minimize liability...

deleted 17d ago

[deleted]

deleted 16d ago

[deleted]