55 ms·

AI can filter through thousands of Legal case files and court documents to pinpoint exactly what cases are relevant to a current case that a lawyer may be advis

AI can filter through thousands of Legal case files and court documents to pinpoint exactly what cases are relevant to a current case that a lawyer may be advising on! It took me exactly 30 minutes to go through 428 case files... :)

bottled_poe 4y ago
How much better is this than a keyword search across the documents?
- WoodenChair 4y ago
  And when it’s a probabilistic LLM instead of a keyword search, how do you know you didn’t miss anything?
  - wickoff 4y ago
    Humans miss things all the time. It's okay to make mistakes.
    - WoodenChair 4y ago
      It’s not replacing a human, it’s replacing a keyword search.
  - _puk 4y ago
    As opposed to manually trawling through thousands of documents, based on a keyword that may, or may not, appear in the context of the case file? It's a huge boon, but nothing is perfect. If it's that important that absolutely nothing is missed, maybe try multiple approaches concurrently. Take this as a value add, not a replacement.
  - ben_w 4y ago
    If you don't already grok all the legal jargon, the LLM is probably better than a keyword search. I'm not a lawyer, so when I search British copyright law, I wonder why those forms of words don't result in the staff of search engines and social media sites being arrested and imprisoned: https://www.gov.uk/government/publications/intellectual-property-offences/intellectual-property-offences https://www.gov.uk/government/publications/intellectual-prop... Although, probably best to ask a lawyer for legal advice even if they end up using GPT-n themselves; they've probably got public liability insurance if they get something wrong.
- dist-epoch 4y ago
  I've seen people use "code words" exactly to avoid them being find by regulatory keyword searches. AI can figure out from context, a keyword search will miss it. And no, you don't know the "code words".
- MagicMoonlight 4y ago
  Keyword search doesn’t understand the context of what you’re asking for.
Jabbles 4y ago
Can it? Or are you just imagining that it might be able to one day?
SCdF 4y ago
Did it? How do you know it didn't confidently miss important documents? My experience so far has just been asking chatgpt questions and then researching it myself to confirm what it says, so maybe I'm missing something. But, it has been confidently wrong on important details a large enough percentage (right now) to make it absolutely not a fire and forget tool. The worst part is the confidence: it's like having a coworker that just straight up lies to your face randomly. Even if it's only 5% of the time you basically can't trust anything they say, and so you need to double check all of it. This doesn't make it useless, but it means it lends itself to "hard to do but easy to verify" tasks. Which afaict your example is not: you can verify the documents it picked out are relevant, but not that the documents that it didn't, weren't.
- asah 4y ago
  coworkers give me wrong answers a lot more than 5% of the time.
  - hiq 4y ago
    On the other hand I can usually come up with my own estimate of how trustworthy the answer is when a human gave it to me, e.g. thanks to: * their reputation with respect to the question domain (if I ask a basic C++ question to a C++ expert I'll trust them) * their own communicated confidence and how good they are at seeing their own shortcomings (if they say "but don't quote me on that, better ask this other person who knows more" it's fine) 5% of bad answers doesn't matter if 99% of these times I knew I should look further. ChatGPT and others are missing this confidence indicator, and they seem to answer just as confidently no matter what. To be clear I don't see a fundamental reason why LLMs couldn't compute some measure of confidence (which will itself be wrong from time to time but with less impact) so I expect this to be solved eventually.
    - famouswaffles 4y ago
      Base gpt-4 already did this.(confidence about something directly correlated with ability to solve problem/answer questions correctly) You can read the technical paper. But the hammer of alignment(RLHF) took it away.
copperx 4y ago
How in the world did you feed GPT those 428 case files? There's nothing in the API to make it ingest that much info.
- fritzo 4y ago
  I'm guessing the author is computing embeddings https://platform.openai.com/docs/api-reference/embeddings https://platform.openai.com/docs/api-reference/embeddings

bottled_poe 4y ago

How much better is this than a keyword search across the documents?

WoodenChair 4y ago
And when it’s a probabilistic LLM instead of a keyword search, how do you know you didn’t miss anything?
- wickoff 4y ago
  Humans miss things all the time. It's okay to make mistakes.
  - WoodenChair 4y ago
    It’s not replacing a human, it’s replacing a keyword search.
- _puk 4y ago
  As opposed to manually trawling through thousands of documents, based on a keyword that may, or may not, appear in the context of the case file? It's a huge boon, but nothing is perfect. If it's that important that absolutely nothing is missed, maybe try multiple approaches concurrently. Take this as a value add, not a replacement.
- ben_w 4y ago
  If you don't already grok all the legal jargon, the LLM is probably better than a keyword search. I'm not a lawyer, so when I search British copyright law, I wonder why those forms of words don't result in the staff of search engines and social media sites being arrested and imprisoned: https://www.gov.uk/government/publications/intellectual-property-offences/intellectual-property-offences https://www.gov.uk/government/publications/intellectual-prop... Although, probably best to ask a lawyer for legal advice even if they end up using GPT-n themselves; they've probably got public liability insurance if they get something wrong.
dist-epoch 4y ago
I've seen people use "code words" exactly to avoid them being find by regulatory keyword searches. AI can figure out from context, a keyword search will miss it. And no, you don't know the "code words".
MagicMoonlight 4y ago
Keyword search doesn’t understand the context of what you’re asking for.

Jabbles 4y ago

Can it? Or are you just imagining that it might be able to one day?

SCdF 4y ago

Did it? How do you know it didn't confidently miss important documents? My experience so far has just been asking chatgpt questions and then researching it myself to confirm what it says, so maybe I'm missing something. But, it has been confidently wrong on important details a large enough percentage (right now) to make it absolutely not a fire and forget tool. The worst part is the confidence: it's like having a coworker that just straight up lies to your face randomly. Even if it's only 5% of the time you basically can't trust anything they say, and so you need to double check all of it. This doesn't make it useless, but it means it lends itself to "hard to do but easy to verify" tasks. Which afaict your example is not: you can verify the documents it picked out are relevant, but not that the documents that it didn't, weren't.

asah 4y ago
coworkers give me wrong answers a lot more than 5% of the time.
- hiq 4y ago
  On the other hand I can usually come up with my own estimate of how trustworthy the answer is when a human gave it to me, e.g. thanks to: * their reputation with respect to the question domain (if I ask a basic C++ question to a C++ expert I'll trust them) * their own communicated confidence and how good they are at seeing their own shortcomings (if they say "but don't quote me on that, better ask this other person who knows more" it's fine) 5% of bad answers doesn't matter if 99% of these times I knew I should look further. ChatGPT and others are missing this confidence indicator, and they seem to answer just as confidently no matter what. To be clear I don't see a fundamental reason why LLMs couldn't compute some measure of confidence (which will itself be wrong from time to time but with less impact) so I expect this to be solved eventually.
  - famouswaffles 4y ago
    Base gpt-4 already did this.(confidence about something directly correlated with ability to solve problem/answer questions correctly) You can read the technical paper. But the hammer of alignment(RLHF) took it away.

copperx 4y ago

How in the world did you feed GPT those 428 case files? There's nothing in the API to make it ingest that much info.

fritzo 4y ago
I'm guessing the author is computing embeddings https://platform.openai.com/docs/api-reference/embeddings https://platform.openai.com/docs/api-reference/embeddings