3 ms·

I'd like to have LawGPT, where it reads all laws of a certain country, and I can ask it things like, "what are my options when the neighbors tree crosses over i

I'd like to have LawGPT, where it reads all laws of a certain country, and I can ask it things like, "what are my options when the neighbors tree crosses over into my property?"

consumer451 3y ago
> I'd like to have LawGPT I tried ChatGPT 4 for a specific TX county's laws. This was after I had spent hours doing traditional search. ChatGPT got everything correct, and even gave me some more direction.
- xmprt 3y ago
  This only shows that ChatGPT is as good as or better than your traditional search and not that it's gotten the law right. Unless you go to a lawyer and confirm this doesn't say a whole lot especially considering how prone LLMs are to hallucinating incorrect answers that look correct (even if you found it matched up with Google search, the answers might still be wrong in whatever blog or article you read).
  - consumer451 3y ago
    That's a fair point. This happened to all be in an area of the law, estate law, which was heavily documented for the public in that jurisdiction. The estate was tiny, and the actual legal advice was locked behind a paywall ($7k minimum in legal fees) which would have otherwise taken the majority of the estate. Still a huge win for $20/month.
    - xmprt 3y ago
      Yeah. The results are impressive and I can't deny the value when ChatGPT is able to get things like this right. But especially given how much disinformation is starting to be spread on the internet these days and how it's all being scraped for training LLMs (and especially in the case of law where different jurisdictions can have very different laws but ChatGPT could easily get its wires crossed), I'm worried that it might lead to more harm than good when people make decisions based on incorrect outputs of LLMs.
- Alupis 3y ago
  > ChatGPT got everything correct My anecdote with technical information is exactly the opposite, unfortunately. I routinely use it to help guide my research - generally about topics I know something about but want to know more. It often will provide some response that is entirely wrong but looks very good. When called out, it apologizes, then proceeds to produce more information that's also not entirely correct. There's an artform to teasing out the right answer... but of course you have to be knowledgeable enough to know what is right in the first place. The subtleties of correct or wrong can be very difficult to determine for a layperson. Without deep knowledge, one might be inclined to believe the first response it provides. That's pretty terrifying when you think of the possible implications revolving around research, law, etc.
  - consumer451 3y ago
    I have had similar experiences in other areas, that's why I didn't try an LLM before spending hours researching. But I must say that I was shocked at the accuracy in that one particular area... Harris County estate law. What is the difference here? No hallucinations, everything correct… is it just random chance?
  - LouisSayers 3y ago
    I'm having the same issues with it getting things confidently incorrect. The flow I've been going through: * Do this task with this constraint > GPT4 Does task * Analyse all of the items and whether they adhere to the constraint > GPT4 "Apologies, I got some things wrong, here's what I did incorrect" * Fix the things you did wrong, remember the constraint is of upmost importance > GPT4 "I have completed the task and fixed the constraint * Analyse all of the items and whether they adhere to the constraint > GPT4 "Apologies, I got some things wrong, here's what I did incorrect" ... At least it can analyse the results, but even then if you don't ask in the right way it will blatantly lie to you and tell you it's all correct when it's not.
  - nicce 3y ago
    I have found the very similar issues. But on programming language and other logical question areas it does not matter much, as you can verify with logic if something is correct, and then it is incredible useful
- Turskarama 3y ago
  The issue is that you cannot _guarantee_ it will get everything correct, the fact that it managed it in your case is largely a matter of luck. In law, being correct 99% of the time is not really good enough.
- skissane 3y ago
  > I tried ChatGPT 4 for a specific TX county's laws. This was after I had spent hours doing traditional search. ChatGPT got everything correct, and even gave me some more direction. Some months back, I tried asking it detailed questions about Australian drug laws. Initially it responded accurately, but then told me some absolute crazy nonsense (that LSD was a completely legal drug in Australia). I just tried it again and it isn't doing that any more. I still wouldn't trust anything it says about legal topics. IBM mainframe assembly language remains a topic on which ChatGPT (the GPT-3.5 version at least) rather consistently hallucinates. For example, I just asked it to explain the difference between SVC (Supervisor Call) and PC (Program Call) instructions. It wrongly claimed PC is used to make calls within the current program. On the contrary, the PC instruction is basically an LPC/IPC mechanism, it is used to make a call to another process running in a different address space.
- tbrownaw 3y ago
  Relying on that too hard might be a bit hazardous: https://reason.com/volokh/2023/08/21/another-note-from-a-judge-about-generative-ai-programs/ https://reason.com/volokh/2023/08/21/another-note-from-a-jud... https://reason.com/volokh/2023/06/22/sanctions-issued-in-case-where-lawyers-cited-chatgpt-hallucinated-precedents/ https://reason.com/volokh/2023/06/22/sanctions-issued-in-cas...
- menro2 3y ago
  We are working on LawGPT. Today's foundational models like GPT 4 have some Laws, but not all. Sadly some US States still require the purchase of printed copies and others are blocked by TOS.
- bronco21016 3y ago
  I’m curious of methods other commenters to you are using. I recently vectorized a PDF of material that contains a lot of complex scheduling language. It covers legal regulations as well as contractual constraints. Using langchain’s QA function I queried the vector DB. I then tested it against a sample of questions from a Facebook group on the subject. It did shockingly well. It seems so much has to do with feeding it the right chunks of source material. When I experimented with just copy and paste small chunks on my own it rarely provided useful feedback. I love the law idea. I just bought a house and would like to do an addition. Feeding local code and regulation for QA could be a god send in navigating this stuff for a non-expert.
PeterisP 3y ago
Lots and lots of organizations would want a strong ChatGPT-like tool which is trained on a specific (likely not open) dataset of their choosing in addition to all the large general data that grants "common sense" and language understanding. Your example for all laws of a certain country is a good one, and the legal department of some (for example) chemical megacorp would want that and also all the court cases that are even so slightly relevant, and all their contracts they have made since 1900, and all their internal policy documents - and that has quite some potential. But it really does need to be a large language model, for such scenarios, the differences between e.g. ChatGPT 3.5 or ChatGPT 4 or Llama or Llama 2 are meaningful.
yreg 3y ago
How do you build that in a way that you can rely on the results? The safest approach currently is to use GPT-4 and tell it something like: "I will give you a block of text and a question. Please answer the question using only the information in the provided text. Don't make any guesses and admit if you are not sure about your answer. If you don't know the answer, say you don't know the answer. Question: X Text: Y" Trouble is that you cannot put the entire books of law into Y because the context is limited. So currently you need to do something else to sieve through the law material and find only the passages most relevant to the question through embeddings or full text search or some other method. It's not very reliable.
shagie 3y ago
There was an example project a bit ago... https://www.chatpdf.com https://www.chatpdf.com - https://news.ycombinator.com/item?id=35626312 https://news.ycombinator.com/item?id=35626312 and also https://askyourpdf.com https://askyourpdf.com https://news.ycombinator.com/item?id=35430409 https://news.ycombinator.com/item?id=35430409 I don't recall which one it was (or if it might have been another one) that had a rental contract as the example document. The interesting part (to me) was the ability to ask it questions like "can you bring a dog to a party" and have it answer and show the passages that restricted pets and parties.
hnfong 3y ago
I suspect having a database of legislation and case law, and a good vector database, then feeding relevant context into a language model with a large context window will work best.

I'd like to have LawGPT, where it reads all laws of a certain country, and I can ask it things like, "what are my options when the neighbors tree crosses over into my property?"

consumer451 3y ago

> I'd like to have LawGPT I tried ChatGPT 4 for a specific TX county's laws. This was after I had spent hours doing traditional search. ChatGPT got everything correct, and even gave me some more direction.

xmprt 3y ago
This only shows that ChatGPT is as good as or better than your traditional search and not that it's gotten the law right. Unless you go to a lawyer and confirm this doesn't say a whole lot especially considering how prone LLMs are to hallucinating incorrect answers that look correct (even if you found it matched up with Google search, the answers might still be wrong in whatever blog or article you read).
- consumer451 3y ago
  That's a fair point. This happened to all be in an area of the law, estate law, which was heavily documented for the public in that jurisdiction. The estate was tiny, and the actual legal advice was locked behind a paywall ($7k minimum in legal fees) which would have otherwise taken the majority of the estate. Still a huge win for $20/month.
  - xmprt 3y ago
    Yeah. The results are impressive and I can't deny the value when ChatGPT is able to get things like this right. But especially given how much disinformation is starting to be spread on the internet these days and how it's all being scraped for training LLMs (and especially in the case of law where different jurisdictions can have very different laws but ChatGPT could easily get its wires crossed), I'm worried that it might lead to more harm than good when people make decisions based on incorrect outputs of LLMs.
Alupis 3y ago
> ChatGPT got everything correct My anecdote with technical information is exactly the opposite, unfortunately. I routinely use it to help guide my research - generally about topics I know something about but want to know more. It often will provide some response that is entirely wrong but looks very good. When called out, it apologizes, then proceeds to produce more information that's also not entirely correct. There's an artform to teasing out the right answer... but of course you have to be knowledgeable enough to know what is right in the first place. The subtleties of correct or wrong can be very difficult to determine for a layperson. Without deep knowledge, one might be inclined to believe the first response it provides. That's pretty terrifying when you think of the possible implications revolving around research, law, etc.
- consumer451 3y ago
  I have had similar experiences in other areas, that's why I didn't try an LLM before spending hours researching. But I must say that I was shocked at the accuracy in that one particular area... Harris County estate law. What is the difference here? No hallucinations, everything correct… is it just random chance?
- LouisSayers 3y ago
  I'm having the same issues with it getting things confidently incorrect. The flow I've been going through: * Do this task with this constraint > GPT4 Does task * Analyse all of the items and whether they adhere to the constraint > GPT4 "Apologies, I got some things wrong, here's what I did incorrect" * Fix the things you did wrong, remember the constraint is of upmost importance > GPT4 "I have completed the task and fixed the constraint * Analyse all of the items and whether they adhere to the constraint > GPT4 "Apologies, I got some things wrong, here's what I did incorrect" ... At least it can analyse the results, but even then if you don't ask in the right way it will blatantly lie to you and tell you it's all correct when it's not.
- nicce 3y ago
  I have found the very similar issues. But on programming language and other logical question areas it does not matter much, as you can verify with logic if something is correct, and then it is incredible useful
Turskarama 3y ago
The issue is that you cannot _guarantee_ it will get everything correct, the fact that it managed it in your case is largely a matter of luck. In law, being correct 99% of the time is not really good enough.
skissane 3y ago
> I tried ChatGPT 4 for a specific TX county's laws. This was after I had spent hours doing traditional search. ChatGPT got everything correct, and even gave me some more direction. Some months back, I tried asking it detailed questions about Australian drug laws. Initially it responded accurately, but then told me some absolute crazy nonsense (that LSD was a completely legal drug in Australia). I just tried it again and it isn't doing that any more. I still wouldn't trust anything it says about legal topics. IBM mainframe assembly language remains a topic on which ChatGPT (the GPT-3.5 version at least) rather consistently hallucinates. For example, I just asked it to explain the difference between SVC (Supervisor Call) and PC (Program Call) instructions. It wrongly claimed PC is used to make calls within the current program. On the contrary, the PC instruction is basically an LPC/IPC mechanism, it is used to make a call to another process running in a different address space.
tbrownaw 3y ago
Relying on that too hard might be a bit hazardous: https://reason.com/volokh/2023/08/21/another-note-from-a-judge-about-generative-ai-programs/ https://reason.com/volokh/2023/08/21/another-note-from-a-jud... https://reason.com/volokh/2023/06/22/sanctions-issued-in-case-where-lawyers-cited-chatgpt-hallucinated-precedents/ https://reason.com/volokh/2023/06/22/sanctions-issued-in-cas...
menro2 3y ago
We are working on LawGPT. Today's foundational models like GPT 4 have some Laws, but not all. Sadly some US States still require the purchase of printed copies and others are blocked by TOS.
bronco21016 3y ago
I’m curious of methods other commenters to you are using. I recently vectorized a PDF of material that contains a lot of complex scheduling language. It covers legal regulations as well as contractual constraints. Using langchain’s QA function I queried the vector DB. I then tested it against a sample of questions from a Facebook group on the subject. It did shockingly well. It seems so much has to do with feeding it the right chunks of source material. When I experimented with just copy and paste small chunks on my own it rarely provided useful feedback. I love the law idea. I just bought a house and would like to do an addition. Feeding local code and regulation for QA could be a god send in navigating this stuff for a non-expert.

PeterisP 3y ago

Lots and lots of organizations would want a strong ChatGPT-like tool which is trained on a specific (likely not open) dataset of their choosing in addition to all the large general data that grants "common sense" and language understanding. Your example for all laws of a certain country is a good one, and the legal department of some (for example) chemical megacorp would want that and also all the court cases that are even so slightly relevant, and all their contracts they have made since 1900, and all their internal policy documents - and that has quite some potential. But it really does need to be a large language model, for such scenarios, the differences between e.g. ChatGPT 3.5 or ChatGPT 4 or Llama or Llama 2 are meaningful.

yreg 3y ago

How do you build that in a way that you can rely on the results? The safest approach currently is to use GPT-4 and tell it something like: "I will give you a block of text and a question. Please answer the question using only the information in the provided text. Don't make any guesses and admit if you are not sure about your answer. If you don't know the answer, say you don't know the answer. Question: X Text: Y" Trouble is that you cannot put the entire books of law into Y because the context is limited. So currently you need to do something else to sieve through the law material and find only the passages most relevant to the question through embeddings or full text search or some other method. It's not very reliable.

shagie 3y ago

There was an example project a bit ago... https://www.chatpdf.com https://www.chatpdf.com - https://news.ycombinator.com/item?id=35626312 https://news.ycombinator.com/item?id=35626312 and also https://askyourpdf.com https://askyourpdf.com https://news.ycombinator.com/item?id=35430409 https://news.ycombinator.com/item?id=35430409 I don't recall which one it was (or if it might have been another one) that had a rental contract as the example document. The interesting part (to me) was the ability to ask it questions like "can you bring a dog to a party" and have it answer and show the passages that restricted pets and parties.

hnfong 3y ago

I suspect having a database of legislation and case law, and a good vector database, then feeding relevant context into a language model with a large context window will work best.