3 ms·

> Even for lawyer it could be the same thing Funny you mention that, since lawyers have already gotten in trouble for citing fictional cases when submitting wo

> Even for lawyer it could be the same thing Funny you mention that, since lawyers have already gotten in trouble for citing fictional cases when submitting work performed by chatgpt. It's useful for rote things, but for anything that you depend on, you still need to give it just as much attention as if you'd done it yourself.

qup 3y ago
I agree, but it does change the shape of the work you would be doing. Validating work isn't the same as generating new work. Whether that's better, or useful, probably differs by situation.
- blibble 3y ago
  > Validating work isn't the same as generating new work. in many cases it's way harder
cloverich 3y ago
> you still need to give it just as much attention as if you'd done it yourself. It's different than a Lawyers case, where the facts require manually cross referencing. In our case, that verification can come via directly and immediately running the code. How good the actual code is varies, but the fundamental difference remains.
skissane 3y ago
> Funny you mention that, since lawyers have already gotten in trouble for citing fictional cases when submitting work performed by chatgpt. For something like legal work, you don’t want a raw LLM. You want an agent integrated with a legal database, in such a way that it can’t cite cases which don’t exist in the database. And it can’t generate a direct quote from a case unless that text actually occurs in the case. You still need a human lawyer familiar with the law to pick up on subtler errors, but better technology can prevent the grosser ones. And the subtler errors (e.g. misrepresenting what a case says through partial or out of context quoting) is the kind of error human lawyers sometimes make too - like every other profession, lawyers vary greatly in their competence, and often only the grosser cases of incompetence incur sanctions
- zmgsabst 3y ago
  Tried Lexis+ AI, and… it’s just not very good yet. Much like ChatGPT, it can only handle recall and summation — and even then, I don’t fully trust it because it often misses key ideas. And much like ChatGPT, it can’t do anything coherent at length without a lot of help and working around its faults. And seems entirely unaware of similar sounding words having distinct legal meanings. Which is not good.
  - skissane 3y ago
    > Tried Lexis+ AI, and… it’s just not very good yet. I wonder to what extent that’s due to current inherent limitations of the technology, and to what extent it is due to quality of implementation issues. It is hard to say because (I presume) there is only limited public information on how it is actually implemented. e.g. which LLM is it using? How much fine-tuning has been done? Are they using other potentially helpful techniques such as guided sampling? Or breaking down the task into parts and having multiple agents each specialised to handle one particular part? > Much like ChatGPT When you compare it to ChatGPT, do you mean GPT3.5 or GPT4? I also would guess that certain areas of law (especially criminal law) may be prone to triggering “safeguards” which result in poorer performance than if those safeguards were absent. Arguing that what your client did was legal (even be it ethically unsavoury) is an essential part of a lawyer’s job

qup 3y ago

I agree, but it does change the shape of the work you would be doing. Validating work isn't the same as generating new work. Whether that's better, or useful, probably differs by situation.

blibble 3y ago
> Validating work isn't the same as generating new work. in many cases it's way harder

cloverich 3y ago

> you still need to give it just as much attention as if you'd done it yourself. It's different than a Lawyers case, where the facts require manually cross referencing. In our case, that verification can come via directly and immediately running the code. How good the actual code is varies, but the fundamental difference remains.

skissane 3y ago

> Funny you mention that, since lawyers have already gotten in trouble for citing fictional cases when submitting work performed by chatgpt. For something like legal work, you don’t want a raw LLM. You want an agent integrated with a legal database, in such a way that it can’t cite cases which don’t exist in the database. And it can’t generate a direct quote from a case unless that text actually occurs in the case. You still need a human lawyer familiar with the law to pick up on subtler errors, but better technology can prevent the grosser ones. And the subtler errors (e.g. misrepresenting what a case says through partial or out of context quoting) is the kind of error human lawyers sometimes make too - like every other profession, lawyers vary greatly in their competence, and often only the grosser cases of incompetence incur sanctions

zmgsabst 3y ago
Tried Lexis+ AI, and… it’s just not very good yet. Much like ChatGPT, it can only handle recall and summation — and even then, I don’t fully trust it because it often misses key ideas. And much like ChatGPT, it can’t do anything coherent at length without a lot of help and working around its faults. And seems entirely unaware of similar sounding words having distinct legal meanings. Which is not good.
- skissane 3y ago
  > Tried Lexis+ AI, and… it’s just not very good yet. I wonder to what extent that’s due to current inherent limitations of the technology, and to what extent it is due to quality of implementation issues. It is hard to say because (I presume) there is only limited public information on how it is actually implemented. e.g. which LLM is it using? How much fine-tuning has been done? Are they using other potentially helpful techniques such as guided sampling? Or breaking down the task into parts and having multiple agents each specialised to handle one particular part? > Much like ChatGPT When you compare it to ChatGPT, do you mean GPT3.5 or GPT4? I also would guess that certain areas of law (especially criminal law) may be prone to triggering “safeguards” which result in poorer performance than if those safeguards were absent. Arguing that what your client did was legal (even be it ethically unsavoury) is an essential part of a lawyer’s job