4 ms·

According to a friend who is a senior partner at a major class action firm many law firms are heavily relying on GPT4 (including his own). However in his firm i

According to a friend who is a senior partner at a major class action firm many law firms are heavily relying on GPT4 (including his own). However in his firm it’s used to generate drafts and summarize large boiler plate documents. Humans always meticulously review and revise the drafts and the summaries done by humans are so likely to be flawed as documents grow just due to signal to noise ratios that the GPT4 analysis is more accurate.

gpderetta 3y ago
> Humans always meticulously review and revise the drafts The risk of course is that they might get complacent, trust the output too much, and errors will start slipping in. Especially if the error rate is lower than human and the throughput is higher than they can effectively review. But this is not necessarily an LLM issue per-se.
- 6510 3y ago
  If humans stop doing it they will get worse at it. A race to the bottom if you like.
  - loa_in_ 3y ago
    Most mistakes come from exhaustion specific to the task, at least in my experience. Also that reading mostly correct material leads to errors being spotted more easily, again in my personal experience.
  - AlecSchueler 3y ago
    Did we get worse at maths by using computers to do our computation?
    - andersa 3y ago
      Worse at doing calculations manually? Absolutely. How many people have done a long division on paper in the last 10 years? Could remember immediately how to do it, not having done it since school? But does that matter, if it lets us get better at applying it to larger and larger problems?
      - 6510 3y ago
        I did one for an arithmetic function some months ago because I wasn't going to use a math lib. I had to watch no less than 3 children's youtube videos to remember how to do it. Not sure if remember is still the right word, I didn't do it since I learn it in school.
    - tivert 3y ago
      > Did we get worse at maths by using computers to do our computation? We did as a matter of fact. If you could get in a time machine, I'm sure people 50 years ago were better at mental arithmetic that current-day people. They got a lot more practice, since they had not other choice. Today you can probably live your life without even knowing how to add or subtract. Just stick your card in the slot and let the computer do everything.
    - fnordpiglet 3y ago
      We are worse at rote arithmetic but better at math. More people complete calculus and beyond now than ever in history. The major inflection appears to be in the late 90’s and early 00’s after the SAT allowed calculators and presumably schools became more permissive over time. https://maa.org/sites/default/files/pdf/cspcc/InsightsandRecommendations.pdf https://maa.org/sites/default/files/pdf/cspcc/InsightsandRec... I often hear people bemoan the loss of arithmetic skill. I will never understand that. Beyond very basic arithmetic for accelerating daily life or saving time from reaching for a calculator it has almost no actual value in life and knowledge of the mechanical shortcuts taught in school teach you absolutely NOTHING about math - like actual math, not extremely limited and specialized edge case of grade school arithmetic.
- fnordpiglet 3y ago
  It’s not just about errors. They have a lot more context than the LLM has, so it’s mostly about building the scaffold of boilerplate and adding in the specific detail of their points to the court. And I think every competent lawyer has heard the news of the complacent lawyers who got called out. He specifically said they don’t use the LLMs as a source of information on case law - they have powerful search tools already and they are skilled lawyers. He sees the LLMs as a powerful tool in their tool box, not as a replacement for junior attorneys. Of course they’re a top firm and there’s a huge continuum in quality and diligence. But that existed before LLMs.

gpderetta 3y ago

> Humans always meticulously review and revise the drafts The risk of course is that they might get complacent, trust the output too much, and errors will start slipping in. Especially if the error rate is lower than human and the throughput is higher than they can effectively review. But this is not necessarily an LLM issue per-se.

6510 3y ago
If humans stop doing it they will get worse at it. A race to the bottom if you like.
- loa_in_ 3y ago
  Most mistakes come from exhaustion specific to the task, at least in my experience. Also that reading mostly correct material leads to errors being spotted more easily, again in my personal experience.
- AlecSchueler 3y ago
  Did we get worse at maths by using computers to do our computation?
  - andersa 3y ago
    Worse at doing calculations manually? Absolutely. How many people have done a long division on paper in the last 10 years? Could remember immediately how to do it, not having done it since school? But does that matter, if it lets us get better at applying it to larger and larger problems?
    - 6510 3y ago
      I did one for an arithmetic function some months ago because I wasn't going to use a math lib. I had to watch no less than 3 children's youtube videos to remember how to do it. Not sure if remember is still the right word, I didn't do it since I learn it in school.
  - tivert 3y ago
    > Did we get worse at maths by using computers to do our computation? We did as a matter of fact. If you could get in a time machine, I'm sure people 50 years ago were better at mental arithmetic that current-day people. They got a lot more practice, since they had not other choice. Today you can probably live your life without even knowing how to add or subtract. Just stick your card in the slot and let the computer do everything.
  - fnordpiglet 3y ago
    We are worse at rote arithmetic but better at math. More people complete calculus and beyond now than ever in history. The major inflection appears to be in the late 90’s and early 00’s after the SAT allowed calculators and presumably schools became more permissive over time. https://maa.org/sites/default/files/pdf/cspcc/InsightsandRecommendations.pdf https://maa.org/sites/default/files/pdf/cspcc/InsightsandRec... I often hear people bemoan the loss of arithmetic skill. I will never understand that. Beyond very basic arithmetic for accelerating daily life or saving time from reaching for a calculator it has almost no actual value in life and knowledge of the mechanical shortcuts taught in school teach you absolutely NOTHING about math - like actual math, not extremely limited and specialized edge case of grade school arithmetic.
fnordpiglet 3y ago
It’s not just about errors. They have a lot more context than the LLM has, so it’s mostly about building the scaffold of boilerplate and adding in the specific detail of their points to the court. And I think every competent lawyer has heard the news of the complacent lawyers who got called out. He specifically said they don’t use the LLMs as a source of information on case law - they have powerful search tools already and they are skilled lawyers. He sees the LLMs as a powerful tool in their tool box, not as a replacement for junior attorneys. Of course they’re a top firm and there’s a huge continuum in quality and diligence. But that existed before LLMs.