3 ms·

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, dra

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

streetfighter64 4mo ago
> analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more The same could be said about programming. Or if you want to be even more reductive, looking at a screen and pressing buttons to make the correct lights light up https://xkcd.com/722/ https://xkcd.com/722/
- aristofun 4mo ago
  Philosophically or metaphorically speaking - yes. But in my comment it is literally what some subset of lawyers do. Literally is much more tangible and risky in terms of real impact on employment etc.
- glitcher 4mo ago
  Oh wow, did Randall Munroe inadvertently predict the employee workload in the show Severance? :)
conartist6 4mo ago
I see the same problem with AI in both programming and law though. AI is like a scab on a wound: it's a temporary filler, it rushes in to fill a void, but it's not going to be the final solution. Models showed us that there was huuuge unmet demand for literacy, both in software and in law. But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab.
- bluefirebrand 4mo ago
  > But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab. Yeah, but in my experience it won't come down to "which is the better solution" but "which is cheaper/easier" So I look forward to lots of layers of papered over AI scabs in the future. It won't be cheaper in the long run, but it will pump someone's quarterly numbers enough that they get a promotion before the problem they introduce come back to them
epolanski 4mo ago
The more I see the evolution, the more it looks to me that any knowledge workers is going to be impacted.
NoboruWataya 4mo ago
These are academics. Not to disparage them or their work at all but it is very different to the transaction or litigation work that is done in BigLaw. It is a lot more focused on analysing and summarising existing texts, which are themselves more easily available for LLMs to train on (statutes, case law, legal journals, textbooks). As such it is probably the easiest legal work to LLM-ify but also the least valuable, because I assume law professors aren't getting paid nearly as much as BigLaw lawyers. So this approach won't scale. Not to say AI won't crack BigLaw but it will be a different challenge.
scotty79 4mo ago
LLMs answered student questions of the top of their heads, without any refresher look into the case law. And systems that were primied with the case law like NotebookLM underperformed when compared to baseline LLMs that you'd as anything about anything. It's not about what LLMs can or are suited to do. This study shows strengths of what's already in them, innately.
nickburns 4mo ago
'Bread *and butter'. The English expression requires the second part—but otherwise fits perfectly in your well-stated point, with which I wholeheartedly agree. Source: AAL.
- deleted 4mo ago
  [deleted]
- aristofun 4mo ago
  Thank you! As a non native speaker I was not sure if “and butter” is a mandatory part but didn’t want (nor had time) to llm the comment for the sake of authenticity :) TIL
  - nickburns 4mo ago
    Ah, great! I 'took a shot in the dark' there. Glad it 'panned out'. ;)
alansaber 4mo ago
"That is literally the bread of LLMs." correct. However, programming has a large number of advantages RE LLM use compare to law: You can execute the logic, and set up loops from the output. You can set up more useful RL. It's easier to generate synthetic training data. It naturally supports tool use and agent parallelism. It's easier to integrate with APIs (with what few APIs the court systems provide). Programming explicitly encodes abstractions at the function, module levels etc that are easier to KG/reason/build upon than text chunks.
iterance 4mo ago
Just because it is theoretically the bread and butter of LLMs does not mean LLMs are capable of doing the job. It still needs to be proven, setting prior beliefs aside. Law is a life-critical system and deserves our highest level of scrutiny.

streetfighter64 4mo ago

> analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more The same could be said about programming. Or if you want to be even more reductive, looking at a screen and pressing buttons to make the correct lights light up https://xkcd.com/722/ https://xkcd.com/722/

aristofun 4mo ago
Philosophically or metaphorically speaking - yes. But in my comment it is literally what some subset of lawyers do. Literally is much more tangible and risky in terms of real impact on employment etc.
glitcher 4mo ago
Oh wow, did Randall Munroe inadvertently predict the employee workload in the show Severance? :)

conartist6 4mo ago

I see the same problem with AI in both programming and law though. AI is like a scab on a wound: it's a temporary filler, it rushes in to fill a void, but it's not going to be the final solution. Models showed us that there was huuuge unmet demand for literacy, both in software and in law. But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab.

bluefirebrand 4mo ago
> But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab. Yeah, but in my experience it won't come down to "which is the better solution" but "which is cheaper/easier" So I look forward to lots of layers of papered over AI scabs in the future. It won't be cheaper in the long run, but it will pump someone's quarterly numbers enough that they get a promotion before the problem they introduce come back to them

epolanski 4mo ago

The more I see the evolution, the more it looks to me that any knowledge workers is going to be impacted.

NoboruWataya 4mo ago

These are academics. Not to disparage them or their work at all but it is very different to the transaction or litigation work that is done in BigLaw. It is a lot more focused on analysing and summarising existing texts, which are themselves more easily available for LLMs to train on (statutes, case law, legal journals, textbooks). As such it is probably the easiest legal work to LLM-ify but also the least valuable, because I assume law professors aren't getting paid nearly as much as BigLaw lawyers. So this approach won't scale. Not to say AI won't crack BigLaw but it will be a different challenge.

scotty79 4mo ago

LLMs answered student questions of the top of their heads, without any refresher look into the case law. And systems that were primied with the case law like NotebookLM underperformed when compared to baseline LLMs that you'd as anything about anything. It's not about what LLMs can or are suited to do. This study shows strengths of what's already in them, innately.

nickburns 4mo ago

'Bread *and butter'. The English expression requires the second part—but otherwise fits perfectly in your well-stated point, with which I wholeheartedly agree. Source: AAL.

deleted 4mo ago
[deleted]
aristofun 4mo ago
Thank you! As a non native speaker I was not sure if “and butter” is a mandatory part but didn’t want (nor had time) to llm the comment for the sake of authenticity :) TIL
- nickburns 4mo ago
  Ah, great! I 'took a shot in the dark' there. Glad it 'panned out'. ;)

alansaber 4mo ago

"That is literally the bread of LLMs." correct. However, programming has a large number of advantages RE LLM use compare to law: You can execute the logic, and set up loops from the output. You can set up more useful RL. It's easier to generate synthetic training data. It naturally supports tool use and agent parallelism. It's easier to integrate with APIs (with what few APIs the court systems provide). Programming explicitly encodes abstractions at the function, module levels etc that are easier to KG/reason/build upon than text chunks.

iterance 4mo ago

Just because it is theoretically the bread and butter of LLMs does not mean LLMs are capable of doing the job. It still needs to be proven, setting prior beliefs aside. Law is a life-critical system and deserves our highest level of scrutiny.