16 ms·

Better Call GPT: Comparing large language models against lawyers [pdf]

389 points by vinnyglennon 3y ago

crakenzak 3y ago
This is one of the domains I'm very very excited about for LLMs to help me with. In 5-10 years (even though this research paper makes me feel its already here), I would feel very confident chatting for a few hours with a "lawyer" LLM that has access to all my relevant taxes/medical/insurance/marriage documents and would be able to give me specialized advice and information without billing me $500 an hour. A wave of (better) legally informed common-person is coming, and I couldn't be more excited!
- OldOneEye 3y ago
  I wouldn't blindly trust what the LLM says, but I take it that it would be mostly right, and that would give me at the very least explorable vocabulary that I can expand on my own, or keep grilling it about. I've already used some LLMs to ask questions about licenses and legal consequences for software related matters, and it gave me a base, without having to involve a very expensive professional into it for what are mostly questions for hobby things I'm doing. If there was a significant amount of money involved in the decision, though, I will of course use the services of a professional. These are the kinds of topics you can't be "mostly right".
  - nprateem 3y ago
    The only problems are it could be convincingly wrong about anything it tells you and isn't liable for its mistakes.
    - sonofaragorn 3y ago
      What if they were liable? Say the company that offers the LLM lawyer is liable. Would that make this feasible? In terms of being convincingly wrong, it's not like lawyers never make mistakes...
      - giantg2 3y ago
        "What if they were liable?" They'd be sued out of existence. "In terms of being convincingly wrong, it's not like lawyers never make mistakes..." They have malpractice insurance, they can potentially defend their position if later sued, and most importantly they have the benefit of appeal to authority image/perception.
        AnimalMuppet 3y ago
        All right, what if legal GPTs had to carry malpractice insurance? Either they give good advice, or the insurance rates will drive them out of business. I guess you'd have to have some way of knowing that the "malpractice insurance ID" that the GPT gave you at the start of the session was in fact valid, and with an insurance company that had the resources to actually cover if needed...
        kulikalov 3y ago
        It's funny how any conversation ends with this question unanswered.
        YetAnotherNick 3y ago
        Weirdly HN is full of anti AI people who just refuses to discuss the point that is being discussed and goes into all the same argument of wrong answer that they got some time. And then they present anecdotal evidence as truth, while there is no clear evidence if AI lawyer has more or less chance to be wrong than human. Surely AI could remember more and has been shown to clear bar examination.
        giantg2 3y ago
        "while there is no clear evidence if AI lawyer has more or less chance to be wrong than human." In the tests they are shown to be pretty close. The point I made wasn't about more mistakes, but about other factors influencing liability and how it would be worse for AI than humans at this point.
    - engineer_22 3y ago
      This is an area for further development and thought... If a LLM can pass the bar, and has a corpus of legal work instantly accessible, what prevents the deployment of the LLM (or other AI structure) to provide legitimate legal services? If the AI is providing legal services, how do we assign responsibility for the work (to the AI, or to its owner)? How to insure the work for Errors and Omissions? More practically, if willing to take on responsibility for yourself, is the use of AI going to save you money?
      - AnimalMuppet 3y ago
        A human that screws up either too often or too spectacularly can be disbarred, even if they passed the bar. They can also be sued. If a GPT screws up, it could in theory be disbarred. But you can't sue it for damages, and you can't tell whether the same model under a different name is the next legal GPT you consult.
        engineer_22 3y ago
        Agreed - which is why this is an area that needs more thought and development
      - nprateem 3y ago
        Re your first point: it's not conscious. It has no understanding. It's perfectly possible the model could successfully answer an exam question but fail to reach the same or similar conclusion when it has to reason it's own way there based on information provided.
        mvdtnz 3y ago
        Careful, there are plenty of True Believers on this website who really think that these "guess the next word" machines really do have consciousness and understanding.
        lewhoo 3y ago
        I incline towards you on the subject but if you call it guessing you open yourself up to all sorts of rebuttals.
    - boplicity 3y ago
      The obvious intermediate step is that you add an actual expert into the workflow, in terms of using LLMs for this purpose. Basically, add a "validate" step. So, you'd first chat with the LLM, create conclusions, then vet those conclusions with an expert specially trained to be skeptical of LLM generated content. I would be shocked if there aren't law agencies that aren't already doing something exactly like this.
      - freejazz 3y ago
        Ah, so have the lawyer do everything the GPT did so the lawyer can be sure the GPT didn't fuck up.
    - deleted 3y ago
      [deleted]
  - thallium205 3y ago
    I wouldn't blindly trust what a lawyer says either so there's no difference there.
    - brk 3y ago
      Sure, but you have a lot less personal risk following advice from a lawyer vs. advice from an LLM.
      - toomuchtodo 3y ago
        When your GPT is wrong, you will be laughed out of the room and sanctioned. When your attorney is wrong, you get to point at the attorney and show a good faith effort was made. Hacks are fun, just keep in mind the domain you're operating in.
        giantg2 3y ago
        "When your attorney is wrong, you get to point at the attorney and show a good faith effort was made." And possibly sue their insurance to correct their mistakes.
        deleted 3y ago
        [deleted]
        Scoundreller 3y ago
        But you’ll have to find a lawyer that specializes in suing lawyers and their own malpractice plans. Maybe that’s where legal AI will find the most demand.
        kulikalov 3y ago
        Can't a tech firm running a "legal gpt" have an insurance?
        toomuchtodo 3y ago
        Do they have a license to practice law?
  - mannykannot 3y ago
    I like the term "explorable vocabulary." I can see using LLMs to get an idea of what the relevant issues are before I approach a professional, without assuming that any particular claim in the model's responses is correct.
  - chaxor 3y ago
    I don't understand how everyone keeps making this mistake over and over. They explicitly just said "in 5-10 years". So many people continually use arguments that revolve around 'I used it once and it wasn't the best and/or me things up', and imply that this will always be the case. There are many solutions already for knowledge editing, there are many solutions for improving performance, and there will very likely continue to be many improvements across the board for this. It took ~5 years from when people in the NLP literature noticed BERT and knew the powerful applications that were coming, until the public at large was aware of the developments via ChatGPT. It may take another 5 before the public sees the developments happening now in the literature hit something in a companies web UI.
    - mjr00 3y ago
      on the other hand the rate of change isn't constant and there isn't a guarantee that the incredible progress in the past ~2 years in the LLM/diffusion/"AI" space will continue. As an example, take computer gaming graphics; compare the evolution between Wolfenstein 3D (1992) and Quake 3 Arena (1999), which is an absolute quantum leap. Now compare Resident Evil 7 (2017) and Alan Wake 2 (2023) and it's an improvement but nowhere near the same scale. We've already seen a fair bit of stagnation in the past year as ChatGPT gets progressively worse as the company is more focusing on neutering results to limit its exposure to legal liability.
      - GaggiX 3y ago
        >ChatGPT gets progressively worse https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar..., In blinded human comparisons, newer models perform better than older ones.
        mvdtnz 3y ago
        That website doesn't load for me but anyone who uses ChatGPT semi regularly can see that it's getting steadily worse if you ever ask for anything that begins to border risque. It has even refused to provide me with things like bolt torque specs because of risk.
  - coffeebeqn 3y ago
    I wonder could GPTs come up with legal loopholes. Like they are expected to come up with security vulnerabilities
- XCSme 3y ago
  Or a LLM that helps you spend less. Imagine a LLM that goes over all your spending, knows all the current laws, benefits, organizations, promotional campaigns, and suggests (or even executes) things like changing electricity provider, insurance provider, buying stuff in bulk from a different shop that you get for 4x the price at your local store, etc.
  - OldOneEye 3y ago
    I love this idea. It would be incredibly useful! I feel LLMs are great at suggestions that you follow up yourself (if only for sanity checking, but nothing you wouldn't do with a human too).
  - p-e-w 3y ago
    That would not be in the interest of anyone with any real power, so you're going to see tanks on the streets before it happens.
- Solvency 3y ago
  This does not make sense to me. ChatGPT is completely nerfed to the point where it's either been conditioned or trained to provide absolutely zero concrete responses to anything. All it does is provide the most baseline, generic possible response followed by some throwaway recommendation to seek the advice of an actual expert.
  - frankfrank13 3y ago
    The way to get around this is to have it "quote" or at least try to quote from input documents. Which is why RAG became so popular. Sure, it won't right you a contract, but it will read one back to you if you've provided one in your prompt. In my experience, this does not get you close to what the top-level comment is describing. But it gets around the "nerfing" you describe
  - baobabKoodaa 3y ago
    It's very easy to get ChatGPT to provide legal advice based on information fed in the prompt. OpenAI is not censoring legal advice anywhere near as hard as they are censoring politics or naughty talk.
    - freejazz 3y ago
      That's because its just advice, not legal advice. Legal advice is something you get from an attorney that represents you. It creates a liability relationship between you and the attorney for the legal advice they did provide.
      - baobabKoodaa 3y ago
        Sure, we can call it "advice" instead of "legal advice" or we can even call it other names, like "potato", if that's what you want. My point is that potato not censored.
        freejazz 3y ago
        Feel free to miss the point as much as you want. You can call it a baked potato then!
- frankfrank13 3y ago
  I think a lot of startups are working on exactly what you are describing, and honestly, I wouldn't hold my breath. Everyone is still bound by token limits and the two best approaches for getting around them are RAG and Knowledge-Graphs, both of which could get you close to what you describe but not close enough to be useful (IMO).
- baobabKoodaa 3y ago
  We are literally building this today! Our core business is legal document generation (rule based logic, no AI). Since we already have the users' legal documents available to us as a result of our core business, we are perfectly positioned to build supplementary AI chat features related to legal documents. We recently deployed a product recommendation AI to prod (partially rule based, but personalized recommendation texts generated by GPT-4). We are currently building AI chat features to help users understand different legal documents and our services. We're intending to replace the first level of customer support with this AI chat (and before you get upset, know that the first level of customer support is currently a very bad rule-based AI). Main website in Finnish: https://aatos.app https://aatos.app (also some services for SE and DK, plus we recently opened UK with just a e-sign service)
  - baobabKoodaa 3y ago
    Here's an example of what our product recommendations look like: Given your ownership in a company and real estate, a lasting power of attorney is a prudent step. This allows you to appoint PARTNER_NAME or another trusted individual to manage your business and property affairs in the event of incapacitation. Additionally, it can also provide tax benefits by allowing tax-free gifts to your children, helping to avoid unnecessary inheritance taxes and secure the financial future of your large family.
  - Closi 3y ago
    > Since we already have the users' legal documents available to us as a result of our core business, we are perfectly positioned to build supplementary AI chat features related to legal documents. Uhh... What are the privacy implications here?!
    - baobabKoodaa 3y ago
      If you look at the example I posted of our product recommendations, you will see that the GPT-4 generated text contains "PARTNER_NAME" instead of actual partner name. That's because we've created anonymized dataset from users in such a way that it's literally impossible for OpenAI to deanonymize users. Of course the same can not be done if we want to provide a service where users can, for example, chat with their legal documents. In that case we will have to send some private details to OpenAI. Not sure how that will pan out (what details we decide to send and what we decide not to send). In any case, all startups today are created on top of a mountain of cloud services. Any one of those services can leak private user data as a result of outsider hack or insider attack or accident. OpenAI is just one more cloud service on top of the mountain.
- nostromo 3y ago
  And not just legal either. I uploaded all of my bloodwork tests and my 23andme data to Chat GPT and it was better at analyzing it than my doctor was.
  - slingnow 3y ago
    This is a really interesting use case for me. I've been envisioning a specially trained LLM that can give useful advice or insights that your average PCP might gloss over or not have the time to investigate. Did you do anything special to achieve this? What were the results like?
  - p-e-w 3y ago
    Yup. Because the doctor doesn't have time and doesn't give a fuck. LLMs don't have to compete against the cutting edge of human professional knowledge. They only have to compete against the disinterested, arrogant, greedy, and overworked professionals that are actually available to people in practice. No wonder they're winning.
throwaway17_17 3y ago
I will reserve judgment of the possibilities of LLMs as applied to the legal field until they are tested on something other than Document/ contract review. Contract review is, in the large business law case, often done by outsourcing to hundreds of recent graduates and act more like proof reading with minimal application of actual lawyering skills to increase a corporation’s bottom line. The more common, for individual purchasers of legal services, lawyering is going to be family law matters, criminal law matters, and small claims court matters. I can not see a time in the near future where an LLM can handle the fact specific and circumstantial analysis required to handle felony criminal litigation, and I see nothing that would imply LLMs can even approach the individualized, case specific and convoluted family dynamics required for custody cases or contested divorces. I’m not unwilling to accept LLMs as a tool an attorney can use, but outside of more rote legal proof reading I don’t think the technology is at all ready for adoption in actual practice.
- giantg2 3y ago
  "and I see nothing that would imply LLMs can even approach the individualized, case specific and convoluted family dynamics required for custody cases or contested divorces." Humans are pretty bad at this. Based on the results, it seems the judges' personal views and emotions are a large part of these cases. I'm not sure what they would look like without emotion, personal views, and the case law built off of those.
  - SkyBelow 3y ago
    The worse judges are at being perfectly removed arbiters of justice, the more room for lawyers to exploit things like emotions and humans connections with those judges, and thus the worse LLMs will be at doing that part of the job. A charismatic lawyer backed by an LLM will be much better than an LLM. At least until the LLMs surpass humans at being charismatic, but that would seem to be its own nightmare scenario.
    - staunton 3y ago
      > At least until the LLMs surpass humans at being charismatic Look into "virtual influencers". Sounds like you should find it interesting.
Workaccount2 3y ago
I wonder what the reach of a legal argument a bunch of lawyers are going to come up with in order to cripple the tech that threatens their industry?
- delichon 3y ago
  Copyright appears to be the primary attack vector.
  - anotherhue 3y ago
    Are the arguments submitted to a court (and made publicly accessible) subject to copyright? I kind of assumed they were in the same space as government documents.
    - delichon 3y ago
      A legal LLM would be significantly crippled without the knowledge stored in the universe of non-legal documents.
      - anotherhue 3y ago
        You're probably right, but the law and reality often seem to be orthogonal.
- zugi 3y ago
  Lawyers control government, at least in the U.S. Expect laws banning or severely restricting the use of AI in the legal field soon. I expect arguments will range from the dangers of ineffective counsel to "but think of the children" - whatever helps them protect their monopoly.
  - deleted 3y ago
    [deleted]
  - axpy906 3y ago
vitiral 3y ago
It feels to me like the law is already a staggering heap of complexity. Isn't using technology going to just enable more of the same, making the situation worse?
- engineer_22 3y ago
  on the contrary, it may help to highlight incongruities in the legal domain and provide lawyers with compelling groundwork to make relevant claims
  - deleted 3y ago
    [deleted]
- urbandw311er 3y ago
  Or you could take the view that, in fact, this is one of the things LLMs are very good at, ie making sense of complexity.
  - vitiral 3y ago
    But the lawyers reading the law won't be the only ones using LLMs. LLMs will also be used to write laws. Then lawmakers will use them to "check" that their 20,000 page law supposedly works. No human can understand the scope of today's laws: how much less when no LLMs can understand the laws created 20 years from now. I'd love to hear that LLMs can be used to trim and simplify complexity, but I don't believe it. They generate content, not reduce it.
delichon 3y ago
In all criminal prosecutions, the accused shall enjoy the right ... to have the Assistance of Counsel for his defense -- 6th amendment to US Constitution When an LLM is more competent than an average human counsel, does this amendment still require assistance of a human counsel?
- urbandw311er 3y ago
  Ironically you might be better asking GPT this question.
- District5524 3y ago
  All the governments in the world would do everything in their power to get people accept this suggestion as a truth and use LLMs instead of human lawyers especially in criminal defense. Now, why is that? Maybe it's not the technical knowledge that is the most important feature of a lawyer?
  - sudden_dystopia 3y ago
    Yea it’s their ability to manipulate language and people to bend the letter of the law to suit their specific cases regardless of any long term potential societal harm.
  - AustinDev 3y ago
    All governments of the world at least in the west appear to mostly consist of attorney's I doubt they'd let it happen. It'd be bad for their guild.
    - District5524 3y ago
      You are mixing up important things and probably that's why you don't see a bigger threat here than the livelihood of a couple of lawyers. People who attended law school/university/have a law degree are not necessarily practising law as lawyers/attorney or whatever they call them in the given country, as a member of the government. But of course, once a marine, always a marine... On the other hand, the parent was trying to refer to criminal defense (Amendment 6 of US Constitution). That's not something active politicians can participate in (conflict of interest, etc.). If one has the means to pay for their criminal defense (or family law, anything that truly matters for the average citizen), they tend to be willing to pay a lot and expect some quality of service in exchange. They will certainly not choose ChatGPT, just because that provides the answer faster for long documents. But a large number of people don't have this kind of money. As long as governments don't have a proper budget for legal aid, they will be inclined to cover the vanishing legal aid budget with false claims that they fulfill "appointment of counsel" with LLM services - while they don't really care about the outcome of the case. (And of course, successful politicians have rarely worked as pro bono counsels before their current job, not that it has any more relevance here.) Non-Western governments are also interested in having this replacement service - the cost of lawyers is also money there. It's much better for all governments to have chatbots they have reliably trained than less controllable humans. That's a hypothetical threat for now, but I hope you understand why there is something more here beyond "guildthink" and protecting the livelihood of lawyers at all costs behind the Sixth Amendment. There are lots of countries (or even within the US, like in Utah) where you can review contracts without being a practising lawyer, like LPOs mentioned in the paper. So far, it failed to make the world a better place. But replacing including LLMs as "counsel" under Sixth Amendment would mean a very different situation.
deleted 3y ago
[deleted]
hansonkd 3y ago
I run a startup that does legal contract generation (contracts written by lawyers turned into templates) and have done some work GPT analysis of the contract for laypersons to interact and ask questions about the contract they are getting. In terms of contract review, what I've found is that GPT is better at analysis of the document than generating the document, which is what this paper supports. However, I have used several startups options of AI document review and they all fall apart with any sort of prodding for specific answers. This paper looks like it just had to locate the section not necessarily have the back and forth conversation about the contract that a lawyer and client would have. There is also no legal liability for GPT for giving the wrong answer. So It works well for someone smart who is doing their own research. Just like if you are smart you could use google before to do your own research. My feelings on contract generation is that for the majority of cases, people are better served if there were simply better boilerplate contracts available. Laywers hoard their contracts and it was very difficult in our journey to find lawyers who would be willing to write contracts we would turn into templates because they are essentially putting themselves and their professional community out of income streams in the future. But people don't need a unique contract generated on the fly from GPT every time when a template of a well written and well reviewed contract does just fine. It cost hundreds of millions to train GPT4. If $10m was just spent building a repository of well reviewed contracts, it would be a more useful than spending the equivalent money training a GPT to generate them. People ask pretty wide range of questions about what they want to do with their documents and GPT didn't do a great job with it, so for the near future, it looks like lawyers still have a job.
- OldOneEye 3y ago
  Which is mostly what I feel also happens with LLMs producing code. Useful to start with, but not more than that. We've still got a job us programmers. For the moment.
  - klabb3 3y ago
    Producing code is like producing syntactically correct algebra. It has very little value on its own. I’ve been trying to pair system design with ChatGPT and it feels just like talking with a person who’s confident and regurgitates trivia, but doesn’t really understand. No sense of self-contradiction, doubt, curiosity. I’m very, very impressed with the language abilities and the regurgitation can be handy, but is there a single novel discovery by LLMs? Even a (semantic) simplification of a complicated theory would be valuable.
deleted 3y ago
[deleted]
williamcotton 3y ago
Lexis has some AI feature built into it: https://www.lexisnexis.com/en-us/products/lexis-plus-ai.page https://www.lexisnexis.com/en-us/products/lexis-plus-ai.page I haven't had a chance to test it out as anyone should be a bit weary to add more paid features to an already insanely expensive software product!
FrustratedMonky 3y ago
Theoretically. The language in laws should be structured similar to code. It has some logical structure. Thus should be more easily adopted to LLMs than other 'natural language'. So despite the early news about lawyers submitting 'fake' cases, it is only a matter of time before the legal profession is up-ended. There are a ton of paralegals, etc.. doing 'grunt' work in firms, that an LLM can do. These are considered white color, and will be gone. It will progress in a similar fashion to coding. It will be like having a junior partner that you have to double check, or that can do some boiler plate for you. You can't trust completely, but you don't trust your junior devs do you, but it gets you 80% there.
- NoboruWataya 3y ago
  > It will progress in a similar fashion to coding. I kind of agree with this, but this is why I am confused that I only ever see people (at least on HN) talk about AI up-ending the legal profession and putting droves of lawyers out of work--I never see the same talk about the coding industry being transformed in this way.
  - FrustratedMonky 3y ago
    I've heard it discussed. A lot more a few months ago when GPT first blew up. Maybe HN is full of coders that still think themselves 'special' and can't be replaced. Or maybe, the law profession has a lot more boilerplate than the coding profession? So legal profession has more that can be replaced? Coders will be replaced, but maybe not at same rate as paralegals.
ulrischa 3y ago
Lawyers have been resisting technological advances for years. No industry rejects technological tools as vehemently as the legal industry, arguing that everything has to be judged by people. Even laws that are available online do not even link to the paragraphs that are referenced. All in all, progress is institutionally suppressed here in order to preserve jobs.
- NoboruWataya 3y ago
  > Even laws that are available online do not even link to the paragraphs that are referenced. It's not lawyers' job to publish the laws online. Lawyers are the ones who would benefit from more easily searchable online laws, as they are the ones whose job is actually to read the laws. That is why there are various commercial tools that provide this functionality, that lawyers pay for. You need to ask your government why public online legal databases are so poor, not your lawyer.
  - ulrischa 3y ago
    Right. 70% of governmental staff are people with a law education. So I mean lawyer in a broader sense
- DanielSantos 3y ago
  I also built an AI contract review ai tool and talked to > 100 lawyers. What I found is that lawyers want technological advances but only if they work 100% of the time. Also helped lawyers looking for a CLM, and they rejected something if it caused any inconvenience.
light_hue_1 3y ago
Talk about a conflict of interest. A company that pushes llms for legal work says they work better. This isn't worth the pdf it wasn't printed on.
- carstenhag 3y ago
  I disagree. The company is mentioned multiple times, a conflict of interest is clearly visible. We also don't complain about Google et al publishing papers about some of their internal systems and why it helps, I hope?
  - drewdevault 3y ago
    Google isn't trying to sell you their internal systems. This is a bullshit AI hype bubble advert masquerading as an academic paper. Bet you 10 bucks it doesn't get through peer review (or isn't even submitted for peer review).
    - famouswaffles 3y ago
      >Google isn't trying to sell you their internal systems. They sometimes are. If you have an issue with the methodology of paper then all well and good but "conflict of interest" is pretty weak. Yes, Google and Microsoft et al regularly publish papers describing Sota performance they sometimes use internally and even sell. I didn't have to think much before wavenet came to mind. Besides, the best performing models here are all Open AI.
    - carstenhag 3y ago
      As some internal systems will be made available as a product, they do sell it (maybe).
- vibeproaaaac21 3y ago
  Agreed. That's most AI research though. They are a mechanism whose entire value proposition lies in laundering externalities. Not that this isn't exactly what all the big "tech innovation" of the last decade were either. It's depressing and everyone involved should be ashamed of themselves.
oldgregg 3y ago
Interesting problem space-- I have a culture jamming legal theory this might work for: What if you had a $99/mo pro-se legal service that does two things, 1) teaches you how to move all of your assets into secure vehicles. 2) At the same time it lets you conduct your own defense pro-se-- but the point is not to win, it's just to jam the system. If you signal to the opposing party that you're legally bankrupt and then you just file motion after motion and make it as excruciating as possible for them they might just say nevermind when they realize it's gonna take them 5 years to get through appeals process. It's true lawyers don't want to give up their legal documents for a template service-- but honestly just going to the court house and ingesting tons of filings might do the trick. With that strategy in mind you don't really need GREAT documents or legal theory anyway. Just docs that comply with court filing requirements. Yeah we're def gonna need to deposition your housekeepers daughters at $400/h and if you have a problem with that I would be happy to have a hearing about it. If enough people did this is would basically bring the legal system to a standstill and give power back to the people. RIP Aaron Swartz who died fighting for these issues :'(
- gee_totes 3y ago
  What you're describing in 2) sounds a lot like paper terrorism[0] [0]https://en.wikipedia.org/wiki/Paper_terrorism https://en.wikipedia.org/wiki/Paper_terrorism
  - oldgregg 3y ago
    So what do you call it when the wealthy and corporations exploit their opponent's inability to afford legal representation? A normal Tuesday in Amerika. Yes, the banality of tuesday terrorism.
    - akira2501 3y ago
      > So what do you call it when the wealthy and corporations exploit their opponent's inability to afford legal representation? It's called a Strategic Lawsuit Against Public Participation. It's also illegal in most states. > A normal Tuesday in Amerika. We already understood your derogatory outlook without it needing to be literal. Anyways, that's why the founders gave us the First, Fourth, and Fifth amendments. It turns out, they recognized, the "law" is literally the worst mechanism for discovering truth and managing outcomes. > Yes, the banality of tuesday terrorism. It's terrorism because it has no logical conclusion nor any ability to positively benefit anyone's life, in particular, the person who would wield it. Your equivocation ignores this.
zehaeva 3y ago
Given the recent legal cases where lawyers did use Chat GPT to do research and help write their brief did not go very well I'm not sold that on all the optimism that's here in the comments.
- minimaxir 3y ago
  The technology is fine, the education and literacy about its flaws and limitations among typical nontechnical users is another story.
- frankfrank13 3y ago
  That was rookie level mistakes though. Not checking a *case* exists? Building a small pipeline of generation->validation isn't trivial, but it isn't impossible. The cases you describe seem to me like very lazy associates matched with a very poor understanding of what LLMs do.
- bpodgursky 3y ago
  They were all idiots too cheap to pay for GPT-4. Got caught by hallucinations.
cwoolfe 3y ago
Where can someone upload a contract and ask the AI questions about it in a secure and private way? It's my understanding that most people and organizations aren't able to do this because it isn't private.
- btbuildem 3y ago
  You can host your own LLM - something like Mixtral for example - then you have full control over the information you submit to it.
- kveton 3y ago
  We do this today (securely upload a file and ask questions or summarize) and part of our promise, and why we're having early success, is because we promise not to train with customer data and we don't run directly on top of OpenAI. https://casemark.ai https://casemark.ai
- kulikalov 3y ago
  ChatGPT enterprise? Or over API. They state that those offerings data is not used for training. I'm not a lawyer but afaik illegally retrieved evidence cannot be used - "exclusionary rule".
- DanielSantos 3y ago
  You can try us [1] . During the upload process you can enable data anonymization. It’s not perfect though. We use open ai but they only get segments of a contract. Not the full one and can’t connect them. You get the review via email and after you can delete the document and keep the review. [1] legalreview.ai
mkrishnan 3y ago
[dead]
adastra22 3y ago
Nice title!
advael 3y ago
It's bizarre how easily we got to the Goodhart's Law event horizon in our comparisons of complex fields to AI models But this is what happens when industries spend a decade brain-draining academia with offers researchers would be insane to refuse
District5524 3y ago
While this paper is clearly not without merits, it intends to be more like an excuse to make a bombastic statements about a whole profession or "industry" (perhaps to raise their visibility and try to sell something later on?). The worst part is that they have actually referenced a single preprint document as "previous art" - and that document itself is not related to contract review, but to legal reasoning in general. (A part of LegalBench is of course "interpretation", and that is built on existing contract review benchmarks, but they could've found more relevant papers). Automating legal document review has been a very active field in NLP for twenty years or so (including in QA tasks) and became a lot more active since 2017. At least e.g. Kira (and Luminance etc., none of which is LLM-based) are already used quite widely in legal departments/firms around the world. So lawyers do have practical experience in their limitations... But Kira & co. are not measuring the performance of the latest and greatest models and they do not use transparent benchmarks etc. So the benchmark results in this paper are indeed a welcome addition in terms of using LLMs. But also considering its limited scope of reviewing 10 (!) documents based on a single review playbook, they should not have written about "implications for the industry". It is very much pretentious and shows more of the lack of knowledge of the authors of the very same industry than about the future of the legal services industry. If you're interested in the capabilities and limitations, I suggest these informative, but still light reads as well: https://kirasystems.com/science/ https://kirasystems.com/science/ https://zuva.ai/blog/ https://zuva.ai/blog/ https://www.atticusprojectai.org/cuad https://www.atticusprojectai.org/cuad
- d0odk 3y ago
  Any particular papers you would recommend? The links are to blogs with lots of papers.
  - District5524 3y ago
    The third one (CUAD) is a single paper, not blogposts like the others. I think this paper is still the best in terms of being done by NLP experts and understanding the possibilities and not being just some and mirror. But there are so many papers published in this area nowadays that I might not even notice a new one. The CUAD paper was still based on BERT, so pretraining was needed - that needs a bit more expertise than just prompting GPT-4-32k like in this paper or feeding prompts back to GPT-4 for another round of refining, or doing RAG. For honest research purposes, "contract review" is not really a good area of approach: the subject field is not standardised, there is no good benchmark yet and your paper can easily get into bad company of snake oil sellers cashing on visceral hatred of average people for all professions.
very_good_man 3y ago
We may finally again get affordable access to the rule of law in the United States.
gogogo_allday 3y ago
I must not be reading this paper correctly because it appears that they only used 10 procurement contracts to do this study. If so, the abstract and title feels misleading. I’d be more interested in a study done on thousands of contracts of different types. I also have my doubts it would perform well on novel clauses or contracts.
- District5524 3y ago
  Your reading is correct. Their primary aim was to compare some real lawyers' work with their wonderprompts to show off an upcoming service and how faster that is compared to humans. But they didn't have the funds to ask lawyers to review 15 or 50 or 1000 contracts, just 10, hence the cheat in methods and in conclusions. This is not about drawing up a benchmark for an industry, let alone validating any useful method in general. Ten docs reviewed based a single review 'playbook' (I think it's not in the paper, but probably max 20 questions per contract?) and compared across 3 different providers/roles + LLMs...
gumby 3y ago
I would not want an transformer-generated contract but I would be delighted if a transformer-generated contract were used as input by an actual lawyer and it saved me money. In my experience current practice (unchanged for the decades I've been using lawyers) is that associates start with an existing contract that's pretty similar to what's needed and just update it as necessary. Also in my experience, a contract of any length ends up with overlooked bugs (changed sections II(a) and IV(c) for the new terms but forgot to update IX(h)) and I doubt this would be any better with a machine-generated first draft.
ok123456 3y ago
Does LPO (Legal Process Outsourcing) mean a paralegal?
- District5524 3y ago
  Could be paralegals as well, but they are usually just human employees of legal service providers that are not licensed lawyers/law firms. It's jurisdiction specific who can do this besides lawyers. In the UK and Australia (and probably NZ as well) they can be practically be anyone, contract review is not a reserved legal activity like in most US states.
graphe 3y ago
Law is very specific. BERT was sufficient for law even before chatGPT. https://towardsdatascience.com/lawbert-towards-a-legal-domain-specific-bert-716886522b49 https://towardsdatascience.com/lawbert-towards-a-legal-domai... https://huggingface.co/nlpaueb/legal-bert-base-uncased https://huggingface.co/nlpaueb/legal-bert-base-uncased
adrianmonk 3y ago
I wonder if this could help regular (non-lawyer) people understand legal documents they run into in everyday life. Things like software license agreements, terms of service, privacy policies, release of liability forms, deposit agreements, apartment leases, and rental car agreements. Many people don't even try to read these because they're too long and you wouldn't necessarily understand what it means even if you did read it. What if, before you signed something, you could have an LLM review it, summarize the key points, and flag anything unusual about it compared to similar kinds of documents? That seems better than not reading it at all.
- wow_its_tru 3y ago
  We're building exactly this today, for common business contracts. We're not building for consumers today, because I think it's vanishingly unlikely that you'll, like, pick a different car rental company once you read their contract :) but leases, employment contracts, options agreements, SaaS agreements... all common, all boilerplate with 5-10 areas to focus on, all ready for LLMs!
  - mattmaroon 3y ago
    Honestly, that type of consumer use case might actually be relevant once LLM’s can do this sort of thing. Certainly, nobody is going to contact their attorney before renting a car, but if this could be integrated into a travel site or something… You never know how consumer behavior may change when something that was either impossible or impractical becomes very easy b
  - DanielSantos 3y ago
    We have also have been building this[1] but struggled to monetize even with 100s of users and 1000s of contracts review. We are live for about 1 year. If you want to share experience feel free to reach out [1] legalreview.ai
adi4213 3y ago
For the auditory learners, here is this paper in a summarized Audiobook format : https://player.oration.app/1960399e-ccb0-44f6-81f0-870ef7600766 https://player.oration.app/1960399e-ccb0-44f6-81f0-870ef7600...
visarga 3y ago
Two reasons why it's a bad idea: 1. ChatGPT can't be held responsible, it has no body, like summoning the genie from the lamp, and about as sneaky with its hard to detect errors 2. ChatGPT is not autonomous, not even a little, these models can't recover from error on their own. And their input buffers are limited in size, and don't work all that well when stretched at maximum Especially the autonomy part is hard. Very hard in general. LLMs need to become agents to collect experience and learn, just training on human text is not good enough, it's our experience not theirs. They need to make their own mistakes to learn how to correct themselves.
itissid 3y ago
GPTs only generate specific answers if they are trained using RHLF to prefer certain answers to others. Won't this mean that coming up with a contract that meets a special individual's case will require that much more fine-tuning? Also how do you reconcile several logical arguments without a solver? Like "If all tweedles must tweed", "If X is a tweedle, therefore it must tweed unless it can meet conditions in para 12". How can it possibly learn to solve many such conjunctions that are staple in legal language?
unyttigfjelltol 3y ago
This is apples and bowling balls. You also probably could replace the CEO and entire executive team with LLMs. And cheaper! Much cheaper! But, if the stochastic analysis was ... wrong ... who would be left to correct it?
sandbx 3y ago
I like that they incl the prompt in the paper
pwmiller_ai 3y ago
I lead data and AI at a tech-enabled commercial insurance brokerage. We have been leveraging GPT-4 to build a deep contract analysis tool specifically for insurance requirements. My teams at Google also built several LLM solutions to support Google's legal team, from patent classification to discovery support. Language models are great at digesting legalese and analyzing what's going on. However, many legal applications involve around pretty important decisions that you don't want to get wrong ("am I contractually covered with my current insurance program?"). Because of that, we've built LLM products in the legal space with the following principles in mind: - Human-in-the-loop tooling -- The product should be built around an expert using it whenever possible, so decision support as opposed to automation. You still see massive time savings with that in place - Transparency / citations -- With a human-in-the-loop tool, you need mechanisms to build trust. Whether that's highlighting clauses in the document that the LLM referred to or explaining why a part of the analysis wasn't provided, citing your work is important - Tuned for precision instead of recall -- False positives (and hallucinations) are especially bad in many of these legal use cases, so tuning the model or prompts for precision helps with mitigation.
- theptip 3y ago
  Do you have any pointers on how to get GPT-4 to do citations? Is a prompt like “quote back the passage you are citing” so you can locate and highlight the original? When you say “tuned for precision” is this your prompt engineering or are you actually fine-tuning GPT-4? Appreciate the insights.
  - pwmiller_ai 3y ago
    For RAG applications, starting simply with a reference to the most relevant chunk(s) is helpful in building transparency. A lot of our contract review task is a data extraction one (e.g. extract this type of insurance language and compare against your policy). As such, it's much easier to pinpoint the exact text from the source doc for citation. In our applications, currently, we are doing citations as a postprocessing task as opposed to as part of the prompt itself. Finding that feeding too many instructions to the LLM results in worse responses. We're not fine-tuning today. Instead, "tuning for precision" is done through prompt chains. A simple example would be returning "I don't know" early on if the document isn't a contract or doesn't have clear insurance requirements in it. We've had success with various guardrail prompts. The citation work we did at Google used model internals to highlight text (path integrated gradients). It's also easier to finetune for precision when you have control over the model itself.
ProllyInfamous 3y ago
Just as reminder, Chief US Justice released his "year in review" on acceptance of LLM / GPT systems within legal practices, and that it will likely enable access to the law by more-and-more commoners. The entire six pages is worth the time reading [1]. [1] [PDF] https://www.supremecourt.gov/publicinfo/year-end/2023year-endreport.pdf https://www.supremecourt.gov/publicinfo/year-end/2023year-en...
notfed 3y ago
> Cost wise, LLMs operate at a fraction of the price, offering a staggering 99.97 percent reduction in cost over traditional methods. This seems like a weirdly arbitrary and forced statistic, when "100% reduction" would be just as valid of a statement.
- TeMPOraL 3y ago
  100% reduction would mean "literally free", which it isn't.
  - notfed 3y ago
    But LLMs are literally free. This paper just chose to focus on the "prominent" ones.
underlines 3y ago
Conflict of interest in the paper, as this mainly is a PR piece from Onit: "Onit Announces Generative AI-Powered Virtual Legal Operations Assistant for In-house Counsel"
happytiger 3y ago
Comparing current generation LLMs, which aren't design for accuracy, against lawyers has some pretty obvious problems. The fact is that only one of the two is set to make non-linear improvements, and that is the only point that matters in this debate.
Sofiaaz 3y ago
[flagged]
Vinaas 3y ago
Life can be so cruel sometimes but even worst when you have a cheating partner. I got to discover my spouse was having an affair through the help of { remote spy wise @ gm ail c o m } who gave me access to his device remotely without his notice. I got access to all social media apps, call logs and sms and deleted datas and messages too. Do not be a victim to lies and mischief. Get in contact with him and stay safe.

389 points by vinnyglennon 3y ago

This is one of the domains I'm very very excited about for LLMs to help me with. In 5-10 years (even though this research paper makes me feel its already here), I would feel very confident chatting for a few hours with a "lawyer" LLM that has access to all my relevant taxes/medical/insurance/marriage documents and would be able to give me specialized advice and information without billing me $500 an hour. A wave of (better) legally informed common-person is coming, and I couldn't be more excited!

OldOneEye 3y ago
I wouldn't blindly trust what the LLM says, but I take it that it would be mostly right, and that would give me at the very least explorable vocabulary that I can expand on my own, or keep grilling it about. I've already used some LLMs to ask questions about licenses and legal consequences for software related matters, and it gave me a base, without having to involve a very expensive professional into it for what are mostly questions for hobby things I'm doing. If there was a significant amount of money involved in the decision, though, I will of course use the services of a professional. These are the kinds of topics you can't be "mostly right".
- nprateem 3y ago
  The only problems are it could be convincingly wrong about anything it tells you and isn't liable for its mistakes.
  - sonofaragorn 3y ago
    What if they were liable? Say the company that offers the LLM lawyer is liable. Would that make this feasible? In terms of being convincingly wrong, it's not like lawyers never make mistakes...
    - giantg2 3y ago
      "What if they were liable?" They'd be sued out of existence. "In terms of being convincingly wrong, it's not like lawyers never make mistakes..." They have malpractice insurance, they can potentially defend their position if later sued, and most importantly they have the benefit of appeal to authority image/perception.
      - AnimalMuppet 3y ago
        All right, what if legal GPTs had to carry malpractice insurance? Either they give good advice, or the insurance rates will drive them out of business. I guess you'd have to have some way of knowing that the "malpractice insurance ID" that the GPT gave you at the start of the session was in fact valid, and with an insurance company that had the resources to actually cover if needed...
        kulikalov 3y ago
        It's funny how any conversation ends with this question unanswered.
        YetAnotherNick 3y ago
        Weirdly HN is full of anti AI people who just refuses to discuss the point that is being discussed and goes into all the same argument of wrong answer that they got some time. And then they present anecdotal evidence as truth, while there is no clear evidence if AI lawyer has more or less chance to be wrong than human. Surely AI could remember more and has been shown to clear bar examination.
        giantg2 3y ago
        "while there is no clear evidence if AI lawyer has more or less chance to be wrong than human." In the tests they are shown to be pretty close. The point I made wasn't about more mistakes, but about other factors influencing liability and how it would be worse for AI than humans at this point.
  - engineer_22 3y ago
    This is an area for further development and thought... If a LLM can pass the bar, and has a corpus of legal work instantly accessible, what prevents the deployment of the LLM (or other AI structure) to provide legitimate legal services? If the AI is providing legal services, how do we assign responsibility for the work (to the AI, or to its owner)? How to insure the work for Errors and Omissions? More practically, if willing to take on responsibility for yourself, is the use of AI going to save you money?
    - AnimalMuppet 3y ago
      A human that screws up either too often or too spectacularly can be disbarred, even if they passed the bar. They can also be sued. If a GPT screws up, it could in theory be disbarred. But you can't sue it for damages, and you can't tell whether the same model under a different name is the next legal GPT you consult.
      - engineer_22 3y ago
        Agreed - which is why this is an area that needs more thought and development
    - nprateem 3y ago
      Re your first point: it's not conscious. It has no understanding. It's perfectly possible the model could successfully answer an exam question but fail to reach the same or similar conclusion when it has to reason it's own way there based on information provided.
      - mvdtnz 3y ago
        Careful, there are plenty of True Believers on this website who really think that these "guess the next word" machines really do have consciousness and understanding.
        lewhoo 3y ago
        I incline towards you on the subject but if you call it guessing you open yourself up to all sorts of rebuttals.
  - boplicity 3y ago
    The obvious intermediate step is that you add an actual expert into the workflow, in terms of using LLMs for this purpose. Basically, add a "validate" step. So, you'd first chat with the LLM, create conclusions, then vet those conclusions with an expert specially trained to be skeptical of LLM generated content. I would be shocked if there aren't law agencies that aren't already doing something exactly like this.
    - freejazz 3y ago
      Ah, so have the lawyer do everything the GPT did so the lawyer can be sure the GPT didn't fuck up.
  - deleted 3y ago
    [deleted]
- thallium205 3y ago
  I wouldn't blindly trust what a lawyer says either so there's no difference there.
  - brk 3y ago
    Sure, but you have a lot less personal risk following advice from a lawyer vs. advice from an LLM.
    - toomuchtodo 3y ago
      When your GPT is wrong, you will be laughed out of the room and sanctioned. When your attorney is wrong, you get to point at the attorney and show a good faith effort was made. Hacks are fun, just keep in mind the domain you're operating in.
      - giantg2 3y ago
        "When your attorney is wrong, you get to point at the attorney and show a good faith effort was made." And possibly sue their insurance to correct their mistakes.
        deleted 3y ago
        [deleted]
        Scoundreller 3y ago
        But you’ll have to find a lawyer that specializes in suing lawyers and their own malpractice plans. Maybe that’s where legal AI will find the most demand.
        kulikalov 3y ago
        Can't a tech firm running a "legal gpt" have an insurance?
        toomuchtodo 3y ago
        Do they have a license to practice law?
- mannykannot 3y ago
  I like the term "explorable vocabulary." I can see using LLMs to get an idea of what the relevant issues are before I approach a professional, without assuming that any particular claim in the model's responses is correct.
- chaxor 3y ago
  I don't understand how everyone keeps making this mistake over and over. They explicitly just said "in 5-10 years". So many people continually use arguments that revolve around 'I used it once and it wasn't the best and/or me things up', and imply that this will always be the case. There are many solutions already for knowledge editing, there are many solutions for improving performance, and there will very likely continue to be many improvements across the board for this. It took ~5 years from when people in the NLP literature noticed BERT and knew the powerful applications that were coming, until the public at large was aware of the developments via ChatGPT. It may take another 5 before the public sees the developments happening now in the literature hit something in a companies web UI.
  - mjr00 3y ago
    on the other hand the rate of change isn't constant and there isn't a guarantee that the incredible progress in the past ~2 years in the LLM/diffusion/"AI" space will continue. As an example, take computer gaming graphics; compare the evolution between Wolfenstein 3D (1992) and Quake 3 Arena (1999), which is an absolute quantum leap. Now compare Resident Evil 7 (2017) and Alan Wake 2 (2023) and it's an improvement but nowhere near the same scale. We've already seen a fair bit of stagnation in the past year as ChatGPT gets progressively worse as the company is more focusing on neutering results to limit its exposure to legal liability.
    - GaggiX 3y ago
      >ChatGPT gets progressively worse https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar..., In blinded human comparisons, newer models perform better than older ones.
      - mvdtnz 3y ago
        That website doesn't load for me but anyone who uses ChatGPT semi regularly can see that it's getting steadily worse if you ever ask for anything that begins to border risque. It has even refused to provide me with things like bolt torque specs because of risk.
- coffeebeqn 3y ago
  I wonder could GPTs come up with legal loopholes. Like they are expected to come up with security vulnerabilities
XCSme 3y ago
Or a LLM that helps you spend less. Imagine a LLM that goes over all your spending, knows all the current laws, benefits, organizations, promotional campaigns, and suggests (or even executes) things like changing electricity provider, insurance provider, buying stuff in bulk from a different shop that you get for 4x the price at your local store, etc.
- OldOneEye 3y ago
  I love this idea. It would be incredibly useful! I feel LLMs are great at suggestions that you follow up yourself (if only for sanity checking, but nothing you wouldn't do with a human too).
- p-e-w 3y ago
  That would not be in the interest of anyone with any real power, so you're going to see tanks on the streets before it happens.
Solvency 3y ago
This does not make sense to me. ChatGPT is completely nerfed to the point where it's either been conditioned or trained to provide absolutely zero concrete responses to anything. All it does is provide the most baseline, generic possible response followed by some throwaway recommendation to seek the advice of an actual expert.
- frankfrank13 3y ago
  The way to get around this is to have it "quote" or at least try to quote from input documents. Which is why RAG became so popular. Sure, it won't right you a contract, but it will read one back to you if you've provided one in your prompt. In my experience, this does not get you close to what the top-level comment is describing. But it gets around the "nerfing" you describe
- baobabKoodaa 3y ago
  It's very easy to get ChatGPT to provide legal advice based on information fed in the prompt. OpenAI is not censoring legal advice anywhere near as hard as they are censoring politics or naughty talk.
  - freejazz 3y ago
    That's because its just advice, not legal advice. Legal advice is something you get from an attorney that represents you. It creates a liability relationship between you and the attorney for the legal advice they did provide.
    - baobabKoodaa 3y ago
      Sure, we can call it "advice" instead of "legal advice" or we can even call it other names, like "potato", if that's what you want. My point is that potato not censored.
      - freejazz 3y ago
        Feel free to miss the point as much as you want. You can call it a baked potato then!
frankfrank13 3y ago
I think a lot of startups are working on exactly what you are describing, and honestly, I wouldn't hold my breath. Everyone is still bound by token limits and the two best approaches for getting around them are RAG and Knowledge-Graphs, both of which could get you close to what you describe but not close enough to be useful (IMO).
baobabKoodaa 3y ago
We are literally building this today! Our core business is legal document generation (rule based logic, no AI). Since we already have the users' legal documents available to us as a result of our core business, we are perfectly positioned to build supplementary AI chat features related to legal documents. We recently deployed a product recommendation AI to prod (partially rule based, but personalized recommendation texts generated by GPT-4). We are currently building AI chat features to help users understand different legal documents and our services. We're intending to replace the first level of customer support with this AI chat (and before you get upset, know that the first level of customer support is currently a very bad rule-based AI). Main website in Finnish: https://aatos.app https://aatos.app (also some services for SE and DK, plus we recently opened UK with just a e-sign service)
- baobabKoodaa 3y ago
  Here's an example of what our product recommendations look like: Given your ownership in a company and real estate, a lasting power of attorney is a prudent step. This allows you to appoint PARTNER_NAME or another trusted individual to manage your business and property affairs in the event of incapacitation. Additionally, it can also provide tax benefits by allowing tax-free gifts to your children, helping to avoid unnecessary inheritance taxes and secure the financial future of your large family.
- Closi 3y ago
  > Since we already have the users' legal documents available to us as a result of our core business, we are perfectly positioned to build supplementary AI chat features related to legal documents. Uhh... What are the privacy implications here?!
  - baobabKoodaa 3y ago
    If you look at the example I posted of our product recommendations, you will see that the GPT-4 generated text contains "PARTNER_NAME" instead of actual partner name. That's because we've created anonymized dataset from users in such a way that it's literally impossible for OpenAI to deanonymize users. Of course the same can not be done if we want to provide a service where users can, for example, chat with their legal documents. In that case we will have to send some private details to OpenAI. Not sure how that will pan out (what details we decide to send and what we decide not to send). In any case, all startups today are created on top of a mountain of cloud services. Any one of those services can leak private user data as a result of outsider hack or insider attack or accident. OpenAI is just one more cloud service on top of the mountain.
nostromo 3y ago
And not just legal either. I uploaded all of my bloodwork tests and my 23andme data to Chat GPT and it was better at analyzing it than my doctor was.
- slingnow 3y ago
  This is a really interesting use case for me. I've been envisioning a specially trained LLM that can give useful advice or insights that your average PCP might gloss over or not have the time to investigate. Did you do anything special to achieve this? What were the results like?
- p-e-w 3y ago
  Yup. Because the doctor doesn't have time and doesn't give a fuck. LLMs don't have to compete against the cutting edge of human professional knowledge. They only have to compete against the disinterested, arrogant, greedy, and overworked professionals that are actually available to people in practice. No wonder they're winning.

throwaway17_17 3y ago

I will reserve judgment of the possibilities of LLMs as applied to the legal field until they are tested on something other than Document/ contract review. Contract review is, in the large business law case, often done by outsourcing to hundreds of recent graduates and act more like proof reading with minimal application of actual lawyering skills to increase a corporation’s bottom line. The more common, for individual purchasers of legal services, lawyering is going to be family law matters, criminal law matters, and small claims court matters. I can not see a time in the near future where an LLM can handle the fact specific and circumstantial analysis required to handle felony criminal litigation, and I see nothing that would imply LLMs can even approach the individualized, case specific and convoluted family dynamics required for custody cases or contested divorces. I’m not unwilling to accept LLMs as a tool an attorney can use, but outside of more rote legal proof reading I don’t think the technology is at all ready for adoption in actual practice.

giantg2 3y ago
"and I see nothing that would imply LLMs can even approach the individualized, case specific and convoluted family dynamics required for custody cases or contested divorces." Humans are pretty bad at this. Based on the results, it seems the judges' personal views and emotions are a large part of these cases. I'm not sure what they would look like without emotion, personal views, and the case law built off of those.
- SkyBelow 3y ago
  The worse judges are at being perfectly removed arbiters of justice, the more room for lawyers to exploit things like emotions and humans connections with those judges, and thus the worse LLMs will be at doing that part of the job. A charismatic lawyer backed by an LLM will be much better than an LLM. At least until the LLMs surpass humans at being charismatic, but that would seem to be its own nightmare scenario.
  - staunton 3y ago
    > At least until the LLMs surpass humans at being charismatic Look into "virtual influencers". Sounds like you should find it interesting.

Workaccount2 3y ago

I wonder what the reach of a legal argument a bunch of lawyers are going to come up with in order to cripple the tech that threatens their industry?

delichon 3y ago
Copyright appears to be the primary attack vector.
- anotherhue 3y ago
  Are the arguments submitted to a court (and made publicly accessible) subject to copyright? I kind of assumed they were in the same space as government documents.
  - delichon 3y ago
    A legal LLM would be significantly crippled without the knowledge stored in the universe of non-legal documents.
    - anotherhue 3y ago
      You're probably right, but the law and reality often seem to be orthogonal.
zugi 3y ago
Lawyers control government, at least in the U.S. Expect laws banning or severely restricting the use of AI in the legal field soon. I expect arguments will range from the dangers of ineffective counsel to "but think of the children" - whatever helps them protect their monopoly.
- deleted 3y ago
  [deleted]
- axpy906 3y ago

vitiral 3y ago

It feels to me like the law is already a staggering heap of complexity. Isn't using technology going to just enable more of the same, making the situation worse?

engineer_22 3y ago
on the contrary, it may help to highlight incongruities in the legal domain and provide lawyers with compelling groundwork to make relevant claims
- deleted 3y ago
  [deleted]
urbandw311er 3y ago
Or you could take the view that, in fact, this is one of the things LLMs are very good at, ie making sense of complexity.
- vitiral 3y ago
  But the lawyers reading the law won't be the only ones using LLMs. LLMs will also be used to write laws. Then lawmakers will use them to "check" that their 20,000 page law supposedly works. No human can understand the scope of today's laws: how much less when no LLMs can understand the laws created 20 years from now. I'd love to hear that LLMs can be used to trim and simplify complexity, but I don't believe it. They generate content, not reduce it.

delichon 3y ago

In all criminal prosecutions, the accused shall enjoy the right ... to have the Assistance of Counsel for his defense -- 6th amendment to US Constitution When an LLM is more competent than an average human counsel, does this amendment still require assistance of a human counsel?

urbandw311er 3y ago
Ironically you might be better asking GPT this question.
District5524 3y ago
All the governments in the world would do everything in their power to get people accept this suggestion as a truth and use LLMs instead of human lawyers especially in criminal defense. Now, why is that? Maybe it's not the technical knowledge that is the most important feature of a lawyer?
- sudden_dystopia 3y ago
  Yea it’s their ability to manipulate language and people to bend the letter of the law to suit their specific cases regardless of any long term potential societal harm.
- AustinDev 3y ago
  All governments of the world at least in the west appear to mostly consist of attorney's I doubt they'd let it happen. It'd be bad for their guild.
  - District5524 3y ago
    You are mixing up important things and probably that's why you don't see a bigger threat here than the livelihood of a couple of lawyers. People who attended law school/university/have a law degree are not necessarily practising law as lawyers/attorney or whatever they call them in the given country, as a member of the government. But of course, once a marine, always a marine... On the other hand, the parent was trying to refer to criminal defense (Amendment 6 of US Constitution). That's not something active politicians can participate in (conflict of interest, etc.). If one has the means to pay for their criminal defense (or family law, anything that truly matters for the average citizen), they tend to be willing to pay a lot and expect some quality of service in exchange. They will certainly not choose ChatGPT, just because that provides the answer faster for long documents. But a large number of people don't have this kind of money. As long as governments don't have a proper budget for legal aid, they will be inclined to cover the vanishing legal aid budget with false claims that they fulfill "appointment of counsel" with LLM services - while they don't really care about the outcome of the case. (And of course, successful politicians have rarely worked as pro bono counsels before their current job, not that it has any more relevance here.) Non-Western governments are also interested in having this replacement service - the cost of lawyers is also money there. It's much better for all governments to have chatbots they have reliably trained than less controllable humans. That's a hypothetical threat for now, but I hope you understand why there is something more here beyond "guildthink" and protecting the livelihood of lawyers at all costs behind the Sixth Amendment. There are lots of countries (or even within the US, like in Utah) where you can review contracts without being a practising lawyer, like LPOs mentioned in the paper. So far, it failed to make the world a better place. But replacing including LLMs as "counsel" under Sixth Amendment would mean a very different situation.

deleted 3y ago

[deleted]

hansonkd 3y ago

I run a startup that does legal contract generation (contracts written by lawyers turned into templates) and have done some work GPT analysis of the contract for laypersons to interact and ask questions about the contract they are getting. In terms of contract review, what I've found is that GPT is better at analysis of the document than generating the document, which is what this paper supports. However, I have used several startups options of AI document review and they all fall apart with any sort of prodding for specific answers. This paper looks like it just had to locate the section not necessarily have the back and forth conversation about the contract that a lawyer and client would have. There is also no legal liability for GPT for giving the wrong answer. So It works well for someone smart who is doing their own research. Just like if you are smart you could use google before to do your own research. My feelings on contract generation is that for the majority of cases, people are better served if there were simply better boilerplate contracts available. Laywers hoard their contracts and it was very difficult in our journey to find lawyers who would be willing to write contracts we would turn into templates because they are essentially putting themselves and their professional community out of income streams in the future. But people don't need a unique contract generated on the fly from GPT every time when a template of a well written and well reviewed contract does just fine. It cost hundreds of millions to train GPT4. If $10m was just spent building a repository of well reviewed contracts, it would be a more useful than spending the equivalent money training a GPT to generate them. People ask pretty wide range of questions about what they want to do with their documents and GPT didn't do a great job with it, so for the near future, it looks like lawyers still have a job.

OldOneEye 3y ago
Which is mostly what I feel also happens with LLMs producing code. Useful to start with, but not more than that. We've still got a job us programmers. For the moment.
- klabb3 3y ago
  Producing code is like producing syntactically correct algebra. It has very little value on its own. I’ve been trying to pair system design with ChatGPT and it feels just like talking with a person who’s confident and regurgitates trivia, but doesn’t really understand. No sense of self-contradiction, doubt, curiosity. I’m very, very impressed with the language abilities and the regurgitation can be handy, but is there a single novel discovery by LLMs? Even a (semantic) simplification of a complicated theory would be valuable.

deleted 3y ago

[deleted]

williamcotton 3y ago

Lexis has some AI feature built into it: https://www.lexisnexis.com/en-us/products/lexis-plus-ai.page https://www.lexisnexis.com/en-us/products/lexis-plus-ai.page I haven't had a chance to test it out as anyone should be a bit weary to add more paid features to an already insanely expensive software product!

FrustratedMonky 3y ago

Theoretically. The language in laws should be structured similar to code. It has some logical structure. Thus should be more easily adopted to LLMs than other 'natural language'. So despite the early news about lawyers submitting 'fake' cases, it is only a matter of time before the legal profession is up-ended. There are a ton of paralegals, etc.. doing 'grunt' work in firms, that an LLM can do. These are considered white color, and will be gone. It will progress in a similar fashion to coding. It will be like having a junior partner that you have to double check, or that can do some boiler plate for you. You can't trust completely, but you don't trust your junior devs do you, but it gets you 80% there.

NoboruWataya 3y ago
> It will progress in a similar fashion to coding. I kind of agree with this, but this is why I am confused that I only ever see people (at least on HN) talk about AI up-ending the legal profession and putting droves of lawyers out of work--I never see the same talk about the coding industry being transformed in this way.
- FrustratedMonky 3y ago
  I've heard it discussed. A lot more a few months ago when GPT first blew up. Maybe HN is full of coders that still think themselves 'special' and can't be replaced. Or maybe, the law profession has a lot more boilerplate than the coding profession? So legal profession has more that can be replaced? Coders will be replaced, but maybe not at same rate as paralegals.

ulrischa 3y ago

Lawyers have been resisting technological advances for years. No industry rejects technological tools as vehemently as the legal industry, arguing that everything has to be judged by people. Even laws that are available online do not even link to the paragraphs that are referenced. All in all, progress is institutionally suppressed here in order to preserve jobs.

NoboruWataya 3y ago
> Even laws that are available online do not even link to the paragraphs that are referenced. It's not lawyers' job to publish the laws online. Lawyers are the ones who would benefit from more easily searchable online laws, as they are the ones whose job is actually to read the laws. That is why there are various commercial tools that provide this functionality, that lawyers pay for. You need to ask your government why public online legal databases are so poor, not your lawyer.
- ulrischa 3y ago
  Right. 70% of governmental staff are people with a law education. So I mean lawyer in a broader sense
DanielSantos 3y ago
I also built an AI contract review ai tool and talked to > 100 lawyers. What I found is that lawyers want technological advances but only if they work 100% of the time. Also helped lawyers looking for a CLM, and they rejected something if it caused any inconvenience.

light_hue_1 3y ago

Talk about a conflict of interest. A company that pushes llms for legal work says they work better. This isn't worth the pdf it wasn't printed on.

carstenhag 3y ago
I disagree. The company is mentioned multiple times, a conflict of interest is clearly visible. We also don't complain about Google et al publishing papers about some of their internal systems and why it helps, I hope?
- drewdevault 3y ago
  Google isn't trying to sell you their internal systems. This is a bullshit AI hype bubble advert masquerading as an academic paper. Bet you 10 bucks it doesn't get through peer review (or isn't even submitted for peer review).
  - famouswaffles 3y ago
    >Google isn't trying to sell you their internal systems. They sometimes are. If you have an issue with the methodology of paper then all well and good but "conflict of interest" is pretty weak. Yes, Google and Microsoft et al regularly publish papers describing Sota performance they sometimes use internally and even sell. I didn't have to think much before wavenet came to mind. Besides, the best performing models here are all Open AI.
  - carstenhag 3y ago
    As some internal systems will be made available as a product, they do sell it (maybe).
vibeproaaaac21 3y ago
Agreed. That's most AI research though. They are a mechanism whose entire value proposition lies in laundering externalities. Not that this isn't exactly what all the big "tech innovation" of the last decade were either. It's depressing and everyone involved should be ashamed of themselves.

oldgregg 3y ago

Interesting problem space-- I have a culture jamming legal theory this might work for: What if you had a $99/mo pro-se legal service that does two things, 1) teaches you how to move all of your assets into secure vehicles. 2) At the same time it lets you conduct your own defense pro-se-- but the point is not to win, it's just to jam the system. If you signal to the opposing party that you're legally bankrupt and then you just file motion after motion and make it as excruciating as possible for them they might just say nevermind when they realize it's gonna take them 5 years to get through appeals process. It's true lawyers don't want to give up their legal documents for a template service-- but honestly just going to the court house and ingesting tons of filings might do the trick. With that strategy in mind you don't really need GREAT documents or legal theory anyway. Just docs that comply with court filing requirements. Yeah we're def gonna need to deposition your housekeepers daughters at $400/h and if you have a problem with that I would be happy to have a hearing about it. If enough people did this is would basically bring the legal system to a standstill and give power back to the people. RIP Aaron Swartz who died fighting for these issues :'(

gee_totes 3y ago
What you're describing in 2) sounds a lot like paper terrorism[0] [0]https://en.wikipedia.org/wiki/Paper_terrorism https://en.wikipedia.org/wiki/Paper_terrorism
- oldgregg 3y ago
  So what do you call it when the wealthy and corporations exploit their opponent's inability to afford legal representation? A normal Tuesday in Amerika. Yes, the banality of tuesday terrorism.
  - akira2501 3y ago
    > So what do you call it when the wealthy and corporations exploit their opponent's inability to afford legal representation? It's called a Strategic Lawsuit Against Public Participation. It's also illegal in most states. > A normal Tuesday in Amerika. We already understood your derogatory outlook without it needing to be literal. Anyways, that's why the founders gave us the First, Fourth, and Fifth amendments. It turns out, they recognized, the "law" is literally the worst mechanism for discovering truth and managing outcomes. > Yes, the banality of tuesday terrorism. It's terrorism because it has no logical conclusion nor any ability to positively benefit anyone's life, in particular, the person who would wield it. Your equivocation ignores this.

zehaeva 3y ago

Given the recent legal cases where lawyers did use Chat GPT to do research and help write their brief did not go very well I'm not sold that on all the optimism that's here in the comments.

minimaxir 3y ago
The technology is fine, the education and literacy about its flaws and limitations among typical nontechnical users is another story.
frankfrank13 3y ago
That was rookie level mistakes though. Not checking a *case* exists? Building a small pipeline of generation->validation isn't trivial, but it isn't impossible. The cases you describe seem to me like very lazy associates matched with a very poor understanding of what LLMs do.
bpodgursky 3y ago
They were all idiots too cheap to pay for GPT-4. Got caught by hallucinations.

cwoolfe 3y ago

Where can someone upload a contract and ask the AI questions about it in a secure and private way? It's my understanding that most people and organizations aren't able to do this because it isn't private.

btbuildem 3y ago
You can host your own LLM - something like Mixtral for example - then you have full control over the information you submit to it.
kveton 3y ago
We do this today (securely upload a file and ask questions or summarize) and part of our promise, and why we're having early success, is because we promise not to train with customer data and we don't run directly on top of OpenAI. https://casemark.ai https://casemark.ai
kulikalov 3y ago
ChatGPT enterprise? Or over API. They state that those offerings data is not used for training. I'm not a lawyer but afaik illegally retrieved evidence cannot be used - "exclusionary rule".
DanielSantos 3y ago
You can try us [1] . During the upload process you can enable data anonymization. It’s not perfect though. We use open ai but they only get segments of a contract. Not the full one and can’t connect them. You get the review via email and after you can delete the document and keep the review. [1] legalreview.ai

mkrishnan 3y ago

[dead]

adastra22 3y ago

Nice title!

advael 3y ago

It's bizarre how easily we got to the Goodhart's Law event horizon in our comparisons of complex fields to AI models But this is what happens when industries spend a decade brain-draining academia with offers researchers would be insane to refuse

District5524 3y ago

While this paper is clearly not without merits, it intends to be more like an excuse to make a bombastic statements about a whole profession or "industry" (perhaps to raise their visibility and try to sell something later on?). The worst part is that they have actually referenced a single preprint document as "previous art" - and that document itself is not related to contract review, but to legal reasoning in general. (A part of LegalBench is of course "interpretation", and that is built on existing contract review benchmarks, but they could've found more relevant papers). Automating legal document review has been a very active field in NLP for twenty years or so (including in QA tasks) and became a lot more active since 2017. At least e.g. Kira (and Luminance etc., none of which is LLM-based) are already used quite widely in legal departments/firms around the world. So lawyers do have practical experience in their limitations... But Kira & co. are not measuring the performance of the latest and greatest models and they do not use transparent benchmarks etc. So the benchmark results in this paper are indeed a welcome addition in terms of using LLMs. But also considering its limited scope of reviewing 10 (!) documents based on a single review playbook, they should not have written about "implications for the industry". It is very much pretentious and shows more of the lack of knowledge of the authors of the very same industry than about the future of the legal services industry. If you're interested in the capabilities and limitations, I suggest these informative, but still light reads as well: https://kirasystems.com/science/ https://kirasystems.com/science/ https://zuva.ai/blog/ https://zuva.ai/blog/ https://www.atticusprojectai.org/cuad https://www.atticusprojectai.org/cuad

d0odk 3y ago
Any particular papers you would recommend? The links are to blogs with lots of papers.
- District5524 3y ago
  The third one (CUAD) is a single paper, not blogposts like the others. I think this paper is still the best in terms of being done by NLP experts and understanding the possibilities and not being just some and mirror. But there are so many papers published in this area nowadays that I might not even notice a new one. The CUAD paper was still based on BERT, so pretraining was needed - that needs a bit more expertise than just prompting GPT-4-32k like in this paper or feeding prompts back to GPT-4 for another round of refining, or doing RAG. For honest research purposes, "contract review" is not really a good area of approach: the subject field is not standardised, there is no good benchmark yet and your paper can easily get into bad company of snake oil sellers cashing on visceral hatred of average people for all professions.

very_good_man 3y ago

We may finally again get affordable access to the rule of law in the United States.

gogogo_allday 3y ago

I must not be reading this paper correctly because it appears that they only used 10 procurement contracts to do this study. If so, the abstract and title feels misleading. I’d be more interested in a study done on thousands of contracts of different types. I also have my doubts it would perform well on novel clauses or contracts.

District5524 3y ago
Your reading is correct. Their primary aim was to compare some real lawyers' work with their wonderprompts to show off an upcoming service and how faster that is compared to humans. But they didn't have the funds to ask lawyers to review 15 or 50 or 1000 contracts, just 10, hence the cheat in methods and in conclusions. This is not about drawing up a benchmark for an industry, let alone validating any useful method in general. Ten docs reviewed based a single review 'playbook' (I think it's not in the paper, but probably max 20 questions per contract?) and compared across 3 different providers/roles + LLMs...

gumby 3y ago

I would not want an transformer-generated contract but I would be delighted if a transformer-generated contract were used as input by an actual lawyer and it saved me money. In my experience current practice (unchanged for the decades I've been using lawyers) is that associates start with an existing contract that's pretty similar to what's needed and just update it as necessary. Also in my experience, a contract of any length ends up with overlooked bugs (changed sections II(a) and IV(c) for the new terms but forgot to update IX(h)) and I doubt this would be any better with a machine-generated first draft.

ok123456 3y ago

Does LPO (Legal Process Outsourcing) mean a paralegal?

District5524 3y ago
Could be paralegals as well, but they are usually just human employees of legal service providers that are not licensed lawyers/law firms. It's jurisdiction specific who can do this besides lawyers. In the UK and Australia (and probably NZ as well) they can be practically be anyone, contract review is not a reserved legal activity like in most US states.

graphe 3y ago

Law is very specific. BERT was sufficient for law even before chatGPT. https://towardsdatascience.com/lawbert-towards-a-legal-domain-specific-bert-716886522b49 https://towardsdatascience.com/lawbert-towards-a-legal-domai... https://huggingface.co/nlpaueb/legal-bert-base-uncased https://huggingface.co/nlpaueb/legal-bert-base-uncased

adrianmonk 3y ago

I wonder if this could help regular (non-lawyer) people understand legal documents they run into in everyday life. Things like software license agreements, terms of service, privacy policies, release of liability forms, deposit agreements, apartment leases, and rental car agreements. Many people don't even try to read these because they're too long and you wouldn't necessarily understand what it means even if you did read it. What if, before you signed something, you could have an LLM review it, summarize the key points, and flag anything unusual about it compared to similar kinds of documents? That seems better than not reading it at all.

wow_its_tru 3y ago
We're building exactly this today, for common business contracts. We're not building for consumers today, because I think it's vanishingly unlikely that you'll, like, pick a different car rental company once you read their contract :) but leases, employment contracts, options agreements, SaaS agreements... all common, all boilerplate with 5-10 areas to focus on, all ready for LLMs!
- mattmaroon 3y ago
  Honestly, that type of consumer use case might actually be relevant once LLM’s can do this sort of thing. Certainly, nobody is going to contact their attorney before renting a car, but if this could be integrated into a travel site or something… You never know how consumer behavior may change when something that was either impossible or impractical becomes very easy b
- DanielSantos 3y ago
  We have also have been building this[1] but struggled to monetize even with 100s of users and 1000s of contracts review. We are live for about 1 year. If you want to share experience feel free to reach out [1] legalreview.ai

adi4213 3y ago

For the auditory learners, here is this paper in a summarized Audiobook format : https://player.oration.app/1960399e-ccb0-44f6-81f0-870ef7600766 https://player.oration.app/1960399e-ccb0-44f6-81f0-870ef7600...

visarga 3y ago

Two reasons why it's a bad idea: 1. ChatGPT can't be held responsible, it has no body, like summoning the genie from the lamp, and about as sneaky with its hard to detect errors 2. ChatGPT is not autonomous, not even a little, these models can't recover from error on their own. And their input buffers are limited in size, and don't work all that well when stretched at maximum Especially the autonomy part is hard. Very hard in general. LLMs need to become agents to collect experience and learn, just training on human text is not good enough, it's our experience not theirs. They need to make their own mistakes to learn how to correct themselves.

itissid 3y ago

GPTs only generate specific answers if they are trained using RHLF to prefer certain answers to others. Won't this mean that coming up with a contract that meets a special individual's case will require that much more fine-tuning? Also how do you reconcile several logical arguments without a solver? Like "If all tweedles must tweed", "If X is a tweedle, therefore it must tweed unless it can meet conditions in para 12". How can it possibly learn to solve many such conjunctions that are staple in legal language?

unyttigfjelltol 3y ago

This is apples and bowling balls. You also probably could replace the CEO and entire executive team with LLMs. And cheaper! Much cheaper! But, if the stochastic analysis was ... wrong ... who would be left to correct it?

sandbx 3y ago

I like that they incl the prompt in the paper

pwmiller_ai 3y ago

I lead data and AI at a tech-enabled commercial insurance brokerage. We have been leveraging GPT-4 to build a deep contract analysis tool specifically for insurance requirements. My teams at Google also built several LLM solutions to support Google's legal team, from patent classification to discovery support. Language models are great at digesting legalese and analyzing what's going on. However, many legal applications involve around pretty important decisions that you don't want to get wrong ("am I contractually covered with my current insurance program?"). Because of that, we've built LLM products in the legal space with the following principles in mind: - Human-in-the-loop tooling -- The product should be built around an expert using it whenever possible, so decision support as opposed to automation. You still see massive time savings with that in place - Transparency / citations -- With a human-in-the-loop tool, you need mechanisms to build trust. Whether that's highlighting clauses in the document that the LLM referred to or explaining why a part of the analysis wasn't provided, citing your work is important - Tuned for precision instead of recall -- False positives (and hallucinations) are especially bad in many of these legal use cases, so tuning the model or prompts for precision helps with mitigation.

theptip 3y ago
Do you have any pointers on how to get GPT-4 to do citations? Is a prompt like “quote back the passage you are citing” so you can locate and highlight the original? When you say “tuned for precision” is this your prompt engineering or are you actually fine-tuning GPT-4? Appreciate the insights.
- pwmiller_ai 3y ago
  For RAG applications, starting simply with a reference to the most relevant chunk(s) is helpful in building transparency. A lot of our contract review task is a data extraction one (e.g. extract this type of insurance language and compare against your policy). As such, it's much easier to pinpoint the exact text from the source doc for citation. In our applications, currently, we are doing citations as a postprocessing task as opposed to as part of the prompt itself. Finding that feeding too many instructions to the LLM results in worse responses. We're not fine-tuning today. Instead, "tuning for precision" is done through prompt chains. A simple example would be returning "I don't know" early on if the document isn't a contract or doesn't have clear insurance requirements in it. We've had success with various guardrail prompts. The citation work we did at Google used model internals to highlight text (path integrated gradients). It's also easier to finetune for precision when you have control over the model itself.

ProllyInfamous 3y ago

Just as reminder, Chief US Justice released his "year in review" on acceptance of LLM / GPT systems within legal practices, and that it will likely enable access to the law by more-and-more commoners. The entire six pages is worth the time reading [1]. [1] [PDF] https://www.supremecourt.gov/publicinfo/year-end/2023year-endreport.pdf https://www.supremecourt.gov/publicinfo/year-end/2023year-en...

notfed 3y ago

> Cost wise, LLMs operate at a fraction of the price, offering a staggering 99.97 percent reduction in cost over traditional methods. This seems like a weirdly arbitrary and forced statistic, when "100% reduction" would be just as valid of a statement.

TeMPOraL 3y ago
100% reduction would mean "literally free", which it isn't.
- notfed 3y ago
  But LLMs are literally free. This paper just chose to focus on the "prominent" ones.

underlines 3y ago

Conflict of interest in the paper, as this mainly is a PR piece from Onit: "Onit Announces Generative AI-Powered Virtual Legal Operations Assistant for In-house Counsel"

happytiger 3y ago

Comparing current generation LLMs, which aren't design for accuracy, against lawyers has some pretty obvious problems. The fact is that only one of the two is set to make non-linear improvements, and that is the only point that matters in this debate.

Sofiaaz 3y ago

[flagged]

Vinaas 3y ago

Life can be so cruel sometimes but even worst when you have a cheating partner. I got to discover my spouse was having an affair through the help of { remote spy wise @ gm ail c o m } who gave me access to his device remotely without his notice. I got access to all social media apps, call logs and sms and deleted datas and messages too. Do not be a victim to lies and mischief. Get in contact with him and stay safe.