4 ms·
I know someone who works in law and deals particularly with an area of US benefits and healthcare law. One of their workflows for lower-level employees at their
by ivraatiems 9d ago
I know someone who works in law and deals particularly with an area of US benefits and healthcare law. One of their workflows for lower-level employees at their firm involves taking in documents from healthcare plans and organizations, analyzing them for certain kinds of data, and then importing that data into an internal system they use to analyze and provide guidance on plans. The internal system can contain hundreds of documents for an individual client. All of the documents have the same information (roughly) but in totally diverse formats and styles. Once it's in the system, it's easy to compare and analyze across documents and the research process is much faster.
They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.
LLMs have great advantages for this kind of work - but not for decision-making. I just don't see OpenAI ever admitting that.
(I've left some details intentionally vague because this is a very specific area of law and I don't want my friends to be identified without their consent.)
- deleted 9d ago[deleted]
- lemonlimetea 9d ago[dead]
- MiroslavPokorny 9d agoMy dog can review documents at an even faster rate. You havent given any proofs or even comments that the work is the same level of quality or accuracy.
- ivraatiems 9d agoThe proof would be that the attorney who did the work before, and who still reviews all ingested data, says it is.
- MiroslavPokorny 8d agoExactly. The statements in the post are opinions, there is no actual PROOF they are true, and thats my point.
- deleted 8d ago[deleted]
- samtp 9d agoYou should free your dog then because a creature that intelligent should not be anyone's pet.
- nradov 9d agoAnd free the biglaw junior associates while we're at it!
- ivraatiems 8d agoThey're lawyers. They don't deserve freedom.
- TallGuyShort 9d agoI'm thinking back to some of my interactions with corporate lawyers and you know what? My bar (<- pun) for trusting Astra more than a lawyer is pretty low.
- newyankee 9d agoVisit a developing or corrupt country and you will realise the LLMs of today analyse better than the judges from purely precedence and literature review POV
- refurb 9d agoI’m curious how this increased throughput happens. You’ve accurately stated that AI isn’t as rigorous as a trained attorney. Doesn’t that mean that every single datapoint must be confirmed by a human? How is that quicker than just using a human to read the content and make the call? Data entry savings?
- juiceland 9d agoYou don’t need a trained attorney to schematize data. The LLMs are used to make the data easier to understand and manipulate.
- cromka 9d agoThey'll also hallucinate and change meaning in the process of extraction and "schematization"
- margalabargala 9d agoNot necessarily. Depends how you use it. "Write a python script that breaks down this PDF by X feature" would not hallucinate anything in the PDF. Certainly you could trivially double check that all text in the extracted JSON was in the text layer of the PDF.
- jeffbee 9d agoHow much experience do you have with LLMs exactly? It would be consistent with my experience if Claude stuck in a line of python that just emits a JSON literal with no justification, potentially buried in a large program where an untrained person might not notice it. I don't even trust them if the output consists of structured data paired with source images from the PDF, because I've experienced LLMs fabricating the source rectangles to match the output. I only use tools like this by asking for programs, because as you note LLMs are good at that, and the verification process consists of tool calls to legitimate PDF manipulation tools so I have some confidence everything is above board. Even then I only do this for hobbies, not anything that matters.
- 2Gkashmiri 9d agoI'm a lawyer myself. I see problems in LLMs doing research, making drafts, etc. I see 3 kinds of potential users. 1. Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs. 2. Fresh out of law school graduates. They think they can do fast work, draft quick replies, do fast case research, enh. They should go the traditional route of interning, learning the "art", they probably won't need it because they will learn "how the law works" 3. Established attorneys. They usually have templates made for most things, over their career, they have established routines of making things fast, maybe it can help them but to say replacing good chunk of work or offloading to LLMs isn't probably going to go well. There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....
- jcims 9d ago>I see 3 kinds of potential users. 3 kinds of users that don't scare the shit out of you. My buddy works with lots of folks that have gone all in on Harvey. There are other kids of users.
- nullsanity 9d ago[dead]
- newyankee 9d agoMight be true today, but I still feel pareto principle applies. These points also sound like gatekeeping a bit and I hope future AI versions prove that law is also essentially something that can be made as programmable as possible for most common answers if the underlying principles and constitution are organised such. If there is no precedence for X happening when Y , Z, P, Q, R are special combinations , the judge decides and sets precedence. But if another judge interprets it differently with similar set of circumstances it is not considered incorrect I presume especially if in different jurisdiction unless overridden by a higher court. This is my layman understanding but I think this design is wrong and essentially where human subjectivity controls the system. Subjectivity is not totally avoidable in law but the goal of the legal system should be reduce it as much as possible along with ambiguity.
- inemesitaffia 8d agoThey'd be better off outsourcing to lawyers in another country.
- stymaar 8d ago> They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work. At my company we're also in the process of deploying a system that does exactly that. And what's interesting is that you absolutely do not need a frontier model for that, a small VLM (vision-language model), with optionally a little bit of fine-tuning, gives you the same output quality at a fraction of the latency and cost.
- dguest 8d agoI'm wondering if the bottom is going to fall out of frontier models when people start to realize this. Sure, as an experienced programmer you can tell the difference between a compact free model and the ones that are 100x bigger and cost billions to train, but 90% of busy work is dead simple: turn a PDF into json fields, or do the inverse and fill out a form. As the tools for this conversion become more widespread you can also imagine an economic shift toward efficient information brokers that make frontier models less necessary. It's also not really clear if everyone is going to want a frontier model when the real implications sink in. Maybe we'll get sick of incomprehensible code optimizations and people wile tire of reading AI prose that feels ever-more-human. There might be a few use cases, but who is going to pay for this when providers start charging enough to be profitable.
- Cthulhu_ 8d agoI think this is the long-term reality of LLM tech and one of the objectives of engineers implementing solutions, finding the right fit for the job - the right model and cost to achieve high enough accuracy at the lowest price. But that's phase two, phase one is finding localized problems to solve using LLMs and productize them. I'm reminded of cloud tech, where phase one was changing software to run in the cloud, and phase two was optimizing costs.
- KoolKat23 8d ago
- ManuelKiessling 8d agoI just realized how refreshing it is to read an honest take like "from 2-3 documents an hour to 8-10 documents an hour" instead of "it's doing the work of a month in 5 minutes!!!!1".
- deleted 8d ago[deleted]
- eru 8d agoAgreed. However: That's a factor of 3 to 5 improvement reported on here. If you keep stacking these kinds of improvements, you end up at month to minutes pretty quickly.
- monax 8d ago10 women can't make a baby in a month
- eru 8d agoThroughput-wise, they can. Latency-wise, you are right.
- egeozcan 8d agoSorry for being so detached from the discussion and your actual point but if someone had made a dystopian sci-fi novel with the tagline following your sentence with "...or can they?", and made it all cheesy, I'd read it in one sitting.
- lukan 8d agoYou mean each women all producing parts of the baby one then one women/machine putting all the parts together?
- egeozcan 8d agoIn the future, because of bad things we did to the atmosphere and stuff, miscarriages are the common ending of a pregnancy, instead of being the exception. One evil company finds a way to accelerate the process by using 10 women, lowering the chances of miscarriage simply by reducing the time in womb. I thought, something like a medication helping the women develop the baby in 30 days, but seriously damages the host in a couple of days, so the fetus gets transferred from host to host. Some die, but the evil rich don't care. Your idea could also work though. I realize that this is totally the wrong corner of the internet to post this but I guess today I'll just go with it.
- armcat 8d agoHow do these use cases stack up (for real) in legal AI tools like Legora and Harvey? Disclaimer: I used to work in legaltech, but not those two companies.
- jorvi 8d agoLLMs are still absolutely horrid at analyzing PDFs so the results they are getting must be chock full of errors..
- Gareth321 8d agoAre they? I've had excellent success. The confusing part of this is that there are two types of PDF. The first is a "normal" digital PDF. The second is a scanned PDF. The first can essentially be read like a document. LLMs have no issues with this. It's the second kind of PDF where the constraint becomes the vision capability, and this is very impressive with Astra. I've had no issues with either. I imagine there could be issues with unusually dense and/or misaligned text on scanned PDFs, but I have not tested this. The bottom line, though, is that PDF OCR is usually regarded as a solved problem. LLMs won't usually do the recognition itself. It will farm it out to established tools which are very good.
- gf000 8d agoWell, I would argue about the first part. Even if they contain "native" text that can be extracted, in most cases their order will be messed up and it is often crucial for correct parsing. So in many cases the visual way is the only one that works correctly, the textual one is just a shortcut that may be walkable in certain cases.
- Otterly99 8d agoIt depends on what you called solved. If the goal is to only extract the unstructured text from the document, it is definitely solved. Extracting a more natural structure like paragraph separation, tables, header, footers (what is referred as document intelligence) is much more complicated and not fully solved, but I would say almost.
- jorvi 8d agoYup, this. It is actually one of my test cases for LLMs: take the weekly discount PDFs of all the big supermarkets and process each of them, creating a nice table per supermarkt, converting discounts like 1+1 and only listing discounts that are interesting value. I then share that with a bunch of people. All models fail this, even the really expensive ones. Even with harness, examples and proper insistent instruction, they'll mix up items and their related discount, which category the item should be in, which page they are on, skipping over items etc. As said above, you can OCR it, but at that point you're not processing a PDF, you're processing an image. And yes, I know the underlying raw PDF data is messy, but that's why it's such a good test.
- Cthulhu_ 8d agoHappy to see measured results that isn't breathless glazing, and I suspect this is the kind of thing happening everywhere all at once. The other thing to consider is "burnout", in quotes because I mean that the work you describe sounds kinda boring, so people would resent doing it after a while and productivity would slow right down. Automation is the solution to tedium / toil, has been long before AI.
- alansaber 8d ago"Sounds kinda boring" - many do say this about lawyers.
- ivraatiems 8d agoYeah, it's sort of odd that even here people are pushing back like it couldn't possibly be doing this, you must have misunderstood it or are overhyping it. I come neither to praise AI nor to bury it. This is a case where the tool is useful. The key is that this is work the attornies themselves do not want to do and which does not require their brains or expertise. It's not just "boring" but a bad use of their time.
- alansaber 8d agoWait what, the conversion is so lossy you use an LLM to analyse it? Or you get the claude analysis into JSON and put that into the system too? I'm not seeing what's causing the 2-3x efficiency bump.
- wafriedemann 8d agothis is one of the largest tasks of any professional services support firm like big law, audit, tax. if you can automate this monkey work you have to cut teams at least in half to stay competitive (probably more).
- habitbit 8d agoyeah, i think this is probably the more interesting use case. i do a bunch of boring data wrangling as a solo dev and the amount of time that disappears into "read this thing, find these 5 bits, put them somewhere else" is ridiculous. if you're going from 2-3 docs/hr to 8-10, even with someone checking the result, that's a pretty substantial improvement. i don't really care whether we call that decision making or not, i'd be happy to never do the copy/paste part again.
- cowpig 8d agoAs someone who has privacy concerns this is horrifying to me. Both companies have settled lawsuits for illegal use of data. I would be extremely upset to learn that anybody is passing my private information to these companies for any reason.
- ivraatiems 8d agoThese documents are not personal private information. They might be client privileged in some cases? But I don't think so.
- camdenreslink 8d agoI don't think the details of the company's benefits plan is considered your private information. But I am concerned about errors in this parsing leading to incorrect results (and that compounding everywhere throughout society when this sort of workflow becomes ubiquitous).