4 ms·

Yes, remember that these are effectively random PDFs in various different designs and formats, some of them not editable or even OCR'd. It took a human attorne

Yes, remember that these are effectively random PDFs in various different designs and formats, some of them not editable or even OCR'd. It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality). Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour. Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.

barrenko 14d ago
I'm doing some public court records processing for bankruptcy cases (interested mostly to seek out corruption in big national cases), and yes, the "variousness" of random PDFs is exactly the issue. Trying to get the cost for a whole case down to a minimum. Sample is around 300 court dates, shy under 1k files. At best I'm building a claude skills file.
newAccount2025 14d ago
Does the human find enough bugs that they stay on guard, or just rubber stamp everything without really looking at it? It’s hard to stay vigilant when stuff looks plausible.
- eru 14d ago
  Complete speculation: you could instruct Claude to hide one random mistake in every document. And Claude should write down the mistake in a sealed envelope, so it doesn't make into the database. A review that doesn't find the mistake counts as invalid.
  - k4tsu 14d ago
    This is what bag scanners at airports do - the hit rate is so low and the job so boring the software projects fake contraband onto the imagery. Fail to spot the knuckledusters and expect a chat with the manager.
  - camdenreslink 14d ago
    You could do this for code reviews. Have Claude include one subtle bug into each PR. If the reviewer does not spot it, then the PR is not mergable.
    - eru 13d ago
      Yes. Or you want to have a potentially more than one flaw planted, so people don't get lazy after the first one.
stymaar 14d ago
> Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense. A better use of these Claude subscription would be to develop the app (which it can pretty much do at that point) and you could iterate to make the workflow even more efficient than your current one.
- ivraatiems 14d ago
  Nobody working there has the requisite experience to do this in a reasonable amount of time. These are not particularly tech-savvy folks, Claude use aside.
  - eru 14d ago
    Yes. And it might not even be worth it, as the AI agents gets cheaper and cheaper. Keep in mind that the task is fixed, so as the frontier of AI advances, you can switch to a cheaper trailing edge system and still get the same or even better performance for this task.
  - stymaar 14d ago
    > These are not particularly tech-savvy folks, Claude use aside. The difference between a tech-savyy person, and a non-tech-savyy person has always been mostly in the later's head, but this is even more true now that we have pocket assistants who can answer pretty much all of our questions in a language tuned to our level of understanding.
ricky54 14d ago
Don't forget that humans have a not insignificant error rate when copy/pasting or copy/typing data.
- tkgally 14d ago
  And it's possible to run each document through the LLM pipeline multiple times, using different models and/or prompts each time, to check for errors and inconsistencies. That will take more time and cost more, but it can reduce the error and hallucination rate significantly.
refurb 14d ago
Thanks for sharing the details. Does the attorney check that the AI copied the data accurately? Or is it just assumed to be correct? Your experience mirrors my own. AI is great for parsing data that can take up a huge amount of time. My only concern is whether or not it’s done accurately. I wouldn’t use it for anything where mistakes cause serious consequences.

barrenko 14d ago

I'm doing some public court records processing for bankruptcy cases (interested mostly to seek out corruption in big national cases), and yes, the "variousness" of random PDFs is exactly the issue. Trying to get the cost for a whole case down to a minimum. Sample is around 300 court dates, shy under 1k files. At best I'm building a claude skills file.

newAccount2025 14d ago

Does the human find enough bugs that they stay on guard, or just rubber stamp everything without really looking at it? It’s hard to stay vigilant when stuff looks plausible.

eru 14d ago
Complete speculation: you could instruct Claude to hide one random mistake in every document. And Claude should write down the mistake in a sealed envelope, so it doesn't make into the database. A review that doesn't find the mistake counts as invalid.
- k4tsu 14d ago
  This is what bag scanners at airports do - the hit rate is so low and the job so boring the software projects fake contraband onto the imagery. Fail to spot the knuckledusters and expect a chat with the manager.
- camdenreslink 14d ago
  You could do this for code reviews. Have Claude include one subtle bug into each PR. If the reviewer does not spot it, then the PR is not mergable.
  - eru 13d ago
    Yes. Or you want to have a potentially more than one flaw planted, so people don't get lazy after the first one.

stymaar 14d ago

> Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense. A better use of these Claude subscription would be to develop the app (which it can pretty much do at that point) and you could iterate to make the workflow even more efficient than your current one.

ivraatiems 14d ago
Nobody working there has the requisite experience to do this in a reasonable amount of time. These are not particularly tech-savvy folks, Claude use aside.
- eru 14d ago
  Yes. And it might not even be worth it, as the AI agents gets cheaper and cheaper. Keep in mind that the task is fixed, so as the frontier of AI advances, you can switch to a cheaper trailing edge system and still get the same or even better performance for this task.
- stymaar 14d ago
  > These are not particularly tech-savvy folks, Claude use aside. The difference between a tech-savyy person, and a non-tech-savyy person has always been mostly in the later's head, but this is even more true now that we have pocket assistants who can answer pretty much all of our questions in a language tuned to our level of understanding.

ricky54 14d ago

Don't forget that humans have a not insignificant error rate when copy/pasting or copy/typing data.

tkgally 14d ago
And it's possible to run each document through the LLM pipeline multiple times, using different models and/or prompts each time, to check for errors and inconsistencies. That will take more time and cost more, but it can reduce the error and hallucination rate significantly.

refurb 14d ago

Thanks for sharing the details. Does the attorney check that the AI copied the data accurately? Or is it just assumed to be correct? Your experience mirrors my own. AI is great for parsing data that can take up a huge amount of time. My only concern is whether or not it’s done accurately. I wouldn’t use it for anything where mistakes cause serious consequences.