2 ms·
You are talking from a top-down point of view. Of course they can make an error any time, after all they stochastically sample the output. All approaches to inf
by visarga 3y ago
You are talking from a top-down point of view. Of course they can make an error any time, after all they stochastically sample the output. All approaches to information extraction are upper bounded at 90-95% accuracy, I have extensive experience with this task and run many evaluations on invoices, receipts, forms and other doctypes. Human in the loop is still required.
But in practice you can rely on good copying and excellent ability to parse names, addresses and various values as good or better than other approaches. LLMs reduce the number of corrections, and they are easier to deploy - instead of labeling hundreds of documents you can just query with the field name.
- z3c0 3y agoI'm talking about an "at scale" point of view. A 90-95% success rate would be at worst 9:1 odds and at best 19:1. That means your best case scenario would be a likely failure on every 20th inference. This may be passable on an adhoc individual basis, but why do this when a deterministic solution can achieve beyond 99%, with proper error handling? Data normalization tools are rigid as a feature - LLMs are not, even at a temp of 0
- pvman 3y agoThanks for the info z3c0. Have you implemented open source solutions? Do you mind sharing some deterministic solutions to detect entities without huge dictionaries (NER)? Or maybe extractive QA if you used that instead?
- z3c0 3y agoDeterministic? Not anything that is "one size fits all". If your documents are of an unpredictable shape, then ML is your best bet. For both NER and QA, BERT models are very capable. To your first question, most of my ML work is proprietary, unfortunately. I am hoping to change that in the near future. Edit: spaCy is a great library for NER, if you're hoping for an open-source solution that can have you hitting the ground quickly. SQuAD for QA.