17 ms·
Examples to compare OCR services: Amazon vs. Google vs. Microsoft
- cdolan 7y agoI was hoping that they had taken the time to test tables, nested tables, and irregular table rows with headers every so often. A good example of what I’m talking about are invoices. In my line of work we extract data from thousands of different types of invoices that are the only place to find key data from your services provider (waste & recycling bills, soon other parts of a building’s expenses), and normalize that information. There is a long-tail of industries that are years away from having an API to transact information…but that information is available on their monthly statements in some sort of crazy table format! In my experience Amazon Textract has been the best in terms of processing speed, ease of use, and table extraction accuracy. However post processing is almost always needed with any OCR implementation. Edit: Its important to note that Microsoft and Google don’t even support table extraction in the APIs listed in this article!
- tiernano 7y agojust on the note of tables, have you tried the table extraction and OCR on Microsoft Excel on iPhone? It may also be on Android, i just tried it on iPhone, and man! it works great!
- cdolan 7y agoI have - its really good but for our use case (thousands of PDFs a day) that wouldn’t be reasonable :-(
- andrejk 7y agoAzure has a separate service to read forms and formatted docs. https://azure.microsoft.com/en-us/services/cognitive-services/form-recognizer/ https://azure.microsoft.com/en-us/services/cognitive-service...
- bpchaps 7y agoDo you know how good it is? I have a LOT of structured documents that I need to OCR.
- CorneliaKara 7y agoMSFT person here - give it a try! sign up, and you get a free trial that can allow you to easily benchmark.
- Havoc 7y agoInteresting. That is actually something I've been looking for and I do have a msdn sub. fyi that defaulted to indian rupee as default currency for me (UK based & zero indian connections). Weird
- dav43 7y agoHave you tried the tabula free program? I use it for some finance work reading filings. http://tabula.technology/ http://tabula.technology/
- bpchaps 7y agoYep! It's great, but is maybe 60% there, so I'm looking for something that can extract much more structure from a document. I doubt what I'm looking for will exist for another 10 years, though.
- leeoniya 7y agois it feasible to create loose templates for where the data is and extract that way? i have a mothballed project that did pretty well. it was able to discern different templates from a mass of documents.
- ocrcustomserver 7y agoI'm curious, if you email me a sample I can tell you what's possible.
- jordoh 7y agoThat's a good point not explained in the article: there are a huge number of use cases for OCR. In this case, the use is extracting words that can be used in full-text search, so structural extraction isn't a key criteria. Edit: and now it's hopefully clarified in the article itself. :)
- ocrcustomserver 7y agoIn this case, the use is extracting words that can be used in full-text search, so structural extraction isn't a key criteria. In case someone wants to know more, the former is known as "full page OCR" and the latter as "data capture"/"document processing" (or IDP, intelligent document processing). Full page OCR for machine printed text is considered a solved problem (but not for handwritten text). Data capture is hard to do and involves extracting specific fields from documents. The first big cloud company going into data capture territory was Amazon with AWS Textract (calling it OCR++). There's also Document Understanding AI (Google) and Azure Form Recognizer in Beta, as mentioned by others in this thread. The big 3 RPA companies (UiPath, Automation Anywhere, Blue Prism) have also gone into data capture (calling it cognitive or intelligent RPA). ABBYY (with FlexiCapture) and Kofax (who recently acquired Nuance's imaging division, the 2nd most popular OCR engine after ABBYY's) are the traditional IDP players.
- formercoder 7y agoThis comes up a lot with financial statements and government data as well. The government stuff is getting much better though.
- eitally 7y agoSo does Google Cloud: https://cloud.google.com/solutions/document-understanding/ https://cloud.google.com/solutions/document-understanding/
- cdolan 7y agoThnaks - looks like table extraction is in Alpha. I submitted a request to check it out!
- itcrowd 7y agoI found a relatively recent (Feb. 2019) comparison of OCR software and services [1]. It may be interesting to you, especially the "Texas Campaign Filing" comparison, which seems to have (nested) tables. [1] https://source.opennews.org/articles/so-many-ocr-options/ https://source.opennews.org/articles/so-many-ocr-options/
- sandGorgon 7y agoWhat kind of post processing? I'm not an expert, but usually we have done a lot of preprocessing (like binarization,etc) before we call cloud ocr services.
- ocrcustomserver 7y agoWhat is the accuracy that you're getting from it?
- SethTro 7y agoThey tested a single image of each type. :/
- tiernano 7y agounless i am having a hard time reading the text on the site, they all are $1500 for 1M images, but Amazon Textract, which has the second lowest average result, is "cheapest" compared to Azure Congnitive Services, which has the highest average result... Did i miss something? How is 4 items, all the same price, ranked arseways?
- bduerst 7y agoWould have been nice to see % accuracy as well as or instead of just % words matched. It seems like the author decided to forego false positives. You can train models that are good at finding words in images but can have terrible accuracy at perceiving the actual text of the word.
- rhizome 7y agoWhat distinction are you making? I think it might be a little too subtle written that way.
- ocrcustomserver 7y agoTypically OCR accuracy is measured in two ways, CER (Character error rate) and WER (Word error rate). If just one number is provided, it's typically CER. "Finding words in images" is a bit ambiguous. It can mean "word spotting" (intended for retrieval rather than transcription) but also "text segmentation" (part of preprocessing step before OCR).
- fartcannon 7y agoMaybe compare to some of the free-non-hosted options? Like where are we with OCR? Last I checked it was CTC magic. Any progress?
- steventhedev 7y agoTesseract[0] is the classic example. There's a bunch of advice for improving your accuracy with it, like making your images larger (literally just scale it up x2 or x4). It would be interesting to the benchmark from the article repeated with different scaling options (or other preprocessing, depending on platform). [0]: https://github.com/tesseract-ocr/tesseract https://github.com/tesseract-ocr/tesseract
- syntaxing 7y agoTesseract works really good for literal OCR but I haven't had much luck before with more common work (like documents with tables and such). Has anything changed as of late?
- jacobolus 7y agoSince October there is a new version, v4: “Added a new OCR engine that uses neural network system based on LSTMs, with major accuracy gains. [..] Added trained data that includes LSTM models to 123 languages.” No idea how well it does for structured content like tables. There seems to be a recent v2 of a javascript port (i.e. Tesseract v4 compiled to wasm), if anyone wants to do OCR in their browser: https://observablehq.com/@tmcw/tesseract-js-v2-alpha https://observablehq.com/@tmcw/tesseract-js-v2-alpha https://github.com/naptha/tesseract.js https://github.com/naptha/tesseract.js
- jordoh 7y agoRunning tesseract (4.0.0 using the LSTM engine) on the same images leaves a lot to be desired for handwriting, but does well on the (non-handwriting) website image (the source images are linked in the "OCR Image Processing Results" section).
- DanHulton 7y agoDang, I was hoping to see a comparison to ABBYY's FineReader Online in there: https://finereaderonline.com https://finereaderonline.com Maybe in the 2020 review?
- jordoh 7y agoI tried running the source images through FineReader Online, but the images with handwriting resulted in "was not processed: the recognized document contains errors". The website image worked, but was missing a few elements, like the other headings on the line with "Minimalist editor".
- ysleepy 7y agoI wonder if some implementations just use nearest neighbour words to increase accuracy in the common case of normal text. - Decreasing performance on random strings considerably. The tested corpus only contains relatively common words, so this aspect is not tested.
- jordoh 7y agoHave a specific image you'd be interested in seeing tested? The article only contains a few examples that could be freely used, but images with sparse random text (e.g. [1]) do tend to have good results across all the services. [1] https://www.gettyimages.com/detail/news-photo/ken-griffey-jr-of-the-seattle-mariners-makes-a-hit-during-news-photo/91124776 https://www.gettyimages.com/detail/news-photo/ken-griffey-jr...
- WalterBright 7y agoIt's nice to see progress on handwriting recognition.
- georgewsinger 7y agoVicarious has text recognition technology that is invariant to, i.e., letter spacing, but they haven't released it due to fear of CAPTCHA fraud. See, i.e., [here](https://i.imgur.com/lN4AzmE.png https://i.imgur.com/lN4AzmE.png). I wonder if they could modify it to release a near 100% accurate OCR service?
- ipsum2 7y agoLetter spacing seems like an arbitrary distinction. IF you design a neural net that contains operators that make certain things invariant (e.g. size) then of course it would perform better than a neural net without it.
- catchmeifyoucan 7y agoAre there any open source alternatives that are competitive in this space?
- noahster11 7y agoGoogle is behind tesseract[0], not sure if they use this for Google Cloud Vision OCR [0]: https://opensource.google.com/projects/tesseract https://opensource.google.com/projects/tesseract
- ocrcustomserver 7y agoAccuracy of Google OCR vs Tesseract v4 (on books and web): https://imgur.com/a/Tj0TASf https://imgur.com/a/Tj0TASf
- johnmalatras 7y agoMy app, Quotable, uses Azure if you were curious about perf. Works great!
- georgewsinger 7y agoFor what it's worth the Wolfram text recognition function does a poor job parsing the whiteboard photo: http://www.wolframcloud.com/obj/user-900a994f-78ab-4931-b18e-353eff4d0b33/text_recognition.png http://www.wolframcloud.com/obj/user-900a994f-78ab-4931-b18e... (Not a critique of Wolfram technologies here; their services are otherwise amazing).
- aw3c2 7y agoFYI that link just gives a log-in screen.
- danso 7y agoOne of the things that Textract purports to do is also detect structured data, e.g. if the scanned images has tables, like a spreadsheet. I tested it out when it became available to the general public, and included the API call and JSON response for their default and relatively simple example: https://gist.github.com/dannguyen/1a71c8fb98ddd1df6abdc08bc746397a https://gist.github.com/dannguyen/1a71c8fb98ddd1df6abdc08bc7... I tried it against what I consider one of the harder real-world types of scanned tabular data, a Senate personal financial disclosure form [0], and it didn't do great. In fact, I found that it did substantially worse than ABBYY FineReader. Or rather, ABBYY did much better than I would've expected for any software, even managing to read the vertically-oriented column headers accurately [1]. [0] https://gist.github.com/dannguyen/1a71c8fb98ddd1df6abdc08bc746397a#is-textract-our-saint-and-savior-of-pdf-to-csv-data https://gist.github.com/dannguyen/1a71c8fb98ddd1df6abdc08bc7... [1] https://github.com/dannguyen/abbyy-finereader-ocr-senate#less-simple-table https://github.com/dannguyen/abbyy-finereader-ocr-senate#les...
- ndm000 7y agoI’m new to the OCR space. Given the the successes in self-driving (given no systems are production ready, Waymo and Tesla have MVPs), what makes reading text so hard that cloud providers struggle to have human-level accuracies?
- visarga 7y agoMany fonts, characters, languages and low quality scans.
- rhizome 7y agoSame problems as with self-driving cars and speech recognition: it's very very hard (if not impossible) for software to parse the universe of scenarios. Said another way, it's never accurate enough to "work" when it's trained on the outside world. Usable output requires the kind of accuracy you only get when the process is trained in controlled environments (limited "vocabulary": fonts, layouts, languages, words, accents, geography, etc.).
- jeffrogers 7y agoHow does iOS Vision + Core ML compare?
- dzink 7y agoToutanova, one of the co-authors of the BERT paper worked for Microsoft for 12+ years until shifting to Google recently.
- LeonB 7y agoComparison should include the Apple Newton.
- mkl 7y agoThat couldn't do OCR at all, right? It had no camera. It could do handwriting recognition on stylus input (where it has full perfect stroke data, not just pixelated images of strokes), which is a very different and easier problem.
- ocrcustomserver 7y agoThe Apple Newton was capable of online OCR (doesn't have to do with internet connectivity in this case). As you mention, online OCR is when you input the strokes directly on the device vs offline OCR where the input is an image. Some trivia: The first version of handwriting recognition engine of the Newton was developed by ParaGraph International (founded by the founder of Evernote). Another version (Print Recognizer) was later developed by Apple. https://web.archive.org/web/20120324055221/http://www.beanblossom.in.us/larryy/Yaegeretal.AIMag.pdf https://web.archive.org/web/20120324055221/http://www.beanbl...
- m0zg 7y agoThis is one of those things that badly needs an open source solution, and for which technology exists to solve it really well, but nobody wants to do it because it's a ton of really boring, high-maintenance work.
- poxrud 7y agoOnce a year when I do my taxes I go through every single bank transaction and classify it into a proper column in a spreadsheet. The problem is that my bank provides CSV transaction data for the past 6 months only. But it does provide years of bank statements in PDF format. I decided I can save a lot of time by using Amazon Textract to extract the tables from the PDF's and convert them to CSV files. The problem is that while Textract works really well for well defined tabular data it does not work for tables where the rows and columns are implied with white space, instead of lines. When I reached out to aws they confirmed this problem and suggested that I draw the table lines into the PDF and then run textract again on this modified pdf. This felt like a dirty hack so I did not proceed with this suggestion. Textract is a great tool when it works well, but unfortunately when it doesn't there are no ways to make adjustments in order to improve the results. In the end I managed to complete my project and get a lot better results by using the excellent Camelot Python table extraction library.
- ocrcustomserver 7y ago"The problem is that while Textract works really well for well defined tabular data it does not work for tables where the rows and columns are implied with white space, instead of lines." This is what Tabula and Camelot call "Stream" and "Lattice" parsing methods. Whitespace between cells is Stream, demarcated lines is Lattice.
- hn23 7y agoMalicious gossip has it that you have to perform these tests with Google/Amazon when the Mechanical Turk is not sleeping.
- gbolcer 7y agoNice! I did the same thing for ASR technologies a while back. Results here. https://drive.google.com/a/bolcer.org/file/d/1CJTHikHldMYTMvhDsfx7dr-IXUTrlp-8/view?usp=drivesdk https://drive.google.com/a/bolcer.org/file/d/1CJTHikHldMYTMv... (methodology: same parts of speech analysis on exact transcripts ran through different services). All OCR services, BTW, have the same threading problem. They work really well for sequential text, but as soon as you start getting into more complicated "marketing" formats, they don't work at all. They best use of the online OCR services I found was figuring out published dates for news articles that typically don't show up in anything other than images. Even w/ both Azure, AWS, and Google, I still needed a post-capture regex to figure the stuff out. https://drive.google.com/a/bolcer.org/file/d/16vujemgD91EbuuI98KCXjUuIGHBGR_3e/view?usp=drivesdk https://drive.google.com/a/bolcer.org/file/d/16vujemgD91Ebuu... We've got a long ways to go.
- citizenpaul 7y agoI did this a while back with speech to text services. Microsoft's offerings by far beat out the others in terms of accuracy and performance.