13 ms·
Creating a Modern OCR Pipeline Using Computer Vision and Deep Learning
- milad_nazari 9y ago> "At some point, we will open source and release this synthetically generated data for others to train and validate their own systems and research on." Glad to hear this.
- amelius 9y agoWhy do they use mechanical turk to train printed text? You could easily build a tool that transforms ASCII into something that looks like photographed printed text, with different levels of quality (using e.g. erosion and dilation, different fonts, etc.)
- menacingly 9y agoIsn't that training for a different task, though?
- dangrossman 9y agoThey were using Mechanical Turk to evaluate existing OCR systems before they started at building their own. If you read further into the article, they did exactly what you suggested to train their own system: "A corpus with words to use...A collection of fonts for drawing the words...A set of geometric and photometric transformations meant to simulate real world distortions...The generation algorithm simply samples from each of these to create a unique training example."
- deleted 9y ago[deleted]
- BHSPitMonkey 9y agoThey discuss exactly this, beginning with the third paragraph under "Word Deep Net". Did you stop reading before then?
- greenpizza13 9y agoKeep reading, you might be pleasantly surprised.
- computerex 9y agoYes. This is called data augmentation by synthesis. They used this.
- whatever_dude 9y agoAll this and yet they still can't implement support for a .dropboxignore file
- dingo_bat 9y agoYou can just place any such file outside the Dropbox directory. Doesn't seem like a big deal.
- Rotten194 9y agoIt's very frustrating if you keep your programming projects backed up through Dropbox, and want to back up the code but not gigabytes of node_modules and python virtualenvs. Not that Dropbox has any incentive to make it easier to use less space...
- pault 9y agoIsn't git or mercurial more appropriate for archiving programming projects?
- Nullabillity 9y agoFor archival, yes. But it's not rare that I want to swap computer without committing WIP changes.
- parthdesai 9y agoWouldn't a branching do the trick? checkout a WIP branch and push it. You can pull it on any other computer you want, keep working on it till you are comfortable enough to merge back on master/develop.
- whatever_dude 9y agoWhile that sort of works and I do that, it's a much more cumbersome solution. I commit WIP code to a separate branch, change, and now if I go back to the original computer, I need to make sure to push, go back, pull again... god forbid I had done interactive rebasing or amended a commit, since it'll be all of of sync... it's just more more complicated than simply saving a file and having it reflected. "Temporary commits" should be a rare thing, not mandatory. I end up using both git and dropbox for some projects and it mostly works, but having projects on dropbox is a mess because of node_modules, having sync running continuously because some tools/projects/IDEs will keep generating temporary files, etc.
- brightball 9y agoHadn't really used the document scanner yet...but it sounds like I might have to now. If they keep moving on this it's going to replace my primary use of Evernote...and since Dropbox has a Linux client but Evernote doesn't it'll end up being a pretty easy switch. Are there any Evernote -> Dropbox Paper migration tools out there?
- webscalist 9y agoWhy would a personal backup company invest in OCR? > To peek at personal data
- nielsbot 9y agoA more innocent explanation would be "make your scans searchable" (Not saying you're wrong)
- blacksmith_tb 9y agoUnless you're encrypting it first (via Cryptomator etc.), they already have lots of your data in easier-to-peek-at formats. You could always do your own OCR, and encrypt the results before storing it with them - though I accept that most won't, or use encryption at all for the files they keep in Dropbox, gDrive, etc.
- barkingcat 9y agoThis statement is obvious. I pay Dropbox specifically so they can hold my data and yes - look at my personal data. And by OCR'ing it make my data even more understandable and searchable. How is this a concern when that is the purpose of the company's software?
- kinkrtyavimoodh 9y agoI understand that you are fine with them looking at your data, but how exactly is it obvious that the primary purpose of a company you are paying to backup your data is to mine that data?
- acdha 9y agoNormal people like to find the things which they store. If they're storing images, 99% of the time there won't be much metadata or text alternatives available — it's the same reason why everyone is investing in image classification and face recognition so you can search for “uncle bob” and find pictures which have them.
- ggg9990 9y ago
- philipkglass 9y ago10 years ago I used Abbyy Finereader to add an OCR text layer to about 500,000 pages of scanned PDF documents. At the time it appeared to be the best OCR software available, at least of the options I could try. Later I played around with the open source Tesseract OCR, but it was way behind commercial desktop packages in accuracy/usability. It lagged especially in dealing with different document layouts. What's the current leader in desktop OCR? Has deep learning yet revolutionized the "traditional" OCR market for desktop document processing, or is it just making it easier for companies like Dropbox to quickly get something reasonably good working?
- pault 9y agoI had to research the commercial OCR market recently for a client project. It's abysmal. Most of the solutions are horrendously overpriced, windows only enterprise packages, and the ones that I was able to try out bordered on unusable (bad results and terrible UI). Since OCR is basically "hello world" for tensorflow, I don't understand why these incumbents haven't been wiped off the map.
- bflesch 9y agoBecause for example Abby replaces the image with an PDF with invisible text overlays. I think it is a bit more complicated once you account for the formatting of documents.
- milesokeefe 9y agoAbbyy has had that ability for a longer period of time so it may be more accurate, but this is something tesseract supports[1]. In fact I'd say most OCR systems support it with varying degrees of accuracy. [1]:https://github.com/tesseract-ocr/tesseract/wiki/FAQ#how-do-i-produce-searchable-pdf-output https://github.com/tesseract-ocr/tesseract/wiki/FAQ#how-do-i...
- deleted 9y ago[deleted]
- saadem3000 9y agoOpen Source or it didn't happen.
- Ninn 9y agoJust like autonomous driving isn't happening cus its not open source?
- Mathnerd314 9y agocomma got open-sourced: https://github.com/commaai/openpilot https://github.com/commaai/openpilot
- Ninn 9y agoOnly due to the fact that geohot got destroyed legally peddling his untested bullshit basic ML with no responsibility for the real world side effects the work might have.
- deleted 9y ago[deleted]
- arnioxux 9y ago> Our Word Detector was also a significant bottleneck. We ended up essentially rewriting OpenCV’s C++ MSER implementation in a more modular way to avoid duplicating slow work when doing two passes (to be able to handle both black on white text as well as white on black text); to expose more to our Python layer (the underlying MSER tree hierarchy) for more efficient processing; and to make the code actually readable. Was this ever contributed back to opencv?
- qume 9y agoLooking forward to seeing this Dropbox. If someone there is reading this, please release this code in any way you like, even if you think it's too rushed or ugly, i'm sure the community can clean it up.
- turblety 9y agoOpenCV is licensed under the BSD license. [1] "BSD (unlike some other licenses) does not require that source code be distributed at all." [2] Looks like there is no legal obligation to provide their source code or changes. But I really hope they did push their changes upstream. It is a real shame when companies that benefit so much from the open source community don't give back. 1. https://github.com/opencv/opencv/blob/master/LICENSE https://github.com/opencv/opencv/blob/master/LICENSE 2. https://en.wikipedia.org/wiki/BSD_licenses https://en.wikipedia.org/wiki/BSD_licenses
- aidenn0 9y agoEven under GPL there would be no requirement, since this happens on the server, not the client. The AGPL is the only license I'm aware of that places this requirement when not distributing the software.
- wiz21c 9y agoHopefully, those who miss that Dropbox code today may think about using a more powerful license (ie AGPL) on the code they'll write themselves in the future so that they make sure their users will contribute back like they want DropBox to. :-)
- WalterBright 9y agoI'd love to see a system that worked on handwriting. With all the image recognition progress in the last few years, I never see a mention of recognizing handwriting.
- derefr 9y agoIIRC, Evernote will index—though not transcribe—handwritten text.
- tobltobs 9y agoCharacter separation is much harder, bordering on impossible, with handwriting. Also look at all all the handwriting where you need to understand the context to eg. differ a 'm' from an 'nn'.
- WalterBright 9y agoI thought facial recognition would border on the impossible, but that seems to be a solved problem today. And yes, context is needed to recognize cursive. At the very least a word frequency combined with a dictionary should get "most likely" recocgnitions. For example, my OCR software (for printed text) often recognizes ? as }. Come on.
- criddell 9y agoI use two apps on my iPad that do a scarily good job of recognizing my handwriting. They are Nebo and GoodNotes. Neither app was expensive. I mostly use GoodNotes. I don't convert my handwriting to text, but it is nice that I can search my notes.
- saagarjha 9y agoThere a couple of apps that do this, notably Evernote and Goodnotes. Recently, the built in Notes app on iOS has added this feature as well, but it's still a bit finicky.
- WalterBright 9y agoDo you mean recognize text as you draw it, or recognize text in a scanned image?
- evv 9y agoAre there any fully-offline alternatives to this? What is the most capable open-source solution?
- SloopJon 9y agoIt's pretty remarkable that they were able to build a system more or less from scratch that "gave us the accuracy, precision, and recall numbers that all met or exceeded the OCR state-of-the-art." There was a recent post on the Google Developers blog about a similar project to recognize bottle-cap codes in a mobile app: https://developers.googleblog.com/2017/09/how-machine-learning-with-tensorflow.html https://developers.googleblog.com/2017/09/how-machine-learni... HN discussion: https://news.ycombinator.com/item?id=15489176 https://news.ycombinator.com/item?id=15489176
- ksk 9y agoVery interesting post. Is there any benchmark data showing how their system compares to the competition?
- cardigan 9y agoA better baseline than an out of the box OCR engine is applying some computer vision techniques before using an out of the box OCR engine. This can significantly out perform a pure OCR engine approach. (speaking from experience working on startups doing this kind of stuff for ~10 months) One dumb but surprisingly effective thing to try is apply a bunch of random binarizing filters before putting your text into an OCR engine, then picking the most common output from those filtered images. Some combination of morphological operations (dilates/erodes in different orders and different strengths), different levels of blur and sharpening, adaptive binary threshold at different levels, resizing the image (hilarious but some OCR engines are sensitive to relative scale), adding white borders of different thicknesses, rotating by small angles, denoising, ... This gave us ~95% word level accuracy on our dataset (with tesseract as the OCR engine), without tuning the filters (we had used the filters individually before with some brittle logic for deciding when to use them: the random filter approach was kind of a joke which ended up working). Tesseract on its own had an abysmal ~40% word level accuracy on that same dataset. Dataset was of words segmented out of scanned bank statements. Main downside was that this was pretty slow (we were generating ~10k filters per word), but we found a way to make it work (hierarchical stacked AWS Lambdas, lol).
- wwarner 9y agoc'mon -- that's the easy way!
- Jhsto 9y agoSelfless plug, but for some examples of these see a side project of mine: https://www.juusohaavisto.com/northern-nike-nabob.html https://www.juusohaavisto.com/northern-nike-nabob.html
- justonepost 9y agoOr post processing with domain specific knowledge. eg, https://www.taggun.io/ https://www.taggun.io/ using google vision for receipts and then applying some ml, nlp, etc.
- rb2018 9y ago...and the result was better than what you could buy from Google cloud vision or ocr.space out of the box?
- tudorw 9y agoseems like a lot of value locked away in these training sets, is there a commercial platform to buy and sell trained sets yet ?, answering my own question, yes, https://algorithmia.com/ https://algorithmia.com/ any others ?
- Roritharr 9y agoI'd kill for a self-hostable solution like that. I just don't have the resources to spend like Dropbox. Can anyone recommend a poor man's approach to this? I can afford a lot of AWS instances, but not the 8 months of engineering.
- romuloab42 9y agoNot self-hostable, but Google Vision API OCR was pretty good when I used last time. AWS also has a offering for that, but I haven't personally tested.
- ocrcustomserver 9y agoI'm offering a self-hosted solution, mail is in profile.
- ayw 9y agoSuper cool! Solving problems like DropTurk with stronger guarantees around turnaround/quality/confidentiality is exactly what we do at Scale. Would love to hear more about DropTurk’s experiences around building this internally and how it compares to our product: https://www.scaleapi.com/ocr-transcription https://www.scaleapi.com/ocr-transcription
- misiti3780 9y agoDropturk sounds useful, any plans of open-sourcing it ?
- verelo 9y agoWas this done by the Endorse team Dropbox acquired a few years back? I know they were big into this same line of work.
- jccalhoun 9y agoI hope some better OCR comes along. I use Nuance and either I have confused it with my training file or it makes some very dumb decisions. It seems like is should scan something that it sees as "wi1l" or "s0mething" and realize that it isn't very likely that someone would put a number in the middle of a word or put random superscript in the middle of a word.