10 ms·

Tesseract.js: Pure JavaScript OCR for 100 Languages

364 points by petercooper 7y ago

smashah 7y ago
Ooh exciting! The main reason why I've needed to leverage the GCP vision API is due to orientation limitations on local OCR. I'll test and migrate to this soon depending on accuracy. Great job so far.
umvi 7y ago
I recently did a project where I OCR'd a very rare book that I could only find in the library of congress so I could read it on my kindle. Tesseract was amazingly powerful and accurate, but it seemed to struggle if the page was warped or tilted even a little. I had to preprocess the images heavily to try to dewarp the natural spine curvature, and even then it could only get about 99% accuracy (which sounds like a lot, but consider a book where every 100th letter was wrong - I basically flagged the errors on my kindle as I went along and manually corrected them later). I guess the point of this comment is that, in my experience, Tesseract.js is probably going to need an accompanying PageDewarp.js for it to be of use scanning books. Not everyone has access to a right angle scanner or can slice the spine and get perfectly straight high-res scans.
- superpermutat0r 7y ago
  That's very interesting given that Tesseract uses Leptonica. I'm not sure if they use it for dewarping but all of my little projects with Leptonica really worked well. Dewarping, binarizing, extracting individual elements etc. https://github.com/DanBloomberg/leptonica https://github.com/DanBloomberg/leptonica
  - umvi 7y ago
    Maybe I wasn't using tesseract to its fullest potential, but I had a really hard time getting it to do accurate OCR on warped paged - straight pages worked perfect.
- dunham 7y ago
  Also, the last time I checked tesseract liked stuff to be 200-300 dpi. (You don’t have to scan it at that resolution, but helps if you scale it up.) Years ago, I dug up a couple of papers on the spine thing, but never got around to implementing it. I think you can estimate the curvature based on the shadow and dewarp. I was just scanning some recipes to save typing, so it wasn’t really worth the efffort.
  - umvi 7y ago
    I ended up using this guy's tool[1] for dewarping, which worked pretty well. The tool was more a prototype than anything, but it was enough to finish my project. [1] https://mzucker.github.io/2016/08/15/page-dewarping.html https://mzucker.github.io/2016/08/15/page-dewarping.html
- khalilravanna 7y ago
  I'm curious what the book was and the context for why you were reading it.
- NicoJuicy 7y ago
  Why didn't you combine it with a dictionary and then ML in it to correct the letter based on the context?
madsohm 7y ago
Nice project! Tried it with a screenshot from an eBook. Unfortunately the Os became Qs and the Is became |s.
- umvi 7y ago
  You never realize how similar characters are until you start an OCR project. ec ij tf Il|1 hb OQ etc. Even the tiniest addition or subtraction of printed ink can transform one character in one of the above rows to any other character in the row. Throw in page tilt/warp/etc. and OCR can frequently confuse them unless you train it specifically on your text. The pipeline I've found that works best is: image -> upscale -> dewarp -> OCR -> spellcheck -> grammar check
  - magicalhippo 7y ago
    I too have been very disappointed in Tesseract for "simple" OCR (converting subtitles). For communication, Turbo Codes[1] for example have the decoder produce an integer value for each bit, rather than just a bit. The value is a measure of how likely the value is 0 or 1. This is then used with previous bit values, which includes parity data, to make a "hard" decision. I wonder if something similar has been tried for OCR? I imagine the OCR front-end could feed a number of probable hits, along with confidence, into a spellchecker. Or something along those lines. [1]: https://en.wikipedia.org/wiki/Turbo_code https://en.wikipedia.org/wiki/Turbo_code
crazygringo 7y ago
In case it's not clear, Tesseract is developed by Google since 2006, having been started at HP in 1985 and open-sourced by HP in 2005. [1] As far as I know, it powers all OCR at Google (e.g. in Keep, Docs, etc.). This (Tesseract.js) is a WASM port of the project by a separate group of people. I investigated using this port a couple years ago, but as you can see from the demo, it's fairly slow to initialize and run, so I never found a practical use for OCR client-side rather than server-side, but I still think it's tremendously cool. In case anyone's interested (shameless plug), because I do a lot of academic research that involves tons of copying from webpages, PDF's and screenshots and pasting into notes documents, I created a tool at https://pastemagic.com https://pastemagic.com that helps selectively remove rich text formatting, remove line breaks and does OCR on screenshots and camera photos. Setting up Tesseract on my server and creating a simple HTTP endpoint for it took less than an hour, and for free I had OCR as powerful as Google's. Pretty cool I thought. [1] https://github.com/tesseract-ocr/tesseract https://github.com/tesseract-ocr/tesseract
- bhanhfo 7y ago
  > As far as I know, it powers all OCR at Google (e.g. in Keep, Docs, etc.). Afaik Google no longer uses Tesseract for any of its products. Googles Clould OCR is much better than Tesseract. I think Google devs still work on Tesseract, but only as their side project (not sure about this, obviously)
  - crazygringo 7y ago
    Oh very interesting. I'd verified the output was identical a couple of years ago, and that Keep and Docs in production were using the 4.0 beta release at the time. But if Cloud OCR is better, makes sense they would have switched since then. Tesseract 4.0 has a brand-new neural engine that totally supersedes the earlier engine, however -- I wonder if there's any relation between that and Cloud OCR?
  - ComputerGuru 7y ago
    “Cloud OCR” is an interface. Something is still doing the OCR behind the scenes (and that may indeed not be Tesseract).
petercooper 7y ago
The reason I posted this is that they've just released v2.0. This isn't highlighted on the homepage, but I assume it has some significance to the project overall.
sankyo 7y ago
Does it work only on books and magazines or would it work on a driver license or ID card as well?
- bhanhfo 7y ago
  Tesseract is optimized for images with white backgrounds. ID cards or movie screenshots do not work well.
  - Certified 7y ago
    I have used tesseract ocr combined with imagemagick and ffmpeg to great success for video text extraction.
    - beagle3 7y ago
      Can you list your script/pipeline? I haven't had much success (though, I only ran ffmpeg's internal tesseract OCR[0], no imagemagick processing or any other processing in between) [0] https://ffmpeg.org/ffmpeg-all.html#ocr https://ffmpeg.org/ffmpeg-all.html#ocr
- crazygringo 7y ago
  Before OCR'ing it converts to black-and-white using a brightness threshold. Keeping an evenly lit background is particularly important, because otherwise a shadow area can easily fall under the threshold of all-black. A license or ID will almost certainly have medium-contrast elements in the background that will show up as dark. But if you were able to manipulate the contrast/brightness appropriately in advance, you could probably get it to work.
bhanhfo 7y ago
Related Chrome extension: https://chrome.google.com/webstore/detail/project-naptha/molncoemjfmpgdkbdlbjmhlcgniigdnf https://chrome.google.com/webstore/detail/project-naptha/mol... Another extension (not using Tesseract.JS): https://chrome.google.com/webstore/detail/copyfish-%F0%9F%90%9F-free-ocr-soft/eenjdnjldapjajjofmldgmkjaienebbj?hl=en https://chrome.google.com/webstore/detail/copyfish-%F0%9F%90...
throwaway_ocr 7y ago
I tested this by taking a screenshot of the introduction blurb. This is what it came up with: Tesseract s is 2 pure Javascript port of the popular Tesseract OCR engine. This library supports more than 100 languages, automatic text orientation and script detection, a simple interface for reading paragraph, word, and character bounding boxes. Tesseract s can run either in 2 browser znd on & server with NodeJS. Not bad, but far from useful.
EGreg 7y ago
I remember reading that Tesseract is the old tech which does worse than current ML based stuff. But since no one gives out their ML data that’s all you get.
- mkl 7y ago
  Your knowledge is out of date. Tesseract 4 added a new OCR engine based on LSTM neural networks.
atum47 7y ago
This is just amazing. I wonder how much work and how many brilliant people it took to develop this. Congratulations to everyone involved. I'm sure a lot of cool stuff will come from this tool.
gwbas1c 7y ago
I'm curious about the performance tradeoffs of Javascript versus native code? A few weeks ago, I tried writing something that was long-running, CPU intense, ect, in a webworker. It was so darn slow that I switched to a native language. (I hope I didn't do something silly that made my code run more slowly than it should.) I see some mention about running in WASM. Does this do something like have ordinary Tesseract compiled for WebAssembly and then fallback to Javascript?
- fyp 7y ago
  A while back I ported several c++ libraries with emscripten and it's usually just 2-10x slower than native. It gets maybe another order of magnitude worse if you're a porting a library that heavily relies on vectorization which isn't available on the web.
gwd 7y ago
What's slightly weird is that the Chinese "example" text mis-reads a character in the first line. The image shows: 冬日平泉路晚归 But the OCR reports: 冬日平柳路晚归 (Note the different character right in the middle)
- bufferoverflow 7y ago
  I tried using Tesseract around a year ago to recognize digits only, very clean images, not some weird font. I had thousands of images, and it failed in around 3% of them. It was so weird, as it would recognize the same digits in other images just fine. I tried 4 or 5 different OCR programs, and none of them worked well enough for my case. I was actually surprised, I thought OCR was a solved problem with ridiculously low error rates.
  - mkl 7y ago
    In my experience Tesseract can get confused near the edges of images, and padding with a wide white border helped a lot. Strange that none of the OCR programs were good enough, though.
dbhattar 7y ago
Even though it says it supports 100 languages, I cannot find the list of supported languages. I am mostly trying to find out if it supports Indic languages.
- severine 7y ago
  https://github.com/tesseract-ocr/tesseract/wiki/Data-Files#data-files-for-version-400-november-29-2016 https://github.com/tesseract-ocr/tesseract/wiki/Data-Files#d... Seems so, lists at least Hindi, Urdu, Bengali, Sanskrit, Urdu, Nepali, Marathi, Sinhala and Punjabi!
dang 7y ago
A thread from 2016: https://news.ycombinator.com/item?id=12694004 https://news.ycombinator.com/item?id=12694004
rolling_roland 7y ago
I investigated tesseract.js for turning images of spreadsheets into data. I didn't mind the initial startup time or run time but unfortunately wasn't able to get good enough accuracy for my case. It seemed to work really well with plain english text though.
ngcc_hk 7y ago
There is another post about tesseract but by python. Same question but I guess it might be closer ... is it compatible with TensorFlow js? or is it ok to run alongside? Or the model can be “simplified” to run on client side like mobile net?
alexcnwy 7y ago
Ugh why does open source OCR still suck so bad?! Why isn’t there an open source OCR engine even half as powerful as the Google Cloud Platform API?!
kbumsik 7y ago
> Tesseract.js wraps an emscripten port of the Tesseract OCR Engine. So it is a wrapper library of a C++ project, which is cool. But saying it is a "Pure" JavaScript is purely misleading.
JeromeWu 7y ago
Wow, I was wondering why there were so many new github stars yesterday, and here I found the reason why. :) Thanks for being interested in tesseract.js, it makes all the work worth. And I have to thank @antimatter15 for creating this library, without him we cannot go this far I have read all the comments and here I would like to provide my two cents for some questions: 1. Is tesseract.js pure JavaScript? Yes, it is 100% JavaScript and it leverages Webassembly port of original tesseract-ocr. (means we compile the C sorce code to JavaScript Webassembly code, powered by Emscripten) 2. The accuracy of tesseract.js is poor. In my experience, it is hard to get perfect results without applying additional techniques to your source images. You may need to some preprocessing and sometimes train a custom traineddata. It is not easy, but it is the price of high accuracy. 3. Cloud OCR service is much more accurate Yes, that's true. But tesseract.js provides an in browser offline option to do your OCR, it is useful for scenarios like PWA and high confidential image content (which you don't want to send to server). Tesseract.js is not a silver bullet, but it is handy sometimes. Hope you enjoy this library and feel free to leave any comment to us!

364 points by petercooper 7y ago

Ooh exciting! The main reason why I've needed to leverage the GCP vision API is due to orientation limitations on local OCR. I'll test and migrate to this soon depending on accuracy. Great job so far.

umvi 7y ago

I recently did a project where I OCR'd a very rare book that I could only find in the library of congress so I could read it on my kindle. Tesseract was amazingly powerful and accurate, but it seemed to struggle if the page was warped or tilted even a little. I had to preprocess the images heavily to try to dewarp the natural spine curvature, and even then it could only get about 99% accuracy (which sounds like a lot, but consider a book where every 100th letter was wrong - I basically flagged the errors on my kindle as I went along and manually corrected them later). I guess the point of this comment is that, in my experience, Tesseract.js is probably going to need an accompanying PageDewarp.js for it to be of use scanning books. Not everyone has access to a right angle scanner or can slice the spine and get perfectly straight high-res scans.

superpermutat0r 7y ago
That's very interesting given that Tesseract uses Leptonica. I'm not sure if they use it for dewarping but all of my little projects with Leptonica really worked well. Dewarping, binarizing, extracting individual elements etc. https://github.com/DanBloomberg/leptonica https://github.com/DanBloomberg/leptonica
- umvi 7y ago
  Maybe I wasn't using tesseract to its fullest potential, but I had a really hard time getting it to do accurate OCR on warped paged - straight pages worked perfect.
dunham 7y ago
Also, the last time I checked tesseract liked stuff to be 200-300 dpi. (You don’t have to scan it at that resolution, but helps if you scale it up.) Years ago, I dug up a couple of papers on the spine thing, but never got around to implementing it. I think you can estimate the curvature based on the shadow and dewarp. I was just scanning some recipes to save typing, so it wasn’t really worth the efffort.
- umvi 7y ago
  I ended up using this guy's tool[1] for dewarping, which worked pretty well. The tool was more a prototype than anything, but it was enough to finish my project. [1] https://mzucker.github.io/2016/08/15/page-dewarping.html https://mzucker.github.io/2016/08/15/page-dewarping.html
khalilravanna 7y ago
I'm curious what the book was and the context for why you were reading it.
NicoJuicy 7y ago
Why didn't you combine it with a dictionary and then ML in it to correct the letter based on the context?

madsohm 7y ago

Nice project! Tried it with a screenshot from an eBook. Unfortunately the Os became Qs and the Is became |s.

umvi 7y ago
You never realize how similar characters are until you start an OCR project. ec ij tf Il|1 hb OQ etc. Even the tiniest addition or subtraction of printed ink can transform one character in one of the above rows to any other character in the row. Throw in page tilt/warp/etc. and OCR can frequently confuse them unless you train it specifically on your text. The pipeline I've found that works best is: image -> upscale -> dewarp -> OCR -> spellcheck -> grammar check
- magicalhippo 7y ago
  I too have been very disappointed in Tesseract for "simple" OCR (converting subtitles). For communication, Turbo Codes[1] for example have the decoder produce an integer value for each bit, rather than just a bit. The value is a measure of how likely the value is 0 or 1. This is then used with previous bit values, which includes parity data, to make a "hard" decision. I wonder if something similar has been tried for OCR? I imagine the OCR front-end could feed a number of probable hits, along with confidence, into a spellchecker. Or something along those lines. [1]: https://en.wikipedia.org/wiki/Turbo_code https://en.wikipedia.org/wiki/Turbo_code

crazygringo 7y ago

In case it's not clear, Tesseract is developed by Google since 2006, having been started at HP in 1985 and open-sourced by HP in 2005. [1] As far as I know, it powers all OCR at Google (e.g. in Keep, Docs, etc.). This (Tesseract.js) is a WASM port of the project by a separate group of people. I investigated using this port a couple years ago, but as you can see from the demo, it's fairly slow to initialize and run, so I never found a practical use for OCR client-side rather than server-side, but I still think it's tremendously cool. In case anyone's interested (shameless plug), because I do a lot of academic research that involves tons of copying from webpages, PDF's and screenshots and pasting into notes documents, I created a tool at https://pastemagic.com https://pastemagic.com that helps selectively remove rich text formatting, remove line breaks and does OCR on screenshots and camera photos. Setting up Tesseract on my server and creating a simple HTTP endpoint for it took less than an hour, and for free I had OCR as powerful as Google's. Pretty cool I thought. [1] https://github.com/tesseract-ocr/tesseract https://github.com/tesseract-ocr/tesseract

bhanhfo 7y ago
> As far as I know, it powers all OCR at Google (e.g. in Keep, Docs, etc.). Afaik Google no longer uses Tesseract for any of its products. Googles Clould OCR is much better than Tesseract. I think Google devs still work on Tesseract, but only as their side project (not sure about this, obviously)
- crazygringo 7y ago
  Oh very interesting. I'd verified the output was identical a couple of years ago, and that Keep and Docs in production were using the 4.0 beta release at the time. But if Cloud OCR is better, makes sense they would have switched since then. Tesseract 4.0 has a brand-new neural engine that totally supersedes the earlier engine, however -- I wonder if there's any relation between that and Cloud OCR?
- ComputerGuru 7y ago
  “Cloud OCR” is an interface. Something is still doing the OCR behind the scenes (and that may indeed not be Tesseract).

petercooper 7y ago

The reason I posted this is that they've just released v2.0. This isn't highlighted on the homepage, but I assume it has some significance to the project overall.

sankyo 7y ago

Does it work only on books and magazines or would it work on a driver license or ID card as well?

bhanhfo 7y ago
Tesseract is optimized for images with white backgrounds. ID cards or movie screenshots do not work well.
- Certified 7y ago
  I have used tesseract ocr combined with imagemagick and ffmpeg to great success for video text extraction.
  - beagle3 7y ago
    Can you list your script/pipeline? I haven't had much success (though, I only ran ffmpeg's internal tesseract OCR[0], no imagemagick processing or any other processing in between) [0] https://ffmpeg.org/ffmpeg-all.html#ocr https://ffmpeg.org/ffmpeg-all.html#ocr
crazygringo 7y ago
Before OCR'ing it converts to black-and-white using a brightness threshold. Keeping an evenly lit background is particularly important, because otherwise a shadow area can easily fall under the threshold of all-black. A license or ID will almost certainly have medium-contrast elements in the background that will show up as dark. But if you were able to manipulate the contrast/brightness appropriately in advance, you could probably get it to work.

bhanhfo 7y ago

Related Chrome extension: https://chrome.google.com/webstore/detail/project-naptha/molncoemjfmpgdkbdlbjmhlcgniigdnf https://chrome.google.com/webstore/detail/project-naptha/mol... Another extension (not using Tesseract.JS): https://chrome.google.com/webstore/detail/copyfish-%F0%9F%90%9F-free-ocr-soft/eenjdnjldapjajjofmldgmkjaienebbj?hl=en https://chrome.google.com/webstore/detail/copyfish-%F0%9F%90...

throwaway_ocr 7y ago

I tested this by taking a screenshot of the introduction blurb. This is what it came up with: Tesseract s is 2 pure Javascript port of the popular Tesseract OCR engine. This library supports more than 100 languages, automatic text orientation and script detection, a simple interface for reading paragraph, word, and character bounding boxes. Tesseract s can run either in 2 browser znd on & server with NodeJS. Not bad, but far from useful.

EGreg 7y ago

I remember reading that Tesseract is the old tech which does worse than current ML based stuff. But since no one gives out their ML data that’s all you get.

mkl 7y ago
Your knowledge is out of date. Tesseract 4 added a new OCR engine based on LSTM neural networks.

atum47 7y ago

This is just amazing. I wonder how much work and how many brilliant people it took to develop this. Congratulations to everyone involved. I'm sure a lot of cool stuff will come from this tool.

gwbas1c 7y ago

I'm curious about the performance tradeoffs of Javascript versus native code? A few weeks ago, I tried writing something that was long-running, CPU intense, ect, in a webworker. It was so darn slow that I switched to a native language. (I hope I didn't do something silly that made my code run more slowly than it should.) I see some mention about running in WASM. Does this do something like have ordinary Tesseract compiled for WebAssembly and then fallback to Javascript?

fyp 7y ago
A while back I ported several c++ libraries with emscripten and it's usually just 2-10x slower than native. It gets maybe another order of magnitude worse if you're a porting a library that heavily relies on vectorization which isn't available on the web.

gwd 7y ago

What's slightly weird is that the Chinese "example" text mis-reads a character in the first line. The image shows: 冬日平泉路晚归 But the OCR reports: 冬日平柳路晚归 (Note the different character right in the middle)

bufferoverflow 7y ago
I tried using Tesseract around a year ago to recognize digits only, very clean images, not some weird font. I had thousands of images, and it failed in around 3% of them. It was so weird, as it would recognize the same digits in other images just fine. I tried 4 or 5 different OCR programs, and none of them worked well enough for my case. I was actually surprised, I thought OCR was a solved problem with ridiculously low error rates.
- mkl 7y ago
  In my experience Tesseract can get confused near the edges of images, and padding with a wide white border helped a lot. Strange that none of the OCR programs were good enough, though.

dbhattar 7y ago

Even though it says it supports 100 languages, I cannot find the list of supported languages. I am mostly trying to find out if it supports Indic languages.

severine 7y ago
https://github.com/tesseract-ocr/tesseract/wiki/Data-Files#data-files-for-version-400-november-29-2016 https://github.com/tesseract-ocr/tesseract/wiki/Data-Files#d... Seems so, lists at least Hindi, Urdu, Bengali, Sanskrit, Urdu, Nepali, Marathi, Sinhala and Punjabi!

dang 7y ago

A thread from 2016: https://news.ycombinator.com/item?id=12694004 https://news.ycombinator.com/item?id=12694004

rolling_roland 7y ago

I investigated tesseract.js for turning images of spreadsheets into data. I didn't mind the initial startup time or run time but unfortunately wasn't able to get good enough accuracy for my case. It seemed to work really well with plain english text though.

ngcc_hk 7y ago

There is another post about tesseract but by python. Same question but I guess it might be closer ... is it compatible with TensorFlow js? or is it ok to run alongside? Or the model can be “simplified” to run on client side like mobile net?

alexcnwy 7y ago

Ugh why does open source OCR still suck so bad?! Why isn’t there an open source OCR engine even half as powerful as the Google Cloud Platform API?!

kbumsik 7y ago

> Tesseract.js wraps an emscripten port of the Tesseract OCR Engine. So it is a wrapper library of a C++ project, which is cool. But saying it is a "Pure" JavaScript is purely misleading.

JeromeWu 7y ago

Wow, I was wondering why there were so many new github stars yesterday, and here I found the reason why. :) Thanks for being interested in tesseract.js, it makes all the work worth. And I have to thank @antimatter15 for creating this library, without him we cannot go this far I have read all the comments and here I would like to provide my two cents for some questions: 1. Is tesseract.js pure JavaScript? Yes, it is 100% JavaScript and it leverages Webassembly port of original tesseract-ocr. (means we compile the C sorce code to JavaScript Webassembly code, powered by Emscripten) 2. The accuracy of tesseract.js is poor. In my experience, it is hard to get perfect results without applying additional techniques to your source images. You may need to some preprocessing and sometimes train a custom traineddata. It is not easy, but it is the price of high accuracy. 3. Cloud OCR service is much more accurate Yes, that's true. But tesseract.js provides an in browser offline option to do your OCR, it is useful for scenarios like PWA and high confidential image content (which you don't want to send to server). Tesseract.js is not a silver bullet, but it is handy sometimes. Hope you enjoy this library and feel free to leave any comment to us!