10 ms·

Show HN: I wrote a free Mac app to OCR any text on screen

321 points by schappim 5y ago

mromanuk 5y ago
I’m frustrated that there is no easy way to capture text from pictures. Thank you for building this. This type of functionality should be integrated or readily available in MacOS. I would love to have a way to do basic math operations, unit conversions, etc. without resorting to write them in spotlight. For example hovering over a price, it should convert it to a different currency. Or compare similar types of informations from different sources. Your solution goes in this direction, thanks!
- schappim 5y ago
  You’re welcome. It should be pretty trivial for someone to hook this into https://insect.sh/ https://insect.sh/.
- replwoacause 5y ago
  There is an app called Text Sniper which works really well and is available on SetApp. I use it constantly and love it.
- maven29 5y ago
  https://antimatter15.com/project/project-naptha/ https://antimatter15.com/project/project-naptha/ https://projectnaptha.com/ https://projectnaptha.com/ We had this fully client-side in the browser all the way back in 2013.
  - eastendguy 5y ago
    The problem with Naptha is that the OCR quality of Tesseract (used by Project Naptha) is not very good for screenshots. Historically Tesseract has been optimized for text documents. So the OCR results of this new macOCR app are significantly better. Another alternative is Copyfish. It is cross-platform and uses cloud ocr: https://github.com/A9T9/Copyfish https://github.com/A9T9/Copyfish
    - duckfang 5y ago
      There is 0 reason to use the cloud, unless exfil and selling data is your thing. The cloud == someone else's computer. Never forget that.
      - eastendguy 5y ago
        My personal reason for using "the cloud" in this specific case is that the OCR results are significantly better. > The cloud == someone else's computer. I think everyone around here knows this and can make their own decisions, if and for what data they use "the cloud".
        duckfang 5y ago
        Be careful speaking about "everyone". You never know who the 10000 are. https://xkcd.com/1053/ https://xkcd.com/1053/
        bori5 5y ago
        Not everyone’s in the US
        stjohnswarts 5y ago
        Sure but it's fairly assured that 80% at least are from the USA, possibly higher. Also it's fairly easy to extrapolate the premise of "the 10000" to include the rest of the world
- loraxclient 5y ago
  So there actually is a very quiet functionality hiding in MacOS Notes.app I noticed recently. If you take any screenshot to clipboard that includes some text, then paste it into Notes, it will silently name the resulting image file using the text pictured. Not terribly useful but I did find it helpful once while taking screenshots for documentation.
jbackus 5y ago
Very nice. I've been using https://screenotate.com/ https://screenotate.com/ for years in part because of the nice OCR support
- seltzered_ 5y ago
  yeah, screenotate has been nice for this, have also used a lesser known feature in Prizmo (https://creaceed.com/prizmo https://creaceed.com/prizmo) too for this.
djrogers 5y ago
This is cool! I’ve been a user of TextSniper to do this for a long time, but will happily try this out.
dheera 5y ago
Not a Mac user or developer, poking at the source code because I might be interested in building a Linux equivalent. Why is code so damn complicated these days? What does all this crap do? Why isn't the source code of things these days 100% human-readable? /* Begin PBXBuildFile section / 0425D1C16E9B7E34F8EBCCFB229F6BCF / Pods-ocr-umbrella.h in Headers / = {isa = PBXBuildFile; fileRef = E52F12A9CD9DA185DB6C7CFAF9971233 / Pods-ocr-umbrella.h /; settings = {ATTRIBUTES = (Project, ); }; }; 69F017594F16B64B4E70E96B863F38D1 / Pods-ocr-dummy.m in Sources / = {isa = PBXBuildFile; fileRef = 812D67335813B22DFC54237ACEB07CC8 / Pods-ocr-dummy.m /; }; 9D8F5FD727B32865EE80BA6ACDA12AF4 / ScreenCapture.swift in Sources / = {isa = PBXBuildFile; fileRef = CAE82544998B753F1708876308FF330D / ScreenCapture.swift /; }; AC8C4224C366FAD03EFFDC427D793373 / ScreenCapture-dummy.m in Sources / = {isa = PBXBuildFile; fileRef = 2BFFD24873C787E751AFC41D8C497ECB / ScreenCapture-dummy.m /; }; BE8E791706F107976678CAA1DE681FA6 / ScreenCapture-umbrella.h in Headers / = {isa = PBXBuildFile; fileRef = B9D6CB7E3F7CD4599F66F1F010D4CADD / ScreenCapture-umbrella.h /; settings = {ATTRIBUTES = (Project, ); }; }; CA9117D8B1C22828347BFE8326E2F7D2 / ScreenRecorder.swift in Sources / = {isa = PBXBuildFile; fileRef = FC34AC3B539E1EFA3B0D1E086E1BA1D9 / ScreenRecorder.swift /; }; / End PBXBuildFile section */
- schappim 5y ago
  My tiny contribution is within main.swift. The heavy lifting is done by the CoreImage/Vision API, and a library that interfaces with Screen Capture. You can achieve the same thing using Tesseract on Linux, or even better quality using Google Vision.
  - florin0x01 5y ago
    I'm a noob using Xcode. Loaded the workspace, removed the signing team, everything builds for all schemes but where is the binary?
    - schappim 5y ago
      Does this screenshot[1] answer your question? [1] https://files.littlebird.com.au/Shared-Image-2021-05-22-17-16-06.png https://files.littlebird.com.au/Shared-Image-2021-05-22-17-1...
vidarh 5y ago
Reminds me of PowerSnap on the Amiga, that did a very simple version of this (it did not run a proper OCR engine, but relied on a very close match with one of the installed fonts, so worked great for UI of apps that didn't support cut and paste, not so great for images unless they were created using the same font and not scaled, but still very useful).
gostsamo 5y ago
This can be a valuable accessibility tool, especially on inaccessible apps. Have you looked into checking its compatibility with Voice Over?
- schappim 5y ago
  You could just pipe the output to the say command eg: /usr/local/bin/ocr | say
  - gostsamo 5y ago
    I can, but not everybody else. 90% of the population does not use terminal commands as a part of their everyday lives.
upbeat_general 5y ago
I had no idea that macOS/iOS now had built in OCR! Any idea on how this compares to tessaract (or other local OCR). I currently have an Alfred workflow that invokes tessaract and it works decently well but the accuracy could be better.
- schappim 5y ago
  I’ve tried with Tesseract, the Vision API, Google Vision and Azure’s equivalent (also kicked off using an Alfred workflow). By far the best for text was Google Vision and then Azure. Whilst Google Cloud and Azure both also do handwriting recognition, Azure did better at this. The cloud platforms performed better than pure on device with Apple’s vision API outperforming Tesseract.
  - upbeat_general 5y ago
    Good to know! I thought most cloud OCR services were paid only but it turns out google vision has 1000 free invocations a month with Azure at 5000. That should be plenty for most people (certainly for me). Do you have the source for those workflows/would you be willing to share them?
TacticalCoder 5y ago
Very nice. Does this work with screen fonts / pixel perfect fonts too? That'd be my biggest gripe with all the OCR tools: what looks like the simplest fonts of them all to OCR are usually the one the less well detected. For example using tesseract on Linux and trying to OCR the "terminus" font I get better result by first resizing the screenshot to something bigger (and blurry) and even then it's far from perfect OCR'ing. When in the first place it's a pixel perfect font... (and, yes, there are cases where OCR'ing screen fonts make sense)
- schappim 5y ago
  I actually use another version with Google Vision (it is in a whole other class to on device OCR), and it is quite challenging to make Google Vision not work. For example it even works for hand written text.
  - TacticalCoder 5y ago
    Oh interesting, I should check that out.
- derefr 5y ago
  I’ve always wondered whether you couldn’t build a not-fully-ML app for screen-font recognition, that just takes a bunch of fonts, renders out every individual glyph at every size, trims them, converts them to an alpha mask, and then generates an indexible image fingerprint of said alpha mask. The OCR software would then just need to be smart enough to recognize “things that look like glyphs”, and put bounding boxes around them; and everything from there could be implemented in logic, rather than a model. (Just apply the same transforms to the thing in the bounding box, and then search the fingerprint DB.)
cheeaun 5y ago
Hey this is very useful! I wrote a Raycast (https://raycast.com/ https://raycast.com/) script to invoke it here https://gist.github.com/cheeaun/1405816e5ceb397cbc9028204f82dc98 https://gist.github.com/cheeaun/1405816e5ceb397cbc9028204f82... I tweeted a GIF on how it looks like https://twitter.com/cheeaun/status/1395973544983425025 https://twitter.com/cheeaun/status/1395973544983425025.
Rebelgecko 5y ago
It's barfing due to missing libraries on my machine, anyone know what OS version this requires?
- schappim 5y ago
  Should be Catalina and above, I’ll add it to the docs. I’ve tested on an i9 running Catalina and M1 forced to be on Big Sur.
deleted 5y ago
[deleted]
SiVal 5y ago
Outstanding. I have a question, and it's not a complaint or a request for anything except a bit of data: What kinds of text does it support, as in, would it work for handwritten English, or German rotated 45 degrees, or Japanese...?
- darcien 5y ago
  I dug into the code, looks like it use VNRecognizeTextRequest[1] from Vision framework by Apple here [2]. And the docs says >By default, a text recognition request first locates all possible glyphs or characters in the input image, then analyzes each string. Since the code doesn't specify any preferred languages, I think it would try to detect any languages supported by the framework. From short googling, I found this thread [3]. Looks like the supported languages depends on the MacOS version, and it only support en, fr, it, de, es, pt, zh on Big Sur. Not sure about the rotation though. [1] https://developer.apple.com/documentation/vision/vnrecognizetextrequest https://developer.apple.com/documentation/vision/vnrecognize... [2] https://github.com/schappim/macOCR/blob/master/ocr/main.swift#L55 https://github.com/schappim/macOCR/blob/master/ocr/main.swif... [3] https://developer.apple.com/forums/thread/121048 https://developer.apple.com/forums/thread/121048
apetrovic 5y ago
Very nice! I'm currently using https://github.com/amebalabs/TRex https://github.com/amebalabs/TRex for the same purpose.
- deleted 5y ago
  [deleted]
- schappim 5y ago
  This looks like exactly what I was after when making this!
- cheeaun 5y ago
  Hmm does this support non-English languages?
Ennea 5y ago
This looks super useful. Is there some equivalent for Windows? A quick search didn't yield anything useful.
greggman3 5y ago
I didn't try the app yet but I use an app called "Yomiwa" to OCR Japanese. The biggest issue I find is that it seems to be trained on black on white images so if I try to OCR a sign with different colors, or some text on a product or menu, it often fails. For example I just tried it on the ingredients of the label of the soda I'm drinking which is white on black and it's completely failing.
- vicedvin 5y ago
  Masking the background and making text black would solve this. I have used OpenCV for that before.
  - rjzzleep 5y ago
    OCRmyPDF does that. Given the name I assume it's only for PDFs given the name, but theoretically it should work for any image without the need to extract it from the PDF. https://github.com/jbarlow83/OCRmyPDF https://github.com/jbarlow83/OCRmyPDF
stormdennis 5y ago
Sorry if this is off topic but as a windows user is there a good one around for that, that people would recommend
- flarg 5y ago
  One note
- eastendguy 5y ago
  Copyfish is cross-platform: https://ocr.space/copyfish https://ocr.space/copyfish
- frogpelt 5y ago
  Microsoft One Note can do it.
- vitorsr 5y ago
  Pardon the late reply. I've been successfully using Mathpix Snip [1] to do general OCR for quite some time. It's not as well communicated as its initial purpose of applying OCR to LaTeX equations, but it currently supports much more than that, such as mixed text/math and tables. On a personal note, I'm actually surprised it wasn't mentioned thus far in this thread. See more discussion here on HN in [2], [3]. [1] https://mathpix.com https://mathpix.com [2] https://news.ycombinator.com/item?id=16535358 https://news.ycombinator.com/item?id=16535358 [3] https://news.ycombinator.com/item?id=21871780 https://news.ycombinator.com/item?id=21871780
gitgud 5y ago
Great project! This is one of those tools that solves a problem that's right in front of you. I can see this being built into screenshot software, just like mark-up tools are these days. Also, does anyone know of a similar project for Linux?
- rossvor 5y ago
  No project I'm aware of, but I did cobble together a script to do something similar. In my I case also wanted it to work with copyq so there's some noise related to it. tmp=/tmp/out maim -s -u | tesseract - "$tmp" # Remove empty lines sed -ir '/^\s*$/d' "$tmp".txt copyq add "$(cat "$tmp".txt)" rm "$tmp".txt rm "$tmp".txtr tesseract insists on adding on txt extension and what I assume is some intermediary file txtr, making it awkward to use with mktemp. Probably explained in the manual which I skipped. But like others have said tesseract is not very reliable, at least with default settings -- it's common for it to add extra spaces or various single quotes, or omit spaces.
  - mdaniel 5y ago
    I wasn't familiar with that "-r", but FWIW the gsed man page says "(for portability use POSIX -E)" Abutting that "r" option against the "i" option is likely why you ended up with a file named .txtr and therefore implies that it did not actually hear the "-r" you intended I've had the best luck picking an actual backup suffix such as "-i.bak" or "-i~" to keep BSD sed and GNU sed on the same page, although I've also seen scripts that go as far as "--version" sniffing and changing the actual invocation as "${SED_I} -E" type stuff
    - rossvor 5y ago
      You are absolutely correct! I wasn't aware that "-i" takes an optional suffix, so it didn't even occur to me to look at the sed line as possible cause for this extension weirdness.
      - mdaniel 5y ago
xlii 5y ago
People here recommended a lot of different solution that does the same. I’m using (and I’d like to recommend) https://www.keyboardmaestro.com/ https://www.keyboardmaestro.com/ for this. It requires self-written macro, however it can do much more than that, including parsing & formatting OCRed text. For one job I went directly Image->OCR->File so I could copy OCRs into text for some non-elegant hardcodes ;-)
- astrostl 5y ago
  Is your macro published anywhere?
jp0d 5y ago
Is the OCR part readily available using this `VNRecognizedTextObservation`? So it suggests possible option and the best one needs to be chosen? Would love to see how it's implemented.
kome 5y ago
that's really cool, congrats! it would be nice to have an UI and a package for installation.
filt 5y ago
I tried a bunch of different solutions but found OwlOCR to be the one with the best result. https://owlocr.com/ https://owlocr.com/ I'll keep an eye on this one too though!
- ChemSpider 5y ago
  This uses the same OCR as this ShowHN, so the OCR results should be identical.
unicornporn 5y ago
Here's another one: https://screenotate.com/ https://screenotate.com/
hendry 5y ago
Flameshot could do with this feature https://github.com/flameshot-org/flameshot/issues/702 https://github.com/flameshot-org/flameshot/issues/702
yawaworht1978 5y ago
Wow this is very nice, if this can be made really solid, then you are very close to a kyc as a service product. Meaning, compare issuing date with the official formats for the time frames, then compare if the fonts and checksums are correct and an API to send the results to the customers. Most platforms dealing with fiat are waiting for something like that, they are suffering from doing this manually or semi manually with prohibitive employee costs. If this can be done platform agnostic and fraud proof, the companies will throw themselves at the product. Are you planning anything towards that direction?
jerieljan 5y ago
Personally, I use a combination of tesseract and MacOS' screen capture to achieve the same thing. Very handy to have if you use an app like BetterTouchTool to run it with a quick hotkey. https://gist.github.com/jerieljan/f86843388c4ee9ecbc44d687a3b3dd27 https://gist.github.com/jerieljan/f86843388c4ee9ecbc44d687a3...
- jftuga 5y ago
  nice script!
- salsadip 5y ago
  nice, thanks for sharing! for anyone having trouble configuring the shell execution: use /bin/sh/ for launch path and "-c" (without quotes) for parameter. Otherwise BTT will autofill "(null)" and then it won't work and it won't tell you why
deleted 5y ago
[deleted]
keepshoutingsir 5y ago
Really nice! Would love if it supported tables. Have struggled to find any OCR that can take a photo of a table and turn it into a spreadsheet. It’s usually an inconsistently tab-separated mess requiring a ton of cleanup.
- sintezcs 5y ago
  Have you tried FineReader? I used it a couple of years ago for table recognition
Cybotron5000 5y ago
Very useful - thanks!
sgt 5y ago
Seems to be a tiny wrapper around VNRecognizeTextRequest which is available on iOS, macOS etc. It's a dozen lines of code to do this - I think someone should build this into a menu item app, perhaps?
- djbeadle 5y ago
  How about xbar? https://github.com/matryer/xbar https://github.com/matryer/xbar
p2hari 5y ago
Very nice and it is super wonderful to use. Now my workflow is so easy. Most of the time, translation on GIF and images don't work and I have to get them and then use a translation tool. Now it is just snap and my shortcut to translation all in one place.
- sorenjan 5y ago
  Yandex supports translation on images, and you can paste it straight from the clipboard. I'm assuming that MacOS has something similar to Windows' Win-Shift-S to take a screenshot a section of the screen, which makes it really fast and easy. https://translate.yandex.com/ocr https://translate.yandex.com/ocr
feikname 5y ago
For those on Windows, I find Capture2Text[0] to be a pretty great FOSS Screen OCR application: you press Win Q and select an area like the snipping tool would, simple as that. It also has some other snipping modes, supports more than english, and has the option to, after the OCR, immediately show a popup window where you can fix what the OCR inevitably failed to properly recognize. I recommend use alongside a clipboard manager. [0] http://capture2text.sourceforge.net http://capture2text.sourceforge.net P.S. I recommend changing/disabling its Win E shortcut as that conflicts with Windows built-in shortcut for file explorer
- beagle3 5y ago
  It is, however, based on Tesseract which works well on clean or scanned documents, but isn’t very good for text “in the wild” like street signs and not-parallel-to-view images. The Mac program uses VisionKit which does handle these cases better (not as well as google cloud vision from my outdated experience but way better than tesseract)
p4l4g4 5y ago
Nice! Although I'm linux user, I got inspired by the idea. I often combine imagemagick import with xclip to quickly snap parts of the screen to clipboard (import png:- | xclip -sel clip). This can easily be extended with tesseract for ocr support (just add 'tesseract - -' between import and xclip in the pipeline)!
- p4l4g4 5y ago
  Forgot a crucial parameter in the first xclip call: -t image/png. Without this, your system don't know its png data and will paste the raw data when you hit ctrl-v.
fdeage 5y ago
That is _exactly_ what I've been looking for the past 5 years! Unfortunately I am still on Mojave, so of course I get an error: dyld: Symbol not found: _OBJC_CLASS_$_VNRecognizeTextRequest Any chance to get it to work on poor macOS 10.14?
Terretta 5y ago
For folks looking for something like this in an app store, I think I found TextSniper here on HN last August, and it's also in the great alt app store "SetApp" collection: “TextSniper is an easy-to-use desktop Mac OCR app that can extract and recognize any non-searchable and non-editable text on your Mac's screen. As an extra feature, it can turn OCR text into speech. It is a super convenient alternative to complicated optical character recognition tools.” https://apps.apple.com/us/app/textsniper-ocr-simplified/id1528890965?mt=12 https://apps.apple.com/us/app/textsniper-ocr-simplified/id15... “Meet lightning-fast text recognition on Mac. TextSniper is an app that can extract text from a selected portion of your screen. Forget taking notes — get TextSniper to capture and save what’s important.” https://setapp.com/apps/textsniper https://setapp.com/apps/textsniper Consolidating into this note, OwlOCR is mentioned elsewhere in this post: “Capture any text on your Mac's screen. Digitize images and PDFs to searchable PDFs using OCR right on your Mac.” https://owlocr.com https://owlocr.com “OwlOCR allows grabbing a part of the screen and having any text in that area be instantaneously recognized and copied to clipboard. Additionally, the application supports recognizing text from PDF files, images and converting the contents to plain text. All conversion is done securely on-device - none of your images or files are sent to third-party services in the cloud.” https://apps.apple.com/us/app/id1499181666 https://apps.apple.com/us/app/id1499181666 // Both process on device, OwlOCR mentions Apple's algo. Users of both in this thread are happy.
- masswerk 5y ago
  If you want a launchable app, you can also use Automator (select new App, drag and drop “Utilities/Run Shell Script”, insert the path "/usr/local/bin/ocr" as the script content, save) and put the resulting app in the Doc. (No terminal or screen space required to launch, and it's always at your finger tips.)

321 points by schappim 5y ago

I’m frustrated that there is no easy way to capture text from pictures. Thank you for building this. This type of functionality should be integrated or readily available in MacOS. I would love to have a way to do basic math operations, unit conversions, etc. without resorting to write them in spotlight. For example hovering over a price, it should convert it to a different currency. Or compare similar types of informations from different sources. Your solution goes in this direction, thanks!

schappim 5y ago
You’re welcome. It should be pretty trivial for someone to hook this into https://insect.sh/ https://insect.sh/.
replwoacause 5y ago
There is an app called Text Sniper which works really well and is available on SetApp. I use it constantly and love it.
maven29 5y ago
https://antimatter15.com/project/project-naptha/ https://antimatter15.com/project/project-naptha/ https://projectnaptha.com/ https://projectnaptha.com/ We had this fully client-side in the browser all the way back in 2013.
- eastendguy 5y ago
  The problem with Naptha is that the OCR quality of Tesseract (used by Project Naptha) is not very good for screenshots. Historically Tesseract has been optimized for text documents. So the OCR results of this new macOCR app are significantly better. Another alternative is Copyfish. It is cross-platform and uses cloud ocr: https://github.com/A9T9/Copyfish https://github.com/A9T9/Copyfish
  - duckfang 5y ago
    There is 0 reason to use the cloud, unless exfil and selling data is your thing. The cloud == someone else's computer. Never forget that.
    - eastendguy 5y ago
      My personal reason for using "the cloud" in this specific case is that the OCR results are significantly better. > The cloud == someone else's computer. I think everyone around here knows this and can make their own decisions, if and for what data they use "the cloud".
      - duckfang 5y ago
        Be careful speaking about "everyone". You never know who the 10000 are. https://xkcd.com/1053/ https://xkcd.com/1053/
        bori5 5y ago
        Not everyone’s in the US
        stjohnswarts 5y ago
        Sure but it's fairly assured that 80% at least are from the USA, possibly higher. Also it's fairly easy to extrapolate the premise of "the 10000" to include the rest of the world
loraxclient 5y ago
So there actually is a very quiet functionality hiding in MacOS Notes.app I noticed recently. If you take any screenshot to clipboard that includes some text, then paste it into Notes, it will silently name the resulting image file using the text pictured. Not terribly useful but I did find it helpful once while taking screenshots for documentation.

jbackus 5y ago

Very nice. I've been using https://screenotate.com/ https://screenotate.com/ for years in part because of the nice OCR support

seltzered_ 5y ago
yeah, screenotate has been nice for this, have also used a lesser known feature in Prizmo (https://creaceed.com/prizmo https://creaceed.com/prizmo) too for this.

djrogers 5y ago

This is cool! I’ve been a user of TextSniper to do this for a long time, but will happily try this out.

dheera 5y ago

Not a Mac user or developer, poking at the source code because I might be interested in building a Linux equivalent. Why is code so damn complicated these days? What does all this crap do? Why isn't the source code of things these days 100% human-readable? /* Begin PBXBuildFile section / 0425D1C16E9B7E34F8EBCCFB229F6BCF / Pods-ocr-umbrella.h in Headers / = {isa = PBXBuildFile; fileRef = E52F12A9CD9DA185DB6C7CFAF9971233 / Pods-ocr-umbrella.h /; settings = {ATTRIBUTES = (Project, ); }; }; 69F017594F16B64B4E70E96B863F38D1 / Pods-ocr-dummy.m in Sources / = {isa = PBXBuildFile; fileRef = 812D67335813B22DFC54237ACEB07CC8 / Pods-ocr-dummy.m /; }; 9D8F5FD727B32865EE80BA6ACDA12AF4 / ScreenCapture.swift in Sources / = {isa = PBXBuildFile; fileRef = CAE82544998B753F1708876308FF330D / ScreenCapture.swift /; }; AC8C4224C366FAD03EFFDC427D793373 / ScreenCapture-dummy.m in Sources / = {isa = PBXBuildFile; fileRef = 2BFFD24873C787E751AFC41D8C497ECB / ScreenCapture-dummy.m /; }; BE8E791706F107976678CAA1DE681FA6 / ScreenCapture-umbrella.h in Headers / = {isa = PBXBuildFile; fileRef = B9D6CB7E3F7CD4599F66F1F010D4CADD / ScreenCapture-umbrella.h /; settings = {ATTRIBUTES = (Project, ); }; }; CA9117D8B1C22828347BFE8326E2F7D2 / ScreenRecorder.swift in Sources / = {isa = PBXBuildFile; fileRef = FC34AC3B539E1EFA3B0D1E086E1BA1D9 / ScreenRecorder.swift /; }; / End PBXBuildFile section */

schappim 5y ago
My tiny contribution is within main.swift. The heavy lifting is done by the CoreImage/Vision API, and a library that interfaces with Screen Capture. You can achieve the same thing using Tesseract on Linux, or even better quality using Google Vision.
- florin0x01 5y ago
  I'm a noob using Xcode. Loaded the workspace, removed the signing team, everything builds for all schemes but where is the binary?
  - schappim 5y ago
    Does this screenshot[1] answer your question? [1] https://files.littlebird.com.au/Shared-Image-2021-05-22-17-16-06.png https://files.littlebird.com.au/Shared-Image-2021-05-22-17-1...

vidarh 5y ago

Reminds me of PowerSnap on the Amiga, that did a very simple version of this (it did not run a proper OCR engine, but relied on a very close match with one of the installed fonts, so worked great for UI of apps that didn't support cut and paste, not so great for images unless they were created using the same font and not scaled, but still very useful).

gostsamo 5y ago

This can be a valuable accessibility tool, especially on inaccessible apps. Have you looked into checking its compatibility with Voice Over?

schappim 5y ago
You could just pipe the output to the say command eg: /usr/local/bin/ocr | say
- gostsamo 5y ago
  I can, but not everybody else. 90% of the population does not use terminal commands as a part of their everyday lives.

upbeat_general 5y ago

I had no idea that macOS/iOS now had built in OCR! Any idea on how this compares to tessaract (or other local OCR). I currently have an Alfred workflow that invokes tessaract and it works decently well but the accuracy could be better.

schappim 5y ago
I’ve tried with Tesseract, the Vision API, Google Vision and Azure’s equivalent (also kicked off using an Alfred workflow). By far the best for text was Google Vision and then Azure. Whilst Google Cloud and Azure both also do handwriting recognition, Azure did better at this. The cloud platforms performed better than pure on device with Apple’s vision API outperforming Tesseract.
- upbeat_general 5y ago
  Good to know! I thought most cloud OCR services were paid only but it turns out google vision has 1000 free invocations a month with Azure at 5000. That should be plenty for most people (certainly for me). Do you have the source for those workflows/would you be willing to share them?

TacticalCoder 5y ago

Very nice. Does this work with screen fonts / pixel perfect fonts too? That'd be my biggest gripe with all the OCR tools: what looks like the simplest fonts of them all to OCR are usually the one the less well detected. For example using tesseract on Linux and trying to OCR the "terminus" font I get better result by first resizing the screenshot to something bigger (and blurry) and even then it's far from perfect OCR'ing. When in the first place it's a pixel perfect font... (and, yes, there are cases where OCR'ing screen fonts make sense)

schappim 5y ago
I actually use another version with Google Vision (it is in a whole other class to on device OCR), and it is quite challenging to make Google Vision not work. For example it even works for hand written text.
- TacticalCoder 5y ago
  Oh interesting, I should check that out.
derefr 5y ago
I’ve always wondered whether you couldn’t build a not-fully-ML app for screen-font recognition, that just takes a bunch of fonts, renders out every individual glyph at every size, trims them, converts them to an alpha mask, and then generates an indexible image fingerprint of said alpha mask. The OCR software would then just need to be smart enough to recognize “things that look like glyphs”, and put bounding boxes around them; and everything from there could be implemented in logic, rather than a model. (Just apply the same transforms to the thing in the bounding box, and then search the fingerprint DB.)

cheeaun 5y ago

Hey this is very useful! I wrote a Raycast (https://raycast.com/ https://raycast.com/) script to invoke it here https://gist.github.com/cheeaun/1405816e5ceb397cbc9028204f82dc98 https://gist.github.com/cheeaun/1405816e5ceb397cbc9028204f82... I tweeted a GIF on how it looks like https://twitter.com/cheeaun/status/1395973544983425025 https://twitter.com/cheeaun/status/1395973544983425025.

Rebelgecko 5y ago

It's barfing due to missing libraries on my machine, anyone know what OS version this requires?

schappim 5y ago
Should be Catalina and above, I’ll add it to the docs. I’ve tested on an i9 running Catalina and M1 forced to be on Big Sur.

deleted 5y ago

[deleted]

SiVal 5y ago

Outstanding. I have a question, and it's not a complaint or a request for anything except a bit of data: What kinds of text does it support, as in, would it work for handwritten English, or German rotated 45 degrees, or Japanese...?

darcien 5y ago
I dug into the code, looks like it use VNRecognizeTextRequest[1] from Vision framework by Apple here [2]. And the docs says >By default, a text recognition request first locates all possible glyphs or characters in the input image, then analyzes each string. Since the code doesn't specify any preferred languages, I think it would try to detect any languages supported by the framework. From short googling, I found this thread [3]. Looks like the supported languages depends on the MacOS version, and it only support en, fr, it, de, es, pt, zh on Big Sur. Not sure about the rotation though. [1] https://developer.apple.com/documentation/vision/vnrecognizetextrequest https://developer.apple.com/documentation/vision/vnrecognize... [2] https://github.com/schappim/macOCR/blob/master/ocr/main.swift#L55 https://github.com/schappim/macOCR/blob/master/ocr/main.swif... [3] https://developer.apple.com/forums/thread/121048 https://developer.apple.com/forums/thread/121048

apetrovic 5y ago

Very nice! I'm currently using https://github.com/amebalabs/TRex https://github.com/amebalabs/TRex for the same purpose.

deleted 5y ago
[deleted]
schappim 5y ago
This looks like exactly what I was after when making this!
cheeaun 5y ago
Hmm does this support non-English languages?

Ennea 5y ago

This looks super useful. Is there some equivalent for Windows? A quick search didn't yield anything useful.

greggman3 5y ago

I didn't try the app yet but I use an app called "Yomiwa" to OCR Japanese. The biggest issue I find is that it seems to be trained on black on white images so if I try to OCR a sign with different colors, or some text on a product or menu, it often fails. For example I just tried it on the ingredients of the label of the soda I'm drinking which is white on black and it's completely failing.

vicedvin 5y ago
Masking the background and making text black would solve this. I have used OpenCV for that before.
- rjzzleep 5y ago
  OCRmyPDF does that. Given the name I assume it's only for PDFs given the name, but theoretically it should work for any image without the need to extract it from the PDF. https://github.com/jbarlow83/OCRmyPDF https://github.com/jbarlow83/OCRmyPDF

stormdennis 5y ago

Sorry if this is off topic but as a windows user is there a good one around for that, that people would recommend

flarg 5y ago
One note
eastendguy 5y ago
Copyfish is cross-platform: https://ocr.space/copyfish https://ocr.space/copyfish
frogpelt 5y ago
Microsoft One Note can do it.
vitorsr 5y ago
Pardon the late reply. I've been successfully using Mathpix Snip [1] to do general OCR for quite some time. It's not as well communicated as its initial purpose of applying OCR to LaTeX equations, but it currently supports much more than that, such as mixed text/math and tables. On a personal note, I'm actually surprised it wasn't mentioned thus far in this thread. See more discussion here on HN in [2], [3]. [1] https://mathpix.com https://mathpix.com [2] https://news.ycombinator.com/item?id=16535358 https://news.ycombinator.com/item?id=16535358 [3] https://news.ycombinator.com/item?id=21871780 https://news.ycombinator.com/item?id=21871780

gitgud 5y ago

Great project! This is one of those tools that solves a problem that's right in front of you. I can see this being built into screenshot software, just like mark-up tools are these days. Also, does anyone know of a similar project for Linux?

rossvor 5y ago
No project I'm aware of, but I did cobble together a script to do something similar. In my I case also wanted it to work with copyq so there's some noise related to it. tmp=/tmp/out maim -s -u | tesseract - "$tmp" # Remove empty lines sed -ir '/^\s*$/d' "$tmp".txt copyq add "$(cat "$tmp".txt)" rm "$tmp".txt rm "$tmp".txtr tesseract insists on adding on txt extension and what I assume is some intermediary file txtr, making it awkward to use with mktemp. Probably explained in the manual which I skipped. But like others have said tesseract is not very reliable, at least with default settings -- it's common for it to add extra spaces or various single quotes, or omit spaces.
- mdaniel 5y ago
  I wasn't familiar with that "-r", but FWIW the gsed man page says "(for portability use POSIX -E)" Abutting that "r" option against the "i" option is likely why you ended up with a file named .txtr and therefore implies that it did not actually hear the "-r" you intended I've had the best luck picking an actual backup suffix such as "-i.bak" or "-i~" to keep BSD sed and GNU sed on the same page, although I've also seen scripts that go as far as "--version" sniffing and changing the actual invocation as "${SED_I} -E" type stuff
  - rossvor 5y ago
    You are absolutely correct! I wasn't aware that "-i" takes an optional suffix, so it didn't even occur to me to look at the sed line as possible cause for this extension weirdness.
    - mdaniel 5y ago

xlii 5y ago

People here recommended a lot of different solution that does the same. I’m using (and I’d like to recommend) https://www.keyboardmaestro.com/ https://www.keyboardmaestro.com/ for this. It requires self-written macro, however it can do much more than that, including parsing & formatting OCRed text. For one job I went directly Image->OCR->File so I could copy OCRs into text for some non-elegant hardcodes ;-)

astrostl 5y ago
Is your macro published anywhere?

jp0d 5y ago

Is the OCR part readily available using this `VNRecognizedTextObservation`? So it suggests possible option and the best one needs to be chosen? Would love to see how it's implemented.

kome 5y ago

that's really cool, congrats! it would be nice to have an UI and a package for installation.

filt 5y ago

I tried a bunch of different solutions but found OwlOCR to be the one with the best result. https://owlocr.com/ https://owlocr.com/ I'll keep an eye on this one too though!

ChemSpider 5y ago
This uses the same OCR as this ShowHN, so the OCR results should be identical.

unicornporn 5y ago

Here's another one: https://screenotate.com/ https://screenotate.com/

hendry 5y ago

Flameshot could do with this feature https://github.com/flameshot-org/flameshot/issues/702 https://github.com/flameshot-org/flameshot/issues/702

yawaworht1978 5y ago

Wow this is very nice, if this can be made really solid, then you are very close to a kyc as a service product. Meaning, compare issuing date with the official formats for the time frames, then compare if the fonts and checksums are correct and an API to send the results to the customers. Most platforms dealing with fiat are waiting for something like that, they are suffering from doing this manually or semi manually with prohibitive employee costs. If this can be done platform agnostic and fraud proof, the companies will throw themselves at the product. Are you planning anything towards that direction?

jerieljan 5y ago

Personally, I use a combination of tesseract and MacOS' screen capture to achieve the same thing. Very handy to have if you use an app like BetterTouchTool to run it with a quick hotkey. https://gist.github.com/jerieljan/f86843388c4ee9ecbc44d687a3b3dd27 https://gist.github.com/jerieljan/f86843388c4ee9ecbc44d687a3...

jftuga 5y ago
nice script!
salsadip 5y ago
nice, thanks for sharing! for anyone having trouble configuring the shell execution: use /bin/sh/ for launch path and "-c" (without quotes) for parameter. Otherwise BTT will autofill "(null)" and then it won't work and it won't tell you why

deleted 5y ago

[deleted]

keepshoutingsir 5y ago

Really nice! Would love if it supported tables. Have struggled to find any OCR that can take a photo of a table and turn it into a spreadsheet. It’s usually an inconsistently tab-separated mess requiring a ton of cleanup.

sintezcs 5y ago
Have you tried FineReader? I used it a couple of years ago for table recognition

Cybotron5000 5y ago

Very useful - thanks!

sgt 5y ago

Seems to be a tiny wrapper around VNRecognizeTextRequest which is available on iOS, macOS etc. It's a dozen lines of code to do this - I think someone should build this into a menu item app, perhaps?

djbeadle 5y ago
How about xbar? https://github.com/matryer/xbar https://github.com/matryer/xbar

p2hari 5y ago

Very nice and it is super wonderful to use. Now my workflow is so easy. Most of the time, translation on GIF and images don't work and I have to get them and then use a translation tool. Now it is just snap and my shortcut to translation all in one place.

sorenjan 5y ago
Yandex supports translation on images, and you can paste it straight from the clipboard. I'm assuming that MacOS has something similar to Windows' Win-Shift-S to take a screenshot a section of the screen, which makes it really fast and easy. https://translate.yandex.com/ocr https://translate.yandex.com/ocr

feikname 5y ago

For those on Windows, I find Capture2Text[0] to be a pretty great FOSS Screen OCR application: you press Win Q and select an area like the snipping tool would, simple as that. It also has some other snipping modes, supports more than english, and has the option to, after the OCR, immediately show a popup window where you can fix what the OCR inevitably failed to properly recognize. I recommend use alongside a clipboard manager. [0] http://capture2text.sourceforge.net http://capture2text.sourceforge.net P.S. I recommend changing/disabling its Win E shortcut as that conflicts with Windows built-in shortcut for file explorer

beagle3 5y ago
It is, however, based on Tesseract which works well on clean or scanned documents, but isn’t very good for text “in the wild” like street signs and not-parallel-to-view images. The Mac program uses VisionKit which does handle these cases better (not as well as google cloud vision from my outdated experience but way better than tesseract)

p4l4g4 5y ago

Nice! Although I'm linux user, I got inspired by the idea. I often combine imagemagick import with xclip to quickly snap parts of the screen to clipboard (import png:- | xclip -sel clip). This can easily be extended with tesseract for ocr support (just add 'tesseract - -' between import and xclip in the pipeline)!

p4l4g4 5y ago
Forgot a crucial parameter in the first xclip call: -t image/png. Without this, your system don't know its png data and will paste the raw data when you hit ctrl-v.

fdeage 5y ago

That is _exactly_ what I've been looking for the past 5 years! Unfortunately I am still on Mojave, so of course I get an error: dyld: Symbol not found: _OBJC_CLASS_$_VNRecognizeTextRequest Any chance to get it to work on poor macOS 10.14?

Terretta 5y ago

For folks looking for something like this in an app store, I think I found TextSniper here on HN last August, and it's also in the great alt app store "SetApp" collection: “TextSniper is an easy-to-use desktop Mac OCR app that can extract and recognize any non-searchable and non-editable text on your Mac's screen. As an extra feature, it can turn OCR text into speech. It is a super convenient alternative to complicated optical character recognition tools.” https://apps.apple.com/us/app/textsniper-ocr-simplified/id1528890965?mt=12 https://apps.apple.com/us/app/textsniper-ocr-simplified/id15... “Meet lightning-fast text recognition on Mac. TextSniper is an app that can extract text from a selected portion of your screen. Forget taking notes — get TextSniper to capture and save what’s important.” https://setapp.com/apps/textsniper https://setapp.com/apps/textsniper Consolidating into this note, OwlOCR is mentioned elsewhere in this post: “Capture any text on your Mac's screen. Digitize images and PDFs to searchable PDFs using OCR right on your Mac.” https://owlocr.com https://owlocr.com “OwlOCR allows grabbing a part of the screen and having any text in that area be instantaneously recognized and copied to clipboard. Additionally, the application supports recognizing text from PDF files, images and converting the contents to plain text. All conversion is done securely on-device - none of your images or files are sent to third-party services in the cloud.” https://apps.apple.com/us/app/id1499181666 https://apps.apple.com/us/app/id1499181666 // Both process on device, OwlOCR mentions Apple's algo. Users of both in this thread are happy.

masswerk 5y ago
If you want a launchable app, you can also use Automator (select new App, drag and drop “Utilities/Run Shell Script”, insert the path "/usr/local/bin/ocr" as the script content, save) and put the resulting app in the Doc. (No terminal or screen space required to launch, and it's always at your finger tips.)