3 ms·
It always surprises and annoys me that these two steps—OCR and translation—are put together inseparably in these cloud APIs. OCR is the much simpler of the two
by derefr 7y ago
It always surprises and annoys me that these two steps—OCR and translation—are put together inseparably in these cloud APIs.
OCR is the much simpler of the two, and is possible with an entirely-offline model; but AFAIK there are no “modern” OCR algorithms (e.g. the tech behind WordLens) available as offline libraries. It’s cloud or nothing if you want to get the best algorithms, and especially if you want an OCR algorithm that works on text containing an arbitrary mixture of Unicode graphemes rather than text in a specific known language.
And, even if I didn’t just want OCRed text—if you separate the two steps of the process (putting things through OCR first and getting a result back, then feeding those results to machine translation separately), then you also get the opportunity to apply domain-specific “cleaning” to the resultant text before feeding it into machine translation. This usually has much better translation results.
One thing I’ve always wanted from web browsers is the ability to just “see into” images, such that I could copy-and-paste text out of any image embedded in a page. (If you’ve ever seen a “translation annotation” done as a floating div on an image, I’d expect to see those, but with text in the original language, not the translated language.) I’ve been consistently surprised when new browsers like Opera or Edge come out and don’t decide to differentiate themselves by including this feature.
But, back on topic, I’d love if emulators could “lift” the bitmap text that gets rendered to the framebuffer into semantic “text” visible to the OS accessibility API, in the same way. This would enable:
• screen-reader software to read the text
• the user to highlight, copy and paste the text
• text transcript dumps to be made of the game, using 100% demo-files + scripting
Certainly, translation could be added on top of this, but just having the original text, in the original language, available to the emulator’s engine, would open all sorts of possibilities.
- polm23 7y ago> One thing I’ve always wanted from web browsers is the ability to just “see into” images, such that I could copy-and-paste text out of any image embedded in a page. Here you go: https://projectnaptha.com/ https://projectnaptha.com/
- derefr 7y agoNaptha has two flaws that exclude it from being what I mean: 1. It relies on the crappy old kind of OCR algorithms, that were only trained on English text. (Yes, even though it uses cloud services. The cloud services themselves use the crappy old OCR algorithms. Literally only the translation services have modern ones.) 2. It isn’t built into a web-browser, in the sense of the model being on-disk as part of the web browser and operating offline, but rather has to upload your images to cloud services. If you want Naptha to be available automatically for any page (such as you would if you were using a screen-reader), then every image on every page you visit is automatically fed to a cloud service. This is not only a horrible privacy leak but also means that with Naptha enabled, pages take forever to fully load. (Which wouldn’t be true with an offline model, given a good computer, or a device that has dedicated ML-model-running circuits like the iPhone.) Also, a third, that’s more of a UX flaw than anything and could be fixed: 3. The model tries to parse the whole image as if it were left-to-right prose, when often the image’s text is in columns, or figure captions, or dialogue balloons. The user can very easily see where the text is, but there’s no way for the user to indicate to the engine where on the image the text is. This means that the model will “connect” text that shouldn’t be connected, just because it’s on the same baseline, even though it’s part of two separate text flows. It would make a lot more sense if the images with this plugin enabled became interactive in the sense of allowing you to draw crop-mark-like rectangles on them, where the text in the rectangle would then be separately OCRed. (As well—even for English!—some text has the characters flowing vertically instead of horizontally. Rather than recognizing these as individual “lines”, it’d be helpful to be able to draw an arrow indicating direction-of-flow.) Yes, you could train a separate model to discover flow-boxes—but nobody’s done it, and AFAIK we don’t have a good training corpus for it (save for rendering documents with TeX and using the output of an intermediate layout step as the training data, but this only goes so far.) It would make a lot of sense to use these users’ indications of flow-box placement (with their opt-in consent) as this training corpus. (This model alone would be valuable enough to make a sustainable business out of—a lack of a flow-box recognizer is the very reason that Evernote, Notes.app, etc. all have handwriting-based indexing, but no handwriting transcription.)