4 ms·

> One thing I’ve always wanted from web browsers is the ability to just “see into” images, such that I could copy-and-paste text out of any image embedded in a

• derefr 7y ago
  Naptha has two flaws that exclude it from being what I mean: 1. It relies on the crappy old kind of OCR algorithms, that were only trained on English text. (Yes, even though it uses cloud services. The cloud services themselves use the crappy old OCR algorithms. Literally only the translation services have modern ones.) 2. It isn’t built into a web-browser, in the sense of the model being on-disk as part of the web browser and operating offline, but rather has to upload your images to cloud services. If you want Naptha to be available automatically for any page (such as you would if you were using a screen-reader), then every image on every page you visit is automatically fed to a cloud service. This is not only a horrible privacy leak but also means that with Naptha enabled, pages take forever to fully load. (Which wouldn’t be true with an offline model, given a good computer, or a device that has dedicated ML-model-running circuits like the iPhone.) Also, a third, that’s more of a UX flaw than anything and could be fixed: 3. The model tries to parse the whole image as if it were left-to-right prose, when often the image’s text is in columns, or figure captions, or dialogue balloons. The user can very easily see where the text is, but there’s no way for the user to indicate to the engine where on the image the text is. This means that the model will “connect” text that shouldn’t be connected, just because it’s on the same baseline, even though it’s part of two separate text flows. It would make a lot more sense if the images with this plugin enabled became interactive in the sense of allowing you to draw crop-mark-like rectangles on them, where the text in the rectangle would then be separately OCRed. (As well—even for English!—some text has the characters flowing vertically instead of horizontally. Rather than recognizing these as individual “lines”, it’d be helpful to be able to draw an arrow indicating direction-of-flow.) Yes, you could train a separate model to discover flow-boxes—but nobody’s done it, and AFAIK we don’t have a good training corpus for it (save for rendering documents with TeX and using the output of an intermediate layout step as the training data, but this only goes so far.) It would make a lot of sense to use these users’ indications of flow-box placement (with their opt-in consent) as this training corpus. (This model alone would be valuable enough to make a sustainable business out of—a lack of a flow-box recognizer is the very reason that Evernote, Notes.app, etc. all have handwriting-based indexing, but no handwriting transcription.)