5 ms·
I'd avoid using this. This is a format that's lossy re: text, so text and numbers can get silently rewritten; it reuses a letter image, placing it throughout a
by pronoiac 8y ago
I'd avoid using this. This is a format that's lossy re: text, so text and numbers can get silently rewritten; it reuses a letter image, placing it throughout a page wherever it's recognized. A low resolution scan or bad ocr results in silently corrupted text.
- exikyut 8y agoDo you have any sources for this? This is very interesting. Particularly because Xerox photocopiers (yes, self-contained photocopiers) had exactly the same kinds of issues due to image compression engine glitches (https://news.ycombinator.com/item?id=6156238 https://news.ycombinator.com/item?id=6156238, https://news.ycombinator.com/item?id=9584172 https://news.ycombinator.com/item?id=9584172)
- azdle 8y agoIt's not sourced, but the Wikipedia page says it uses something called JB2 compression which is susceptible to this: https://en.wikipedia.org/wiki/DjVu#Compression https://en.wikipedia.org/wiki/DjVu#Compression
- bb88 8y agoJBIG2 was the problem algorithm with the photocopiers. JBIG2 was based upon JB2.
- jacquesm 8y agoI've been reading quite a few DjVu encoded documents and have yet to see a single occurrence of what you describe. If there are such errors they will be visually indistinguishable from the original, and a good OCR package would likely identify the right letter given the context. Do you have some concrete examples of corrupted texts? I'd very much like them to use as test cases in a document processing pipeline.
- reaperducer 8y agohave yet to see a single occurrence of what you describe How would you know? Maybe if a document had an unusual misspelling. But how would you know if DjVu munged a 3 into an 8?
- LeoPanthera 8y agoBut this is an entirely hypothetical weakness. Without at least one example of it happening in the wild, it may as well not exist.
- monista 8y agoLet me assure you, this does exist in the wild. I spotted it many times, especially in documents scanned 15+ years ago, when space and bandwidth were an issue and people did anything to reduce file size, at the cost of quality. It was rare enough to not make the format unusable, but not just once, making you worry "was it 3 sp. or 8 sp.?" It seems also that the problem happens more often with cyrillic glyphs which have less descending/accending elements that typical latin ones.
- jacquesm 8y agoWith DjVu or with ocr'd documents in general?
- monista 8y agoWhile of course it's an issue with any scanned docs, the DjVu compression method makes it harsher, as, say, you can end with _all_ '8's in document replaced with '3' (or, say, 'h's with 'n's), at the same time keeping seemingly decent appearance so that you don't suspect that something is wrong.
- jacquesm 8y agoI'd love to see some concrete examples. I totally understand how this could theoretically happen but it all hinges on it actually happening in the documents that I'm working with and to have a test case where it happens for sure would make all the difference in determining whether or not the documents I'm working with are at risk or not and if so if I can detect which ones are at risk (that would be half the battle won). It would have to be a case where a human would see a 3 or an 8 (or a similar transposition with other glyphs) resulting in a document that is corrupted in such a way that afterwards the human would see the (invalid) alternative. Even a single instance of this happening would be very relevant.
- magic_quotes 8y agoCJB2 from DjVuLibre has lossless mode and it's enabled by default.