5 ms·
JBIG2 Undetectable Data Corruption: Destroying Our Past, One Character at a Time
- tsujamin 4y agostill capable of emulating CPU circuits for gucci iOS 0day: https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-into-nso-zero-click.html https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-i...
- nullc 4y agoWait till you get ML based compression and you get replacements with entirely different but plausibly looking alternatives that go beyond single character substitutions. "A new research report demonstrates the connection between the rising amount of mass produced housing featuring a dedicated sex dungeon and biased content in the training set used for the compression algorithm in a popular line of architectural plotters"
- Agingcoder 4y agoThat was funny! Thanks.
- Dr_ReD 4y agoYeah, this kind of lossyness can be very bad for serious archival purposes of critical documents, because this small corruption wouldn't be self-evident nor detectable after the fact in an easy way. Most probably its raison d'être is for non-critical stuff, such as literature, where character flips may be easily detectable during encoding and after the fact (eg. with a spell-checker) and won't possibly subvert the meaning of a whole literary work anyway.
- richard_todd 4y agoThis potential problem is overstated if the quality of the scans is decent. I have scanned literally hundreds of books, and encoded them in both djvu and pdf/jbig2 formats, and have never once found a bad character. (Yes, having heard of this issue, I did initially try to find examples) The free djvu tools have parameters you can adjust to make them more or less aggressive at combining similar characters, and above 200DPI it's hard to even force the issue to happen for something like an 8 versus a 6. That was my experience anyway. It's also telling that the example trotted out is always a decades-old photocopier implementation. Unless someone does some comparisons of recent scans and finds a bunch of altered characters, I just don't buy this concern. MRC+JBig2 is really great and the size savings have not been trivial for me personally. (I hesitate to add that the alternative would be, I assume, DCT/jpeg page images, which introduce lots of noisy artifacts of their own... destroying our history one DCT-block at a time!)
- 8n4vidtmkvmk 4y agoI've been reading some books on my ereader and every now and then I've come across a word that has been misspelled with similar looking characters. now I'm wondering if it's this jbig2 issue
- jimjimjim 4y agoThere are a number of scans I've come across on archive.org where the person uploading has used jbig2 and there are pages where letters like 'e' get swapped for an 's'. My general rule when creating pdfs with scans is: If it's mostly text or line drawings and it will be used in a professional setting use png/flate. If it's a photo use jpg (especially if the source is a jpg).
- richard_todd 4y agoI'd love to see some examples (especially if the scan appeared to be of good quality and not a grainy 100dpi affair). Do you have any idea what kinds of documents you were browsing when you found them?
- deleted 4y ago[deleted]
- jwilk 4y agohttps://news.ycombinator.com/item?id=17435514 https://news.ycombinator.com/item?id=17435514 (found via https://en.wikipedia.org/wiki/DjVu https://en.wikipedia.org/wiki/DjVu) There's apparently 22 pages of fluff at the beggining, so you should add that the page numbers. Also "the third line" was meant to be the 3rd line from the bottom. Direct links (to be opened in DjView): https://ia800204.us.archive.org/28/items/cu31924026442156/cu31924026442156.djvu?djvuopts&page=80&highlight=960,277,173,39 https://ia800204.us.archive.org/28/items/cu31924026442156/cu... https://ia800204.us.archive.org/28/items/cu31924026442156/cu31924026442156.djvu?djvuopts&page=442&highlight=983,1544,146,41 https://ia800204.us.archive.org/28/items/cu31924026442156/cu...
- 4y ago
- AreYouSirius 4y agohttps://www.youtube.com/watch?v=c0O6UXrOZJo https://www.youtube.com/watch?v=c0O6UXrOZJo
- joosters 4y agoIf this is such a serious problem, please demonstrate at least one example in Google Books where a scan shows the wrong letter or number. That really can’t be hard to do, since there are lots of common books in GB, there are even OCRs available so you could even do the error checking automatically for thousands of books. If you can’t show a single case of JBIG data corruption in GOogle Books, you have absolutely no justification in calling it such a serious problem!
- j16sdiz 4y agoIt is quite common on archive.org. Google know what they are doing and don't have these error.
- adhesive_wombat 4y agoGoogle Book has atrocious scan quality in general, though (compared to, say, the Microsoft-sponsored archive.org ones). Some pages are completely unreadable and sometimes images are completely deleted as the compressor mistakes them for background.
- vlovich123 4y agoWouldn’t you need to have an original scan not compressed in JBIG? I’m not sure I understand the suggestion on how to find it in Google Books…
- kfichter 4y agoPresumably OCR on the Google Books scans then compare to some known text from a different source (Gutenberg or something)
- real_joschi 4y agoSee also https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... and https://news.ycombinator.com/from?site=dkriesel.com https://news.ycombinator.com/from?site=dkriesel.com
- pronoiac 4y agoI checked: jbig2 is used in ocrmypdf, set to lossless by default. It reduced my book size by 20%.
- EarlKing 4y agoWhat? You didn't entitle your post JBUG? What a wasted opportunity for a good pun.
- thunfisch 4y agohttps://media.ccc.de/v/31c3_-_6558_-_de_-_saal_g_-_201412282300_-_traue_keinem_scan_den_du_nicht_selbst_gefalscht_hast_-_david_kriesel https://media.ccc.de/v/31c3_-_6558_-_de_-_saal_g_-_201412282... Immediately this talk from David Kriesel comes to mind. :)
- deleted 4y ago[deleted]
- userbinator 4y agoSorry, this article is low on content and high on fearmongering. This issue was known years ago and the only example of it seems to be one implementation from Xerox, which was also set to lossy mode --- and I believe that was subsequently fixed too. The other comments here have linked to the previous articles about this, which do give far more detailed information about the problem. JBIG2 in lossless mode won't do this.
- Brian_K_White 4y agoSaying "jbig2 in lossless mode won't do this" is as relevant as saying "png won't do this" The problem is the existence and use of a codec for a purpose, that harms that purpose. It doesn't matter what it's name is. Are scans being produced with this codec? And is it not true that not only is there data loss or corruption like with jpg, but that unlike jpg artifacts you can sometimes not know that the data loss or corruption occurred? That does make it not merely a problem of "I wish they scanned this in higher quality" but a far worse problem of "this document looks good so I trust it" when in fact it was corrupt.
- iforgotpassword 4y agoThe according talk is linked here in the comments - no, it wasn't just lossy mode. It was much harder, but in the end it was reproduced on other quality levels. But if it happened once, who's gonna guarantee this won't happen again? When it happens with numbers, worst case it can have fatal consequences.
- adhesive_wombat 4y agoIf you can't reconstruct the original bit-for-bit, for whatever reason, it must be lossy.
- deleted 4y ago[deleted]
- jwilk 4y ago> "JBIG2" and its derivative "JB2" I don't belive that's correct; it's rather the other way round. https://github.com/barak/djvulibre/blob/release.3.5.28/libdjvu/JB2Image.h#L75 https://github.com/barak/djvulibre/blob/release.3.5.28/libdj... > JB2 has strong similarities with the forthcoming JBIG2 standard developed by the "ISO/IEC JTC1 SC29 Working Group 1" which is responsible for both the JPEG and JBIG standards. This is hardly surprising since JB2 was our own proposal for the JBIG2 standard and remained the only proposal for years. The full JBIG2 standard however is significantly more complex and slighlty less efficient than JB2 because it addresses a broader range of applications.
- cxcorp 4y agoThere has been previous discussion about this issue with JBIG2: https://news.ycombinator.com/item?id=6156238 https://news.ycombinator.com/item?id=6156238 https://news.ycombinator.com/item?id=29223815 https://news.ycombinator.com/item?id=29223815