7 ms·

By "lossy" you mean "17" may look like a crappier "17" with reasonable confidence, but will never, ever, become "21" at any compression setting, then I don't mi

by eksith 13y ago

By "lossy" you mean "17" may look like a crappier "17" with reasonable confidence, but will never, ever, become "21" at any compression setting, then I don't mind lossy. That's not asking for too much, is it?

gwright 13y ago
But the scanner isn't starting off with "17" (as in two ascii characters) it is starting off with a bit mapped image that your brain happens to interpret as the number 17. It is too much to ask that a lossy image compression algorithm never result in a compressed bitmap that your brain interprets exactly as the original. Having various compression/quality options allows you to pick the tradeoff (file size/resulting quality) that is acceptable for your situtation. There is no perfect setting for all situations. Even the original bitmap is an imperfect (i.e. lossy) rendering of the original document.
- mikeash 13y ago
  I don't believe that e.g. JPEG compression can ever take a clearly readable 17 and turn it into a clearly readable 21. Lossy compression implies changing the data, obviously, but it does not have to imply changing the data in that particular way.
  - mansr 13y ago
    JPEG will not do that, but MPEG at high compression levels might.
- eksith 13y ago
  Let's take that premise to OCR then. This whole debacle started with JBIG2 settings that I guess duplicated(?) one section and inserted it where similar text exists. Only it was marginally similar. According to Adam (https://news.ycombinator.com/item?id=6156418 https://news.ycombinator.com/item?id=6156418) this is a known problem that Xerox, who call themselves document people for crying out loud, should have known and compensated for.
- Gormo 13y ago
  It seems a bit too coincidental that images to which human beings assign semantic value are being transformed into images to which human beings assign different semantic value. I don't expect the scanner to have any semantic awareness of the document content, so when I hear "lossy compression", my expectation is "image may become illegible", and not "image may remain legible, but become inaccurate".
  - DrStalker 13y ago
    It's not a coincidence that the thing that looks most like a blurred number is another blurred number. A document will be covered in numbers, and the compression algorithm looks for similar blocks it can re-use; the side effect is sometimes it says "that blurry 4 looks pretty close to this blurry two, so I'll just store that block once and reuse it" The problem is that this is a minor side effect to a programmer and an absolutely massive issue to an end user that no-one had thought of previously, and now we all have to be worried that all our scanned documents might be incorrect. (just because this was found in fuji-xerox scanners doesn't mean other brands don't also have the issue)
  - jessedhillon 13y ago
    This is hacker news -- I don't expect everyone to know how jbig2 or other compression scheme works. But before you insinuate that the scanner has semantic awareness of the document and is altering that meaning in a less-than-coincidental way, I would hope that you could have a cursory look at how such compression works. The issue only involves small letters, because the compression scheme breaks up the image into patches and then tries to identify visually similar blocks and reuse them. Certain settings can allow for small blocks of text to be deemed identical, within a threshold, and thus replaced. That's all. Coincidence, not semantic awareness. Hence the advisory notice to use a higher resolution -- smaller block sizes.
    - enraged_camel 13y ago
      >>This is hacker news -- I don't expect everyone to know how jbig2 or other compression scheme works. As opposed to what, ImageCompression News where you can expect everyone to know it?
deleted 13y ago
[deleted]

by eksith 13y ago

gwright 13y ago

But the scanner isn't starting off with "17" (as in two ascii characters) it is starting off with a bit mapped image that your brain happens to interpret as the number 17. It is too much to ask that a lossy image compression algorithm never result in a compressed bitmap that your brain interprets exactly as the original. Having various compression/quality options allows you to pick the tradeoff (file size/resulting quality) that is acceptable for your situtation. There is no perfect setting for all situations. Even the original bitmap is an imperfect (i.e. lossy) rendering of the original document.

mikeash 13y ago
I don't believe that e.g. JPEG compression can ever take a clearly readable 17 and turn it into a clearly readable 21. Lossy compression implies changing the data, obviously, but it does not have to imply changing the data in that particular way.
- mansr 13y ago
  JPEG will not do that, but MPEG at high compression levels might.
eksith 13y ago
Let's take that premise to OCR then. This whole debacle started with JBIG2 settings that I guess duplicated(?) one section and inserted it where similar text exists. Only it was marginally similar. According to Adam (https://news.ycombinator.com/item?id=6156418 https://news.ycombinator.com/item?id=6156418) this is a known problem that Xerox, who call themselves document people for crying out loud, should have known and compensated for.
Gormo 13y ago
It seems a bit too coincidental that images to which human beings assign semantic value are being transformed into images to which human beings assign different semantic value. I don't expect the scanner to have any semantic awareness of the document content, so when I hear "lossy compression", my expectation is "image may become illegible", and not "image may remain legible, but become inaccurate".
- DrStalker 13y ago
  It's not a coincidence that the thing that looks most like a blurred number is another blurred number. A document will be covered in numbers, and the compression algorithm looks for similar blocks it can re-use; the side effect is sometimes it says "that blurry 4 looks pretty close to this blurry two, so I'll just store that block once and reuse it" The problem is that this is a minor side effect to a programmer and an absolutely massive issue to an end user that no-one had thought of previously, and now we all have to be worried that all our scanned documents might be incorrect. (just because this was found in fuji-xerox scanners doesn't mean other brands don't also have the issue)
- jessedhillon 13y ago
  This is hacker news -- I don't expect everyone to know how jbig2 or other compression scheme works. But before you insinuate that the scanner has semantic awareness of the document and is altering that meaning in a less-than-coincidental way, I would hope that you could have a cursory look at how such compression works. The issue only involves small letters, because the compression scheme breaks up the image into patches and then tries to identify visually similar blocks and reuse them. Certain settings can allow for small blocks of text to be deemed identical, within a threshold, and thus replaced. That's all. Coincidence, not semantic awareness. Hence the advisory notice to use a higher resolution -- smaller block sizes.
  - enraged_camel 13y ago
    >>This is hacker news -- I don't expect everyone to know how jbig2 or other compression scheme works. As opposed to what, ImageCompression News where you can expect everyone to know it?

deleted 13y ago

[deleted]