5 ms·

I can't quite see the reason why you would lossily compress something when your machine's purpose is to duplicate things. Anyone got a reasonable reason for do

by ElliotH 13y ago

I can't quite see the reason why you would lossily compress something when your machine's purpose is to duplicate things. Anyone got a reasonable reason for doing this?

dietrichepp 13y ago
The article has been updated with the probable cause for this error.
wahnfrieden 13y ago
Cheaper components, maybe? (If it lets them get by with less memory for example.)
hga 13y ago
Good point. Looking at a product page (http://www.office.xerox.com/multifunction-printer/color-multifunction/workcentre-7545-7556/spec-enus.html http://www.office.xerox.com/multifunction-printer/color-mult...), I see that the first model mentioned is multifunction, it can "Copy, email, fax, print, [and] scan". So it sounds like there's one code path and it's seriously broken. I looked at the first settings page, and while it's in German I can see it's 200 DPI. There's no excuse for default lossy compression when you're at 200 DPI and doing office sized paper. We didn't do that in 1991, we got CCITT Group 4 lossless compression of around 50KB per image plus or more generally minus for 8.5x11 inch paper, although we did do thinks like noise reduction and straightening documents (that makes them compress better, among other things).
- linohh 13y ago
  CCITT Group 4 also known as Modified Modified Read is in no way something you'd be wanting to use now ever. I'm telling you why MMR sucks hard: 1. It's monochrome. No greyscale, no color. This works for text and lines, but nothing else. No big surprise, it was designed for Fax. But this makes CCITT G3 and G4 lossy. 2. It has no defined endianess. This adds another fault risk which you won't see coming as long as you're working on an isolated platform but can hit you in the nuts when you change hardware or software. 3. The data does not contain resolution or dimensional information, as well as no information about endianess This means that you have to rely on a container providing these informations. It could be TIFF, it could be PDF, it could be something an intern coded during coffee break. This is good on one hand, but evil on the other. Software is sold, saying CCITT G4 compression (a standard, after all) is used, while the data can be embedded in proprietary containers. 4. It's a 2D compression, meaning the compression is applied on a matrix of binary pixel data. As the standard does not specify the dimensions, you depend on another image container like TIFF to provide information. Because G4 removed EOL markers, there is no way to reconstruct image dimensions from the compressed data alone. 5. It's not exactly fault tolerant. Transmission errors can influence larger areas of the image up to making the picture totally unreadable. Flipped bits are not too critical, missing bits are, due to the 2D compression. There are many excellent, fault tolerant, standardized Image formats ready to use for document processing and archiving, CCITT G4 isn't exactly one of them.
  - hga 13y ago
    Errr, I didn't communicate clearly. What I meant to say was that CCITT Group IV gave acceptable sizes for early-'90s computing power, CPU and disk, and something at or better than its level of lossless compression today should be even more acceptable. And in light of this screwup, I suspect we'd agree that Xerox would have been better off to use lossless (well, after the scanning, as you point out, but then again no one was willing to pay for color) CCITT Group IV than overly clever lossy JBIG2. "It could be TIFF, it could be PDF, it could be something an intern coded during coffee break." It could be something a journeyman software engineer edging to expect coded in a Saturday afternoon in a very fast paced project; for me, 3 weeks on the "engine". And, oh my, I can't remember encoding endiness, except of course for the leading TIFF bytes. But I had a guy who knew this cold telling me what to do, he was the one who debugged all our raw compressed data problems bit by bit. And, yeah, it was an "Intel" little endian TIFF, and I think I recall the Kodak Powescans produced that (600 pound monsters that could scan 18 inches per second at 200 DPI). Hmmm, at least back then, "TIFF" was the selling point, and, oh yeah, it's Group IV compressed (except of course when it wasn't, we once dealt with some weird enhanced Group III).
    - linohh 13y ago
      Of course they would have been better off with T.6, as Group 4 at least did not modify the image content. However especially with TIFF there are/were countless implementations of viewers, components, libraries and every single one of them had their own habits. Some would not regard endianess, some would assume payload endianess is the same as the TIFF, some did respect the Tag for byte order specific to the image. When I coded my first TIFF Library, I was around 14 and the most troublesome part of doing it was keeping myself from bashing my head against the next available wall due to stupidity of other people who thought interpreting a standard according to their wishes was ok, because there'd never be someone trying to display the images with a viewer different from theirs. I don't know how deep you have dived into TIFF, but maybe you remember the TIFF6 Standard way of embedding JPEG. It was the biggest pain in the ass imaginable, having to parse JPEG files, splitting them and packaging it into different TIFF Tags. Before TTN2 and easy embedding of JPEG Images, everyone invented their own way of avoiding the standard. Some defined their own compression type, some used the standard compression type, but used it in a nonstandard way, ah, I'm starting to lose my hair again ;-)
simonster 13y ago
If you're scanning a long document to a PDF, compression makes a lot of sense. It's the difference between being able to email the PDF as an attachment and having to find a place to put the file online.
- algorias 13y ago
  Exactly, and that is why there should be a compression step on the code path that handles the paper -> pdf case. This doesn't make any sense in a paper -> paper case, however, as any electronic version of the image will only be stored internally, for a very brief time.
- lostlogin 13y ago
  It is amazing the things people expect to be emailed. Can you email me the MRI scan? It's over 1500 images. Can you email it?
  - sneak 13y ago
    It's amazing to me the engineers who refuse to update their worldviews about normal people's mental models for "sending data" that still get amazed by this. The size of the files or the number of them are totally irrelevant.
    - lotharbot 13y ago
      Normal people seem to get that it's considerably harder to ship a barn than a letter, and that if you want to move a barn you use a specialty service rather than the post office. The size and number of files are and should be totally relevant even to "normal" people. When someone asks for something in e-mail, it's perfectly reasonable to say "no, it's much too big" and expect them to understand.
      - michaelt 13y ago
        I think we can all agree that e-mails should have finite size - it's not a very good protocol for transferring multi-gigabyte files, for sure! Where we would disagree is where that limit should be drawn. I've seen systems in this day and age that fail in the face of e-mails as small as 5 megabytes (e.g. Yahoo Popgate) which IMHO is far too low - but evidently some sysadmins disagree with me!
rly_ItsMe 13y ago
In the good old days of analog copiers this would be impossible - the scanner send the light through a system of mirrors to the drum, the drum gets static charged, the toner is pulled on the charged parts and gets transferred to the transfer belt, here the paper has the opposite charge and pulls the toner off of the transfer belt, goes through the fusing unit and here is the toner 'burned' to the paper. End of Story On a modern copier the scanner transfers the data first to RAM and than usually to a hard disk (the most of the people do not even know that the "copy machine" has one and saves the scanned stuff to it). From that hard disk the data where transmitted via laser to the drum Tadaaa - you have the reason for having data be compressed on a modern copier.
- simonster 13y ago
  I might have missed something, but my reading is that the article doesn't state or imply this happens with regular photocopies, only with scans to PDF.
- blt 13y ago
  Yup, and those old analog copiers - good ones at least - had beautiful crisp output. The resolution was good enough to reproduce printing dots so they could even duplicate photos from books. Continuous tone of an analog photograph didn't work as well. They sure were expensive though.
Someone 13y ago
Others have pointed out a credible explanation: to have the document take less space on their hard disk. However, it does not have to be compression, per se. Modern copiers want to correct all kinds of errors such as creases and staples. They also want to optimize the colors. To do that, they have logic for detecting what areas of the page are full-color and which are black and white, which are half-tone printed, which are text, line art, photograph, whether the paper might have aged, etc. I don't know what tricks they use, but I do not rule out that they will replace 'looks somewhat dirty' patches with an 'obviously higher quality version' of them, and use too aggressive parameters in some of those heuristics.
- harrytuttle 13y ago
  Well we have 14TiB of financial documents archived on our kit. There is no way we even would consider such compression!!! The whole thing is dangerous and wholly illogical. This is akin to a crappy crime flick where someone hits the "enhance!" button on a CCTV still a few times and gets to see the dirt on the guy's teeth. In this case, the computer decides the guy is female and has no teeth.
  - deleted 13y ago
    [deleted]
  - anonymfus 13y ago
    IIRC when security cameras moved from individual frame compression algorithms like M-JPEG to modern codecs which could sometimes replace small movement in background with still image if there is a bigger change in foreground there were news reports about some problems with investigations.

by ElliotH 13y ago

I can't quite see the reason why you would lossily compress something when your machine's purpose is to duplicate things. Anyone got a reasonable reason for doing this?

dietrichepp 13y ago

The article has been updated with the probable cause for this error.

wahnfrieden 13y ago

Cheaper components, maybe? (If it lets them get by with less memory for example.)

hga 13y ago

Good point. Looking at a product page (http://www.office.xerox.com/multifunction-printer/color-multifunction/workcentre-7545-7556/spec-enus.html http://www.office.xerox.com/multifunction-printer/color-mult...), I see that the first model mentioned is multifunction, it can "Copy, email, fax, print, [and] scan". So it sounds like there's one code path and it's seriously broken. I looked at the first settings page, and while it's in German I can see it's 200 DPI. There's no excuse for default lossy compression when you're at 200 DPI and doing office sized paper. We didn't do that in 1991, we got CCITT Group 4 lossless compression of around 50KB per image plus or more generally minus for 8.5x11 inch paper, although we did do thinks like noise reduction and straightening documents (that makes them compress better, among other things).

linohh 13y ago
CCITT Group 4 also known as Modified Modified Read is in no way something you'd be wanting to use now ever. I'm telling you why MMR sucks hard: 1. It's monochrome. No greyscale, no color. This works for text and lines, but nothing else. No big surprise, it was designed for Fax. But this makes CCITT G3 and G4 lossy. 2. It has no defined endianess. This adds another fault risk which you won't see coming as long as you're working on an isolated platform but can hit you in the nuts when you change hardware or software. 3. The data does not contain resolution or dimensional information, as well as no information about endianess This means that you have to rely on a container providing these informations. It could be TIFF, it could be PDF, it could be something an intern coded during coffee break. This is good on one hand, but evil on the other. Software is sold, saying CCITT G4 compression (a standard, after all) is used, while the data can be embedded in proprietary containers. 4. It's a 2D compression, meaning the compression is applied on a matrix of binary pixel data. As the standard does not specify the dimensions, you depend on another image container like TIFF to provide information. Because G4 removed EOL markers, there is no way to reconstruct image dimensions from the compressed data alone. 5. It's not exactly fault tolerant. Transmission errors can influence larger areas of the image up to making the picture totally unreadable. Flipped bits are not too critical, missing bits are, due to the 2D compression. There are many excellent, fault tolerant, standardized Image formats ready to use for document processing and archiving, CCITT G4 isn't exactly one of them.
- hga 13y ago
  Errr, I didn't communicate clearly. What I meant to say was that CCITT Group IV gave acceptable sizes for early-'90s computing power, CPU and disk, and something at or better than its level of lossless compression today should be even more acceptable. And in light of this screwup, I suspect we'd agree that Xerox would have been better off to use lossless (well, after the scanning, as you point out, but then again no one was willing to pay for color) CCITT Group IV than overly clever lossy JBIG2. "It could be TIFF, it could be PDF, it could be something an intern coded during coffee break." It could be something a journeyman software engineer edging to expect coded in a Saturday afternoon in a very fast paced project; for me, 3 weeks on the "engine". And, oh my, I can't remember encoding endiness, except of course for the leading TIFF bytes. But I had a guy who knew this cold telling me what to do, he was the one who debugged all our raw compressed data problems bit by bit. And, yeah, it was an "Intel" little endian TIFF, and I think I recall the Kodak Powescans produced that (600 pound monsters that could scan 18 inches per second at 200 DPI). Hmmm, at least back then, "TIFF" was the selling point, and, oh yeah, it's Group IV compressed (except of course when it wasn't, we once dealt with some weird enhanced Group III).
  - linohh 13y ago
    Of course they would have been better off with T.6, as Group 4 at least did not modify the image content. However especially with TIFF there are/were countless implementations of viewers, components, libraries and every single one of them had their own habits. Some would not regard endianess, some would assume payload endianess is the same as the TIFF, some did respect the Tag for byte order specific to the image. When I coded my first TIFF Library, I was around 14 and the most troublesome part of doing it was keeping myself from bashing my head against the next available wall due to stupidity of other people who thought interpreting a standard according to their wishes was ok, because there'd never be someone trying to display the images with a viewer different from theirs. I don't know how deep you have dived into TIFF, but maybe you remember the TIFF6 Standard way of embedding JPEG. It was the biggest pain in the ass imaginable, having to parse JPEG files, splitting them and packaging it into different TIFF Tags. Before TTN2 and easy embedding of JPEG Images, everyone invented their own way of avoiding the standard. Some defined their own compression type, some used the standard compression type, but used it in a nonstandard way, ah, I'm starting to lose my hair again ;-)

simonster 13y ago

If you're scanning a long document to a PDF, compression makes a lot of sense. It's the difference between being able to email the PDF as an attachment and having to find a place to put the file online.

algorias 13y ago
Exactly, and that is why there should be a compression step on the code path that handles the paper -> pdf case. This doesn't make any sense in a paper -> paper case, however, as any electronic version of the image will only be stored internally, for a very brief time.
lostlogin 13y ago
It is amazing the things people expect to be emailed. Can you email me the MRI scan? It's over 1500 images. Can you email it?
- sneak 13y ago
  It's amazing to me the engineers who refuse to update their worldviews about normal people's mental models for "sending data" that still get amazed by this. The size of the files or the number of them are totally irrelevant.
  - lotharbot 13y ago
    Normal people seem to get that it's considerably harder to ship a barn than a letter, and that if you want to move a barn you use a specialty service rather than the post office. The size and number of files are and should be totally relevant even to "normal" people. When someone asks for something in e-mail, it's perfectly reasonable to say "no, it's much too big" and expect them to understand.
    - michaelt 13y ago
      I think we can all agree that e-mails should have finite size - it's not a very good protocol for transferring multi-gigabyte files, for sure! Where we would disagree is where that limit should be drawn. I've seen systems in this day and age that fail in the face of e-mails as small as 5 megabytes (e.g. Yahoo Popgate) which IMHO is far too low - but evidently some sysadmins disagree with me!

rly_ItsMe 13y ago

In the good old days of analog copiers this would be impossible - the scanner send the light through a system of mirrors to the drum, the drum gets static charged, the toner is pulled on the charged parts and gets transferred to the transfer belt, here the paper has the opposite charge and pulls the toner off of the transfer belt, goes through the fusing unit and here is the toner 'burned' to the paper. End of Story On a modern copier the scanner transfers the data first to RAM and than usually to a hard disk (the most of the people do not even know that the "copy machine" has one and saves the scanned stuff to it). From that hard disk the data where transmitted via laser to the drum Tadaaa - you have the reason for having data be compressed on a modern copier.

simonster 13y ago
I might have missed something, but my reading is that the article doesn't state or imply this happens with regular photocopies, only with scans to PDF.
blt 13y ago
Yup, and those old analog copiers - good ones at least - had beautiful crisp output. The resolution was good enough to reproduce printing dots so they could even duplicate photos from books. Continuous tone of an analog photograph didn't work as well. They sure were expensive though.

Someone 13y ago

Others have pointed out a credible explanation: to have the document take less space on their hard disk. However, it does not have to be compression, per se. Modern copiers want to correct all kinds of errors such as creases and staples. They also want to optimize the colors. To do that, they have logic for detecting what areas of the page are full-color and which are black and white, which are half-tone printed, which are text, line art, photograph, whether the paper might have aged, etc. I don't know what tricks they use, but I do not rule out that they will replace 'looks somewhat dirty' patches with an 'obviously higher quality version' of them, and use too aggressive parameters in some of those heuristics.

harrytuttle 13y ago
Well we have 14TiB of financial documents archived on our kit. There is no way we even would consider such compression!!! The whole thing is dangerous and wholly illogical. This is akin to a crappy crime flick where someone hits the "enhance!" button on a CCTV still a few times and gets to see the dirt on the guy's teeth. In this case, the computer decides the guy is female and has no teeth.
- deleted 13y ago
  [deleted]
- anonymfus 13y ago
  IIRC when security cameras moved from individual frame compression algorithms like M-JPEG to modern codecs which could sometimes replace small movement in background with still image if there is a bigger change in foreground there were news reports about some problems with investigations.