5 ms·

You want an idea of how tough it is, write a universal TIFF[0] reader (not writer -writers are simple). Fun stuff. BTDT. Got the T-shirt. [0] https://www.itu.

You want an idea of how tough it is, write a universal TIFF[0] reader (not writer -writers are simple). Fun stuff. BTDT. Got the T-shirt. [0] https://www.itu.int/itudoc/itu-t/com16/tiff-fx/docs/tiff6.pdf https://www.itu.int/itudoc/itu-t/com16/tiff-fx/docs/tiff6.pd...

mananaysiempre 2y ago
Is that even possible? I once tried to make a matched pyramid (JPEG-in-)TIFF reader/writer pair and to retain some compatibility with other people who do this sort of thing (GIS, medical images, etc.). Virtually nobody agrees on how you’re supposed to do it[1]: some prescribe storing smaller versions as siblings (a NextIFD chain started by the main image), some as children of the largest one (a NextIFD chain pointed to by the SubIFD link of the main image), some as children each other (a SubIFD chain started by the main image), some as both simultaneously (a chain of identical SubIFD and NextIFD fields started by the main image). And I mean, I could decide on something for my writer. But now I’m a reader and I get a TIFF file with some IFDs somehow linked by the NextIFD and/or SubIFD fields. WTF am I supposed to do with it? Is it a pyramid? Is it a multipage document? Is it a birdplane^W a sequence of pyramidal pages? I suppose I can walk the whole thing and construct a DAG, but again, how the hell can I tell what the DAG means? (And don’t take this as a knock against TIFF in general—as far as I know, it’s one of the few image formats that takes the possibility of large and larger-than-memory images seriously. I think HEIF also does? But ISO paywalled it after first making it publicly available, so, hard pass.) [1] Here’s a writeup that comes to similar conclusions: https://dpb587.me/entries/tiff-ifd-and-subifd-20240226 https://dpb587.me/entries/tiff-ifd-and-subifd-20240226
- ChrisMarshallNY 2y ago
  This was in the ‘90s. About the time the ink was still wet on that spec. It was in C++, and I couldn’t do 100%, but I probably got about 80% (but not so performant). The weirdest thing, if I remember correctly, was pixel data with different sizes between stored components.
- ChrisMarshallNY 2y ago
  We did a similar format, to allow editable "JPEGs." The IFD of a JFIF container was normal, except that we added a second entry, with the raw source of the image. Traditional readers saw a JPEG (although an unusually obese one), but our software could access the second entry, which contained the raw source, and the control parameters for all the processing steps that resulted in the JPEG, so we could treat the image as nondestructive, and reversible. It was never actually released, if I remember, but it may well be patented. TIFF was originally designed to store drum scanner data, in realtime, so it uses strips, as opposed to tiles.
edflsafoiewq 2y ago
BMP is pretty gross too, though fortunately far less useful.
- secondcoming 2y ago
  What's gross about BMP, it's one of the easiest image formats out there.
  - genewitch 2y ago
    my limited understanding is it's basically just raw RGB values as binary, with just enough metadata to put it on the screen. BMP from my understanding is, like most things microsoft from that era, just a memory dump of the structure/array holding the image in memory. I wonder if grandparent meant something else, but i don't know enough this instant to guess at what other format?
    - edflsafoiewq 2y ago
      Haha, if only. Read the wiki article on BMP sometime. There's RLE, huffman coding, all kinds of bitdepths and channel masks. Configurable halftoning algorithms. Embedded JPEGs and PNGs? 64-bit BMPs in some kind of fixed point format? And that's on top of the usual Microsoft DWORD BITMAPV2INFOHEADER; struct-dump gunk. The real kicker is it's not even useful. Just say "unsupported" to everything that isn't a dumb RGB dump and no one will ever notice!
      - genewitch 2y ago
        huh, i've been living under a misapprehension for a very long time. Now that you mention it i do remember run length and Huffman. I wonder if i am confusing it with a different file format that is basically just the RGB values with the WxH metadata?
        rhines 2y ago
        You can create BMP files like that - I wrote some image generators that do that and they render fine in all image viewers. But the spec also supports much more, so if you're writing a BMP viewer then things get hairy.
astrange 2y ago
Isn't TIFF one of those file formats that can contain just about anything, making this just about impossible? Like, DNG files are TIFFs, so now you need a raw camera decoder, which is basically subjective.
- ChrisMarshallNY 2y ago
  Yup. Couldn’t do the whole spec, even after about six months of continuous work. I was young and stupid, back then. I learned about not biting off more than I could chew. Important lesson in humility.
- tialaramex 2y ago
  My favourite part of TIFF is what they do about y-ordering. See, sometimes people think images start at the top, so the first data at y=0 is at the top of the image, other people think they start at the bottom like a graph you draw in maths, were y=0 is clearly at the bottom of the image. So TIFF says: That should be a parameter of the image. Why? Well, TIFF came into existence because of scanners, when scanners were first invented each scanner would have its own data format - you're not going to store all this data because that's expensive - who owns that much tape? But when it's scanned clearly the bits you get have some sort of arrangement, maybe a bright white area is 1 and black is 0, maybe the opposite. That's kind of annoying, lets agree a standard. OK, so as Scanner Maker A my proposed standard is: Exactly what my popular A9 Scanner does No! As scanner maker B, clearly the standard should be what our BZ-20 model does No! Everybody at scanner maker C knows the obvious thing to do is derive the standard from the behaviour of our popular C5 and C10 scanners! Result: The TIFF standard says all of the above are OK, just add header data explaining what's going on. Since some scanners would scan a page from the top, those say y=0 is at the top, those vendors whose scanner works the other way say y=0 is at the bottom!
  - nayuki 2y ago
    > My favourite part of TIFF is what they do about y-ordering. Windows BMP (DIB) also has the convention that y=0 is at the bottom. https://en.wikipedia.org/wiki/BMP_file_format https://en.wikipedia.org/wiki/BMP_file_format
aardvark179 2y ago
Ah yes, that was always entertaining. All the different ways additional metadata could be encoded was so much fun if you were dealing with geographical data.

mananaysiempre 2y ago

Is that even possible? I once tried to make a matched pyramid (JPEG-in-)TIFF reader/writer pair and to retain some compatibility with other people who do this sort of thing (GIS, medical images, etc.). Virtually nobody agrees on how you’re supposed to do it[1]: some prescribe storing smaller versions as siblings (a NextIFD chain started by the main image), some as children of the largest one (a NextIFD chain pointed to by the SubIFD link of the main image), some as children each other (a SubIFD chain started by the main image), some as both simultaneously (a chain of identical SubIFD and NextIFD fields started by the main image). And I mean, I could decide on something for my writer. But now I’m a reader and I get a TIFF file with some IFDs somehow linked by the NextIFD and/or SubIFD fields. WTF am I supposed to do with it? Is it a pyramid? Is it a multipage document? Is it a birdplane^W a sequence of pyramidal pages? I suppose I can walk the whole thing and construct a DAG, but again, how the hell can I tell what the DAG means? (And don’t take this as a knock against TIFF in general—as far as I know, it’s one of the few image formats that takes the possibility of large and larger-than-memory images seriously. I think HEIF also does? But ISO paywalled it after first making it publicly available, so, hard pass.) [1] Here’s a writeup that comes to similar conclusions: https://dpb587.me/entries/tiff-ifd-and-subifd-20240226 https://dpb587.me/entries/tiff-ifd-and-subifd-20240226

ChrisMarshallNY 2y ago
This was in the ‘90s. About the time the ink was still wet on that spec. It was in C++, and I couldn’t do 100%, but I probably got about 80% (but not so performant). The weirdest thing, if I remember correctly, was pixel data with different sizes between stored components.
ChrisMarshallNY 2y ago
We did a similar format, to allow editable "JPEGs." The IFD of a JFIF container was normal, except that we added a second entry, with the raw source of the image. Traditional readers saw a JPEG (although an unusually obese one), but our software could access the second entry, which contained the raw source, and the control parameters for all the processing steps that resulted in the JPEG, so we could treat the image as nondestructive, and reversible. It was never actually released, if I remember, but it may well be patented. TIFF was originally designed to store drum scanner data, in realtime, so it uses strips, as opposed to tiles.

edflsafoiewq 2y ago

BMP is pretty gross too, though fortunately far less useful.

secondcoming 2y ago
What's gross about BMP, it's one of the easiest image formats out there.
- genewitch 2y ago
  my limited understanding is it's basically just raw RGB values as binary, with just enough metadata to put it on the screen. BMP from my understanding is, like most things microsoft from that era, just a memory dump of the structure/array holding the image in memory. I wonder if grandparent meant something else, but i don't know enough this instant to guess at what other format?
  - edflsafoiewq 2y ago
    Haha, if only. Read the wiki article on BMP sometime. There's RLE, huffman coding, all kinds of bitdepths and channel masks. Configurable halftoning algorithms. Embedded JPEGs and PNGs? 64-bit BMPs in some kind of fixed point format? And that's on top of the usual Microsoft DWORD BITMAPV2INFOHEADER; struct-dump gunk. The real kicker is it's not even useful. Just say "unsupported" to everything that isn't a dumb RGB dump and no one will ever notice!
    - genewitch 2y ago
      huh, i've been living under a misapprehension for a very long time. Now that you mention it i do remember run length and Huffman. I wonder if i am confusing it with a different file format that is basically just the RGB values with the WxH metadata?
      - rhines 2y ago
        You can create BMP files like that - I wrote some image generators that do that and they render fine in all image viewers. But the spec also supports much more, so if you're writing a BMP viewer then things get hairy.

astrange 2y ago

Isn't TIFF one of those file formats that can contain just about anything, making this just about impossible? Like, DNG files are TIFFs, so now you need a raw camera decoder, which is basically subjective.

ChrisMarshallNY 2y ago
Yup. Couldn’t do the whole spec, even after about six months of continuous work. I was young and stupid, back then. I learned about not biting off more than I could chew. Important lesson in humility.
tialaramex 2y ago
My favourite part of TIFF is what they do about y-ordering. See, sometimes people think images start at the top, so the first data at y=0 is at the top of the image, other people think they start at the bottom like a graph you draw in maths, were y=0 is clearly at the bottom of the image. So TIFF says: That should be a parameter of the image. Why? Well, TIFF came into existence because of scanners, when scanners were first invented each scanner would have its own data format - you're not going to store all this data because that's expensive - who owns that much tape? But when it's scanned clearly the bits you get have some sort of arrangement, maybe a bright white area is 1 and black is 0, maybe the opposite. That's kind of annoying, lets agree a standard. OK, so as Scanner Maker A my proposed standard is: Exactly what my popular A9 Scanner does No! As scanner maker B, clearly the standard should be what our BZ-20 model does No! Everybody at scanner maker C knows the obvious thing to do is derive the standard from the behaviour of our popular C5 and C10 scanners! Result: The TIFF standard says all of the above are OK, just add header data explaining what's going on. Since some scanners would scan a page from the top, those say y=0 is at the top, those vendors whose scanner works the other way say y=0 is at the bottom!
- nayuki 2y ago
  > My favourite part of TIFF is what they do about y-ordering. Windows BMP (DIB) also has the convention that y=0 is at the bottom. https://en.wikipedia.org/wiki/BMP_file_format https://en.wikipedia.org/wiki/BMP_file_format

aardvark179 2y ago

Ah yes, that was always entertaining. All the different ways additional metadata could be encoded was so much fun if you were dealing with geographical data.