11 ms·

PSD is not my favourite file format (2009)

235 points by kruuuder 6y ago

krylon 6y ago
While I totally empathize with the poor author who had to suffer through this madness, I thoroughly enjoyed reading this rant. My favorite part was this: PSD is not a good format. PSD is not even a bad format. Calling it such would be an insult to other bad formats, such as ...
- stagger87 6y ago
  I'd love to hear why they think JPEG is a worse format.
  - user-the-name 6y ago
    Not worse, that is the point. It is merely a bad format, which is better than what PSD is. As for why JPEG is bad: Did you know there is no one defined way to store the width and height of a JPEG image?
    - stagger87 6y ago
      Thank you, I totally misread that!
    - userbinator 6y ago
      As someone who has written a JPEG decoder, I am compelled to say that you are wrong, and the width and height are stored in the SOF marker segment. http://vip.sugovica.hu/Sardi/kepnezo/JPEG%20File%20Layout%20and%20Format.htm http://vip.sugovica.hu/Sardi/kepnezo/JPEG%20File%20Layout%20...
      - user-the-name 6y ago
        I might be misremembering, but at least, SOF does not tell you what is actually in the channels of the image, so it is not enough to figure out how to actually display an image.
    - just_for_you 6y ago
      To my knowledge, I see most JPEG images encode their height in an "APP1" EXIF segment, and in their SOF1 segment (which is basically just a basic-info segment for the core JPEG bitstream). It's also possible to store the size in an APP1 XMP section, but that's mostly non-essential metadata for interchange purposes, and I usually don't see the size duplicated there for JPEGs anyway (PNG does though, if it has an XMP chunk). It's possible that some of the other officially-defined segments (see https://exiftool.org/TagNames/JPEG.html https://exiftool.org/TagNames/JPEG.html) might also contain duplicate info, but then that'd be specific to the app that uses it, not for general-purpose use. Storing the size in easily-parseable EXIF metadata and then once again in the JPEG bitstream seems pretty reasonable to me. Not to mention if you run a metadata/EXIF stripper on a JPEG file, then you remove all duplicate info and the only remaining image-dimensions will be stored in the JPEG bitstream anyway.
  - eyesee 6y ago
    Well for one thing it's not a file format. For that there's JFIF and EXIF.
    - stagger87 6y ago
      I actually thought about that when I typed my reply, but decided the meaning of my question would still be clear the way it was phrased. Despite that, my question is more or less irrelevant since I misunderstood the author.
jolmg 6y ago
So, what's this project about? There's no README nor description nor anything. I'm curious as to why they'd find it so worthwhile to try to parse PSD despite those troubles. They seem to already have parsers for other image formats.
- krylon 6y ago
  Xee is an image viewer. If you follow the link to the landing page of the repo, it says in the About section it is "Xee source code for xCode 4.5"
- folkrav 6y ago
  I've seen that rant a while ago. IIRC, Xee is an image viewer for macOS.
- masklinn 6y ago
  > So, what's this project about? There's no README nor description nor anything. It's Xee, a lightweight (and excellent) image viewer for macOS. That's not the original repo (or author), used to be on google code, moved to bitbucket, then abandoned / sold the project: it was open-source until version 2.2 or something, v3 is is closed source.
  - Hamuko 6y ago
    Unfortunately Xee3 is basically abandoned at this point. MacPaw bought it alongside The Unarchiver and they haven't really been doing anything with it. Last update was three years ago. I wonder if it's just going to stop working in some future macOS version.
    - aasasd 6y ago
      > bought it alongside The Unarchiver and they haven't really been doing anything with it Good, because IIRC The Unarchiver was caught gathering stats or sending something on the network, after an updated release.
hprotagonist 6y ago
This is one of my favourite rants. The other one is the SO answer about X/HTML parsing with regex. https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454 https://stackoverflow.com/questions/1732348/regex-match-open...
- dspillett 6y ago
  "Please do not flag it for our attention." is the perfect finishing touch too.
- Blikkentrekker 6y ago
  I find it weird that the comment is locked rather than deleted. Funny it be, it does not seem à propos for answering a quæstion.
  - _jal 6y ago
    At this point, it is a cultural artifact.
  - Alupis 6y ago
    It's from a time before StackOverflow (and StackExchange at large) got very ridged with all the rules. Back then, there were a lot of "fun" questions and answers, including the infamous "What is the best comment in source code you have ever encountered?"[1] Some old timers might remember those days... SO was a much more fun place back then. Now it's rife with down-voters, close-voters and hostile-towards-newbie folks. [1] https://stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered https://stackoverflow.com/questions/184618/what-is-the-best-...
  - NobodyNada 6y ago
    The post was locked because it had received hundreds of flags and comments complaining that the post was broken -- even with the moderator note at the bottom -- and several edits trying to "fix" the post. See https://meta.stackoverflow.com/questions/250099#comment637_250099 https://meta.stackoverflow.com/questions/250099#comment637_2...
deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
rdtsc 6y ago
> Why, for instance, did it suddenly decide that these particular chunks should be aligned to four bytes, and that this alignement (sic) should not be included in the size? Not saying this happened here, but I have seen this type of mistake before. It was because they "simply" cast a C/C++ struct to a binary blob https://en.wikipedia.org/wiki/Data_structure_alignment#Typical_alignment_of_C_structs_on_x86 https://en.wikipedia.org/wiki/Data_structure_alignment#Typic... and wrote that to disk (or sent over the network in my case). So that particular compiler version and architecture-specific struct field alignment became the "official" format. It just takes one goofy mistake like that and everyone has to deal with it for years to come.
- masklinn 6y ago
  Note that the issue here is less the part that you quoted and more the two sentences surrounding it, PSD has all of * unaligned chunks * aligned chunks with alignment included in size * aligned chunks with alignment not included in size The problem is not the specific choice, "Either one of these three behaviours would be fine", it's that "PSD, of course, uses all three, and more."
- wolrah 6y ago
  As I understand it that's more or less what the older MS Office formats were, just a dump of the entire OLE object representing the document as-is. I believe SimCity used a similar strategy as well, the .cty and I think SC2k's .s2k formats just dumped the in-memory representation to disk,.
  - krylon 6y ago
    I worked at a company once where they, too, used raw memory dumps as their file format. The compiler flags telling the compiler how to align structs were different for debug and non-debug builds. So, of course, the first thing I did, was to create non-debug build and try to open a file created by a debug-build-executable, the program crashed and burned without giving me a meaningful error message, it took me hours to understand my mistake. Raw memory dumps are very neat efficiency-wise, but they are extremely fragile.
mmastrac 6y ago
This used to show up often on HN but I haven't seen it in years. The best comment was from this old thread [1]: > I enjoyed the commit message most: > > r11 by paracelsus on Sep 11, 2007 Diff > Photoshop loader is DONE for now, fuck you Adobe https://news.ycombinator.com/item?id=575122 https://news.ycombinator.com/item?id=575122
deleted 6y ago
[deleted]
riffraff 6y ago
I remember reading this years ago, I think it was my first encounter with the "... fierce passion of a thousand suns". It still made me chuckle even today, thanks for sharing it.
bluedino 6y ago
Joel on Software, "Why are Microsoft Office File Formats So Complicated" https://www.joelonsoftware.com/2008/02/19/why-are-the-microsoft-office-file-formats-so-complicated-and-some-workarounds/ https://www.joelonsoftware.com/2008/02/19/why-are-the-micros... They were designed to be fast on very old computers. They were designed to use libraries. They were not designed with interoperability in mind. They have to reflect the history of the applications,
ericol 6y ago
> this Rube Goldberg of a file format Concise, to the point, and a good enough insult as to keep it in file. I love it.
egonschiele 6y ago
This came up a couple months ago. I really wish there was a standard open format that did most of the things PSD does. I had asked the folks at Procreate if they'd design something, since they are sort of a challenger to Photoshop, but they said it wasn't something they wanted to do.
- iggldiggl 6y ago
  Where this might get tricky is if you seriously want to support non-destructive editing as well, because in that case any filter that can be applied in non-destructive mode effectively needs to become part of the file format specification, too.
  - wongarsu 6y ago
    You could store filters generic enough that any software can skip over filters it doesn't recognize (e.g. you store filter type uuid, filter data length, then arbitrary filter parameters), have some open registry where you can get official filter uuids in return for an example implementation. Of course not everyone will register every filter, so the format should probably store the image once unfiltered and once with all filter applied, that way if a program doesn't recognize or implement some filter it can still fall back to the destructively filtered version. But of course that increases file size, which may or may not be a concern.
- preommr 6y ago
  Procreate is 10$ for a one-time purchase. PS is 13$/mo. Actually, if you got just PS it would be 30$/mo. And even then I've seen people complain that procreate is too expensive. I don't really have a point other than design tools and their pricing is a pet issue of mine and I like bringing up how insane it is at any given chance.
- hallarempt 6y ago
  We started working on OpenRaster for that: https://www.openraster.org/ https://www.openraster.org/ -- but it's not really moving along very well.
Triv888 6y ago
Plain text file is my favorite file format... because in the end, portability is what matters the most.
- gjvr 6y ago
  ASCII? UTF-8? UTF-16? ... ...
  - Triv888 6y ago
    lol... yes. UTF-512 should be enough for everybody!
    - _kst_ 6y ago
      UTF-640K?
  - user-the-name 6y ago
    Shift-JIS? GB18030? KOI8-R? "Plain text" does not really exist.
  - Dylan16807 6y ago
    Anyone using the words "plain text" for a file had better mean either ASCII or UTF-8. And for almost all purposes these days it should be the latter.
- Arnt 6y ago
  By plain text, do you have in mind CRLF or LF, and for structured data, do you have in mind YAML, JSON or one of the umpty others?
SimianLogic2 6y ago
I have deep sympathy for this, after having done a ton of work with the SWF format and the (even more poorly documented) AEPX format.
1f60c 6y ago
For posterity's sake, the link should probably be changed to https://github.com/zepouet/Xee-xCode-4.5/blob/83394493f51991748b9b4706e6d37a8ed874bc05/XeePhotoshopLoader.m#L108-L136 https://github.com/zepouet/Xee-xCode-4.5/blob/83394493f51991...
bloudermilk 6y ago
I had a good laugh reading this. Of course the irony is that the author, after having gone through the hellish process of learning the spec, didn't documented any of the code (in that file).
hallarempt 6y ago
PSD is horrible, that is certainly true. And underdocumented. And dumb... And we needed to implement read/write support for that in Krita. But Painttool Sai's format is actively evil, with different kinds of encryption for different size layers and things like that: https://github.com/Wunkolo/libsai/issues/6 https://github.com/Wunkolo/libsai/issues/6 (We tried to use that library to implement sai file format support for Krita, but then ran into trouble...) Manga Studio/Clip Studio Paint's file format is interesting as well: it's just an SQLite database, with raster data stored as blobs. Not going to implement that for Krita either.
- Gualdrapo 6y ago
  Thank you for working on Krita.
- Jasper_ 6y ago
  That doesn't look like encryption or differences in algorithms, just the usual confusion and edge cases when you don't have a full mental model of the file format's fixed-size vfs entries.
- pfranz 6y ago
  > Manga Studio/Clip Studio Paint's file format is interesting as well: it's just an SQLite database, with raster data stored as blobs. Not going to implement that for Krita either. I'm curious why? I'm not trying to start anything, but I remember this discussion from a few months ago: https://news.ycombinator.com/item?id=23509914 https://news.ycombinator.com/item?id=23509914 > I wish my predecessor at Krita had made that choice (sqlite), instead of choosing to re-use the KOffice xml-based file format that's basically a zip file with stuff in it. It would have made adding the animation feature so much easier. To summarize the benefits to something like sqlite over zip+xml (not all were made by that author) is; update in place means not having to overwrite the whole file, not inventing xml format for describing an image, not needing to keep the entire representation in-memory, a longer saving process is more fragile, incremental loading.
  - hallarempt 6y ago
mst 6y ago
Personally, rather than printing things out and then setting them on fire, I've preferred printing things out then when I'm done with it donating them to be shredded and used as horse bedding. Some code just deserves to end its life being shat on by a horse.
just_for_you 6y ago
I would actually beg to disagree, as someone who was bored a few weekends ago and decided to go through a few reasonably-complex PSD files (eg, ones with different layer types, bezier curves, etc.) in a hex editor with the official specification in-hand in an attempt to understand the format. (The specification can be found here: https://www.adobe.com/devnet-apps/photoshop/fileformatashtml/ https://www.adobe.com/devnet-apps/photoshop/fileformatashtml...). After a few hours, I was able to grok it without too much difficulty, and found the format was reasonably well laid-out. Yes, it does suffer from some "let's take the in-memory representation and bake it into the on-disk format"-isms, and there were a few things were not covered by the specification (eg, a couple Resource IDs aren't mentioned at all, or they are, but there's no documentation on how to interpret their content) - but it's not anywhere close to insane. To give an example, partially going off memory from what I remember seeing in real files, but mostly skimming the spec, a very basic gist of a typical PSD files might look something like this. (I can only assume HN is going to mangle the formatting of the following text, so expect to see multiple edits to this comment): EDIT: I was right about the formatting getting messed up. I've posted it to Pastebin and will call it a day: https://pastebin.com/raw/WZNrCSAP https://pastebin.com/raw/WZNrCSAP So that (the Pastebin link above) is the basic and oversimplified gist of a PSD file. It's mostly just reading 4-byte tags and length fields. Now, rendering a complex PSD file (or worse: generating a complex one) is another matter, but the on-disk format is pretty understandable IMO. Creating a bitmap-to-PSD conversion program from scratch is also something I could see being doable in a weekend if you followed the spec, and, as long as the PSD only contained simple rasterized data (eg, no filters and other fanciness) and only the bare-minimum ResourceID/Layer tags. Honestly, I really trust PSD as a general interchange format for images. It's quirky in some places, but it's pretty logically laid-out, is universally-recognized, and I can rest assured that a rasterized copy of my images will always be stapled to the end of the file if worse comes to worse. There really is no alternative, because while you can contain Photoshop data in a format like TIFF (Photoshop just staples the entire PSD file into a TIFF/EXIF tag anyway), and if an application unaware of that PSD data opens the TIFF files and saves it, the Photoshop data might go poof or have a mismatch with the other TIFF data. And then there's Gimp's XCF format, where the on-disk format is allowed to be changed willy-nilly because you are only supposed to use Gimp's official library for reading/writing to it (not to mention no application really supports XCF aside from Gimp and maybe a few open-source projects). And let's not even bother with $application-specific formats, because they are meant only for that specific application. The moral of the story is PSD is an alright format.
w0mbat 6y ago
I wrote the classic-Mac image-display app, "Shomi" (originally DePICT) which tried to display anything as fast as possible. In modern times such an app would be just a bunch of API or library calls, but in the early days I had to write all the image format readers myself. At the time TIFF was the worst, so complicated, so many options, and you even had to write your own LZW codec that matched theirs. Next worse was BMP. It's upside-down and you get random zeroes where you expect sensible values. I managed with a minimalist Photoshop parser (the 8BIM format then) that didn't support everything but coped with the real world files my designer friends tried. It's got more complicated since then.
- just_for_you 6y ago
  I would agree on BMP being slightly insane, since there's a number of things you have to magically know about it: 1) If using 8 bits per pixel or less, there's always a color palette between the header and bitmap. If more than 8bpp, then there's none. 2) If more or equal to 8BPP, you must know that you must always use Bitfields to specify which bytes correspond to which RGB(A) channels, and that there cannot be any overlap between the channel masks. 3) There's no intuitive way to tell which version of the Bitmap file you have (there's like 5 major versions). 4) You can specify, inline, the Chromaticity and Gamma (basically an inline color profile) somewhere in the headers. 5) You can also append an ICC profile directly into the file, OR, store a Windows-style filesystem path to the profile (in which case, you must know to ignore the in-header color profile, if you have one). 6) You must know that RLE compression only works if the RGB channels cleanly map to one-byte-each (no packing). 7) RLE compression doesn't work if you use a negative height (to indicate the image is encoded from top-to-bottom, rather than the standard bottom-to-top). 8) The (un)official documentation Microsoft hosts states that BMP files can also store PNG or JPEG bitstream data, but yet no application that I've seen has ever explicitly supported this. 9) You have to know that Windows only supports 2 or so variations of packed 16-bit BMP files (565, etc). For the most part you can just assume you're dealing with a v5 BMP file, read or write the most pertinent parts like the dimensions (skipping dumb stuff like ICC profiles, pixels-per-meter and whatnot), and just have your way with the raw data. But there is still some dumb stuff in there that shouldn't be there, considering it should be a straight-forward format. What also baffles me is that even with the bloat Microsoft added to the format, BMP isn't even extensible. So it's kinda the worst of both: It's slightly complicated to the point of not being straight-foward to work with, and yet, you can't add extra info/metadata to it. On that note, Targa is a dreamy image format. It's got a simple 18-byte or so header, and then the raw data. And, optionally, for Targa v2 you can append a few bytes in a footer that indicate an offset for an optional standard-targa-metadata area, as well as an option metadata area for your data you wish to add to the file. Followed by a magic string, that explicitly indicate you're using v2 of the Targa spec.
luc_ 6y ago
more of these posts please 10/10
shamyl_zakariya 6y ago
I wrote a PSD v5 parser for BeOS back in 1999 or 2000 (it acted as a plug-in for the OS's image format subsystem, and was also usable by 3rd party apps to write PSD files). I was one of the suckers who faxed a formal request to Adobe to get the file spec. I was young, and foolish, and spent the better part of a month or so in a hex editor trying to understand why a single file format could have like 3 different string encodings.
- user-the-name 6y ago
  Oh wow, did faxing actually work? And I feel your pain, but you were only on v5. It just gets worse the more versions you add.
oseph 6y ago
Here's a "cool" PSD quirk. Take a PSD that has many layers. Look at its filesize (mine is ~70mb). Add one layer to the PSD, fill it with white, and make it the topmost layer. Save it as a new PSD and compare the filesizes. The new PSD with the white layer is 55mb. Why?
- mockery 6y ago
  Most* PSD files contain a "preview" copy of the fully-flattened document (which is compressed.) Flat white image compresses far better, so that portion of the file doesn't take as much space. Depending on what your layers look like (how many, how much they cover, etc.) it's not too surprising that the preview image could take a substantial fraction of the total file size (sounds like ~20% in this case!) * I believe this behavior can be toggled off with an option.
  - Dylan16807 6y ago
    Is the preview the full resolution? That seems very overkill.
    - mockery 6y ago
      Perhaps, but it's quite convenient for any software that wants to ingest PSD files without the (nigh-intractable) challenge of reimplementing the entirety of Photoshop's parser and rendering systems!
    - user-the-name 6y ago
      The "preview" is actually the main image data from the version of the file format that was used before Photoshop supported layers. It remains there, I guess, so that you can display the image even if you don't support loading and compositing layers.
  - oseph 6y ago
    Interesting. I want to believe this is the case, but I feel like it is still a bug/quirk. In Photoshop's preferences there is an option under File Handling to disable image previews, and toggling this seems to have no effect on what I described above. Strange!
sam_goody 6y ago
You can see some other great code comments at: https://stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered https://stackoverflow.com/questions/184618/what-is-the-best-... It may be good that SO removed all similar questions (as they would otherwise have become Reddit) but we sure lost some great humoristic collections!

235 points by kruuuder 6y ago

While I totally empathize with the poor author who had to suffer through this madness, I thoroughly enjoyed reading this rant. My favorite part was this: PSD is not a good format. PSD is not even a bad format. Calling it such would be an insult to other bad formats, such as ...

stagger87 6y ago
I'd love to hear why they think JPEG is a worse format.
- user-the-name 6y ago
  Not worse, that is the point. It is merely a bad format, which is better than what PSD is. As for why JPEG is bad: Did you know there is no one defined way to store the width and height of a JPEG image?
  - stagger87 6y ago
    Thank you, I totally misread that!
  - userbinator 6y ago
    As someone who has written a JPEG decoder, I am compelled to say that you are wrong, and the width and height are stored in the SOF marker segment. http://vip.sugovica.hu/Sardi/kepnezo/JPEG%20File%20Layout%20and%20Format.htm http://vip.sugovica.hu/Sardi/kepnezo/JPEG%20File%20Layout%20...
    - user-the-name 6y ago
      I might be misremembering, but at least, SOF does not tell you what is actually in the channels of the image, so it is not enough to figure out how to actually display an image.
  - just_for_you 6y ago
    To my knowledge, I see most JPEG images encode their height in an "APP1" EXIF segment, and in their SOF1 segment (which is basically just a basic-info segment for the core JPEG bitstream). It's also possible to store the size in an APP1 XMP section, but that's mostly non-essential metadata for interchange purposes, and I usually don't see the size duplicated there for JPEGs anyway (PNG does though, if it has an XMP chunk). It's possible that some of the other officially-defined segments (see https://exiftool.org/TagNames/JPEG.html https://exiftool.org/TagNames/JPEG.html) might also contain duplicate info, but then that'd be specific to the app that uses it, not for general-purpose use. Storing the size in easily-parseable EXIF metadata and then once again in the JPEG bitstream seems pretty reasonable to me. Not to mention if you run a metadata/EXIF stripper on a JPEG file, then you remove all duplicate info and the only remaining image-dimensions will be stored in the JPEG bitstream anyway.
- eyesee 6y ago
  Well for one thing it's not a file format. For that there's JFIF and EXIF.
  - stagger87 6y ago
    I actually thought about that when I typed my reply, but decided the meaning of my question would still be clear the way it was phrased. Despite that, my question is more or less irrelevant since I misunderstood the author.

jolmg 6y ago

So, what's this project about? There's no README nor description nor anything. I'm curious as to why they'd find it so worthwhile to try to parse PSD despite those troubles. They seem to already have parsers for other image formats.

krylon 6y ago
Xee is an image viewer. If you follow the link to the landing page of the repo, it says in the About section it is "Xee source code for xCode 4.5"
folkrav 6y ago
I've seen that rant a while ago. IIRC, Xee is an image viewer for macOS.
masklinn 6y ago
> So, what's this project about? There's no README nor description nor anything. It's Xee, a lightweight (and excellent) image viewer for macOS. That's not the original repo (or author), used to be on google code, moved to bitbucket, then abandoned / sold the project: it was open-source until version 2.2 or something, v3 is is closed source.
- Hamuko 6y ago
  Unfortunately Xee3 is basically abandoned at this point. MacPaw bought it alongside The Unarchiver and they haven't really been doing anything with it. Last update was three years ago. I wonder if it's just going to stop working in some future macOS version.
  - aasasd 6y ago
    > bought it alongside The Unarchiver and they haven't really been doing anything with it Good, because IIRC The Unarchiver was caught gathering stats or sending something on the network, after an updated release.

hprotagonist 6y ago

This is one of my favourite rants. The other one is the SO answer about X/HTML parsing with regex. https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454 https://stackoverflow.com/questions/1732348/regex-match-open...

dspillett 6y ago
"Please do not flag it for our attention." is the perfect finishing touch too.
Blikkentrekker 6y ago
I find it weird that the comment is locked rather than deleted. Funny it be, it does not seem à propos for answering a quæstion.
- _jal 6y ago
  At this point, it is a cultural artifact.
- Alupis 6y ago
  It's from a time before StackOverflow (and StackExchange at large) got very ridged with all the rules. Back then, there were a lot of "fun" questions and answers, including the infamous "What is the best comment in source code you have ever encountered?"[1] Some old timers might remember those days... SO was a much more fun place back then. Now it's rife with down-voters, close-voters and hostile-towards-newbie folks. [1] https://stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered https://stackoverflow.com/questions/184618/what-is-the-best-...
- NobodyNada 6y ago
  The post was locked because it had received hundreds of flags and comments complaining that the post was broken -- even with the moderator note at the bottom -- and several edits trying to "fix" the post. See https://meta.stackoverflow.com/questions/250099#comment637_250099 https://meta.stackoverflow.com/questions/250099#comment637_2...

deleted 6y ago

[deleted]

deleted 6y ago

[deleted]

rdtsc 6y ago

> Why, for instance, did it suddenly decide that these particular chunks should be aligned to four bytes, and that this alignement (sic) should not be included in the size? Not saying this happened here, but I have seen this type of mistake before. It was because they "simply" cast a C/C++ struct to a binary blob https://en.wikipedia.org/wiki/Data_structure_alignment#Typical_alignment_of_C_structs_on_x86 https://en.wikipedia.org/wiki/Data_structure_alignment#Typic... and wrote that to disk (or sent over the network in my case). So that particular compiler version and architecture-specific struct field alignment became the "official" format. It just takes one goofy mistake like that and everyone has to deal with it for years to come.

masklinn 6y ago
Note that the issue here is less the part that you quoted and more the two sentences surrounding it, PSD has all of * unaligned chunks * aligned chunks with alignment included in size * aligned chunks with alignment not included in size The problem is not the specific choice, "Either one of these three behaviours would be fine", it's that "PSD, of course, uses all three, and more."
wolrah 6y ago
As I understand it that's more or less what the older MS Office formats were, just a dump of the entire OLE object representing the document as-is. I believe SimCity used a similar strategy as well, the .cty and I think SC2k's .s2k formats just dumped the in-memory representation to disk,.
- krylon 6y ago
  I worked at a company once where they, too, used raw memory dumps as their file format. The compiler flags telling the compiler how to align structs were different for debug and non-debug builds. So, of course, the first thing I did, was to create non-debug build and try to open a file created by a debug-build-executable, the program crashed and burned without giving me a meaningful error message, it took me hours to understand my mistake. Raw memory dumps are very neat efficiency-wise, but they are extremely fragile.

mmastrac 6y ago

This used to show up often on HN but I haven't seen it in years. The best comment was from this old thread [1]: > I enjoyed the commit message most: > > r11 by paracelsus on Sep 11, 2007 Diff > Photoshop loader is DONE for now, fuck you Adobe https://news.ycombinator.com/item?id=575122 https://news.ycombinator.com/item?id=575122

deleted 6y ago

[deleted]

riffraff 6y ago

I remember reading this years ago, I think it was my first encounter with the "... fierce passion of a thousand suns". It still made me chuckle even today, thanks for sharing it.

bluedino 6y ago

Joel on Software, "Why are Microsoft Office File Formats So Complicated" https://www.joelonsoftware.com/2008/02/19/why-are-the-microsoft-office-file-formats-so-complicated-and-some-workarounds/ https://www.joelonsoftware.com/2008/02/19/why-are-the-micros... They were designed to be fast on very old computers. They were designed to use libraries. They were not designed with interoperability in mind. They have to reflect the history of the applications,

ericol 6y ago

> this Rube Goldberg of a file format Concise, to the point, and a good enough insult as to keep it in file. I love it.

egonschiele 6y ago

This came up a couple months ago. I really wish there was a standard open format that did most of the things PSD does. I had asked the folks at Procreate if they'd design something, since they are sort of a challenger to Photoshop, but they said it wasn't something they wanted to do.

iggldiggl 6y ago
Where this might get tricky is if you seriously want to support non-destructive editing as well, because in that case any filter that can be applied in non-destructive mode effectively needs to become part of the file format specification, too.
- wongarsu 6y ago
  You could store filters generic enough that any software can skip over filters it doesn't recognize (e.g. you store filter type uuid, filter data length, then arbitrary filter parameters), have some open registry where you can get official filter uuids in return for an example implementation. Of course not everyone will register every filter, so the format should probably store the image once unfiltered and once with all filter applied, that way if a program doesn't recognize or implement some filter it can still fall back to the destructively filtered version. But of course that increases file size, which may or may not be a concern.
preommr 6y ago
Procreate is 10$ for a one-time purchase. PS is 13$/mo. Actually, if you got just PS it would be 30$/mo. And even then I've seen people complain that procreate is too expensive. I don't really have a point other than design tools and their pricing is a pet issue of mine and I like bringing up how insane it is at any given chance.
hallarempt 6y ago
We started working on OpenRaster for that: https://www.openraster.org/ https://www.openraster.org/ -- but it's not really moving along very well.

Triv888 6y ago

Plain text file is my favorite file format... because in the end, portability is what matters the most.

gjvr 6y ago
ASCII? UTF-8? UTF-16? ... ...
- Triv888 6y ago
  lol... yes. UTF-512 should be enough for everybody!
  - _kst_ 6y ago
    UTF-640K?
- user-the-name 6y ago
  Shift-JIS? GB18030? KOI8-R? "Plain text" does not really exist.
- Dylan16807 6y ago
  Anyone using the words "plain text" for a file had better mean either ASCII or UTF-8. And for almost all purposes these days it should be the latter.
Arnt 6y ago
By plain text, do you have in mind CRLF or LF, and for structured data, do you have in mind YAML, JSON or one of the umpty others?

SimianLogic2 6y ago

I have deep sympathy for this, after having done a ton of work with the SWF format and the (even more poorly documented) AEPX format.

1f60c 6y ago

For posterity's sake, the link should probably be changed to https://github.com/zepouet/Xee-xCode-4.5/blob/83394493f51991748b9b4706e6d37a8ed874bc05/XeePhotoshopLoader.m#L108-L136 https://github.com/zepouet/Xee-xCode-4.5/blob/83394493f51991...

bloudermilk 6y ago

I had a good laugh reading this. Of course the irony is that the author, after having gone through the hellish process of learning the spec, didn't documented any of the code (in that file).

hallarempt 6y ago

PSD is horrible, that is certainly true. And underdocumented. And dumb... And we needed to implement read/write support for that in Krita. But Painttool Sai's format is actively evil, with different kinds of encryption for different size layers and things like that: https://github.com/Wunkolo/libsai/issues/6 https://github.com/Wunkolo/libsai/issues/6 (We tried to use that library to implement sai file format support for Krita, but then ran into trouble...) Manga Studio/Clip Studio Paint's file format is interesting as well: it's just an SQLite database, with raster data stored as blobs. Not going to implement that for Krita either.

Gualdrapo 6y ago
Thank you for working on Krita.
Jasper_ 6y ago
That doesn't look like encryption or differences in algorithms, just the usual confusion and edge cases when you don't have a full mental model of the file format's fixed-size vfs entries.
pfranz 6y ago
> Manga Studio/Clip Studio Paint's file format is interesting as well: it's just an SQLite database, with raster data stored as blobs. Not going to implement that for Krita either. I'm curious why? I'm not trying to start anything, but I remember this discussion from a few months ago: https://news.ycombinator.com/item?id=23509914 https://news.ycombinator.com/item?id=23509914 > I wish my predecessor at Krita had made that choice (sqlite), instead of choosing to re-use the KOffice xml-based file format that's basically a zip file with stuff in it. It would have made adding the animation feature so much easier. To summarize the benefits to something like sqlite over zip+xml (not all were made by that author) is; update in place means not having to overwrite the whole file, not inventing xml format for describing an image, not needing to keep the entire representation in-memory, a longer saving process is more fragile, incremental loading.
- hallarempt 6y ago

mst 6y ago

Personally, rather than printing things out and then setting them on fire, I've preferred printing things out then when I'm done with it donating them to be shredded and used as horse bedding. Some code just deserves to end its life being shat on by a horse.

just_for_you 6y ago

I would actually beg to disagree, as someone who was bored a few weekends ago and decided to go through a few reasonably-complex PSD files (eg, ones with different layer types, bezier curves, etc.) in a hex editor with the official specification in-hand in an attempt to understand the format. (The specification can be found here: https://www.adobe.com/devnet-apps/photoshop/fileformatashtml/ https://www.adobe.com/devnet-apps/photoshop/fileformatashtml...). After a few hours, I was able to grok it without too much difficulty, and found the format was reasonably well laid-out. Yes, it does suffer from some "let's take the in-memory representation and bake it into the on-disk format"-isms, and there were a few things were not covered by the specification (eg, a couple Resource IDs aren't mentioned at all, or they are, but there's no documentation on how to interpret their content) - but it's not anywhere close to insane. To give an example, partially going off memory from what I remember seeing in real files, but mostly skimming the spec, a very basic gist of a typical PSD files might look something like this. (I can only assume HN is going to mangle the formatting of the following text, so expect to see multiple edits to this comment): EDIT: I was right about the formatting getting messed up. I've posted it to Pastebin and will call it a day: https://pastebin.com/raw/WZNrCSAP https://pastebin.com/raw/WZNrCSAP So that (the Pastebin link above) is the basic and oversimplified gist of a PSD file. It's mostly just reading 4-byte tags and length fields. Now, rendering a complex PSD file (or worse: generating a complex one) is another matter, but the on-disk format is pretty understandable IMO. Creating a bitmap-to-PSD conversion program from scratch is also something I could see being doable in a weekend if you followed the spec, and, as long as the PSD only contained simple rasterized data (eg, no filters and other fanciness) and only the bare-minimum ResourceID/Layer tags. Honestly, I really trust PSD as a general interchange format for images. It's quirky in some places, but it's pretty logically laid-out, is universally-recognized, and I can rest assured that a rasterized copy of my images will always be stapled to the end of the file if worse comes to worse. There really is no alternative, because while you can contain Photoshop data in a format like TIFF (Photoshop just staples the entire PSD file into a TIFF/EXIF tag anyway), and if an application unaware of that PSD data opens the TIFF files and saves it, the Photoshop data might go poof or have a mismatch with the other TIFF data. And then there's Gimp's XCF format, where the on-disk format is allowed to be changed willy-nilly because you are only supposed to use Gimp's official library for reading/writing to it (not to mention no application really supports XCF aside from Gimp and maybe a few open-source projects). And let's not even bother with $application-specific formats, because they are meant only for that specific application. The moral of the story is PSD is an alright format.

w0mbat 6y ago

I wrote the classic-Mac image-display app, "Shomi" (originally DePICT) which tried to display anything as fast as possible. In modern times such an app would be just a bunch of API or library calls, but in the early days I had to write all the image format readers myself. At the time TIFF was the worst, so complicated, so many options, and you even had to write your own LZW codec that matched theirs. Next worse was BMP. It's upside-down and you get random zeroes where you expect sensible values. I managed with a minimalist Photoshop parser (the 8BIM format then) that didn't support everything but coped with the real world files my designer friends tried. It's got more complicated since then.

just_for_you 6y ago
I would agree on BMP being slightly insane, since there's a number of things you have to magically know about it: 1) If using 8 bits per pixel or less, there's always a color palette between the header and bitmap. If more than 8bpp, then there's none. 2) If more or equal to 8BPP, you must know that you must always use Bitfields to specify which bytes correspond to which RGB(A) channels, and that there cannot be any overlap between the channel masks. 3) There's no intuitive way to tell which version of the Bitmap file you have (there's like 5 major versions). 4) You can specify, inline, the Chromaticity and Gamma (basically an inline color profile) somewhere in the headers. 5) You can also append an ICC profile directly into the file, OR, store a Windows-style filesystem path to the profile (in which case, you must know to ignore the in-header color profile, if you have one). 6) You must know that RLE compression only works if the RGB channels cleanly map to one-byte-each (no packing). 7) RLE compression doesn't work if you use a negative height (to indicate the image is encoded from top-to-bottom, rather than the standard bottom-to-top). 8) The (un)official documentation Microsoft hosts states that BMP files can also store PNG or JPEG bitstream data, but yet no application that I've seen has ever explicitly supported this. 9) You have to know that Windows only supports 2 or so variations of packed 16-bit BMP files (565, etc). For the most part you can just assume you're dealing with a v5 BMP file, read or write the most pertinent parts like the dimensions (skipping dumb stuff like ICC profiles, pixels-per-meter and whatnot), and just have your way with the raw data. But there is still some dumb stuff in there that shouldn't be there, considering it should be a straight-forward format. What also baffles me is that even with the bloat Microsoft added to the format, BMP isn't even extensible. So it's kinda the worst of both: It's slightly complicated to the point of not being straight-foward to work with, and yet, you can't add extra info/metadata to it. On that note, Targa is a dreamy image format. It's got a simple 18-byte or so header, and then the raw data. And, optionally, for Targa v2 you can append a few bytes in a footer that indicate an offset for an optional standard-targa-metadata area, as well as an option metadata area for your data you wish to add to the file. Followed by a magic string, that explicitly indicate you're using v2 of the Targa spec.

luc_ 6y ago