5 ms·

> Why, for instance, did it suddenly decide that these particular chunks should be aligned to four bytes, and that this alignement (sic) should not be included

• masklinn 6y ago
  Note that the issue here is less the part that you quoted and more the two sentences surrounding it, PSD has all of * unaligned chunks * aligned chunks with alignment included in size * aligned chunks with alignment not included in size The problem is not the specific choice, "Either one of these three behaviours would be fine", it's that "PSD, of course, uses all three, and more."
• wolrah 6y ago
  As I understand it that's more or less what the older MS Office formats were, just a dump of the entire OLE object representing the document as-is. I believe SimCity used a similar strategy as well, the .cty and I think SC2k's .s2k formats just dumped the in-memory representation to disk,.
  • krylon 6y ago
    I worked at a company once where they, too, used raw memory dumps as their file format. The compiler flags telling the compiler how to align structs were different for debug and non-debug builds. So, of course, the first thing I did, was to create non-debug build and try to open a file created by a debug-build-executable, the program crashed and burned without giving me a meaningful error message, it took me hours to understand my mistake. Raw memory dumps are very neat efficiency-wise, but they are extremely fragile.
    • rwallace 6y ago
      I'm curious, what was the reason for the difference? While the C language standard doesn't guarantee anything about alignment, I'm used to the behavior of compilers in practice being consistently 'align everything on its own size'; what changed between debug and release builds in that case?
      • ender341341 6y ago
        They mentioned compiler flags, so my guess would be that they were doing something like tightly packing structs in the release build to lower memory usage or something similar.
        • saagarjha 6y ago
          Or the opposite, as packing usually requires slower, unaligned accesses.
          • sumtechguy 6y ago
            That would depend on how much that struct is created in memory vs the total code that uses it. If you have one instance in memory then yeah the code is probably bigger and slower (depending on arch). But if you have thousands of the structs in memory the speed and code size trade off may be worth it on a memory constrained system. Debug also sometimes turns on overrun buffers so you can check for over/under runs in your code at debug time. Some compilers have this others dont.
            • deleted 6y ago
              [deleted]
        • krylon 6y ago
          Yes, I think that was the main reason. It was kind if pointless, though, the biggest file I saw in my time there was around 10MB. How much bigger would that have been with padding? Even an overhead of 100% probably would not have had much of a performance impact and saving a few megabytes of disk space or RAM was not a very strong argument at the time any more (roughly 2007 - 2009) On the other hand, this was a natural outgrowth of the other programmers' tendency to keep it simple and avoid bloat, which all in all I appreciated very much.
  • banana_giraffe 6y ago
    For what it's worth, I think that's true of the pre-1997 version of the format. For the format from 1997 to 2007, it's a proper format, albeit a bizarre one (looking over the spec makes me think the person tasked with making it was yanked off the filesystem team against their will) Nowadays, it's a XML/ZIP thing. With things like Flatbuffers, I sometimes feel we've regressed to these old formats that are just memory dumps.
    • ryanianian 6y ago
      > With things like Flatbuffers, I sometimes feel we've regressed to these old formats Sorta, but they're expressed in IDLs that are independent of a particular compiler. The downside is the drift between internal structs and the IDL structs, but the upside is you can use the same "memory dump" on interpreted runtimes or entirely different platforms (even those with different endians). Plus impls like protobuf help guard against breaking backward-compatibility by numbering fields and not "allowing" you to remove fields in ways that would change the structs' ABIs.
    • DonHopkins 6y ago
      A story I heard at Sun, which may be apocryphal but was fucking hilarious enough to be a repeatable rumor, was that a release of an early operating system in BETA was determined to be solid and tested and ready to release and ship to customers, so they simply changed the version string from something like "SunOS2.1BETA" to "SunOS2.1FCS" (First Customer Ship), and recompiled. But the change from a 12 character version to an 11 character version threw off the alignment of some important data structures somewhere in the kernel, and the entire OS ran MUCH SLOWER because of 68k unaligned memory accesses!
    • klodolph 6y ago
      > With things like Flatbuffers, I sometimes feel we've regressed to these old formats that are just memory dumps. It depends on what your application requirements are, but there are compelling arguments that on-disk / on-wire representations should match in-memory representations. It’s not too hard to end up with in a scenario where encoding / decoding times are a significant contribution to overall performance.
  • DonHopkins 6y ago
    FWIW here's the original 68k SimCity "Classic" (open source Micropolis) save function, which I've long since cleaned up and put in byte swapping to make it portable to SPARC and x86, but yes it is just writing out some big buffers of memory with a function that now swaps bytes: https://github.com/SimHacker/micropolis/blob/master/MicropolisCore/src/MicropolisEngine/src/fileio.cpp#L340 https://github.com/SimHacker/micropolis/blob/master/Micropol... Here's some original SimCity 2000 Mac code that saves the city into Mac resources (not the flat part of the file, but the Mac resource fork) -- CompGameWrite actually does some simple run length compression of the raw memory: Boolean DoSave(short VolNum,Byte *name){ long count,CountTotal; short i,j,filnum; short x, y; Byte *SaveText[] = {"\pGame Saved As:",NIL,NIL}; FInfo info; Byte *NotCity[] = { "\pSimCity 2000ª will not save over a non-city file.", "\pTry to save again using a different name.", NIL, }; if (name[0]>20) name[0] = 20; #ifndef DEBUG if (GetFInfo(name,VolNum,&info)==0 && info.fdType!=CITYTYPE_ID) { MessageDialog(NotCity); return FALSE; } #endif if (FSOpen(name,VolNum,&filnum)!=noErr) { if (GameError(Create(name,VolNum,APPLICATION_ID,CITYTYPE_ID))) return FALSE; if (GameError(FSOpen(name,VolNum,&filnum))) return FALSE; } if (!WriteHeader(filnum,0L)) return FALSE; WriteLength = 4; // includes 'SCDH' but not header //*** Write *** if (!MiscWrite(filnum)) return FALSE; if (!GameWrite(filnum,'ALTM',(Ptr)AltMap[0])) return FALSE; if (!CompGameWrite(filnum,'XTER',(Ptr)TerrainMap[0])) return FALSE; if (!CompGameWrite(filnum,'XBLD',(Ptr)BuildMap[0])) return FALSE; if (!CompGameWrite(filnum,'XZON',(Ptr)ZoneMap[0])) return FALSE; if (!CompGameWrite(filnum,'XUND',(Ptr)UnderMap[0])) return FALSE; if (!CompGameWrite(filnum,'XTXT',(Ptr)TextMap[0])) return FALSE; if (!CompGameWrite(filnum,'XLAB',(Ptr)LabelArray)) return FALSE; if (!CompGameWrite(filnum,'XMIC',(Ptr)MicroRecord)) return FALSE; if (!CompGameWrite(filnum,'XTHG',(Ptr)ThingList)) return FALSE; if (!CompGameWrite(filnum,'XBIT',(Ptr)BitsMap[0])) return FALSE; if (!CompGameWrite(filnum,'XTRF',(Ptr)TrafficMap[0])) return FALSE; if (!CompGameWrite(filnum,'XPLT',(Ptr)PolluteMap[0])) return FALSE; if (!CompGameWrite(filnum,'XVAL',(Ptr)ValueMap[0])) return FALSE; if (!CompGameWrite(filnum,'XCRM',(Ptr)CrimeMap[0])) return FALSE; if (!CompGameWrite(filnum,'XPLC',(Ptr)PoliceMap[0])) return FALSE; if (!CompGameWrite(filnum,'XFIR',(Ptr)FireMap[0])) return FALSE; if (!CompGameWrite(filnum,'XPOP',(Ptr)PopMap[0])) return FALSE; if (!CompGameWrite(filnum,'XROG',(Ptr)ROGMap[0])) return FALSE; if (!CompGameWrite(filnum,'XGRP',(Ptr)GraphData[0])) return FALSE; //*** Close *** WriteHeader(filnum,WriteLength); GameError(SetEOF(filnum,WriteLength+8)); // 8 bytes for file header GameError(FSClose(filnum)); GameError(FlushVol(NIL,VolNum)); BlockMove(name,CityStr,21); DateCashTitle(); FilVolNum = VolNum; //*** SUCCESS! *** SaveText[1] = CityStr; MessageDialog(SaveText); return TRUE; } And here's the Mac SimEarth save function, which looks like it just write out raw uncompressed memory -- the Mac memory manager has a GetPtrSize function that (obviously) tells you the size of the memory allocated for a pointer: DoSave(VolNum,name) int VolNum; Str255 name; { long count,CountTotal; int i,FilNum; if (EqualString(name,"\pSimEarth",FALSE,TRUE)) { MessageDialog( "\pERROR!!", "\pThe name 'SimEarth' is reserved", "\pfor this application.",1); return; } if (FSOpen(name,VolNum,&FilNum)) { if (GAIAError(Create(name,VolNum,'MYCR','SAVE'))) return; if (GAIAError(FSOpen(name,VolNum,&FilNum))) return; } else if (GAIAError(SetFPos(FilNum,fsFromStart,0L))) { GAIAError(FSClose(FilNum)); return; } /**Write**/ CountTotal = (count=GetPtrSize(Map[0])); if (GAIAError(FSWrite(FilNum,&count,Map[0]))) return; CountTotal += (count=GetPtrSize(Life[0])); if (GAIAError(FSWrite(FilNum,&count,Life[0]))) return; CountTotal += (count=GetPtrSize(OcTempMap[0])); if (GAIAError(FSWrite(FilNum,&count,OcTempMap[0]))) return; CountTotal += (count=GetPtrSize(OcCurrentMap[0])); if (GAIAError(FSWrite(FilNum,&count,OcCurrentMap[0]))) return; CountTotal += (count=GetPtrSize(DriftMap[0])); if (GAIAError(FSWrite(FilNum,&count,DriftMap[0]))) return; CountTotal += (count=GetPtrSize(EventMap[0])); if (GAIAError(FSWrite(FilNum,&count,EventMap[0]))) return; CountTotal += (count=GetPtrSize(SAirTempMap[0])); if (GAIAError(FSWrite(FilNum,&count,SAirTempMap[0]))) return; CountTotal += (count=GetPtrSize(SCloudDensity[0])); if (GAIAError(FSWrite(FilNum,&count,SCloudDensity[0]))) return; CountTotal += (count=GetPtrSize(SAirCurrentMap[0])); if (GAIAError(FSWrite(FilNum,&count,SAirCurrentMap[0]))) return; PutParameters(); CountTotal += (count=GetPtrSize(MiscHis)); if (GAIAError(FSWrite(FilNum,&count,MiscHis))) return; /**Close**/ GAIAError(SetEOF(FilNum,CountTotal)); GAIAError(FSClose(FilNum)); GAIAError(FlushVol(NIL,VolNum)); BlockMove(name,EwinStr,256); SetWTitle(editWindow,EwinStr); MessageDialog("\pSave Complete:",EwinStr,"",1); EnableItem(GetMHandle(301),3); /* Save */ FilVolNum = VolNum; }
• nradov 6y ago
  That was very common not long ago. Even the JPEG/EXIF image file format is designed that way. So it's efficient for reading and writing, but introduces a lot of potential bugs with alignment and chunk size issues. Inserting an additional EXIF tag is a huge hassle because then you have to recalculate all the pointers, even those in other data chunks.
  • just_for_you 6y ago
    One form of true insanity is probably JPEG-encoded TIFF files. Apparently it's so bad that for the next version of the TIFF specification they are taking that insane approach out completely. I can't do justice to the article on TIFF's problems with JPEG (http://www.simplesystems.org/libtiff/TIFFTechNote2.html http://www.simplesystems.org/libtiff/TIFFTechNote2.html), but my understanding is that: 1) Some JPEG-specific data is moved outside of the actual JPEG bitstream tag, and into separate TIFF tags, making editing JPEG-in-TIFF files non-trivial. 2) Size is not encoded in some of these fields, so the TIFF editor you're writing will have to partially implement a JPEG decoder just to know the size of some of those TIFF tags, and 3) Some tags/fields are pointers into other parts of the TIFF file, meaning if you edit the file, you'll have to update the file in many places. (As a quick aside on insane formats, may I also mention the EPWING dictionary format?)
• grishka 6y ago
  That's why you version your file formats and network protocols. Actually, designing file formats and network protocols is one of the few areas in software engineering where you do really need future-proof design and extreme extensibility, because once you release the thing, these are set in stone. Yet not many people seem to realize this. They instead "future-proof" their code with useless abstraction layers. Anyway. Versioning helps you avoid ugly workarounds if you need to extend your format in the future in ways that its current version doesn't allow. You then keep the code for older versions as a backwards-compatibility-only kind of deal and move on to the new one.
  • klodolph 6y ago
    Versioning is one approach, but I favor making the format extensible in the first place. For example, if you pick an XML format you can add new attributes and tags, if you pick a Protobuf format you can add new fields. “Extensible” sounds like it can be a real mess but there are effective strategies to minimize the mess. There are also various chunked formats like AIFC (AIFF) and PNG which can be extended by defining new chunk types, without needing to change versioning. AIFC includes the APPL/stoc chunk and PNG has various mechanisms. There are a few problems with versioning file formats. One is that you often end up with 'if (version > 3)' scattered across your code base or other nonsense. Another problem is that it is easy to accidentally mark the wrong version, either a version which is too low (because you wrote something to the file and forgot to make your encoder bump the version properly) or a version which is too high (because you didn’t bother to use the minimum version your data requires).
    • grishka 6y ago
      I'd say do both, actually. But versioning is more important IMO because it gives you the freedom to potentially start from scratch keeping only a small portion of the header. On mainstream file formats... It's a mixed bag. Image formats — JPEG and PNG especially — are extensible and reasonably easy to parse. It's fairly trivial to get the image dimensions out of one of these without decoding the compressed data. I did as well write a JPEG decoder out of curiosity once to understand the compression algorithm better — it's an interesting exercise, really, every software developer should try it at some point. But the worst format I've ever worked with is MP3. It's an absolute mess. First, there are two kinds of ID3 tags. These store metadata that your player displays. ID3v1 is a fixed-length, fixed-layout thing that goes on the end of the file. ID3v2 is an extensible, I'd say way too extensible, chunked thing capable of storing literally anything, including jpegs of cover arts, that goes on the beginning of the file. But none of them store the duration of the file. You're supposed to chop the tags off the ends of it, then find the first frame of encoded data by searching for the pattern 0xFFFx, read its header, and determine the byte length, bitrate, the sampling rate and ultimately the duration of a that frame using several lookup tables. Now that you know how much audio each frame contains, and how long it is, you take the size of the file (minus tags of course) and divide it by the frame length, then multiply by the frame duration. That's how you get the duration of an MP3. A constant-bitrate one. And to seek within an MP3, you calculate the offset into the file and round it to the nearest frame size and just start playing from there. It gets even worse with VBR, because now you can no longer rely on frames being the same byte length, but I don't really remember the details any more. The gist of it is that there's "header" encoded into the very first frame in the file, and there are two kinds of these headers, and there's a sort of lookup table in it, among other things, to help you seek into the right part of the file because the byte offsets don't linearly correspond to the playback time in a VBR file. After you seek, you have to go back and forth to find the 0xFFFx and play from there. Or not, because sometimes there's a 0xFFFx in the middle of a frame too, so you have to have some heuristics to detect that it's the real one.
• userbinator 6y ago
  In practice, it's the simplest and most efficient way, since the majority of the time you're not going to be dealing with any insanely weird architectures and stuff like 9-bit bytes has thankfully disappeared from common use. I do wish they'd pack their structures, however.