9 ms·
Blender (the 3D modelling software) is a fascinating case of this. To make loading and saving files fast and lossless, it uses identical on-disk and in-memory r
by TazeTSchnitzel 3y ago
Blender (the 3D modelling software) is a fascinating case of this. To make loading and saving files fast and lossless, it uses identical on-disk and in-memory representations. In other words, everything is in an arena, and saving and loading is just a memcpy of the entire arena. Considering the massive potential complexity of a Blender project and the myriad issues you could have with serialisation and deserialisation, this seems like a great design to me.
The trade-off of course is that your data structure design is kinda sticky, insofar as you need to be able to open files for older versions.
- jeffreygoesto 3y agoWasn't that also the way Word used to dump structs onto disc in the "doc" days? I think I read that way also a major problem in writing converters, you baobab needed to decode Word's internal data structures which were of course undocumented.
- orthoxerox 3y agoI remember Spolsky saying Excel .xls worked like this, but I guess this should apply to .doc files as well. This is usually orders of magnitude faster than serializing to/deserializing from a different storage format.
- bestouff 3y agoI don't think so. Serializing is IO-bound, if your format is "simple" (e.g. CBOR) you can't measure the difference. All this look like BS to justify laziness to me, and makes loading/saving operations fragile and setting in-memory structures in stone.
- Joker_vD 3y ago> Serializing is IO-bound Once upon a time, DMA did not meaningfully exist in the x86 world, so an IO-bound task was also CPU-bound.
- bestouff 3y agoYes of course, but the original example was Blender, running on today's hardware. IMO this is just sloppy engineering in this case.
- orthoxerox 3y agoDon't forget that Excel file format is literally 30 years old. When you have 4MB of RAM in total for your OS, Excel and whatever part of the file you can fit into the rest and your CPU is an i486 if you're lucky, minimizing parsing starts to make lots of sense.
- kragen 3y agothe cbor implementations i've had the misfortune to have to use are ridiculously slow, as slow as json parsers, perhaps because everything in cbor is variable-length just because it's unreadable in a text editor doesn't mean it'll be fast
- masklinn 3y agoIt's very common in old software and video games. Pre-97 office did that though with more formalisation than most (the binary format is now called Binary Interchange File Format, or BIFF, and is relatively generic, at the time MS almost certainly had utility libraries for working with BIFF). This was a common source of portability issues for game saves (as well as blowing your saves on updates), as they'd commonly just blit internal data structure, with no formalised interface.
- jamesfinlayson 3y agoYep, pretty sure Valve's brief experiment with a 64-bit Source engine in late 2005 resulted in save games that didn't work with 32-bit Half-Life 2.
- agumonkey 3y agoAnd potentially photoshop.
- glandium 3y agoIt also means you can't share files between little endians and big endians.
- WJW 3y agoIs this a real problem though? It seems to me that any system you would realistically want to run Blender on will be little-endian.
- tjalfi 3y agoIt isn't a real problem. Blender supports only little-endian targets and the code byte swaps data files as needed. I guess someone could port it to AIX, z/OS, or Linux/zSeries, but that's unlikely at best.
- rollcat 3y agoThere's almost certainly byte order+version markers in the header, a "happy path" that just memcpy's the structs, and a "safe path" that does byte order swapping, migrates the old structs to the current format, etc. You can version the structs/loaders by keeping the relevant header files in their own version subfolders, and copious usage of sed. I've done this in a Django project to maintain support for ancient untouchable clients using an old version of the API, it's pretty manageable.
- tjalfi 3y agoI checked the repo. Blender detects files with a different endian and byte swaps if necessary.
- TazeTSchnitzel 3y agoIt doesn't, you just need to be careful to always read and write properties via accessor functions.
- seventhson 3y agoI once worked on a very large commercial application that was based on a home-grown I/O framework which operated on similar principles. It was a complete pain in the ass. You were constantly future-proofing your data structures because you knew you were going to be stuck with them for all eternity because the I/O framework was going to serialize them verbatim whether you liked it or not. Those were dark days...
- Solvency 3y agoI don't get it. Surely if you ever needed to, you could write an interpreter for loading tbe older files if you decided to abandon this 1:1 representation?
- amomchilov 3y agoI think the effort involved disincentivizes against it, and nudges people towards defensively adding spare fields or whatever. The other comment pointed out that you can make a fall back migration code path that migrates over older file versions. That's the escape hatch if you have no other options
- dragontamer 3y agoYou ever hear of .doc files? https://learn.microsoft.com/en-us/openspecs/office_file_formats/ms-doc/ccd7b486-7881-484c-a137-51170af7cc22 https://learn.microsoft.com/en-us/openspecs/office_file_form... Yeah. There is a reason XML was seen as the future back in the 90s. Custom databases often get extended into custom filesystems. And these systems on top of systems get obscure features (like embedding excel sheets inside of Word) and... Thing get hairy.
- 3cats-in-a-coat 3y agoYou can have the benefits of fast save and load without having the same representation. Stream changes to disk.
- TazeTSchnitzel 3y agoStoring changes and the originals blew up badly for Microsoft Word in the late 1990's due to privacy concerns.
- giovannibonetti 3y ago> The trade-off of course is that your data structure design is kinda sticky, insofar as you need to be able to open files for older versions. The missing piece is a way to evolve those data structures over time, in a similar way to database migrations. Only when loading data from an older version of the app those operations would need to be issued, and then the app could save the updated version in disk immediately to avoid incurring that cost again. An example implementation of this concept is this project [1] that was created in the context or CRDTs. If it is not directly applicable, at least it should be a good inspiration. [1] https://www.inkandswitch.com/cambria/ https://www.inkandswitch.com/cambria/
- pavlov 3y agoI believe Microsoft Word originally did this, and it was a major pain as the file format evolved.
- deleted 3y ago[deleted]
- bjourne 3y agoThat is of course not Kosher at all. Compilers are generally free to layout and pad structs in whichever way they prefer, meaning that two compilers for the same platform might very well use incompatible layouts. It can lead to major problems if structs saved by say a 64-bit version of the program is to be loaded by the 32-bit version.
- deleted 3y ago[deleted]
- azornathogron 3y agoCompilers (for C and C++) typically provide mechanisms to control padding and other layout requirements as necessary to make this work. It's not that unusual in C and C++ code to define a struct that has a specific and well-defined memory layout. It's kosher, as long as you accept that you're working with a specific set of real compilers and use the appropriate #pragmas or other controls as needed to avoid undefined behaviour.
- bjourne 3y agoYep, but then you force the compiler to choose a specific struct-layout which may be suboptimal if, for example, coding for simd. You also can't control the values of the struct padding bytes which may cause very interesting bugs.
- grogenaut 3y agoThis is exactly how a PlayStation game I worked on worked and because everything was signed they were in no way worried about attackers. If you were able to alter the files on disk you had already owned your device. It meant loading was absurdly fast we pulled everything in the memory and then just patched pointer locations to where they actually got loaded