4 ms·

Thinking about a file format is a good way to clarify your vision. Even if you don’t want to facilitate interop, you’d get some benefits for free—if you can enc

Thinking about a file format is a good way to clarify your vision. Even if you don’t want to facilitate interop, you’d get some benefits for free—if you can encapsulate the state of a particular thing that the user is working on, you could, for example, easily restore their work when they return, etc. Some cop-out (not necessarily in a bad way) file formats: 1. Don’t have a file format, just specify a directory layout instead. Example: CinemaDNG. Throw a bunch of particularly named DNGs (a file for each frame of the footage) in a directory, maybe add some metadata file or a marker, and you’re good. Compared to the likes of CRAW or BRAW, you lose in compression, but gain in interop. 2. Just dump runtime data. Example: Mnemosyne’s old format. Do you use Python? Just dump your state as a Python pickle. (Con: dependency on a particular runtime, good luck rewriting it in Rust.) 3. Almost dump runtime data. Example: Anki, newer Mnemosyne with their SQLite dumps. (Something suggests to me that they might be using SQLite at runtime.) A step up from a pickle in terms of interop, somewhat opens yourself (but also others) to alternative implementations, at least in any runtime that has the means to read SQLite. I hope if you use this you don’t think that the presence of SQL schema makes the format self-documenting. 4. One or more of the above, except also zip or tar it up. Example: VCV, Anki.

3036e4 1y ago
About 1, directory of files, many formats these days are just a bunch of files in a ZIP. One thing most applications lack unfortunately is a way to instead just read and write the part files from/to a directory. For one thing it makes it much better for version control, but also just easier to access in general when experimenting. I don't understand why this is not more common, since as a developer it is much more fun to debug things when each thing is its own file rather than an entry in an archive. Most times it is also trivial to support both, since any API for accessing directory entries will be close to 1:1 to an API for accessing ZIP entries anyway. When editing a file locally I would prefer to just have it split up in a directory 99% of the time, only exporting to a ZIP to publish it. Of course it is trivial to write wrapper scripts to keep zipping and unzipping files, and I have done that, but it does feel a bit hacky and should be an unnecessary extra step.
- strogonoff 1y ago
  Yes, the zipped version is number four. It’s not great for the reason you noted. Some people come up with smudge/clean filters that handle the (de)compression, letting Git store the more structured version of the data even though your working directory contains the compressed files your software can read and write—but I don’t know how portable these things are. I agree with you in general, and it is also why my number one example is that you might not need a single-file format at all. macOS app bundles is a great example of this approach in the wild. One question I was hoping to ask anyone who thought about these matters: what accepted approaches do exist out there when it comes to documenting/speccing out file formats? Ideally, including the cases where the “file” is in fact a directory with a specific layout.
strogonoff 1y ago
(Correction: instead of “CRAW”, I should have written “Canon Cinema Raw Light”. Apparently, those are different things.)
trinix912 1y ago
> 2. Just dump runtime data. Example: Mnemosyne’s old format. Do you use Python? Just dump your state as a Python pickle. (Con: dependency on a particular runtime, good luck rewriting it in Rust.) Be particularly careful with this one as it can potentially vastly expand the attack surface of your program. Not that you shouldn't ever do it, just make sure the deserializer doesn't accept objects/values outside of your spec.
- strogonoff 1y ago
  I certainly hope no one takes my list as an endorsement… It’s just some formats seen in the wild. It should be noted (the article does not) that parsing and deserialisation is generally a known weak area and a common source of CVEs, even when pickling is not used. Being more disciplined about it helps, of course.

3036e4 1y ago

About 1, directory of files, many formats these days are just a bunch of files in a ZIP. One thing most applications lack unfortunately is a way to instead just read and write the part files from/to a directory. For one thing it makes it much better for version control, but also just easier to access in general when experimenting. I don't understand why this is not more common, since as a developer it is much more fun to debug things when each thing is its own file rather than an entry in an archive. Most times it is also trivial to support both, since any API for accessing directory entries will be close to 1:1 to an API for accessing ZIP entries anyway. When editing a file locally I would prefer to just have it split up in a directory 99% of the time, only exporting to a ZIP to publish it. Of course it is trivial to write wrapper scripts to keep zipping and unzipping files, and I have done that, but it does feel a bit hacky and should be an unnecessary extra step.

strogonoff 1y ago
Yes, the zipped version is number four. It’s not great for the reason you noted. Some people come up with smudge/clean filters that handle the (de)compression, letting Git store the more structured version of the data even though your working directory contains the compressed files your software can read and write—but I don’t know how portable these things are. I agree with you in general, and it is also why my number one example is that you might not need a single-file format at all. macOS app bundles is a great example of this approach in the wild. One question I was hoping to ask anyone who thought about these matters: what accepted approaches do exist out there when it comes to documenting/speccing out file formats? Ideally, including the cases where the “file” is in fact a directory with a specific layout.

strogonoff 1y ago

(Correction: instead of “CRAW”, I should have written “Canon Cinema Raw Light”. Apparently, those are different things.)

trinix912 1y ago

> 2. Just dump runtime data. Example: Mnemosyne’s old format. Do you use Python? Just dump your state as a Python pickle. (Con: dependency on a particular runtime, good luck rewriting it in Rust.) Be particularly careful with this one as it can potentially vastly expand the attack surface of your program. Not that you shouldn't ever do it, just make sure the deserializer doesn't accept objects/values outside of your spec.

strogonoff 1y ago
I certainly hope no one takes my list as an endorsement… It’s just some formats seen in the wild. It should be noted (the article does not) that parsing and deserialisation is generally a known weak area and a common source of CVEs, even when pickling is not used. Being more disciplined about it helps, of course.