6 ms·

On File Formats

159 points by ibobev 1y ago

leiserfg 1y ago
If binary, consider just using SQLite.
- lifthrasiir 1y ago
  Using SQLite as a container format is only beneficial when the file format itself is a composite, like word processor files which will include both the textual data and any attachments. SQLite is just a hinderance otherwise, like image file formats or archival/compressed file formats [1]. [1] SQLite's own sqlar format is a bad idea for this reason.
  - frainfreeze 1y ago
    sqlar proved a great solution in the past for me. Where does it fall short in your experience?
    - lifthrasiir 1y ago
      Unless you are using the container file as a database too, sqlar is strictly inferior to ZIP in terms of pretty much everything [1]. I'm actually more interested in the context sqlar did prove useful for you. [1] https://news.ycombinator.com/item?id=28670418 https://news.ycombinator.com/item?id=28670418
      - frainfreeze 1y ago
        I remember seeing the comment you linked few years back, and back then comments were already locked so I couldn't reply, and this time I sadly don't have the time to get deeper into this, however - I recommend you to research more about sqlar/using sqlite db as _file format_ in general, or at minimum looking at SQLite Encryption Extension (SEE) (https://www.sqlite.org/see/doc/trunk/www/readme.wiki https://www.sqlite.org/see/doc/trunk/www/readme.wiki). You can get a lot out of the box with very little investment. IMHO sqlar is not competing with ZIP (can zip do metadata and transactions?)
        lifthrasiir 1y ago
        > [...] at minimum looking at SQLite Encryption Extension (SEE) (https://www.sqlite.org/see/doc/trunk/www/readme.wiki https://www.sqlite.org/see/doc/trunk/www/readme.wiki). SEE is a proprietary extension, however generous its license is. So it is not very meaningful when sqlar is compared against ZIP. Not to say that I necessarily see encryption as a fundamental feature for compressed archive formats though---I'm advocating for age [1] integration instead. [1] https://github.com/FiloSottile/age https://github.com/FiloSottile/age > IMHO sqlar is not competing with ZIP (can zip do metadata and transactions?) In my understanding SQLite's support for sqlar and ZIP occurred at the same time, so I believe that sqlar was created to demonstrate an alternative to ZIP (and that the demonstration wasn't good enough). I'm aware that this is just a circumstantial evidence, so let me know if you have some concrete one. ZIP can of course do metadata in the form of per-file and archive comments. For more structured metadata, you can make use of extra fields if you really, really want, but at that point SQLite would indeed be a better choice. I however doubt it's a typical use case. ZIP can be partially updated in place but can't do any transaction. But it should be noted that SQLite handles transaction by additional files (`-journal` or `-wal` files). So both sqlar and ZIP would write to an additional file during the update process, though SQLite would write much less data compared to ZIP. Any remaining differences are invisible to end users, unless the in-place update is common enough in which case the use of SQLite is justified.
        frainfreeze 1y ago
        Point is that SEE exists, and so do free alternatives. > In my understanding SQLite's support for sqlar and ZIP occurred at the same time I believe so too. I agree with you on SQLAR being poor general-purpose archive or compression format compared to ZIP; what I'm arguing is that its very good file format for certain applications, offering structured, modifiable, and searchable file storage. We had great success using it as db/file format for PLM solution packed both as desktop and web app. Same database can then be used to power the web ui (single tenant SaaS deployments), and for desktop app (web export is simply a working file for desktop app). This file being just a simple sqlite db lets users play with data, do their own imports, migrations etc., while having all files & docs in one place.
  - SyrupThinker 1y ago
    From my own experience SQLite works just fine as the container for an archive format. It ends up having some overhead compared to established ones, but the ability to query over the attributes of 10000s of files is pretty nice, and definitely faster than the worst case of tar. My archiver could even keep up with 7z in some cases (for size and access speed). Implementing it is also not particularly tricky, and SQLite even allows streaming the blobs. Making readers for such a format seems more accessible to me.
    - lifthrasiir 1y ago
      SQLite format itself is not very simple, because it is a database file format in its heart. By using SQLite you are unknowingly constraining your use case; for example you can indeed stream BLOBs, but you can't randomly access BLOBs because the SQLite format puts a large BLOB into pages in a linked list, at least when I checked last. And BLOBs are limited in size anyway (4GB AFAIK) so streaming itself might not be that useful. The use of SQLite also means that you have to bring SQLite into your code base, and SQLite is not very small if you are just using it as a container. > My archiver could even keep up with 7z in some cases (for size and access speed). 7z might feel slow because it enables solid compression by default, which trades decompression speed with compression ratio. I can't imagine 7z having a similar compression ratio with correct options though, was your input incompressible?
      - duskwuff 1y ago
        BLOBs support random access - the handles aren't stateful. https://www.sqlite.org/c3ref/blob_read.html https://www.sqlite.org/c3ref/blob_read.html You're right that their size is limited, though, and it's actually worse than you even thought (1 GB).
        lifthrasiir 1y ago
        My statement wasn't precise enough, you are correct that random access API is provided. But it is ultimately connected to the `accessPayload` function in btree.c which comment mentions that: ** The content being read or written might appear on the main page ** or be scattered out on multiple overflow pages. In the other words, the API can read from multiple scattered pages unknowingly to the caller. That said I see this can be considered enough for being random accessible, as the underlying file system would use similarly structured indices behind the scene anyway... (But modern file systems do have consecutively allocated pages for performance.)
  - sureglymop 1y ago
    I think it's fine as an image format. I've used the mbtiles format which is basically just a table filled with map tiles. Sqlite makes it super easy to deal with it, e.g. to dump individual blobs and save them as image files. It just may not always be the most performant option. For example, for map tiles there is alternatively the pmtiles binary format which is optimized for http range requests.
  - InsideOutSanta 1y ago
    The Mac image editor Acorn uses SQLite as its file format. It's described here: https://shapeof.com/archives/2025/4/acorn_file_format.html https://shapeof.com/archives/2025/4/acorn_file_format.html The author notes that an advantage is that other programs can easily read the file format and extract information from it.
    - lifthrasiir 1y ago
      It is clearly a composite file format [1]: > Acorn’s native file format is used to losslessly store layer data, editable text, layer filters, an optional composite of the image, and various metadata. Its advantage over other common formats such as PNG or JPEG is that it preserves all this native information without flattening the layer data or vector graphics. As I've mentioned, this is a good use case for SQLite as a container. But ZIP would work equally well here. [1] https://flyingmeat.com/acorn/docs/technotes/ACTN002.html https://flyingmeat.com/acorn/docs/technotes/ACTN002.html
  - aidenn0 1y ago
    Except image formats and archival formats are composites (data+metadata). We have Exif for images, and you might be surprised by how much metadata the USTar format has.
    - lifthrasiir 1y ago
      With that reasoning almost every format is a composite, which doesn't sound like a useful distinction. Such metadata should be fine as long as the metadata itself is isolated and can be updated without the parent format.
      - aidenn0 1y ago
        I agree that almost every format is a composite; you seem to not, which makes me think you mean something different than I by "composite." Your reply suggests that, if all the metadata is auxiliary it can be segregated from the data and doesn't count as a composite. However, that doesn't exclude archives (in many use-cases the file metadata is as important as the data itself; consider e.g. hardlinks in TAR files) Nor does it exclude certain vital metadata for images: resolution, color-space, and bit-depth come to mind.
        lifthrasiir 1y ago
        My reasoning for Exif was that it is not only auxiliary but also post-hoc. Exif was defined independently from image formats and only got adopted later because those formats provided extension points (JPEG APP# markers, PNG chunks). You've got a good point that there are multiple types of metadata and some metadata might be crucial for interpreting data. I would say such "structural" metadata should be considered as a part of data. I'm not saying it is not a metadata; it is a metadata inside some data, so doesn't count for our purpose of defining a composite. I also don't think tar hardlinks are metadata for our purpose, because it technically consists of the linked path instead of the file contents and the information that the file is a hardlink, where the former is clearly a data and the latter is a metadata used to reconstruct the original file system so should be considered as a part of larger data (in this case, a logical notion of "file"). I believe these examples should be enough to derive my own definition of "composite". Please let me know otherwise.
- paulddraper 1y ago
  Did you read the article? That wouldn’t support partial parsing.
  - HelloNurse 1y ago
    On the contrary, loading everything from a database is the limit case of "partial parsing" with queries that read only a few pages of a few tables and indices. From the point of view of the article, a SQLite file is similar to a chunked file format: the compact directory of what tables etc. it contains is more heavyweight than listing chunk names and lengths/offsets, but at least as fast, and loading only needed portions of the file is automatically managed.
adelpozo 1y ago
I would add to make it streamable or at least allow to be read remotely efficiently.
- flowerthoughts 1y ago
  Agreed on that one. With a nice file format, streamable is hopefully just a matter of ordering things appropriately once you know the sizes of the individual chunks. You want to write the index last, but you want to read it first. Perhaps you want the most influential values first if you're building something progressive (level-of-detail split.) Similar is the discussion of delimited fields vs. length prefix. Delimited fields are nicer to write, but length prefixed fields are nicer to read. I think most new formats use length prefixes, so I'd start there. I wrote a blog post about combining the value and length into a VLI that also handles floating point and bit/byte strings: https://tommie.github.io/a/2024/06/small-encoding https://tommie.github.io/a/2024/06/small-encoding
  - lifthrasiir 1y ago
    I don't think a single encoding is generally useful. A good encoding for given application would depend on the value distribution and neighboring data. For example any variable-length scalar encoding would make vectorization much harder.
    - flowerthoughts 1y ago
      Depends if you're optimizing for storage size or code size, and in-memory vs transfer. This encoding was meant to optimize transfer (and perhaps storage.)
mjevans 1y ago
Most of that's pretty good. Compression: For anything that ends up large it's probably desired. Though consider both algorithm and 'strength' based on the use case carefully. Even a simple algorithm might make things faster when it comes time to transfer or write to permanent storage. A high cost search to squeeze out yet more redundancy is probably worth it if something will be copied and/or decompressed many times, but might not be worth it for that locally compiled kernel you'll boot at most 10 times before replacing it with another.
thasso 1y ago
For archive formats, or anything that has a table of contents or an index, consider putting the index at the end of the file so that you can append to it without moving a lot of data around. This also allows for easy concatenation.
- charcircuit 1y ago
  Why not put it at the beginning so that it is available at the start of the filestream that way it is easier to get first so you know what other ranges of the file you may need? >This also allows for easy concatenation. How would it be easier than putting it at the front?
  - lifthrasiir 1y ago
    If the archive is being updated in place, turning ABC# into ABCD#' (where # and #' are indices) is easier than turning #ABC into #'ABCD. The actual position of indices doesn't matter much if the stream is seekable. I don't think the concatenation is a good argument though.
  - shakna 1y ago
    Files are... Flat streams. Sort of. So if you rewrite an index at the head of the file, you may end up having to rewrite everything that comes afterwards, to push it further down in the file, if it overflows any padding offset. Which makes appending an extremely slow operation. Whereas seeking to end, and then rewinding, is not nearly as costly.
    - charcircuit 1y ago
      Most workflows do not modify files in place but rather create new files as its safer and allows you to go back to the original if you made a mistake.
      - shakna 1y ago
        If you're writing twice, you don't care about the performance to begin with. Or the size of the files being produced. But if you're writing indices, there's a good chance that you do care about performance.
shakna 1y ago
Spent the weekend with an untagged chunked format, and... I rather hate it. A friend wanted a newer save viewer/editor for Dragonball Xenoverse 2, because there's about a total of two, and they're slow to update. I thought it'd be fairly easy to spin up something to read it, because I've spun up a bunch of save editors before, and they're usually trivial. XV2 save files change over versions. They're also just arrays of structs [0], that don't properly identify themselves, so some parts of them you're just guessing. Each chunk can also contain chunks - some of which are actually a network request to get more chunks from elsewhere in the codebase! [0] Also encrypted before dumping to disk, but the keys have been known since about the second release, and they've never switched them.
lifthrasiir 1y ago
Generally good points. Unfortunately existing file formats are rarely following these rules. In fact these rules should form naturally when you are dealing with many different file formats anyway. Specific points follow: - Agreed that human-readable formats have to be dead simple, otherwise binary formats should be used. Note that textual numbers are surprisingly complex to handle, so any formats with significant number uses should just use binary. - Chunking is generally good for structuring and incremental parsing, but do not expect it to provide reorderability or back/forward compatibility somehow. Unless explicitly designed, they do not exist. Consider PNG for example; PNG chunks were designed to be quite robust, but nowadays some exceptions [1] do exist. Versioning is much more crucial for that. [1] https://www.w3.org/TR/png/#animation-information https://www.w3.org/TR/png/#animation-information - Making a new file format from scratch is always difficult. Already mentioned, but you should really consider using existing file formats as a container first. Some formats are even explicitly designed for this purpose, like sBOX [2] or RFC 9277 CBOR-labeled data tags [3]. [2] https://nothings.org/computer/sbox/sbox.html https://nothings.org/computer/sbox/sbox.html [3] https://www.rfc-editor.org/rfc/rfc9277.html https://www.rfc-editor.org/rfc/rfc9277.html
- mort96 1y ago
  > Note that textual numbers are surprisingly complex to handle, so any formats with significant number uses should just use binary. Especially true of floats! With binary formats, it's usually enough to only support machines whose floating point representation conforms to IEEE 754, which means you can just memcpy a float variable to or from the file (maybe with some endianness conversion). But writing a floating point parser and serializer which correctly round-trips all floats and where the parser guarantees that it parses to the nearest possible float... That's incredibly tricky. What I've sometimes done when I'm writing a parser for textual floats is, I parse the input into separate parts (so the integer part, the floating point part, the exponent part), then serialize those parts into some other format which I already have a parser for. So I may serialize them into a JSON-style number and use a JSON library to parse it if I have that handy, or if I don't, I serialize it into a format that's guaranteed to work with strtod regardless of locale. (The C standard does, surprisingly, quite significantly constrain how locales can affect strtod's number parsing.)
constantcrying 1y ago
Also you should consider the context in which you are developing. Often there are "standard" tools and methods to deal with the kind of data you want to store. E.g. if you are interested in storing significant amounts of structured floating point data, choosing something like HDF5 will not only make your life easier it will also make it easy to communicate what you have done to others.
deleted 1y ago
[deleted]
InsideOutSanta 1y ago
>Most extensions have three characters, which means the search space is pretty crowded. You may want to consider using four letters. Is there a reason not to use a lot more characters? If your application's name is MustacheMingle, call the file foo.mustachemingle instead of foo.mumi? This will decrease the probability of collision to almost zero. I am unaware of any operating systems that don't allow it, and it will be 100% clear to the user which application the file belongs to. It will be less aesthetically pleasing than a shorter extension, but that's probably mainly a matter of habit. We're just not used to longer file name extensions. Any reason why this is a bad idea?
- delusional 1y ago
  > it will be 100% clear to the user which application the file belongs to. The most popular operating system hides it from the user, so clarity would not improve in that case. At leat one other (Linux) doesn't really use "extensions" and instead relies on magic headers inside the files to determine the format. Otherwise I think the decision is largely aestethic. If you value absolute clarity, then I don't see any reason it won't work, it'll just be a little "ugly"
  - hiAndrewQuinn 1y ago
    I don't even think it's ugly. I'm incredibly thankful every time I see someone make e.g. `db.sqlite`, it immediately sets me at ease to know I'm not accidentally dealing with a DuckDB file or something.
    - wvbdmp 1y ago
      Yes, oh my god. Stop using .db for Sqlite files!!! It’s too generic and it’s already used by Windows for those thumbnail system files.
  - dist-epoch 1y ago
    > At leat one other (Linux) doesn't really use "extensions" and instead relies on magic headers inside the files to determine the format. mostly for executable files. I doubt many Linux apps look inside a .py file to see if it's actually a JPEG they should build a thumbnail for.
strogonoff 1y ago
Thinking about a file format is a good way to clarify your vision. Even if you don’t want to facilitate interop, you’d get some benefits for free—if you can encapsulate the state of a particular thing that the user is working on, you could, for example, easily restore their work when they return, etc. Some cop-out (not necessarily in a bad way) file formats: 1. Don’t have a file format, just specify a directory layout instead. Example: CinemaDNG. Throw a bunch of particularly named DNGs (a file for each frame of the footage) in a directory, maybe add some metadata file or a marker, and you’re good. Compared to the likes of CRAW or BRAW, you lose in compression, but gain in interop. 2. Just dump runtime data. Example: Mnemosyne’s old format. Do you use Python? Just dump your state as a Python pickle. (Con: dependency on a particular runtime, good luck rewriting it in Rust.) 3. Almost dump runtime data. Example: Anki, newer Mnemosyne with their SQLite dumps. (Something suggests to me that they might be using SQLite at runtime.) A step up from a pickle in terms of interop, somewhat opens yourself (but also others) to alternative implementations, at least in any runtime that has the means to read SQLite. I hope if you use this you don’t think that the presence of SQL schema makes the format self-documenting. 4. One or more of the above, except also zip or tar it up. Example: VCV, Anki.
- 3036e4 1y ago
  About 1, directory of files, many formats these days are just a bunch of files in a ZIP. One thing most applications lack unfortunately is a way to instead just read and write the part files from/to a directory. For one thing it makes it much better for version control, but also just easier to access in general when experimenting. I don't understand why this is not more common, since as a developer it is much more fun to debug things when each thing is its own file rather than an entry in an archive. Most times it is also trivial to support both, since any API for accessing directory entries will be close to 1:1 to an API for accessing ZIP entries anyway. When editing a file locally I would prefer to just have it split up in a directory 99% of the time, only exporting to a ZIP to publish it. Of course it is trivial to write wrapper scripts to keep zipping and unzipping files, and I have done that, but it does feel a bit hacky and should be an unnecessary extra step.
  - strogonoff 1y ago
ahoka 1y ago
Also, "Don't try to be clever to save a few bits.", like using the lower and upper 4 bits of a byte for different things (I'm looking at you, ONFI).
- vrighter 1y ago
  I've had to do just that to retrofit features I wasn't allowed to think about up front (we must get the product out the door.... we'll cross that bridge when we get to it)
- Dwedit 1y ago
  iNES file format is guilty of badly designed bit packing. Four flags were packed into the lower 4 bits, then Mapper Number was assigned to the high 4 bits. But then they needed more than 16 mappers. They used 4 high bits of the next byte to store the remaining 4 bits, and that was enough... until they needed over 256 mappers.
teddyh 1y ago
Designing your file (and data) formats well is important. “Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.” — Fred Brooks
zzo38computer 1y ago
Consider DER format. Partial parsing is possible; you can easily ignore any part of the file that you do not care about, since the framing is consistent. Additionally, it works like the "chunked" formats mentioned in the article, and one of the bits of the header indicates whether it includes other chunks or includes data. (Furthermore, I made up a text-based format called TER which is intended to be converted to DER. TER is not intended to be used directly; it is only intended to be converted to DER for then use in other programs. I had also made up some additional data types, and one of these (called ASN1_IDENTIFIED_DATA) can be used for identifying the format of a file (which might conform to multiple formats, and it allows this too).) I dislike JSON and some other modern formats (even binary formats); they often are just not as good in my opinion. One problem is they tend to insist on using Unicode, and/or on other things (e.g. 32-bit integers where you might need 64-bits). When using a text-based format where binary would do better, it can also be inefficient especially if binary data is included within the text as well, especially if the format does not indicate that it is meant to represent binary data. However, even if you use an existing format, you should avoid using the existing format badly; using existing formats badly seems to be common. There is also the issue of if the existing format is actually good or not; many formats are not good, for various reasons (some of which I mentioned above, but there are others, depending on the application). About target hardware, not all software is intended for a specific target hardware, although some is. For compression, another consideration is: there are general compression schemes as well as being able to make up a compression scheme that is specific for the kind of data that is being compressed. They also mention file names. However, this can also depend on the target system; e.g. for DOS files you will need to be limited to three characters after the dot. Also, some programs would not need to care about file names in some or all cases (many programs I write don't care about file names).
- aidenn0 1y ago
  Maybe it's just because I've never needed the complexity, but ASN.1 seems a bit much for any of the formats I've created.
  - zzo38computer 1y ago
    For me too, although you only need to use (and implement) the parts which are relevant for your application and not all of them, so it is not really the problem. (I also never needed to write ASN.1 schemas, and a full implementation of ASN.1 is not necessary for my purpose.) (This is also a reason I use DER instead of BER, even if canonical form is not required; DER is simpler to handle than all of the possibilities of BER.)
fake-name 1y ago
I have a rather ideosyncratic opinion here: For Open-Source projects, human readable file formats are actively harmful. This mostly is motivated by my experience with KiCad. Principally, there are multiple things that the UI does not expose at all (slots in PCB footprint files) where the only way to add them is to manually edit the footprint file in a text editor. There are some other similar annoyances in the same vein. Basically, human readable (and therefore editable) file formats wind up being a way for some things to never be exposed thru the UI. This actively leads to the software being less capable.
- zzo38computer 1y ago
  Not exposing things in the UI is not necessarily a problem (it depends on the program and on other stuff), although it can be (especially if it is not documented). I had not used the program you mention, although it does seem a problem in the way you mention, although someone who wants to add it into the UI could hopefully do so if it is FOSS. However, one potential problem is sometimes if it is a text-based format, writing such a format (in a way which still remains readable rather than messy) can sometimes be more complicated than reading it. (The TEMPLATE.DER lump (which is a binary file format and not plain text) in Super ZZ Zero is not exposed anywhere in the UI; you must use an external program to create this lump if you want it. Fortunately that lump is not actually mandatory, and only affects the automatic initial modifications of a new world file based on an existing template.) However, I think that human readable file formats are harmful for other reasons.
layer8 1y ago
> However, it's cleaner to have a field in your header that states where the first sub-chunk starts; that way you can expand your header as much as you like in future versions, with old code being able to ignore those fields and jump to the good stuff. That’s assuming that parsers will honor this, and not just use the fixed offset that worked for the past ten hears. This has happened often enough in the past.
nikeee 1y ago
Depending on the use-case, it might be good to have a file format "diffable", especially when it's going to be checked in in some version control.
afsdef3f3wafew 1y ago
[dead]
Dwedit 1y ago
Sometimes, you'll need to pack multiple files inside of a single file. Those files will need to grow, and be able to be deleted. At that point, you're asking for a filesystem inside of a file. And you can literally do exactly that with a filesystem library (FAT32, etc).
codingjerk 1y ago
Good overview. The "Chunk your binaries" point is spot on. Creating a huge binary blob that contains everything makes it hard to work with in constrained environments. Also, +1 for "Document your format". More like "Document everything". Future you will thank you for it for sure.
Const-me 1y ago
Good article, but I’d add one more paragraph. If your data format contains multiple streams inside, consider ZIP for the container. Enables standard tools, and libraries available in all languages. The compression support is built-in but optional, can be enabled selectively for different entries. The approach is widely used in practice. MS office files, Java binaries, iOS app store binaries, Android binaries, epub books, chm documentation are all using ZIP container format.
mixcocam 1y ago
If you find yourself building a file format - you should read this page carefully and make sure that you have very good arguments which support why it does not apply to you. https://www.sqlite.org/appfileformat.html https://www.sqlite.org/appfileformat.html

On File Formats

159 points by ibobev 1y ago

leiserfg 1y ago

If binary, consider just using SQLite.

lifthrasiir 1y ago
Using SQLite as a container format is only beneficial when the file format itself is a composite, like word processor files which will include both the textual data and any attachments. SQLite is just a hinderance otherwise, like image file formats or archival/compressed file formats [1]. [1] SQLite's own sqlar format is a bad idea for this reason.
- frainfreeze 1y ago
  sqlar proved a great solution in the past for me. Where does it fall short in your experience?
  - lifthrasiir 1y ago
    Unless you are using the container file as a database too, sqlar is strictly inferior to ZIP in terms of pretty much everything [1]. I'm actually more interested in the context sqlar did prove useful for you. [1] https://news.ycombinator.com/item?id=28670418 https://news.ycombinator.com/item?id=28670418
    - frainfreeze 1y ago
      I remember seeing the comment you linked few years back, and back then comments were already locked so I couldn't reply, and this time I sadly don't have the time to get deeper into this, however - I recommend you to research more about sqlar/using sqlite db as _file format_ in general, or at minimum looking at SQLite Encryption Extension (SEE) (https://www.sqlite.org/see/doc/trunk/www/readme.wiki https://www.sqlite.org/see/doc/trunk/www/readme.wiki). You can get a lot out of the box with very little investment. IMHO sqlar is not competing with ZIP (can zip do metadata and transactions?)
      - lifthrasiir 1y ago
        > [...] at minimum looking at SQLite Encryption Extension (SEE) (https://www.sqlite.org/see/doc/trunk/www/readme.wiki https://www.sqlite.org/see/doc/trunk/www/readme.wiki). SEE is a proprietary extension, however generous its license is. So it is not very meaningful when sqlar is compared against ZIP. Not to say that I necessarily see encryption as a fundamental feature for compressed archive formats though---I'm advocating for age [1] integration instead. [1] https://github.com/FiloSottile/age https://github.com/FiloSottile/age > IMHO sqlar is not competing with ZIP (can zip do metadata and transactions?) In my understanding SQLite's support for sqlar and ZIP occurred at the same time, so I believe that sqlar was created to demonstrate an alternative to ZIP (and that the demonstration wasn't good enough). I'm aware that this is just a circumstantial evidence, so let me know if you have some concrete one. ZIP can of course do metadata in the form of per-file and archive comments. For more structured metadata, you can make use of extra fields if you really, really want, but at that point SQLite would indeed be a better choice. I however doubt it's a typical use case. ZIP can be partially updated in place but can't do any transaction. But it should be noted that SQLite handles transaction by additional files (`-journal` or `-wal` files). So both sqlar and ZIP would write to an additional file during the update process, though SQLite would write much less data compared to ZIP. Any remaining differences are invisible to end users, unless the in-place update is common enough in which case the use of SQLite is justified.
        frainfreeze 1y ago
        Point is that SEE exists, and so do free alternatives. > In my understanding SQLite's support for sqlar and ZIP occurred at the same time I believe so too. I agree with you on SQLAR being poor general-purpose archive or compression format compared to ZIP; what I'm arguing is that its very good file format for certain applications, offering structured, modifiable, and searchable file storage. We had great success using it as db/file format for PLM solution packed both as desktop and web app. Same database can then be used to power the web ui (single tenant SaaS deployments), and for desktop app (web export is simply a working file for desktop app). This file being just a simple sqlite db lets users play with data, do their own imports, migrations etc., while having all files & docs in one place.
- SyrupThinker 1y ago
  From my own experience SQLite works just fine as the container for an archive format. It ends up having some overhead compared to established ones, but the ability to query over the attributes of 10000s of files is pretty nice, and definitely faster than the worst case of tar. My archiver could even keep up with 7z in some cases (for size and access speed). Implementing it is also not particularly tricky, and SQLite even allows streaming the blobs. Making readers for such a format seems more accessible to me.
  - lifthrasiir 1y ago
    SQLite format itself is not very simple, because it is a database file format in its heart. By using SQLite you are unknowingly constraining your use case; for example you can indeed stream BLOBs, but you can't randomly access BLOBs because the SQLite format puts a large BLOB into pages in a linked list, at least when I checked last. And BLOBs are limited in size anyway (4GB AFAIK) so streaming itself might not be that useful. The use of SQLite also means that you have to bring SQLite into your code base, and SQLite is not very small if you are just using it as a container. > My archiver could even keep up with 7z in some cases (for size and access speed). 7z might feel slow because it enables solid compression by default, which trades decompression speed with compression ratio. I can't imagine 7z having a similar compression ratio with correct options though, was your input incompressible?
    - duskwuff 1y ago
      BLOBs support random access - the handles aren't stateful. https://www.sqlite.org/c3ref/blob_read.html https://www.sqlite.org/c3ref/blob_read.html You're right that their size is limited, though, and it's actually worse than you even thought (1 GB).
      - lifthrasiir 1y ago
        My statement wasn't precise enough, you are correct that random access API is provided. But it is ultimately connected to the `accessPayload` function in btree.c which comment mentions that: ** The content being read or written might appear on the main page ** or be scattered out on multiple overflow pages. In the other words, the API can read from multiple scattered pages unknowingly to the caller. That said I see this can be considered enough for being random accessible, as the underlying file system would use similarly structured indices behind the scene anyway... (But modern file systems do have consecutively allocated pages for performance.)
- sureglymop 1y ago
  I think it's fine as an image format. I've used the mbtiles format which is basically just a table filled with map tiles. Sqlite makes it super easy to deal with it, e.g. to dump individual blobs and save them as image files. It just may not always be the most performant option. For example, for map tiles there is alternatively the pmtiles binary format which is optimized for http range requests.
- InsideOutSanta 1y ago
  The Mac image editor Acorn uses SQLite as its file format. It's described here: https://shapeof.com/archives/2025/4/acorn_file_format.html https://shapeof.com/archives/2025/4/acorn_file_format.html The author notes that an advantage is that other programs can easily read the file format and extract information from it.
  - lifthrasiir 1y ago
    It is clearly a composite file format [1]: > Acorn’s native file format is used to losslessly store layer data, editable text, layer filters, an optional composite of the image, and various metadata. Its advantage over other common formats such as PNG or JPEG is that it preserves all this native information without flattening the layer data or vector graphics. As I've mentioned, this is a good use case for SQLite as a container. But ZIP would work equally well here. [1] https://flyingmeat.com/acorn/docs/technotes/ACTN002.html https://flyingmeat.com/acorn/docs/technotes/ACTN002.html
- aidenn0 1y ago
  Except image formats and archival formats are composites (data+metadata). We have Exif for images, and you might be surprised by how much metadata the USTar format has.
  - lifthrasiir 1y ago
    With that reasoning almost every format is a composite, which doesn't sound like a useful distinction. Such metadata should be fine as long as the metadata itself is isolated and can be updated without the parent format.
    - aidenn0 1y ago
      I agree that almost every format is a composite; you seem to not, which makes me think you mean something different than I by "composite." Your reply suggests that, if all the metadata is auxiliary it can be segregated from the data and doesn't count as a composite. However, that doesn't exclude archives (in many use-cases the file metadata is as important as the data itself; consider e.g. hardlinks in TAR files) Nor does it exclude certain vital metadata for images: resolution, color-space, and bit-depth come to mind.
      - lifthrasiir 1y ago
        My reasoning for Exif was that it is not only auxiliary but also post-hoc. Exif was defined independently from image formats and only got adopted later because those formats provided extension points (JPEG APP# markers, PNG chunks). You've got a good point that there are multiple types of metadata and some metadata might be crucial for interpreting data. I would say such "structural" metadata should be considered as a part of data. I'm not saying it is not a metadata; it is a metadata inside some data, so doesn't count for our purpose of defining a composite. I also don't think tar hardlinks are metadata for our purpose, because it technically consists of the linked path instead of the file contents and the information that the file is a hardlink, where the former is clearly a data and the latter is a metadata used to reconstruct the original file system so should be considered as a part of larger data (in this case, a logical notion of "file"). I believe these examples should be enough to derive my own definition of "composite". Please let me know otherwise.
paulddraper 1y ago
Did you read the article? That wouldn’t support partial parsing.
- HelloNurse 1y ago
  On the contrary, loading everything from a database is the limit case of "partial parsing" with queries that read only a few pages of a few tables and indices. From the point of view of the article, a SQLite file is similar to a chunked file format: the compact directory of what tables etc. it contains is more heavyweight than listing chunk names and lengths/offsets, but at least as fast, and loading only needed portions of the file is automatically managed.

adelpozo 1y ago

I would add to make it streamable or at least allow to be read remotely efficiently.

flowerthoughts 1y ago
Agreed on that one. With a nice file format, streamable is hopefully just a matter of ordering things appropriately once you know the sizes of the individual chunks. You want to write the index last, but you want to read it first. Perhaps you want the most influential values first if you're building something progressive (level-of-detail split.) Similar is the discussion of delimited fields vs. length prefix. Delimited fields are nicer to write, but length prefixed fields are nicer to read. I think most new formats use length prefixes, so I'd start there. I wrote a blog post about combining the value and length into a VLI that also handles floating point and bit/byte strings: https://tommie.github.io/a/2024/06/small-encoding https://tommie.github.io/a/2024/06/small-encoding
- lifthrasiir 1y ago
  I don't think a single encoding is generally useful. A good encoding for given application would depend on the value distribution and neighboring data. For example any variable-length scalar encoding would make vectorization much harder.
  - flowerthoughts 1y ago
    Depends if you're optimizing for storage size or code size, and in-memory vs transfer. This encoding was meant to optimize transfer (and perhaps storage.)

mjevans 1y ago

Most of that's pretty good. Compression: For anything that ends up large it's probably desired. Though consider both algorithm and 'strength' based on the use case carefully. Even a simple algorithm might make things faster when it comes time to transfer or write to permanent storage. A high cost search to squeeze out yet more redundancy is probably worth it if something will be copied and/or decompressed many times, but might not be worth it for that locally compiled kernel you'll boot at most 10 times before replacing it with another.

thasso 1y ago

For archive formats, or anything that has a table of contents or an index, consider putting the index at the end of the file so that you can append to it without moving a lot of data around. This also allows for easy concatenation.

charcircuit 1y ago
Why not put it at the beginning so that it is available at the start of the filestream that way it is easier to get first so you know what other ranges of the file you may need? >This also allows for easy concatenation. How would it be easier than putting it at the front?
- lifthrasiir 1y ago
  If the archive is being updated in place, turning ABC# into ABCD#' (where # and #' are indices) is easier than turning #ABC into #'ABCD. The actual position of indices doesn't matter much if the stream is seekable. I don't think the concatenation is a good argument though.
- shakna 1y ago
  Files are... Flat streams. Sort of. So if you rewrite an index at the head of the file, you may end up having to rewrite everything that comes afterwards, to push it further down in the file, if it overflows any padding offset. Which makes appending an extremely slow operation. Whereas seeking to end, and then rewinding, is not nearly as costly.
  - charcircuit 1y ago
    Most workflows do not modify files in place but rather create new files as its safer and allows you to go back to the original if you made a mistake.
    - shakna 1y ago
      If you're writing twice, you don't care about the performance to begin with. Or the size of the files being produced. But if you're writing indices, there's a good chance that you do care about performance.

shakna 1y ago

Spent the weekend with an untagged chunked format, and... I rather hate it. A friend wanted a newer save viewer/editor for Dragonball Xenoverse 2, because there's about a total of two, and they're slow to update. I thought it'd be fairly easy to spin up something to read it, because I've spun up a bunch of save editors before, and they're usually trivial. XV2 save files change over versions. They're also just arrays of structs [0], that don't properly identify themselves, so some parts of them you're just guessing. Each chunk can also contain chunks - some of which are actually a network request to get more chunks from elsewhere in the codebase! [0] Also encrypted before dumping to disk, but the keys have been known since about the second release, and they've never switched them.

lifthrasiir 1y ago

Generally good points. Unfortunately existing file formats are rarely following these rules. In fact these rules should form naturally when you are dealing with many different file formats anyway. Specific points follow: - Agreed that human-readable formats have to be dead simple, otherwise binary formats should be used. Note that textual numbers are surprisingly complex to handle, so any formats with significant number uses should just use binary. - Chunking is generally good for structuring and incremental parsing, but do not expect it to provide reorderability or back/forward compatibility somehow. Unless explicitly designed, they do not exist. Consider PNG for example; PNG chunks were designed to be quite robust, but nowadays some exceptions [1] do exist. Versioning is much more crucial for that. [1] https://www.w3.org/TR/png/#animation-information https://www.w3.org/TR/png/#animation-information - Making a new file format from scratch is always difficult. Already mentioned, but you should really consider using existing file formats as a container first. Some formats are even explicitly designed for this purpose, like sBOX [2] or RFC 9277 CBOR-labeled data tags [3]. [2] https://nothings.org/computer/sbox/sbox.html https://nothings.org/computer/sbox/sbox.html [3] https://www.rfc-editor.org/rfc/rfc9277.html https://www.rfc-editor.org/rfc/rfc9277.html

mort96 1y ago
> Note that textual numbers are surprisingly complex to handle, so any formats with significant number uses should just use binary. Especially true of floats! With binary formats, it's usually enough to only support machines whose floating point representation conforms to IEEE 754, which means you can just memcpy a float variable to or from the file (maybe with some endianness conversion). But writing a floating point parser and serializer which correctly round-trips all floats and where the parser guarantees that it parses to the nearest possible float... That's incredibly tricky. What I've sometimes done when I'm writing a parser for textual floats is, I parse the input into separate parts (so the integer part, the floating point part, the exponent part), then serialize those parts into some other format which I already have a parser for. So I may serialize them into a JSON-style number and use a JSON library to parse it if I have that handy, or if I don't, I serialize it into a format that's guaranteed to work with strtod regardless of locale. (The C standard does, surprisingly, quite significantly constrain how locales can affect strtod's number parsing.)

constantcrying 1y ago

Also you should consider the context in which you are developing. Often there are "standard" tools and methods to deal with the kind of data you want to store. E.g. if you are interested in storing significant amounts of structured floating point data, choosing something like HDF5 will not only make your life easier it will also make it easy to communicate what you have done to others.

deleted 1y ago

[deleted]

InsideOutSanta 1y ago

>Most extensions have three characters, which means the search space is pretty crowded. You may want to consider using four letters. Is there a reason not to use a lot more characters? If your application's name is MustacheMingle, call the file foo.mustachemingle instead of foo.mumi? This will decrease the probability of collision to almost zero. I am unaware of any operating systems that don't allow it, and it will be 100% clear to the user which application the file belongs to. It will be less aesthetically pleasing than a shorter extension, but that's probably mainly a matter of habit. We're just not used to longer file name extensions. Any reason why this is a bad idea?

delusional 1y ago
> it will be 100% clear to the user which application the file belongs to. The most popular operating system hides it from the user, so clarity would not improve in that case. At leat one other (Linux) doesn't really use "extensions" and instead relies on magic headers inside the files to determine the format. Otherwise I think the decision is largely aestethic. If you value absolute clarity, then I don't see any reason it won't work, it'll just be a little "ugly"
- hiAndrewQuinn 1y ago
  I don't even think it's ugly. I'm incredibly thankful every time I see someone make e.g. `db.sqlite`, it immediately sets me at ease to know I'm not accidentally dealing with a DuckDB file or something.
  - wvbdmp 1y ago
    Yes, oh my god. Stop using .db for Sqlite files!!! It’s too generic and it’s already used by Windows for those thumbnail system files.
- dist-epoch 1y ago
  > At leat one other (Linux) doesn't really use "extensions" and instead relies on magic headers inside the files to determine the format. mostly for executable files. I doubt many Linux apps look inside a .py file to see if it's actually a JPEG they should build a thumbnail for.

strogonoff 1y ago

Thinking about a file format is a good way to clarify your vision. Even if you don’t want to facilitate interop, you’d get some benefits for free—if you can encapsulate the state of a particular thing that the user is working on, you could, for example, easily restore their work when they return, etc. Some cop-out (not necessarily in a bad way) file formats: 1. Don’t have a file format, just specify a directory layout instead. Example: CinemaDNG. Throw a bunch of particularly named DNGs (a file for each frame of the footage) in a directory, maybe add some metadata file or a marker, and you’re good. Compared to the likes of CRAW or BRAW, you lose in compression, but gain in interop. 2. Just dump runtime data. Example: Mnemosyne’s old format. Do you use Python? Just dump your state as a Python pickle. (Con: dependency on a particular runtime, good luck rewriting it in Rust.) 3. Almost dump runtime data. Example: Anki, newer Mnemosyne with their SQLite dumps. (Something suggests to me that they might be using SQLite at runtime.) A step up from a pickle in terms of interop, somewhat opens yourself (but also others) to alternative implementations, at least in any runtime that has the means to read SQLite. I hope if you use this you don’t think that the presence of SQL schema makes the format self-documenting. 4. One or more of the above, except also zip or tar it up. Example: VCV, Anki.

3036e4 1y ago
About 1, directory of files, many formats these days are just a bunch of files in a ZIP. One thing most applications lack unfortunately is a way to instead just read and write the part files from/to a directory. For one thing it makes it much better for version control, but also just easier to access in general when experimenting. I don't understand why this is not more common, since as a developer it is much more fun to debug things when each thing is its own file rather than an entry in an archive. Most times it is also trivial to support both, since any API for accessing directory entries will be close to 1:1 to an API for accessing ZIP entries anyway. When editing a file locally I would prefer to just have it split up in a directory 99% of the time, only exporting to a ZIP to publish it. Of course it is trivial to write wrapper scripts to keep zipping and unzipping files, and I have done that, but it does feel a bit hacky and should be an unnecessary extra step.
- strogonoff 1y ago

ahoka 1y ago

Also, "Don't try to be clever to save a few bits.", like using the lower and upper 4 bits of a byte for different things (I'm looking at you, ONFI).

vrighter 1y ago
I've had to do just that to retrofit features I wasn't allowed to think about up front (we must get the product out the door.... we'll cross that bridge when we get to it)
Dwedit 1y ago
iNES file format is guilty of badly designed bit packing. Four flags were packed into the lower 4 bits, then Mapper Number was assigned to the high 4 bits. But then they needed more than 16 mappers. They used 4 high bits of the next byte to store the remaining 4 bits, and that was enough... until they needed over 256 mappers.

teddyh 1y ago

Designing your file (and data) formats well is important. “Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.” — Fred Brooks

zzo38computer 1y ago

Consider DER format. Partial parsing is possible; you can easily ignore any part of the file that you do not care about, since the framing is consistent. Additionally, it works like the "chunked" formats mentioned in the article, and one of the bits of the header indicates whether it includes other chunks or includes data. (Furthermore, I made up a text-based format called TER which is intended to be converted to DER. TER is not intended to be used directly; it is only intended to be converted to DER for then use in other programs. I had also made up some additional data types, and one of these (called ASN1_IDENTIFIED_DATA) can be used for identifying the format of a file (which might conform to multiple formats, and it allows this too).) I dislike JSON and some other modern formats (even binary formats); they often are just not as good in my opinion. One problem is they tend to insist on using Unicode, and/or on other things (e.g. 32-bit integers where you might need 64-bits). When using a text-based format where binary would do better, it can also be inefficient especially if binary data is included within the text as well, especially if the format does not indicate that it is meant to represent binary data. However, even if you use an existing format, you should avoid using the existing format badly; using existing formats badly seems to be common. There is also the issue of if the existing format is actually good or not; many formats are not good, for various reasons (some of which I mentioned above, but there are others, depending on the application). About target hardware, not all software is intended for a specific target hardware, although some is. For compression, another consideration is: there are general compression schemes as well as being able to make up a compression scheme that is specific for the kind of data that is being compressed. They also mention file names. However, this can also depend on the target system; e.g. for DOS files you will need to be limited to three characters after the dot. Also, some programs would not need to care about file names in some or all cases (many programs I write don't care about file names).

aidenn0 1y ago
Maybe it's just because I've never needed the complexity, but ASN.1 seems a bit much for any of the formats I've created.
- zzo38computer 1y ago
  For me too, although you only need to use (and implement) the parts which are relevant for your application and not all of them, so it is not really the problem. (I also never needed to write ASN.1 schemas, and a full implementation of ASN.1 is not necessary for my purpose.) (This is also a reason I use DER instead of BER, even if canonical form is not required; DER is simpler to handle than all of the possibilities of BER.)

fake-name 1y ago

I have a rather ideosyncratic opinion here: For Open-Source projects, human readable file formats are actively harmful. This mostly is motivated by my experience with KiCad. Principally, there are multiple things that the UI does not expose at all (slots in PCB footprint files) where the only way to add them is to manually edit the footprint file in a text editor. There are some other similar annoyances in the same vein. Basically, human readable (and therefore editable) file formats wind up being a way for some things to never be exposed thru the UI. This actively leads to the software being less capable.

zzo38computer 1y ago
Not exposing things in the UI is not necessarily a problem (it depends on the program and on other stuff), although it can be (especially if it is not documented). I had not used the program you mention, although it does seem a problem in the way you mention, although someone who wants to add it into the UI could hopefully do so if it is FOSS. However, one potential problem is sometimes if it is a text-based format, writing such a format (in a way which still remains readable rather than messy) can sometimes be more complicated than reading it. (The TEMPLATE.DER lump (which is a binary file format and not plain text) in Super ZZ Zero is not exposed anywhere in the UI; you must use an external program to create this lump if you want it. Fortunately that lump is not actually mandatory, and only affects the automatic initial modifications of a new world file based on an existing template.) However, I think that human readable file formats are harmful for other reasons.

layer8 1y ago

> However, it's cleaner to have a field in your header that states where the first sub-chunk starts; that way you can expand your header as much as you like in future versions, with old code being able to ignore those fields and jump to the good stuff. That’s assuming that parsers will honor this, and not just use the fixed offset that worked for the past ten hears. This has happened often enough in the past.

nikeee 1y ago

Depending on the use-case, it might be good to have a file format "diffable", especially when it's going to be checked in in some version control.

afsdef3f3wafew 1y ago

[dead]

Dwedit 1y ago

Sometimes, you'll need to pack multiple files inside of a single file. Those files will need to grow, and be able to be deleted. At that point, you're asking for a filesystem inside of a file. And you can literally do exactly that with a filesystem library (FAT32, etc).

codingjerk 1y ago

Good overview. The "Chunk your binaries" point is spot on. Creating a huge binary blob that contains everything makes it hard to work with in constrained environments. Also, +1 for "Document your format". More like "Document everything". Future you will thank you for it for sure.

Const-me 1y ago

Good article, but I’d add one more paragraph. If your data format contains multiple streams inside, consider ZIP for the container. Enables standard tools, and libraries available in all languages. The compression support is built-in but optional, can be enabled selectively for different entries. The approach is widely used in practice. MS office files, Java binaries, iOS app store binaries, Android binaries, epub books, chm documentation are all using ZIP container format.

mixcocam 1y ago

If you find yourself building a file format - you should read this page carefully and make sure that you have very good arguments which support why it does not apply to you. https://www.sqlite.org/appfileformat.html https://www.sqlite.org/appfileformat.html