3 ms·

Why not just a zero followed by a UUID? UUIDs are the obvious standard everyone knows for identifying stuff. Maybe a zero, the UUID as ASCII, then another zero

Why not just a zero followed by a UUID? UUIDs are the obvious standard everyone knows for identifying stuff. Maybe a zero, the UUID as ASCII, then another zero, then a human readable description for debugging and search, or a structured metadata header. But first, ask yourself why you are designing a binary format, unless maybe it's a new media container. When would someone ever want a binary file that's not zip, SQLite, or version controllable text?

addaon 2y ago
> When would someone ever want a binary file that's not zip, SQLite, or version controllable text? It feels like there’s an infinite number of answers to this, but to choose one: when choosing the format to allow memory mapping makes some operations simpler or more performant?
Dwedit 2y ago
Tagged files are too useful. 4 byte tag name, 4 byte length of the object, then the binary data of the object. You see these all the time. Sometimes you see the size before the tag name. Occasionally, you also see a file header, followed by a size, and an "x", that often indicates a block of ZLIB compressed data.
- lelanthran 2y ago
  They're called TLV (Tag, Length, Value) and are used extensively in payment transaction systems.
lelanthran 2y ago
> But first, ask yourself why you are designing a binary format, unless maybe it's a new media container. > When would someone ever want a binary file that's not zip, SQLite, or version controllable text? Maybe I'm not getting the humour here, but in case you are being serious binary files do have a few advantages over text formats. 1. Quick detection (say, for dispatching to a handler) 2. Rapid serialisation, both into and out of a running program (program state, in-memory data, etc) 3. Better and safer handling of binary data (no clunky roundtrips of binary blobs to text and back again) 4. Much better checksumming.
- eternityforest 2y ago
  Binary files are useful, but binary files that aren't either zip, sqlite, or a media container seem pretty niche. It makes sense for model weights and media and opaque blobs where you don't need to load just a part of it, but I see a lot of custom binary save files that don't seem to make any sense. If it's a server, everything is probably in a database, and if it's a desktop app, eventually something is going to make an 8GB file and it's probably going to be slow unless you have indexing. People are also likely to want to incrementally update the file as well. If you're sure nobody will ever make a giant file, then VCSability is probably something someone will want.
petertodd 2y ago
> But first, ask yourself why you are designing a binary format, unless maybe it's a new media container. Binary formats are ideal for critical applications where you want to either 1) parse the file correctly, 2) fail to parse the file at all. Non-binary formats (and re-use of existing binary formats) tend to have failure modes where you parse the file incorrectly due to a bug, resulting in something bad happening like a security exploit.

addaon 2y ago

> When would someone ever want a binary file that's not zip, SQLite, or version controllable text? It feels like there’s an infinite number of answers to this, but to choose one: when choosing the format to allow memory mapping makes some operations simpler or more performant?

Dwedit 2y ago

Tagged files are too useful. 4 byte tag name, 4 byte length of the object, then the binary data of the object. You see these all the time. Sometimes you see the size before the tag name. Occasionally, you also see a file header, followed by a size, and an "x", that often indicates a block of ZLIB compressed data.

lelanthran 2y ago
They're called TLV (Tag, Length, Value) and are used extensively in payment transaction systems.

lelanthran 2y ago

> But first, ask yourself why you are designing a binary format, unless maybe it's a new media container. > When would someone ever want a binary file that's not zip, SQLite, or version controllable text? Maybe I'm not getting the humour here, but in case you are being serious binary files do have a few advantages over text formats. 1. Quick detection (say, for dispatching to a handler) 2. Rapid serialisation, both into and out of a running program (program state, in-memory data, etc) 3. Better and safer handling of binary data (no clunky roundtrips of binary blobs to text and back again) 4. Much better checksumming.

eternityforest 2y ago
Binary files are useful, but binary files that aren't either zip, sqlite, or a media container seem pretty niche. It makes sense for model weights and media and opaque blobs where you don't need to load just a part of it, but I see a lot of custom binary save files that don't seem to make any sense. If it's a server, everything is probably in a database, and if it's a desktop app, eventually something is going to make an 8GB file and it's probably going to be slow unless you have indexing. People are also likely to want to incrementally update the file as well. If you're sure nobody will ever make a giant file, then VCSability is probably something someone will want.

petertodd 2y ago

> But first, ask yourself why you are designing a binary format, unless maybe it's a new media container. Binary formats are ideal for critical applications where you want to either 1) parse the file correctly, 2) fail to parse the file at all. Non-binary formats (and re-use of existing binary formats) tend to have failure modes where you parse the file incorrectly due to a bug, resulting in something bad happening like a security exploit.