3 ms·

Generally good points. Unfortunately existing file formats are rarely following these rules. In fact these rules should form naturally when you are dealing with

Generally good points. Unfortunately existing file formats are rarely following these rules. In fact these rules should form naturally when you are dealing with many different file formats anyway. Specific points follow: - Agreed that human-readable formats have to be dead simple, otherwise binary formats should be used. Note that textual numbers are surprisingly complex to handle, so any formats with significant number uses should just use binary. - Chunking is generally good for structuring and incremental parsing, but do not expect it to provide reorderability or back/forward compatibility somehow. Unless explicitly designed, they do not exist. Consider PNG for example; PNG chunks were designed to be quite robust, but nowadays some exceptions [1] do exist. Versioning is much more crucial for that. [1] https://www.w3.org/TR/png/#animation-information https://www.w3.org/TR/png/#animation-information - Making a new file format from scratch is always difficult. Already mentioned, but you should really consider using existing file formats as a container first. Some formats are even explicitly designed for this purpose, like sBOX [2] or RFC 9277 CBOR-labeled data tags [3]. [2] https://nothings.org/computer/sbox/sbox.html https://nothings.org/computer/sbox/sbox.html [3] https://www.rfc-editor.org/rfc/rfc9277.html https://www.rfc-editor.org/rfc/rfc9277.html

mort96 1y ago
> Note that textual numbers are surprisingly complex to handle, so any formats with significant number uses should just use binary. Especially true of floats! With binary formats, it's usually enough to only support machines whose floating point representation conforms to IEEE 754, which means you can just memcpy a float variable to or from the file (maybe with some endianness conversion). But writing a floating point parser and serializer which correctly round-trips all floats and where the parser guarantees that it parses to the nearest possible float... That's incredibly tricky. What I've sometimes done when I'm writing a parser for textual floats is, I parse the input into separate parts (so the integer part, the floating point part, the exponent part), then serialize those parts into some other format which I already have a parser for. So I may serialize them into a JSON-style number and use a JSON library to parse it if I have that handy, or if I don't, I serialize it into a format that's guaranteed to work with strtod regardless of locale. (The C standard does, surprisingly, quite significantly constrain how locales can affect strtod's number parsing.)
- hyperbolablabla 1y ago
  Couldn't you just write the hex bytes? That would be unambiguous, and it wouldn't lose precision.
  - lifthrasiir 1y ago
    At that point human-readability would suffer.
    - adgjlsfhk1 1y ago
      reading hex floats isn't the most intuitive, but isn't that bad.
      - analog31 1y ago
        An amusing anecdote is that Alan Turing taught himself to read numbers in the internal representation used by the computer, rather than allowing the waste of equipment, time, or labor to translate them.
  - mort96 1y ago
    That's still a binary format in my eyes, it's not human readable or writable. You've just written the binary format as hex.
- HappMacDonald 1y ago
  To me this just sounds like an endianness nightmare waiting to happen.
  - dwattttt 1y ago
    You do have to be aware of endianness, although it's really not hard to handle it: pick one. Your code always knows what endian the file format is, so it always knows how to read it.
  - mort96 1y ago

mort96 1y ago

> Note that textual numbers are surprisingly complex to handle, so any formats with significant number uses should just use binary. Especially true of floats! With binary formats, it's usually enough to only support machines whose floating point representation conforms to IEEE 754, which means you can just memcpy a float variable to or from the file (maybe with some endianness conversion). But writing a floating point parser and serializer which correctly round-trips all floats and where the parser guarantees that it parses to the nearest possible float... That's incredibly tricky. What I've sometimes done when I'm writing a parser for textual floats is, I parse the input into separate parts (so the integer part, the floating point part, the exponent part), then serialize those parts into some other format which I already have a parser for. So I may serialize them into a JSON-style number and use a JSON library to parse it if I have that handy, or if I don't, I serialize it into a format that's guaranteed to work with strtod regardless of locale. (The C standard does, surprisingly, quite significantly constrain how locales can affect strtod's number parsing.)

hyperbolablabla 1y ago
Couldn't you just write the hex bytes? That would be unambiguous, and it wouldn't lose precision.
- lifthrasiir 1y ago
  At that point human-readability would suffer.
  - adgjlsfhk1 1y ago
    reading hex floats isn't the most intuitive, but isn't that bad.
    - analog31 1y ago
      An amusing anecdote is that Alan Turing taught himself to read numbers in the internal representation used by the computer, rather than allowing the waste of equipment, time, or labor to translate them.
- mort96 1y ago
  That's still a binary format in my eyes, it's not human readable or writable. You've just written the binary format as hex.
HappMacDonald 1y ago
To me this just sounds like an endianness nightmare waiting to happen.
- dwattttt 1y ago
  You do have to be aware of endianness, although it's really not hard to handle it: pick one. Your code always knows what endian the file format is, so it always knows how to read it.
- mort96 1y ago