4 ms·

That's basically how I designed the magic bytes for the OpenTimestamps proof files: $ hexdump -C foo.ots 00000000 00 4f 70 65 6e 54 69 6d 65 73 74 6

That's basically how I designed the magic bytes for the OpenTimestamps proof files: $ hexdump -C foo.ots 00000000 00 4f 70 65 6e 54 69 6d 65 73 74 61 6d 70 73 00 |.OpenTimestamps.| 00000010 00 50 72 6f 6f 66 00 bf 89 e2 e8 84 e8 92 94 01 |.Proof..........| 0) Magic is at the beginning of the file. 1) Starts with a null-byte to make it clear this is binary, not text. 2) Includes a human-readable part to make it easy to figure out what the file is in hex dumps. 3) 8 bytes of randomly chosen bytes, all of which greater than 0x7F to ensure they're not ASCII. 3) Finally, a one-byte major version number. 4) Total length (including major version) is 32 bytes to fit nicely in a hex dump.

RustyRussell 2y ago
These days I generally advise that you interpret the version number as odd and even bits: odd means it's compatible with readers, even means it isn't.
- quag 2y ago
  That sounds interesting. Can you say a little more about how this works?
  - RustyRussell 2y ago
    It's a trick I stole from ext2, and simplified. In that filesystem there are three bitsets: one for reading, one for writing, one for fsck. If you don't understand a bit you can't do that action. For most protocols there's only reading and writing, so you can use odd bits to mean "backwards compatible features, you can read even if you don't understand" and even for "stop, we broke compat".
    - petertodd 2y ago
      That's a good idea for filesystems. But OpenTimestamps Proofs aren't really "written to". They're created, and then later validated. Also, being cryptographic proofs, my philosophy is the validator should almost always understand them 100%, or not at all, to avoid any false proofs. That's also why I picked a binary encoding: it's difficult to parse an OTS proof incorrectly. An incorrect implementation will almost always fail to parse the proof at all, with a clear error, rather than silently parse the proof incorrectly.
      - RustyRussell 2y ago
        We use the same for Lightning: even bits for incompatible changes, odd for backwards compatible changes.
kazinator 2y ago
I wouldn't waste the first byte on a null; make use the all four bytes for a "fourcc" code. Then have a null byte soon after that somewhere.
- petertodd 2y ago
  Well, like I said, I wanted a 32-byte magic so it'd look nice in hex-dumps. So I had plenty of room. OTS proof files typically contain lots of 32-byte hash digests. So "wasting" 32 bytes on magic bytes isn't a big deal.

RustyRussell 2y ago

These days I generally advise that you interpret the version number as odd and even bits: odd means it's compatible with readers, even means it isn't.

quag 2y ago
That sounds interesting. Can you say a little more about how this works?
- RustyRussell 2y ago
  It's a trick I stole from ext2, and simplified. In that filesystem there are three bitsets: one for reading, one for writing, one for fsck. If you don't understand a bit you can't do that action. For most protocols there's only reading and writing, so you can use odd bits to mean "backwards compatible features, you can read even if you don't understand" and even for "stop, we broke compat".
  - petertodd 2y ago
    That's a good idea for filesystems. But OpenTimestamps Proofs aren't really "written to". They're created, and then later validated. Also, being cryptographic proofs, my philosophy is the validator should almost always understand them 100%, or not at all, to avoid any false proofs. That's also why I picked a binary encoding: it's difficult to parse an OTS proof incorrectly. An incorrect implementation will almost always fail to parse the proof at all, with a clear error, rather than silently parse the proof incorrectly.
    - RustyRussell 2y ago
      We use the same for Lightning: even bits for incompatible changes, odd for backwards compatible changes.

kazinator 2y ago

I wouldn't waste the first byte on a null; make use the all four bytes for a "fourcc" code. Then have a null byte soon after that somewhere.

petertodd 2y ago
Well, like I said, I wanted a 32-byte magic so it'd look nice in hex-dumps. So I had plenty of room. OTS proof files typically contain lots of 32-byte hash digests. So "wasting" 32 bytes on magic bytes isn't a big deal.