3 ms·

I think your format is closer to ideal than a few others I have seen. Like json, you don't have to prefix size of lists or objects, which is a plus (or in some

I think your format is closer to ideal than a few others I have seen. Like json, you don't have to prefix size of lists or objects, which is a plus (or in some cases needed) for streaming. Strings can be chunked, which can serve the same purpose for huge strings. The power of json is in its simplicity though. And a binary format should retain that a much as possible. The big advantage of binary formats is not the 'conciseness' (there are compression algorithms that can help with that, where needed) but the reduction in lines of code and CPU cycles needed to serialize and deserialize. Json requires escaping/unescaping of strings and base-2 vs base-10 conversion of numbers. A binary format should mostly solve these and beyond that be as simple to implement bug-free and secure as possible.

kstenerud 5y ago
Yes, that is exactly the aim of the binary format. It has a few basic concepts like being byte-oriented, 1-byte headers for most things, ULEB128 encoding for large values, same chunking mechanism for all arrays and string-likes, same "open/close" mechanism for all container types, etc. The binary codec is VERY simple, and can be trivially implemented for an async-safe or otherwise constrained environment. In fact, I expect that many implementations will only build the binary codec, since you could just pass any recorded binary data through enctool [1] or whatever to see or manipulate its contents as a human. Most systems would have no need to process the text format. [1]https://github.com/kstenerud/enctool https://github.com/kstenerud/enctool
- stkdump 5y ago
  My ideal binary format would contain exactly the types supported by json + binary - arbitrary size numbers. It would be something like this, maybe: - 'n' - null - 'f' - false - 't' - true - 'i' - int (64 bit, little endian, two's complement) - 'd' - double (64 bit, ieee-754) - 'a' - array start - 'o' - object start (must contain even number of values) - 'e' - array/object end - 's' - utf-8 string chunk (64 bytes, must be followed by string chunk or string) - 'b' - binary chunk (64 bytes, must be followed by binary chunk or binary) - 0x80-0xbf - utf-8 string (0-63 bytes) - 0xc0-0xff - binary (0-63 bytes) It is at least an order of magnitude simpler to implement than almost all of its contenders. It is just not as concise and doesn't support a rich set of data types (but then again, json does quite well without a timestamp type). It supports streaming. It has a unique representation for every value (if you ignore that ieee-754 can represent many integers and objects have no defined ordering). Reading a single byte can tell you in every case the type and the amount of payload data that follows.

kstenerud 5y ago

Yes, that is exactly the aim of the binary format. It has a few basic concepts like being byte-oriented, 1-byte headers for most things, ULEB128 encoding for large values, same chunking mechanism for all arrays and string-likes, same "open/close" mechanism for all container types, etc. The binary codec is VERY simple, and can be trivially implemented for an async-safe or otherwise constrained environment. In fact, I expect that many implementations will only build the binary codec, since you could just pass any recorded binary data through enctool [1] or whatever to see or manipulate its contents as a human. Most systems would have no need to process the text format. [1]https://github.com/kstenerud/enctool https://github.com/kstenerud/enctool

stkdump 5y ago
My ideal binary format would contain exactly the types supported by json + binary - arbitrary size numbers. It would be something like this, maybe: - 'n' - null - 'f' - false - 't' - true - 'i' - int (64 bit, little endian, two's complement) - 'd' - double (64 bit, ieee-754) - 'a' - array start - 'o' - object start (must contain even number of values) - 'e' - array/object end - 's' - utf-8 string chunk (64 bytes, must be followed by string chunk or string) - 'b' - binary chunk (64 bytes, must be followed by binary chunk or binary) - 0x80-0xbf - utf-8 string (0-63 bytes) - 0xc0-0xff - binary (0-63 bytes) It is at least an order of magnitude simpler to implement than almost all of its contenders. It is just not as concise and doesn't support a rich set of data types (but then again, json does quite well without a timestamp type). It supports streaming. It has a unique representation for every value (if you ignore that ieee-754 can represent many integers and objects have no defined ordering). Reading a single byte can tell you in every case the type and the amount of payload data that follows.