3 ms·
> but it does not do anything about one of the big omissions in JSON: it does not specify the range of integers or the precision of fractions Could you give an
by mehrdadn 9y ago
> but it does not do anything about one of the big omissions in JSON: it does not specify the range of integers or the precision of fractions
Could you give an example of a storage format that does do this and the problems that doing so avoids?
- based2 9y agohttps://stackoverflow.com/questions/6175592/xml-schema-precision-of-decimal-value https://stackoverflow.com/questions/6175592/xml-schema-preci... http://www.ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/field_level_documentation_files/schemas/codebook_xsd/elements/range.html http://www.ddialliance.org/Specification/DDI-Codebook/2.5/XM... https://www.obj-sys.com/docs/xbv23/JavaCSUsersGuide/ch06s03s02.html https://www.obj-sys.com/docs/xbv23/JavaCSUsersGuide/ch06s03s...
- majewsky 9y agoI'm not working with binary protocols that often, so I cannot point to an existing protocol. However, if I had to implement a serialization format that doesn't need to be human-readable, I would never use number literals. I would write numbers as one byte whose value means e.g. "the next 4 bytes are an uint32 in little-endian", followed by just these 4 bytes. Then you also don't have to worry about whitespace, because there is none. The next field can follow directly afterwards, because the type header byte defines how long exactly the current field is. (For variable-length strings, use something like netstrings.) The only problem that I can see with this is that Son wants to ensure deterministic encoding, but in my example, 0 could be encoded as an integer of any size or signedness. You could add a rule like "encode every integer as the smallest possible type", but that may have practical implications elsewhere. I wouldn't be too worried about it, seeing how half-baked Son appears to be. For example: > Object members must be sorted by ascending lexicographic order of their keys. Yeah right, because there is only one lexicographic order in the world.
- pegasuscollins 9y ago> Yeah right, because there is only one lexicographic order in the world. The specification tells you all strings (keys) must be UTF8. There is only one lexicographic ordering of UTF8 strings (RFC3629, page 2). This should probably be explicitly mentioned in the linked specification though.
- majewsky 9y ago> There is only one lexicographic ordering of UTF8 strings. Then why is LC_COLLATE still a thing? Also, you don't get to second-guess a spec. If it does not specify everything that it needs to, it deserves the "half-baked" label.
- pegasuscollins 9y ago> The only problem that I can see with this is that Son wants to ensure deterministic encoding, but in my example, 0 could be encoded as an integer of any size or signedness. To be pedantic: Not really. Assuming your example uses twos complement (what else would it use) and you meant "int32" instead of "uint32", negative zero and positive zero would still be the same encoded value. Also since you said "[u]int32", there is only one size the integer can be: 32 bit. So, in your example, the literal value zero has one and only one unique binary representation, which is "0x00000000"
- majewsky 9y agoI implied that a useful serialization format would include integers of multiple size. When your field only needs a uint16, you don't want to pay the extra 6 bytes to encode as, say, uint64.
- pegasuscollins 9y agoYou could either stipulate (as you suggested) to always use the smallest width that still fits a number and then prefix with a length byte but that would be wasteful. Ideally something like LEB128 would be used. At any rate, it's not an issue in terms of guaranteeing a bijective mapping.
- majewsky 9y agoNice, I didn't know about LEB128.
- pegasuscollins 9y agoPretty much every binary format will encode integers using a fixed width or varlen scheme in "base 2". This generally is done for two major reasons: First of all, such an encoding is significantly easier and cheaper to parse than a base10+ascii (human-readable) encoding. I encourage you to write a parser that reads a fixed, 32-bit binary number (1) and another parser that reads a JSON-formatted number string into an internal variable in a classic language like Java or C++. You will immediately see the big difference in complexity. Make sure your parser can also deal with a message that contains more than just one number, i.e. the parser should be able to tell at which byte index an encoded number begins and ends. Even if you're using a language or library where this is hidden from you (e.g. by using parseInt or std::stod) the same work still happens behind the scenes. The other reason is that for most numbers and fixed/varint encoding schemes, the "binary" representation will be much more compact. Storing the number "1000000" in base10+ascii (human readable) takes at least eight bytes. Storing the same number in a fixed 32-bit integer encoding takes four bytes. Using a varint encoding scheme might allow you to get down to three bytes. (1) Ignoring stuff like byte order and representation of negative numbers; this is usually fixed in the protocol/format specification.