4 ms·

Six things: 1) "Lack of full precision 64-bit integers" is bullshit. Numeric precision is not specified by JSON. If a parser can't deal with 64-bit integer v

Six things: 1) "Lack of full precision 64-bit integers" is bullshit. Numeric precision is not specified by JSON. If a parser can't deal with 64-bit integer values, it's a poor parser. 2) "s: UTF-8 string" What does this mean? JSON strings are strings of Unicode code points; JSON itself may be encoded as UTF-8, -16, or -32. So does this mean "encode the string as UTF-8, then represent as Unicode code points"? That makes no sense. Does this mean "encode the string as UTF-8 and output directly regardless of the encoding of the rest of the JSON output"? That makes no sense either. So I'm guessing the author just conflated "UTF-8" with "Unicode", which is concerning given that he is attempting to define an interchange protocol. 3) "i: signed integer (base 10, 64-bit range)" What does this mean? (-2^64,2^64)? (-2^63,2^63)? [-2^63,2^63)? 4) "t: timestamp (Z-normalized)" What does that mean? There are literally dozens of timestamp formats. Does he mean full ISO 8601, restricted to UTC? 5) What is the point of TJSON anyway? When you deserialize, you still have to check that the data is of the type you expect. At best this saves a bit of parsing, since the deserializer can do that automatically. Various JSON schema languages already exist, which give you this richer typechecking. The only use case I can think of for this is exactly what the author mentions further down the article: canonicalization for content-aware hashing. But this only works if the only types you care about fall into the small handful he thought of. What about, say, IP addresses? Case-insensitive strings (such as e-mail addresses)? 6) If we're talking about canonicalization, TJSON does not say how to canonicalize decimal numbers. I suppose this stems from the author's mistaken belief that numbers in JSON are IEEE floats (they're not, regardless of what common broken parsers do). I hate to be so negative, but this really comes off as half-baked. EDIT: Looking at the spec [1] it seems to address some of these, but still indicates a strong confusion between data types (Unicode, rational numeric) and data representations (UTF-8, IEEE double). [1] https://github.com/tjson/tjson-spec/blob/master/draft-tjson-spec.md https://github.com/tjson/tjson-spec/blob/master/draft-tjson-...

bascule 10y ago
1) Numeric precision of integers is (under)specified in RFC 7159 section 6: https://tools.ietf.org/html/rfc7159#section-6 https://tools.ietf.org/html/rfc7159#section-6 Note that when such software is used, numbers that are integers and are in the range [-(2**53)+1, (2**53)-1] are interoperable in the sense that implementations will agree exactly on their numeric values. There is no contract that JSON integers give you full 64-bit precision. TJSON has such a contract, and tests for support for full-precision 64-bit integers (and expected failure in the boundary cases) is specified in the canonical test cases/examples file: https://github.com/tjson/tjson-spec/blob/master/draft-tjson-examples.txt#L210 https://github.com/tjson/tjson-spec/blob/master/draft-tjson-... 2) Please see https://github.com/tjson/tjson-spec/issues/27 https://github.com/tjson/tjson-spec/issues/27 3) Yes, these specific ranges are covered in the spec: https://www.tjson.org/spec/#rfc.section.3.3 https://www.tjson.org/spec/#rfc.section.3.3 4) Z-normalized RFC3339. See: https://www.tjson.org/spec/#rfc.section.3.4 https://www.tjson.org/spec/#rfc.section.3.4 5) TJSON provides a repertoire of types which approximates what's available in the scalar types of a format like Protobufs: https://developers.google.com/protocol-buffers/docs/proto3#scalar https://developers.google.com/protocol-buffers/docs/proto3#s... What about, say, IP addresses Simple solution for that case: IP addresses have canonical representations as strings, so use their string representations. Or, if you prefer, represent them as a TJSON object. 6) objecthash provides an alternative to canonicalization: we can use a "content-aware" hash algorithm to produce a digest of the content rather than trying to arrange the content into a canonical form. See: https://github.com/tjson/tjson-spec/issues/24 https://github.com/tjson/tjson-spec/issues/24 I hate to be so negative, but this really comes off as half-baked. As far as I can tell, you didn't read the spec. All of your perceived ambiguities are addressed.
- colanderman 10y ago
  1) That paragraph is discussing interoperability, not the semantics of JSON. JSON "integers" have no such concept as "precision". They are just a sequence of digits. Just like most environments have no problem with very large strings, many environments also have no problem with very large numbers. Dictating "numbers can only be this big" is quite a step backward. 5) Ignoring for a second that IP addresses (particularly IPv6) don't have a universally-accepted canonical format, that's a great solution. But it's one that applies equally to every other data type, even those TJSON special-cases. TJSON is picking a handful of "privileged" types that won't be enough for everyone, so we'll just hit the same problem again.
  - bascule 10y ago
    1) TJSON imposes precision requirements on parsers which JSON lacks. It gives you guarantees where JSON doesn't. JSON may or may not lose precision when you go outside the RFC 7159 range [-(2^53)+1, (2^53)-1]. This is a potential silent failure that mangles data and is unacceptable in a security context, and one present in popular language environments such as JavaScript and Go. 5) The set of scalar types provided by TJSON is not too far off from that provided by protos. As I explained in my previous response, if you want to go beyond those, use a non-scalar type: https://developers.google.com/protocol-buffers/docs/proto3#scalar https://developers.google.com/protocol-buffers/docs/proto3#s... This is par for the course for most typed languages and serialization formats. You don't magically define new scalar types de novo: you build them as sum/product types from scalars and other non-scalars. TJSON's objects are self-describing product types.
bascule 10y ago
Responding to: EDIT: Looking at the spec [1] it seems to address some of these, but still indicates a strong confusion between data types (Unicode, rational numeric) and data representations (UTF-8, IEEE double). The format is described in terms of the tags (which act as type annotations), each of which corresponds to a specific on-the-wire format. Different tagged serializations of the same data may correspond to data of the same type. A better place to discuss ambiguities in the spec regarding this issue is here: https://github.com/tjson/tjson-spec/issues/27 https://github.com/tjson/tjson-spec/issues/27 The idea that different on-the-wire representations of an object correspond to the same typed data object (and can therefore result in the same hash) is core to understanding content-aware hashing. So to your I'm guessing the author just conflated accusations, I don't think you fully understand what's going on here.
- colanderman 10y ago
  JSON is not defined in terms of UTF-8. That would be patently ridiculous, since UTF-8 is a serialization. JSON is defined in terms of Unicode code points. A string in JSON is a sequence of code points, some of which are (necessarily) escaped, others of which may be. So, to say "the string must be UTF-8" makes no sense. The JSON serialization itself can be UTF-8 (which I presume is what the author means). But nowhere does JSON talk about the encoding of a string within JSON, because it is not encoded. Furthermore, what does the author intend for escaped characters? Are they allowed? Presumably not, since that would provide for non-canonical representations. But some escapes must be allowed, since control characters (i.e. code points less than U+0020) must be escaped per the JSON spec. Nowhere does he address this; just a technically meaningless "strings must be UTF-8".
  - bascule 10y ago
    JSON is not defined in terms of UTF-8. That would be patently ridiculous, since UTF-8 is a serialization. TJSON is defined as a serialization format on top of a JSON-like data model. The TJSON spec originally used the terminology "Unicode String", but moved to using "UTF-8 String", the rationale for which is given here: https://github.com/tjson/tjson-spec/issues/27 https://github.com/tjson/tjson-spec/issues/27 If your intent is to actually effect a change in the specification, that is the proper place to do it, but specific criticisms of the exact wording of the specification, preferably in the form of pull-requests, would be the best way to affect such changes. If your intent is not to effect a change in the specification, you're entitled to your opinion, but I'm done discussing the matter as the discussion has ceased to be meaningful to me. Generic criticisms like "You used 'UTF-8' instead of 'Unicode'" outside the context of specific sections of the specification aren't particularly helpful. Furthermore, what does the author intend for escaped characters? Are they allowed? Presumably not, since that would provide for non-canonical representations. You are continuing to miss the point: TJSON intends to provide a foundation to use content-aware content hashing in lieu of a canonicalization scheme as an alternative solution which works across multiple encodings of the same data, sidesteps the exact problems you're talking about, and also allows arbitrary subsets of an object graph to be authenticated without requiring rehashing/resigning. Please see this closed issue on canonicalization ("won't do"): https://github.com/tjson/tjson-spec/issues/24 https://github.com/tjson/tjson-spec/issues/24 From what I can gather, TJSON is offering a degree of abstraction you have not yet fully gleaned. The core idea is: many serializations, one underlying data structure/object graph. TJSON is a mere serialization layer, and indeed many TJSON documents may refer to the same underlying data structure, but all will have the same "objecthash": https://github.com/benlaurie/objecthash https://github.com/benlaurie/objecthash

bascule 10y ago

1) Numeric precision of integers is (under)specified in RFC 7159 section 6: https://tools.ietf.org/html/rfc7159#section-6 https://tools.ietf.org/html/rfc7159#section-6 Note that when such software is used, numbers that are integers and are in the range [-(2**53)+1, (2**53)-1] are interoperable in the sense that implementations will agree exactly on their numeric values. There is no contract that JSON integers give you full 64-bit precision. TJSON has such a contract, and tests for support for full-precision 64-bit integers (and expected failure in the boundary cases) is specified in the canonical test cases/examples file: https://github.com/tjson/tjson-spec/blob/master/draft-tjson-examples.txt#L210 https://github.com/tjson/tjson-spec/blob/master/draft-tjson-... 2) Please see https://github.com/tjson/tjson-spec/issues/27 https://github.com/tjson/tjson-spec/issues/27 3) Yes, these specific ranges are covered in the spec: https://www.tjson.org/spec/#rfc.section.3.3 https://www.tjson.org/spec/#rfc.section.3.3 4) Z-normalized RFC3339. See: https://www.tjson.org/spec/#rfc.section.3.4 https://www.tjson.org/spec/#rfc.section.3.4 5) TJSON provides a repertoire of types which approximates what's available in the scalar types of a format like Protobufs: https://developers.google.com/protocol-buffers/docs/proto3#scalar https://developers.google.com/protocol-buffers/docs/proto3#s... What about, say, IP addresses Simple solution for that case: IP addresses have canonical representations as strings, so use their string representations. Or, if you prefer, represent them as a TJSON object. 6) objecthash provides an alternative to canonicalization: we can use a "content-aware" hash algorithm to produce a digest of the content rather than trying to arrange the content into a canonical form. See: https://github.com/tjson/tjson-spec/issues/24 https://github.com/tjson/tjson-spec/issues/24 I hate to be so negative, but this really comes off as half-baked. As far as I can tell, you didn't read the spec. All of your perceived ambiguities are addressed.

colanderman 10y ago
1) That paragraph is discussing interoperability, not the semantics of JSON. JSON "integers" have no such concept as "precision". They are just a sequence of digits. Just like most environments have no problem with very large strings, many environments also have no problem with very large numbers. Dictating "numbers can only be this big" is quite a step backward. 5) Ignoring for a second that IP addresses (particularly IPv6) don't have a universally-accepted canonical format, that's a great solution. But it's one that applies equally to every other data type, even those TJSON special-cases. TJSON is picking a handful of "privileged" types that won't be enough for everyone, so we'll just hit the same problem again.
- bascule 10y ago
  1) TJSON imposes precision requirements on parsers which JSON lacks. It gives you guarantees where JSON doesn't. JSON may or may not lose precision when you go outside the RFC 7159 range [-(2^53)+1, (2^53)-1]. This is a potential silent failure that mangles data and is unacceptable in a security context, and one present in popular language environments such as JavaScript and Go. 5) The set of scalar types provided by TJSON is not too far off from that provided by protos. As I explained in my previous response, if you want to go beyond those, use a non-scalar type: https://developers.google.com/protocol-buffers/docs/proto3#scalar https://developers.google.com/protocol-buffers/docs/proto3#s... This is par for the course for most typed languages and serialization formats. You don't magically define new scalar types de novo: you build them as sum/product types from scalars and other non-scalars. TJSON's objects are self-describing product types.

bascule 10y ago

Responding to: EDIT: Looking at the spec [1] it seems to address some of these, but still indicates a strong confusion between data types (Unicode, rational numeric) and data representations (UTF-8, IEEE double). The format is described in terms of the tags (which act as type annotations), each of which corresponds to a specific on-the-wire format. Different tagged serializations of the same data may correspond to data of the same type. A better place to discuss ambiguities in the spec regarding this issue is here: https://github.com/tjson/tjson-spec/issues/27 https://github.com/tjson/tjson-spec/issues/27 The idea that different on-the-wire representations of an object correspond to the same typed data object (and can therefore result in the same hash) is core to understanding content-aware hashing. So to your I'm guessing the author just conflated accusations, I don't think you fully understand what's going on here.

colanderman 10y ago
JSON is not defined in terms of UTF-8. That would be patently ridiculous, since UTF-8 is a serialization. JSON is defined in terms of Unicode code points. A string in JSON is a sequence of code points, some of which are (necessarily) escaped, others of which may be. So, to say "the string must be UTF-8" makes no sense. The JSON serialization itself can be UTF-8 (which I presume is what the author means). But nowhere does JSON talk about the encoding of a string within JSON, because it is not encoded. Furthermore, what does the author intend for escaped characters? Are they allowed? Presumably not, since that would provide for non-canonical representations. But some escapes must be allowed, since control characters (i.e. code points less than U+0020) must be escaped per the JSON spec. Nowhere does he address this; just a technically meaningless "strings must be UTF-8".
- bascule 10y ago
  JSON is not defined in terms of UTF-8. That would be patently ridiculous, since UTF-8 is a serialization. TJSON is defined as a serialization format on top of a JSON-like data model. The TJSON spec originally used the terminology "Unicode String", but moved to using "UTF-8 String", the rationale for which is given here: https://github.com/tjson/tjson-spec/issues/27 https://github.com/tjson/tjson-spec/issues/27 If your intent is to actually effect a change in the specification, that is the proper place to do it, but specific criticisms of the exact wording of the specification, preferably in the form of pull-requests, would be the best way to affect such changes. If your intent is not to effect a change in the specification, you're entitled to your opinion, but I'm done discussing the matter as the discussion has ceased to be meaningful to me. Generic criticisms like "You used 'UTF-8' instead of 'Unicode'" outside the context of specific sections of the specification aren't particularly helpful. Furthermore, what does the author intend for escaped characters? Are they allowed? Presumably not, since that would provide for non-canonical representations. You are continuing to miss the point: TJSON intends to provide a foundation to use content-aware content hashing in lieu of a canonicalization scheme as an alternative solution which works across multiple encodings of the same data, sidesteps the exact problems you're talking about, and also allows arbitrary subsets of an object graph to be authenticated without requiring rehashing/resigning. Please see this closed issue on canonicalization ("won't do"): https://github.com/tjson/tjson-spec/issues/24 https://github.com/tjson/tjson-spec/issues/24 From what I can gather, TJSON is offering a degree of abstraction you have not yet fully gleaned. The core idea is: many serializations, one underlying data structure/object graph. TJSON is a mere serialization layer, and indeed many TJSON documents may refer to the same underlying data structure, but all will have the same "objecthash": https://github.com/benlaurie/objecthash https://github.com/benlaurie/objecthash