3 ms·

Sorry, as who have already designed yet another JSON alternative. Many things about this format are just wrong (as of the first edition): - (EDIT: Mistakenly r

Sorry, as who have already designed yet another JSON alternative. Many things about this format are just wrong (as of the first edition): - (EDIT: Mistakenly read SHOULD as MUST, ignore this item please) The requirement for the mandatory BOM is unacceptable for most non-Windows users, as BOM is by definition invisible. - An arbitrary name is equally questionable, even XML doesn't do that. Do you accept tabs for example? - There is absolutely no way to indicate the data type. The document accidentally mixes the wire format and serialization protocol; never a good sign. - Do you accept 3,000 or 3,,000 or 30,00 or 3,0,0,0 or ,,,3000 or 3000,,, or 3,e,3 or 0.000,3e7 or ,,,? - How many digits in each group are recommended for the encoder? - What on earth is "IEEE 64 bit double precision floating point numbers" in the context of textual format? - Do you require a specific rounding or not in the decimal-to-binary conversion? - Is `\u{d800}` accepted or not? (JSON famously has this issue.) - If you have to escape the base64 padding, maybe you should just drop the padding? - Graph is generally a bad thing to encode at this level, because many applications do not even expect graphs and that can lead to DoS attacks. - No clear definition of the data model. The document starts with objects, arrays and scalars (untyped? stringy? I dunno), then only reveals that objects can be typed and shared and specific types of scalars should be written in certain ways much later. Define the data model first and describe possible encodings of that model instead. - Not allowing anything besides from space and tab is okay, but that doesn't speed processing to be exact. - While technically a matter of choice, it uses a lot of unmatching angle brackets and that's just... ugly. - In fact, I don't really see why other grouping characters were unused in the first place. - No canonical representation. Maybe you should review tons of other alternatives in this space (I recall at least 2--30 of them, probably much more) before your own design.

Fannon 2y ago
Thanks for writing this up! Sounds like you really dig deeply in to this. Not entirely sure what you mean with canonical representation (I've heard this in the context of JSON-LD before, though). Can you explain what you mean here? Where do you see the problem with Graphs and Dos? A reference is just a pointer. You just have to be careful when doing recursive code. I actually like the idea to explicitly define how a reference / association is made, because otherwise people will have to re-invent ID and association concepts and there's no shared understanding. In JSON Schema, you cannot properly express an association or graph structure and people start using overloaded and not well-defined concepts like `$ref` which is a separate standard.
- lifthrasiir 2y ago
  Here the canonical representation refers to one single definite and unambiguous encoding for given data. This requirement is very common in cryptographic applications and also commonly demanded when the deterministic processing is desired. Technically the "canonical" and "deterministic" encoding can differ (e.g. ASN.1 CER vs. DER), but there is not much value to have two distinct encodings. On graphs: as you've said recursive code has to be careful, but the recursion itself is not very frequent in normal applications and they are more susceptible to attacks if the recursion is built into the serialization format. XML billion laughs attack is a famous demonstration for this issue. It is still worthwhile to have parallel standards to specify how recursive structures should be encoded in the basic format, and possibly to tweak the basic format to better accommodate such standards, but I believe such needs can be met without making the basic format bigger.
  - GeneThomas 2y ago
    Is canonilization not irrelevant. Any format with comments is not canonical; so xᴍʟ is not, ᴊꜱᴏɴ has escapes options for string characters so it not either. Response to graph conjecture stated https://news.ycombinator.com/item?id=42072133 https://news.ycombinator.com/item?id=42072133.
    - lifthrasiir 2y ago
      Almost no serialization format is canonical by default---AFAIK bencode was the sole example that mandates the canonicalization. Instead, a canonical subset of the format is usually defined, which would of course exclude comments, unless comments themselves are considered semantic like XML. Yes, even XML has a canonical subset [1]! [1] https://www.w3.org/TR/xml-c14n/ https://www.w3.org/TR/xml-c14n/
      - GeneThomas 2y ago
        There is the concept of “well formed xᴇɴᴏɴ” as outputted by the ᴅᴏᴍ and serializer which is deterministic as will suffice for canonicalization.
zzo38computer 2y ago
> The requirement for the mandatory BOM is unacceptable for most non-Windows users I think the requirement for Unicode is bad in general, whether it uses BOM or not. > What on earth is "IEEE 64 bit double precision floating point numbers" in the context of textual format? I assume it means that numbers are expected to be IEEE 64-bit floating point numbers represented in the decimal format. > If you have to escape the base64 padding, maybe you should just drop the padding? I agree. > While technically a matter of choice, it uses a lot of unmatching angle brackets and that's just... ugly. I agree with that too. > No canonical representation. I think that a binary format would be better as a canonical representation anyways. However, the lack of indicating the data type would seem to make it difficult to know how to convert it unless you already know the schema. > Maybe you should review tons of other alternatives in this space Just a few days ago, and still working on it today, I had been making something called TER. It might also be worth to look, and I wrote another comment relating to this. Xenon does have a reference type to other nodes, which is something that other similar formats don't have, though.
- lifthrasiir 2y ago
  > I think the requirement for Unicode is bad in general, whether it uses BOM or not. While there exist legitimate complaints about Unicode, I believe there is no other feasible encoding than UTF-8 for textual formats now. > I assume it means that numbers are expected to be IEEE 64-bit floating point numbers represented in the decimal format. I too believe so (hence the next item), but that just doesn't make sense if you ponder. IEEE 754 doesn't define any textual format while it does have binary decimal formats. The correct wording should have been that numeric scalars follow a specific grammar to be interpreted as an IEEE 754 binary64 number in the data model. > I think that a binary format would be better as a canonical representation anyways. However, the lack of indicating the data type would seem to make it difficult to know how to convert it unless you already know the schema. That's fair. But we have already observed textual formats being... "abused" for the cryptographic purpose (e.g. JWT), so it's not too bad either to have a canonical representation or to explain why there is no canonical representation defined. > Xenon does have a reference type to other nodes, which is something that other similar formats don't have, though. There are several formats that do try to support native graph types, including YAML and Concise [1]. So that is hardly new. I think Concise actually tried very hard to make it fine! But it became quite more complex as a result. [1] https://concise-encoding.org/ https://concise-encoding.org/
eviks 2y ago
There is no mandatory bom, it's recommended, not a must
- lifthrasiir 2y ago
  Thank you for noticing that, my mistake. Fixed by now.
GeneThomas 2y ago
- The flexibility in name is to support the plethora of programming languages. - Commas are for readability as per English. - Three digits are recommended for the encoder. - Textual data is decoded into binary information in software so setting expectations, e.g. supporting ∞, facilitates interoperability by reducing impedance mismatches. - If the implementation uses ɪᴇᴇᴇ doubles the rounding shall be as expected. - Unicode surrogates are disallowed. - We are following the Base64 standard exactly. - It is essential for data to be structured as a graph, it simply occurs. Serialization on most data formats has the required support for graphs awkwardly layered on top of the encoding. - The C# implementation defines the data model; essentially objects, array and scalars with multiple parents for nodes. The Formats section is to, as stated, facilitate interoperability. - Limiting whitespace to spaces and tabs does speed processing. The ᴜᴛꜰ-8 bytes can be left in the input buffer rather than copied. All significant bytes are ᴀꜱᴄɪɪ. - There are no unmatched angle brackets (that was a typo). - I know the reasoning behind XML.
- lifthrasiir 2y ago
  > The flexibility in name is to support the plethora of programming languages. As far as aware, there is no language that allows a tab in its identifier. I'm aware of some (uncommon) languages that allow a space in identifiers though. After all, the goal of language is firstly to set what's valid or not and secondly to give a meaning to valid one. The valid set should be as maximally different from the invalid set as possible, but allowing virtually invalid characters in names is against this goal. Consider the approach taken by XML 1.1 (not 1.0) if you need an idea. > Commas are for readability as per English. A lot of English-speaking countries actually have `.` and `,` swapped: 3,141.592 vs. 3.141,592 for example. Due to this ambiguity, a comma as a grouping separator is heavily discouraged. You can instead use a underscore `_` (very common in programming languages) or a space (more preferred in human texts) without such concerns. > Three digits are recommended for the encoder. Also, some English-speaking countries use different group sizes (notably India). I'm personally fine with three-digit groupings despite of that, but that should be clearly specified at the very least. > Textual data is decoded into binary information in software so setting expectations, e.g. supporting ∞, facilitates interoperability by reducing impedance mismatches. This procedure should have been made explicit. In fact, the single worst thing you can do in serialization formats is an unclear definition of data model. > If the implementation uses ɪᴇᴇᴇ doubles the rounding shall be as expected. There is nothing like "as expected" in IEEE 754. The current rounding mode is a part of the execution state, so leaving it unspecified risks a non-deterministic interpretation even in the same execution. Either you should specify some rounding mode (most likely round-to-even), or you should state that encoders should pick a long enough decimal representation to avoid any such issue. > We are following the Base64 standard exactly. Which base64 standard in [1]? The padding in base64 is vestigial anyway and serves no practical need by now, so there is no strong reason to use a particular standard with the required padding. It is much more important to decide what to do with incorrect padded bits. [1] https://en.wikipedia.org/wiki/Base64#Variants_summary_table https://en.wikipedia.org/wiki/Base64#Variants_summary_table > It is essential for data to be structured as a graph, it simply occurs. Serialization on most data formats has the required support for graphs awkwardly layered on top of the encoding. I don't question that graph structures often occur naturally and existing schemes are often awkward, but I think that's more of the lack of co-developed standards. I have outlined my rationale for layering in other comments. > The C# implementation defines the data model; essentially objects, array and scalars with multiple parents for nodes. The Formats section is to, as stated, facilitate interoperability. The data model should be abstract enough to be truly interoperable. JSON suffered a lot from having no defined data model to this day, even when there was a soft-of-reference implementation by Crockford. It is not too hard to define a data model in prose rather than code. > Limiting whitespace to spaces and tabs does speed processing. The ᴜᴛꜰ-8 bytes can be left in the input buffer rather than copied. All significant bytes are ᴀꜱᴄɪɪ. May have been true in the past, but it's no longer true since SIMD-based parsers. Also the very existence of escape sequence prevents the true non-destructive zero-copy (aka in-situ) parsing anyway. With such sequences, zero-copy/in-situ parsing has to be destructive to be performant and that can preclude some use cases. Allowing additional space characters is much easier than that. > There are no unmatched angle brackets (that was a typo). I meant to refer to `<<Name>...<<$>`. Multiple grouping characters can allow for simpler syntaxes.

Fannon 2y ago

Thanks for writing this up! Sounds like you really dig deeply in to this. Not entirely sure what you mean with canonical representation (I've heard this in the context of JSON-LD before, though). Can you explain what you mean here? Where do you see the problem with Graphs and Dos? A reference is just a pointer. You just have to be careful when doing recursive code. I actually like the idea to explicitly define how a reference / association is made, because otherwise people will have to re-invent ID and association concepts and there's no shared understanding. In JSON Schema, you cannot properly express an association or graph structure and people start using overloaded and not well-defined concepts like `$ref` which is a separate standard.

lifthrasiir 2y ago
Here the canonical representation refers to one single definite and unambiguous encoding for given data. This requirement is very common in cryptographic applications and also commonly demanded when the deterministic processing is desired. Technically the "canonical" and "deterministic" encoding can differ (e.g. ASN.1 CER vs. DER), but there is not much value to have two distinct encodings. On graphs: as you've said recursive code has to be careful, but the recursion itself is not very frequent in normal applications and they are more susceptible to attacks if the recursion is built into the serialization format. XML billion laughs attack is a famous demonstration for this issue. It is still worthwhile to have parallel standards to specify how recursive structures should be encoded in the basic format, and possibly to tweak the basic format to better accommodate such standards, but I believe such needs can be met without making the basic format bigger.
- GeneThomas 2y ago
  Is canonilization not irrelevant. Any format with comments is not canonical; so xᴍʟ is not, ᴊꜱᴏɴ has escapes options for string characters so it not either. Response to graph conjecture stated https://news.ycombinator.com/item?id=42072133 https://news.ycombinator.com/item?id=42072133.
  - lifthrasiir 2y ago
    Almost no serialization format is canonical by default---AFAIK bencode was the sole example that mandates the canonicalization. Instead, a canonical subset of the format is usually defined, which would of course exclude comments, unless comments themselves are considered semantic like XML. Yes, even XML has a canonical subset [1]! [1] https://www.w3.org/TR/xml-c14n/ https://www.w3.org/TR/xml-c14n/
    - GeneThomas 2y ago
      There is the concept of “well formed xᴇɴᴏɴ” as outputted by the ᴅᴏᴍ and serializer which is deterministic as will suffice for canonicalization.

zzo38computer 2y ago

> The requirement for the mandatory BOM is unacceptable for most non-Windows users I think the requirement for Unicode is bad in general, whether it uses BOM or not. > What on earth is "IEEE 64 bit double precision floating point numbers" in the context of textual format? I assume it means that numbers are expected to be IEEE 64-bit floating point numbers represented in the decimal format. > If you have to escape the base64 padding, maybe you should just drop the padding? I agree. > While technically a matter of choice, it uses a lot of unmatching angle brackets and that's just... ugly. I agree with that too. > No canonical representation. I think that a binary format would be better as a canonical representation anyways. However, the lack of indicating the data type would seem to make it difficult to know how to convert it unless you already know the schema. > Maybe you should review tons of other alternatives in this space Just a few days ago, and still working on it today, I had been making something called TER. It might also be worth to look, and I wrote another comment relating to this. Xenon does have a reference type to other nodes, which is something that other similar formats don't have, though.

lifthrasiir 2y ago
> I think the requirement for Unicode is bad in general, whether it uses BOM or not. While there exist legitimate complaints about Unicode, I believe there is no other feasible encoding than UTF-8 for textual formats now. > I assume it means that numbers are expected to be IEEE 64-bit floating point numbers represented in the decimal format. I too believe so (hence the next item), but that just doesn't make sense if you ponder. IEEE 754 doesn't define any textual format while it does have binary decimal formats. The correct wording should have been that numeric scalars follow a specific grammar to be interpreted as an IEEE 754 binary64 number in the data model. > I think that a binary format would be better as a canonical representation anyways. However, the lack of indicating the data type would seem to make it difficult to know how to convert it unless you already know the schema. That's fair. But we have already observed textual formats being... "abused" for the cryptographic purpose (e.g. JWT), so it's not too bad either to have a canonical representation or to explain why there is no canonical representation defined. > Xenon does have a reference type to other nodes, which is something that other similar formats don't have, though. There are several formats that do try to support native graph types, including YAML and Concise [1]. So that is hardly new. I think Concise actually tried very hard to make it fine! But it became quite more complex as a result. [1] https://concise-encoding.org/ https://concise-encoding.org/

eviks 2y ago

There is no mandatory bom, it's recommended, not a must

lifthrasiir 2y ago
Thank you for noticing that, my mistake. Fixed by now.

GeneThomas 2y ago

- The flexibility in name is to support the plethora of programming languages. - Commas are for readability as per English. - Three digits are recommended for the encoder. - Textual data is decoded into binary information in software so setting expectations, e.g. supporting ∞, facilitates interoperability by reducing impedance mismatches. - If the implementation uses ɪᴇᴇᴇ doubles the rounding shall be as expected. - Unicode surrogates are disallowed. - We are following the Base64 standard exactly. - It is essential for data to be structured as a graph, it simply occurs. Serialization on most data formats has the required support for graphs awkwardly layered on top of the encoding. - The C# implementation defines the data model; essentially objects, array and scalars with multiple parents for nodes. The Formats section is to, as stated, facilitate interoperability. - Limiting whitespace to spaces and tabs does speed processing. The ᴜᴛꜰ-8 bytes can be left in the input buffer rather than copied. All significant bytes are ᴀꜱᴄɪɪ. - There are no unmatched angle brackets (that was a typo). - I know the reasoning behind XML.

lifthrasiir 2y ago
> The flexibility in name is to support the plethora of programming languages. As far as aware, there is no language that allows a tab in its identifier. I'm aware of some (uncommon) languages that allow a space in identifiers though. After all, the goal of language is firstly to set what's valid or not and secondly to give a meaning to valid one. The valid set should be as maximally different from the invalid set as possible, but allowing virtually invalid characters in names is against this goal. Consider the approach taken by XML 1.1 (not 1.0) if you need an idea. > Commas are for readability as per English. A lot of English-speaking countries actually have `.` and `,` swapped: 3,141.592 vs. 3.141,592 for example. Due to this ambiguity, a comma as a grouping separator is heavily discouraged. You can instead use a underscore `_` (very common in programming languages) or a space (more preferred in human texts) without such concerns. > Three digits are recommended for the encoder. Also, some English-speaking countries use different group sizes (notably India). I'm personally fine with three-digit groupings despite of that, but that should be clearly specified at the very least. > Textual data is decoded into binary information in software so setting expectations, e.g. supporting ∞, facilitates interoperability by reducing impedance mismatches. This procedure should have been made explicit. In fact, the single worst thing you can do in serialization formats is an unclear definition of data model. > If the implementation uses ɪᴇᴇᴇ doubles the rounding shall be as expected. There is nothing like "as expected" in IEEE 754. The current rounding mode is a part of the execution state, so leaving it unspecified risks a non-deterministic interpretation even in the same execution. Either you should specify some rounding mode (most likely round-to-even), or you should state that encoders should pick a long enough decimal representation to avoid any such issue. > We are following the Base64 standard exactly. Which base64 standard in [1]? The padding in base64 is vestigial anyway and serves no practical need by now, so there is no strong reason to use a particular standard with the required padding. It is much more important to decide what to do with incorrect padded bits. [1] https://en.wikipedia.org/wiki/Base64#Variants_summary_table https://en.wikipedia.org/wiki/Base64#Variants_summary_table > It is essential for data to be structured as a graph, it simply occurs. Serialization on most data formats has the required support for graphs awkwardly layered on top of the encoding. I don't question that graph structures often occur naturally and existing schemes are often awkward, but I think that's more of the lack of co-developed standards. I have outlined my rationale for layering in other comments. > The C# implementation defines the data model; essentially objects, array and scalars with multiple parents for nodes. The Formats section is to, as stated, facilitate interoperability. The data model should be abstract enough to be truly interoperable. JSON suffered a lot from having no defined data model to this day, even when there was a soft-of-reference implementation by Crockford. It is not too hard to define a data model in prose rather than code. > Limiting whitespace to spaces and tabs does speed processing. The ᴜᴛꜰ-8 bytes can be left in the input buffer rather than copied. All significant bytes are ᴀꜱᴄɪɪ. May have been true in the past, but it's no longer true since SIMD-based parsers. Also the very existence of escape sequence prevents the true non-destructive zero-copy (aka in-situ) parsing anyway. With such sequences, zero-copy/in-situ parsing has to be destructive to be performant and that can preclude some use cases. Allowing additional space characters is much easier than that. > There are no unmatched angle brackets (that was a typo). I meant to refer to `<<Name>...<<$>`. Multiple grouping characters can allow for simpler syntaxes.