3 ms·

This is the problem space I'm targeting with https://concise-encoding.org/ https://concise-encoding.org/ * Text AND binary so that humans can edit easily, and

This is the problem space I'm targeting with https://concise-encoding.org/ https://concise-encoding.org/ * Text AND binary so that humans can edit easily, and machines can transmit energy and bandwidth efficiently. * Carefully designed spec to avoid ambiguities (and their security implications). * Strong type support so you're not using all kinds of incompatible hacks to serialize your data. * Versioned, because there's no such thing as the perfect format. * Also, the website is 32k bytes ;-)

anticristi 6y ago
Nice! I like some concepts that this format proposes, but the `@` and `|` modifier feels a bit too "loaded".
- kstenerud 6y ago
  It's a compromise; there are only so many letters, numbers, and symbols available in a single keystroke on all keyboards, and I don't want there to be any ambiguity with numbers and unquoted strings (e.g. interpreting the unquoted string value true as the boolean value true). So everything else needs some kind of initiator and/or container syntax to logically separate it from the other objects when interpreted by a human or machine.
chousuke 6y ago
I'm skimming through the human readable spec, and it seems decent, but I noticed the spec allows unquoted strings. What's the reasoning for this? In my experience unquoted strings cause nothing but trouble, and are confusing to humans who may interpret them as keywords. Any reason for not using RFC2119 keywords in the spec? Using them should make the spec easier to read.
- kstenerud 6y ago
  > I noticed the spec allows unquoted strings. What's the reasoning for this? In my experience unquoted strings cause nothing but trouble, and are confusing to humans who may interpret them as keywords. Unquoted strings are much nicer for humans to work with. All special keywords and object encodings are prefixed with sigils (@, &, $, #, etc), so any bare text starting with a letter is either a string or an invalid document, and any bare text starting with a numeral is either a number or an invalid document. > Any reason for not using RFC2119 keywords in the spec? Using them should make the spec easier to read. I use a superset of those keywords to give more precision in meaning: https://github.com/kstenerud/concise-encoding/blob/master/ce-structure.md#terms https://github.com/kstenerud/concise-encoding/blob/master/ce...
  - chousuke 6y ago
    If strings are always unambiquously detectable, why allow quoting them at all? Having two representations for the same data means you can't normalize a document unambiguously. I can understand having barewords seems cleaner for things like map keys, but I am not convinced that it's a worthwhile tradeoff. An important feature of RFC2119 keywords is that they're always capitalized (ie. the keyword is "MUST", not "Must", or "must"). This makes requirements and recommendations stand out amid explanatory text, improving legibility. For example, RFC2119 itself uses MUST and must with different meanings.
    - kstenerud 6y ago
      > If strings are always unambiquously detectable, why allow quoting them at all? Because strings can contain whitespace and other structural characters that would confuse a parser. > Having two representations for the same data means you can't normalize a document unambiguously. The document will always be normalized unambiguously in binary format. The text format is a bit more lenient because humans are involved. The idea is that the binary format is the source of truth, and is what is used in 90% of situations. The text format is only needed as a conduit for human input, or as a human readable representation of the binary data when you need to see what's going on. > An important feature of RFC2119 keywords is that they're always capitalized (ie. the keyword is "MUST", not "Must", or "must"). Hmm good point. I'll add that.
    - kstenerud 6y ago
      Update: I'm removing unquoted strings. Thanks for the critique!
yyyk 6y ago
+ Has binary format. + Avoids ambiguities. - The format seems to feel the need to support everything, including things I am not sure are actual usecases (what's the point of Markup element for example? What does Metadata save us compared to just including it in document, given that parsers must parse it anyway?). This must make implementation most complex and costly, and makes reading the text format more difficult. - Not a fan of octal notation. At 3am not sure I can't confuse 0 and o given certain fonts. Does anyone even use it these days? - Unquoted string were discussed in the thread, I'd like to point out that it's very easy to make an unquoted string not "text-safe" (according to the spec) without noticing it, at which point document is invalid. Just add white-space (maybe a user pasted a string from somewhere without noticing whitespace at the end or forgot the rules), a dot, an exclamation or a question mark. Having surprises like that is IMHO worse than a consistent quoting method. Basically all the things I don't like are about the format supporting a bit too much. YAML 1.1 should teach us more is sometimes less.
- kstenerud 6y ago
  Alright that's two votes against unquoted strings so far (plus my wife agrees so that's three against!) I put in octal because it was trivial to implement after the others. The canonical format when it's stored or being sent is binary, and a decoder shouldn't be presenting integers in octal (that would just be weird). But a human might want octal when inputting data that will be converted to the binary format. Markup is for presentation data, UI layouts, etc, but with full type support rather than all the hacky XML+whatever solutions that many UI toolkits are adopting. Also, having presentation data in binary form is nice to have.
  - yyyk 6y ago
    Well, unquoted strings work when a format is built for that. If the default was "it's text unless we see the special sequences" it would be better for unquoted strings. But even then there are too many special characters in this format IMHO. I saw there's a 'Media' type in the spec. It's seems the type is actually for serializing files. But there's no "name" (or we can call it "description") field. Of course we could accomplish this with a separate field - but than again the entire type's functionality could be accomplished with a u8x array and a string field. So if you're specifying this type at all, might as well add a name field to make it useful.

anticristi 6y ago

Nice! I like some concepts that this format proposes, but the `@` and `|` modifier feels a bit too "loaded".

kstenerud 6y ago
It's a compromise; there are only so many letters, numbers, and symbols available in a single keystroke on all keyboards, and I don't want there to be any ambiguity with numbers and unquoted strings (e.g. interpreting the unquoted string value true as the boolean value true). So everything else needs some kind of initiator and/or container syntax to logically separate it from the other objects when interpreted by a human or machine.

chousuke 6y ago

I'm skimming through the human readable spec, and it seems decent, but I noticed the spec allows unquoted strings. What's the reasoning for this? In my experience unquoted strings cause nothing but trouble, and are confusing to humans who may interpret them as keywords. Any reason for not using RFC2119 keywords in the spec? Using them should make the spec easier to read.

kstenerud 6y ago
> I noticed the spec allows unquoted strings. What's the reasoning for this? In my experience unquoted strings cause nothing but trouble, and are confusing to humans who may interpret them as keywords. Unquoted strings are much nicer for humans to work with. All special keywords and object encodings are prefixed with sigils (@, &, $, #, etc), so any bare text starting with a letter is either a string or an invalid document, and any bare text starting with a numeral is either a number or an invalid document. > Any reason for not using RFC2119 keywords in the spec? Using them should make the spec easier to read. I use a superset of those keywords to give more precision in meaning: https://github.com/kstenerud/concise-encoding/blob/master/ce-structure.md#terms https://github.com/kstenerud/concise-encoding/blob/master/ce...
- chousuke 6y ago
  If strings are always unambiquously detectable, why allow quoting them at all? Having two representations for the same data means you can't normalize a document unambiguously. I can understand having barewords seems cleaner for things like map keys, but I am not convinced that it's a worthwhile tradeoff. An important feature of RFC2119 keywords is that they're always capitalized (ie. the keyword is "MUST", not "Must", or "must"). This makes requirements and recommendations stand out amid explanatory text, improving legibility. For example, RFC2119 itself uses MUST and must with different meanings.
  - kstenerud 6y ago
    > If strings are always unambiquously detectable, why allow quoting them at all? Because strings can contain whitespace and other structural characters that would confuse a parser. > Having two representations for the same data means you can't normalize a document unambiguously. The document will always be normalized unambiguously in binary format. The text format is a bit more lenient because humans are involved. The idea is that the binary format is the source of truth, and is what is used in 90% of situations. The text format is only needed as a conduit for human input, or as a human readable representation of the binary data when you need to see what's going on. > An important feature of RFC2119 keywords is that they're always capitalized (ie. the keyword is "MUST", not "Must", or "must"). Hmm good point. I'll add that.
  - kstenerud 6y ago
    Update: I'm removing unquoted strings. Thanks for the critique!

yyyk 6y ago

+ Has binary format. + Avoids ambiguities. - The format seems to feel the need to support everything, including things I am not sure are actual usecases (what's the point of Markup element for example? What does Metadata save us compared to just including it in document, given that parsers must parse it anyway?). This must make implementation most complex and costly, and makes reading the text format more difficult. - Not a fan of octal notation. At 3am not sure I can't confuse 0 and o given certain fonts. Does anyone even use it these days? - Unquoted string were discussed in the thread, I'd like to point out that it's very easy to make an unquoted string not "text-safe" (according to the spec) without noticing it, at which point document is invalid. Just add white-space (maybe a user pasted a string from somewhere without noticing whitespace at the end or forgot the rules), a dot, an exclamation or a question mark. Having surprises like that is IMHO worse than a consistent quoting method. Basically all the things I don't like are about the format supporting a bit too much. YAML 1.1 should teach us more is sometimes less.

kstenerud 6y ago
Alright that's two votes against unquoted strings so far (plus my wife agrees so that's three against!) I put in octal because it was trivial to implement after the others. The canonical format when it's stored or being sent is binary, and a decoder shouldn't be presenting integers in octal (that would just be weird). But a human might want octal when inputting data that will be converted to the binary format. Markup is for presentation data, UI layouts, etc, but with full type support rather than all the hacky XML+whatever solutions that many UI toolkits are adopting. Also, having presentation data in binary form is nice to have.
- yyyk 6y ago
  Well, unquoted strings work when a format is built for that. If the default was "it's text unless we see the special sequences" it would be better for unquoted strings. But even then there are too many special characters in this format IMHO. I saw there's a 'Media' type in the spec. It's seems the type is actually for serializing files. But there's no "name" (or we can call it "description") field. Of course we could accomplish this with a separate field - but than again the entire type's functionality could be accomplished with a u8x array and a string field. So if you're specifying this type at all, might as well add a name field to make it useful.