3 ms·

Doing things is the easy part. Getting anyone to notice is the hard part. I've been pushing https://concise-encoding.org/ https://concise-encoding.org/ for awh

Doing things is the easy part. Getting anyone to notice is the hard part. I've been pushing https://concise-encoding.org/ https://concise-encoding.org/ for awhile now, and it's a LOT harder to attract people than it has been with my other projects. Probably the nature of it is part of the issue. After all, who gets excited over a data format? The big security scandals recently have helped somewhat since people are starting to take security more seriously, and this format is all about security. But still, it's fighting against inertia, and the going is slow...

jmillikin 5y ago
The first question I have when I see a new generic data encoding -- a question not answered anywhere on your site's front page -- is what advantages it has over the current incumbent (Protocol Buffers). I can write text-formatted Protobuf in any text editor, convert it to efficiently encoded binary, and parse it in dozens of languages with both first- and third-party libraries. Protobuf also has a JSON encoding for use in browsers, an ecosystem of tools such as linters and formatters, and the continued support of a large commercial entity. For use cases where Protobuf is too slow, there is Cap'n Proto and Flatbuffers. Why would I switch to your format, which seems to have extremely limited language support (Go only?), is significantly more complex (look at all those built-in data types!) and is supported by a single person?
randomsilence 5y ago
Just my impressions: 1) It wasn't obvious what you are offering until I scrolled down to the examples. You stress that you offer something secure before you show what it is. 2) Why is JSON not good enough? I can also encode trees in JSON. What's the advantage of your format?
abainbridge 5y ago
I skimmed your site and didn't really get it. Maybe that's what everyone else does. In case it helps, here are the thoughts that occurred to me, in the order they occurred. 0. To me "data format" makes me think of things like PNG, or the DWARF debug symbols format, or the MPEG transport stream, not general purpose text formats like JSON and XML (I'm not a web person). 1. A secure data format. OK, weird. I thought it was always the programs/libraries that dealt with the data formats that were guilty of the security bugs, not the format itself. 2. All the bullet points under simple and efficient are already true for all the data formats I know/use/care about. So already I've dismissed your project as interesting - ie the "simple" and "efficient" are already solved and I don't believe "secure" is a real problem. But I'm aware I might be jumping to conclusions, so I keep reading. 3. I skipped straight to "Security - Protecting your data". I thought the sentence explaining why security matters was superfluous - your audience already knows why security matters. 4. "The existing ad-hoc data formats are too loosely defined to be secure, and can't be fixed because they're not versioned". OK, this looks like the meat. I click the link. 5. "There are many vectors that attackers could take advantage of when they control the data your system is receiving, the most common of which are induced data loss, field omission, key collisions, and exploitation of algorithmic complexity". If the data is from an attacker, data loss and field omission sound like good things - I don't want their data or fields because they are an attacker. 6. '"change user" command with a group of admin\U+D800'. I'm still confused. It still sounds like the "admin\U+D800" string is processed by my program. Your data format doesn't know whether that is a valid string or not. Telling your data format is no easier than telling my program. Oh! Maybe it is. Because you only need to specify it in the data format, not in every program/library that implements the format. Is this the point of the system?
- kstenerud 5y ago
  Hmm ok I'm definitely not communicating properly what it is then... I'm not sure what nomenclature I could use besides "data format" to describe what this is, any more than one could describe JSON as anything other than a data format... The security aspect is where I'm having the most trouble communicating, because it's just not a well known issue. The problem comes from the different ways that codecs deal with the data they receive (for example Java Codec A vs Golang codec B vs Python codec C vs Python codec D, etc). If the spec isn't strict enough, a conformant codec could behave in ways that are insecure when coupled with other systems. Security really does start at the specification, in that the spec has to define what a codec can and cannot do. For example: "A codec MUST reject invalid characters. It MUST NOT truncate them or replace them with the Unicode replacement character". A rule like this would prevent the admin\U+D800 vulnerability because a codec is not allowed to behave this way (truncating or replacing). With JSON, this isn't specified, so systems where the same data is parsed by multiple subsystems are vulnerable. Specifically regarding the admin\U+D800 problem: Any sufficiently complex system will have multiple subsytems. So say you have a user-facing system that accepts a "new account" request in JSON. The attacker passes in "admin\U+D800" as the group (meaning the 5 character "admin" plus the unicode character D800, which is invalid). Your account creator's JSON decoder ignores bad characters, so when it checks the group vs its blacklist, it finds that "admin\U+D800" != "admin". Validation passed, it sends the new user info to the account storage subsystem (as a JSON message). Your account storage subsystem's JSON decoder silently strips bad characters, so your account subsystem only sees "admin", not "admin\U+D800". Now this new user is an admin, thanks to a privilege escalation vulnerability brought on by a lax JSON spec (all codecs involved were conformant). If the JSON spec had specified that "A codec MUST reject invalid characters. It MUST NOT truncate them or replace them with the Unicode replacement character", this kind of vulnerability could not happen with a conformant codec. Thanks for the feedback!
  - abainbridge 5y ago
    > I'm not sure what nomenclature I could use besides "data format" Yeah, I don't know either. I see that the Wikipedia page for JSON describes it as a "data interchange format". Maybe that helps. I'd probably side-step the problem by having your home page start with "Like Protocol Buffers but more secure", followed immediately by the example you just gave me.
rixed 5y ago
I'm very interested in this topic, having authored several such tools. Some remarks after a first glance at the README, it's very possible I missed some obvious answers and I apologize in advance: - You ask for a review on your site but that just links to the github project page. Have you done a show-hn by the way? - I believe schema representation and encoding should be two separate things; In other words, a good data manipulation tool should support several schema formats and several encoding formats - I personalty prefer the types to be a little less opinionated (for instance, there are many legitimate definitions for a "date" so I do not want the data layer to favor one over the others, although I reckon having some support for dates is convenient) - having a type for markup is particularly suspicious. - You do not mention extensibility in the features list, which is a very important topic when a data format is used for RPCs in a distributed system, for instance - Are strings UTF-8? UTF-8 only? - Why do you need specific types for edges and nodes? - No type for records/structures apart from the top-level one?? - No sum types? How do you handle nulls? - Lists are of heterogeneous types? Can we have a type for "lists of some type" - How come comments ended up being types? - The front-page should display a comparison in encoding and decoding speed and encoded size compared to the major contenders (which are protobuf and json, I guess) - It should also display the corresponding go code for each presented schema examples
- kstenerud 5y ago
  Thanks! I'll try to answer as best I can. > - You ask for a review on your site but that just links to the github project page. Have you done a show-hn by the way? I'm asking for a review of the specifications, which is what the main github page is for. There's also a reference implementation, but that's in a seprate repo. I've done a show HN in the past. It might be time for a fresh one since it's been a couple of years. > - I believe schema representation and encoding should be two separate things; In other words, a good data manipulation tool should support several schema formats and several encoding formats The schema representation will not be tied to the encoding. I'm leaning towards maybe https://cuelang.org/ https://cuelang.org/ as the officially endorsed schema format, but there's nothing stopping someone from using something else. > - I personalty prefer the types to be a little less opinionated (for instance, there are many legitimate definitions for a "date" so I do not want the data layer to favor one over the others, although I reckon having some support for dates is convenient) I put a date in there because it's such a fundamental data type that everyone wants one, and if it's not specified in an opinionated manner, everyone comes up with their own (likely incompatible) interpretation, which is what I want to avoid. The idea is that a user of the format shouldn't have to worry about HOW to encode their data unless they're using exotic types. > - having a type for markup is particularly suspicious. I'm still not 100% decided on whether markup will stay or go. I've already tried and dropped dozens of other types already so it might go before I release... > - Are strings UTF-8? UTF-8 only? Yes, UTF-8 only. > - Why do you need specific types for edges and nodes? Because nodes can't represent weighted or other complex or non-directed graphs, and edges are by their nature too bulky for representing trees. > - No type for records/structures apart from the top-level one?? Not sure what you mean? structures are represented using the map type and a schema. > - No sum types? How do you handle nulls? Null is allowed. You can pass {"result" = 500} or {"result" = null} or {"result" = [1 2 3 4]} if you want. Sum types would be enforced by the schema. > - Lists are of heterogeneous types? Can we have a type for "lists of some type" This would be the job of the schema. There are typed arrays for primitive types like int and float since that's a fairly common use case and would be too bulky otherwise. > - How come comments ended up being types? Comments are "types" in the sense of what types of data a document can physically contain. They aren't "types" in the sense of actual data to be passed to applications (although an application could listen for it - for example a CTE reformatter or sanitizer). > - The front-page should display a comparison in encoding and decoding speed and encoded size compared to the major contenders (which are protobuf and json, I guess) Encoding and decoding speed would depend on the implementation. This is just the specification. > - It should also display the corresponding go code for each presented schema examples Not sure how useful that would be since every single example would be cbe.Marshal(myobject, stream) and myobject = ce.Unmarshal(mytype, stream)

jmillikin 5y ago

The first question I have when I see a new generic data encoding -- a question not answered anywhere on your site's front page -- is what advantages it has over the current incumbent (Protocol Buffers). I can write text-formatted Protobuf in any text editor, convert it to efficiently encoded binary, and parse it in dozens of languages with both first- and third-party libraries. Protobuf also has a JSON encoding for use in browsers, an ecosystem of tools such as linters and formatters, and the continued support of a large commercial entity. For use cases where Protobuf is too slow, there is Cap'n Proto and Flatbuffers. Why would I switch to your format, which seems to have extremely limited language support (Go only?), is significantly more complex (look at all those built-in data types!) and is supported by a single person?

randomsilence 5y ago

Just my impressions: 1) It wasn't obvious what you are offering until I scrolled down to the examples. You stress that you offer something secure before you show what it is. 2) Why is JSON not good enough? I can also encode trees in JSON. What's the advantage of your format?

abainbridge 5y ago

I skimmed your site and didn't really get it. Maybe that's what everyone else does. In case it helps, here are the thoughts that occurred to me, in the order they occurred. 0. To me "data format" makes me think of things like PNG, or the DWARF debug symbols format, or the MPEG transport stream, not general purpose text formats like JSON and XML (I'm not a web person). 1. A secure data format. OK, weird. I thought it was always the programs/libraries that dealt with the data formats that were guilty of the security bugs, not the format itself. 2. All the bullet points under simple and efficient are already true for all the data formats I know/use/care about. So already I've dismissed your project as interesting - ie the "simple" and "efficient" are already solved and I don't believe "secure" is a real problem. But I'm aware I might be jumping to conclusions, so I keep reading. 3. I skipped straight to "Security - Protecting your data". I thought the sentence explaining why security matters was superfluous - your audience already knows why security matters. 4. "The existing ad-hoc data formats are too loosely defined to be secure, and can't be fixed because they're not versioned". OK, this looks like the meat. I click the link. 5. "There are many vectors that attackers could take advantage of when they control the data your system is receiving, the most common of which are induced data loss, field omission, key collisions, and exploitation of algorithmic complexity". If the data is from an attacker, data loss and field omission sound like good things - I don't want their data or fields because they are an attacker. 6. '"change user" command with a group of admin\U+D800'. I'm still confused. It still sounds like the "admin\U+D800" string is processed by my program. Your data format doesn't know whether that is a valid string or not. Telling your data format is no easier than telling my program. Oh! Maybe it is. Because you only need to specify it in the data format, not in every program/library that implements the format. Is this the point of the system?

kstenerud 5y ago
Hmm ok I'm definitely not communicating properly what it is then... I'm not sure what nomenclature I could use besides "data format" to describe what this is, any more than one could describe JSON as anything other than a data format... The security aspect is where I'm having the most trouble communicating, because it's just not a well known issue. The problem comes from the different ways that codecs deal with the data they receive (for example Java Codec A vs Golang codec B vs Python codec C vs Python codec D, etc). If the spec isn't strict enough, a conformant codec could behave in ways that are insecure when coupled with other systems. Security really does start at the specification, in that the spec has to define what a codec can and cannot do. For example: "A codec MUST reject invalid characters. It MUST NOT truncate them or replace them with the Unicode replacement character". A rule like this would prevent the admin\U+D800 vulnerability because a codec is not allowed to behave this way (truncating or replacing). With JSON, this isn't specified, so systems where the same data is parsed by multiple subsystems are vulnerable. Specifically regarding the admin\U+D800 problem: Any sufficiently complex system will have multiple subsytems. So say you have a user-facing system that accepts a "new account" request in JSON. The attacker passes in "admin\U+D800" as the group (meaning the 5 character "admin" plus the unicode character D800, which is invalid). Your account creator's JSON decoder ignores bad characters, so when it checks the group vs its blacklist, it finds that "admin\U+D800" != "admin". Validation passed, it sends the new user info to the account storage subsystem (as a JSON message). Your account storage subsystem's JSON decoder silently strips bad characters, so your account subsystem only sees "admin", not "admin\U+D800". Now this new user is an admin, thanks to a privilege escalation vulnerability brought on by a lax JSON spec (all codecs involved were conformant). If the JSON spec had specified that "A codec MUST reject invalid characters. It MUST NOT truncate them or replace them with the Unicode replacement character", this kind of vulnerability could not happen with a conformant codec. Thanks for the feedback!
- abainbridge 5y ago
  > I'm not sure what nomenclature I could use besides "data format" Yeah, I don't know either. I see that the Wikipedia page for JSON describes it as a "data interchange format". Maybe that helps. I'd probably side-step the problem by having your home page start with "Like Protocol Buffers but more secure", followed immediately by the example you just gave me.

rixed 5y ago

I'm very interested in this topic, having authored several such tools. Some remarks after a first glance at the README, it's very possible I missed some obvious answers and I apologize in advance: - You ask for a review on your site but that just links to the github project page. Have you done a show-hn by the way? - I believe schema representation and encoding should be two separate things; In other words, a good data manipulation tool should support several schema formats and several encoding formats - I personalty prefer the types to be a little less opinionated (for instance, there are many legitimate definitions for a "date" so I do not want the data layer to favor one over the others, although I reckon having some support for dates is convenient) - having a type for markup is particularly suspicious. - You do not mention extensibility in the features list, which is a very important topic when a data format is used for RPCs in a distributed system, for instance - Are strings UTF-8? UTF-8 only? - Why do you need specific types for edges and nodes? - No type for records/structures apart from the top-level one?? - No sum types? How do you handle nulls? - Lists are of heterogeneous types? Can we have a type for "lists of some type" - How come comments ended up being types? - The front-page should display a comparison in encoding and decoding speed and encoded size compared to the major contenders (which are protobuf and json, I guess) - It should also display the corresponding go code for each presented schema examples

kstenerud 5y ago
Thanks! I'll try to answer as best I can. > - You ask for a review on your site but that just links to the github project page. Have you done a show-hn by the way? I'm asking for a review of the specifications, which is what the main github page is for. There's also a reference implementation, but that's in a seprate repo. I've done a show HN in the past. It might be time for a fresh one since it's been a couple of years. > - I believe schema representation and encoding should be two separate things; In other words, a good data manipulation tool should support several schema formats and several encoding formats The schema representation will not be tied to the encoding. I'm leaning towards maybe https://cuelang.org/ https://cuelang.org/ as the officially endorsed schema format, but there's nothing stopping someone from using something else. > - I personalty prefer the types to be a little less opinionated (for instance, there are many legitimate definitions for a "date" so I do not want the data layer to favor one over the others, although I reckon having some support for dates is convenient) I put a date in there because it's such a fundamental data type that everyone wants one, and if it's not specified in an opinionated manner, everyone comes up with their own (likely incompatible) interpretation, which is what I want to avoid. The idea is that a user of the format shouldn't have to worry about HOW to encode their data unless they're using exotic types. > - having a type for markup is particularly suspicious. I'm still not 100% decided on whether markup will stay or go. I've already tried and dropped dozens of other types already so it might go before I release... > - Are strings UTF-8? UTF-8 only? Yes, UTF-8 only. > - Why do you need specific types for edges and nodes? Because nodes can't represent weighted or other complex or non-directed graphs, and edges are by their nature too bulky for representing trees. > - No type for records/structures apart from the top-level one?? Not sure what you mean? structures are represented using the map type and a schema. > - No sum types? How do you handle nulls? Null is allowed. You can pass {"result" = 500} or {"result" = null} or {"result" = [1 2 3 4]} if you want. Sum types would be enforced by the schema. > - Lists are of heterogeneous types? Can we have a type for "lists of some type" This would be the job of the schema. There are typed arrays for primitive types like int and float since that's a fairly common use case and would be too bulky otherwise. > - How come comments ended up being types? Comments are "types" in the sense of what types of data a document can physically contain. They aren't "types" in the sense of actual data to be passed to applications (although an application could listen for it - for example a CTE reformatter or sanitizer). > - The front-page should display a comparison in encoding and decoding speed and encoded size compared to the major contenders (which are protobuf and json, I guess) Encoding and decoding speed would depend on the implementation. This is just the specification. > - It should also display the corresponding go code for each presented schema examples Not sure how useful that would be since every single example would be cbe.Marshal(myobject, stream) and myobject = ce.Unmarshal(mytype, stream)