4 ms·

I really wonder why is it so common for people to default to JSON even when designing complex cryptographic protocols like this. I was skimming the spec for th

by treyd 3y ago

I really wonder why is it so common for people to default to JSON even when designing complex cryptographic protocols like this. I was skimming the spec for the script interpreter, which is something you'd expect would be focused on compactness, but it seems like it's designed entirely around JSON semantics and concepts, which means any implementation necessarily carries along this complexity. This isn't to say that JSON is bad, but to put it at the absolute core of the data model seems problematic. This problem happened with Matrix when they started having to layer JSON around base64-encoded-encrypted-JSON and generally the looseness of the format infecting the entire data model of the protocol. Like the SPMessage format here too, uses JSON.stringify to indicate structure encoding, which necessarily implies that any implementation of the protocol is bringing along needing to handle JSON semantics. JSON does not seem to be well-suited here as it doesn't composing cleanly in the way a data structure defined in terms of bytes would.

GoblinSlayer 3y ago
Well, it's a webapp, so everything there is designed with web technologies.
- eptcyka 3y ago
  Yes, for all the great things browsers do, it sure is a shame that the only data format they can handle is JSON.
- treyd 3y ago
  But just because it's a webapp doesn't mean you have to use JSON for everything, it's the other way around, using JSON for everything means that the only place the data model feels natural is the web. Which is strange because JavaScript has had the Uint8Array that makes manipulating arbitrary byte structures easy for a while, it doesn't feel any less natural than other dynamic languages in that regard. Defining a "signed message with a header" structure in terms of bytes like I described above seems like a no-brainer. If you want the payload data of your protocol to use JSON then that's fine, but that shouldn't dictate the cryptographic layers underneath.
ygouzerh 3y ago
Interesting remark! Genuine question here: by what would you replace JSON if you can re-architecture it?
- treyd 3y ago
  I wish there was some language-agnostic tool for defining byte-oriented data strucrures that could emit code to manipulate them in different languages. Protobuf and similar tools exist but they still have their own weirdnesses and aren't entirely zero-overhead. I've seen plenty of pseudo-schemas used in standards documents, in particular the MLS spec which I think was finalized recently.
  - lsaferite 3y ago
    Something similar to this? https://bnfc.digitalgrammars.com/ https://bnfc.digitalgrammars.com/ I agree, having/using a well-defined standard that can have automated tooling to general language-specific implementations would be ideal.
- crest 3y ago
  How about bencode? c-sexp? netstrings? The important features I would demand are: able to represent the types you need (e.g. integers instead of strings of digits or floats) and designed to have a canonical presentation from the beginning. There are so many bugs around ASN.1/XML/JSON parsing and "normalisation" that it's simply inexcusable to repeat this mistake. If the designers didn't come up with a really convincing explanation for selecting the wrong serialisation format for a cryptographic protocol how likely is it they read and understood the prior art in other fields like cryptography? Just because a canonical JSON (de-)serialiser is only a npm/pip/gem install is no excuse to burden a specification with the bloat and attack surface. Even fix sized, flat, packed structs are a better solution. K.I.S.S.!
neilalexander 3y ago
> This problem happened with Matrix when they started having to layer JSON around base64-encoded-encrypted-JSON and generally the looseness of the format infecting the entire data model of the protocol. Indeed, this has already created considerable problems for the request and event signing and getting different Matrix implementations written in different languages to interoperate correctly. I now firmly believe that cryptographically-signed JSON is always a disaster waiting to happen, no matter how hard you try to canonicalise it.
gslepak 3y ago
As others have said, there are unfortunately not that many good alternatives for serializing data in web browsers (that are both performant and don't require bulking your app with the serialization implementation). We've done our best to make sure that the way we're serializing objects can either be done in a consistent manner across implementations or, if done inconsistently, it doesn't matter.
- treyd 3y ago
  I don't really buy the argument that additional serialization logic adds bloat to the application when you compare it to the many thousands of lines of code in modern JS frameworks for manipulating DOM elements. The consistency is really an important part here, are you spending less time doing the testing and bugfixing to ensure that JSON ambiguity is not possible / is harmless than you would writing serialization for cryptographic types in unambiguous byte-oriented structures?
  - gslepak 3y ago
    SPMessage serializations are never regenerated after they leave the client, so it's not an issue as far as I'm aware. JSON is supported universally in all languages and all platforms, and natively in the browser. Is there a specific serialization format you're advocating for?
    - Groxx 3y ago
      I suspect it's more that JSON is (rather explicitly) not designed to be stable across all implementations, so there are subtle differences. Numbers don't have a specified precision, strings don't have a specified encoding, maps have no canonical order nor do they require unique keys[1], etc. Nailing all that down so you can e.g. have a truly deterministic signing process often requires custom encoders/decoders and is rather fraught with problems even then, at which point why even bother with JSON. Use something designed to be consistent. [1]: https://stackoverflow.com/a/23195243 https://stackoverflow.com/a/23195243
      - 3y ago

by treyd 3y ago

GoblinSlayer 3y ago

Well, it's a webapp, so everything there is designed with web technologies.

eptcyka 3y ago
Yes, for all the great things browsers do, it sure is a shame that the only data format they can handle is JSON.
treyd 3y ago
But just because it's a webapp doesn't mean you have to use JSON for everything, it's the other way around, using JSON for everything means that the only place the data model feels natural is the web. Which is strange because JavaScript has had the Uint8Array that makes manipulating arbitrary byte structures easy for a while, it doesn't feel any less natural than other dynamic languages in that regard. Defining a "signed message with a header" structure in terms of bytes like I described above seems like a no-brainer. If you want the payload data of your protocol to use JSON then that's fine, but that shouldn't dictate the cryptographic layers underneath.

ygouzerh 3y ago

Interesting remark! Genuine question here: by what would you replace JSON if you can re-architecture it?

treyd 3y ago
I wish there was some language-agnostic tool for defining byte-oriented data strucrures that could emit code to manipulate them in different languages. Protobuf and similar tools exist but they still have their own weirdnesses and aren't entirely zero-overhead. I've seen plenty of pseudo-schemas used in standards documents, in particular the MLS spec which I think was finalized recently.
- lsaferite 3y ago
  Something similar to this? https://bnfc.digitalgrammars.com/ https://bnfc.digitalgrammars.com/ I agree, having/using a well-defined standard that can have automated tooling to general language-specific implementations would be ideal.
crest 3y ago
How about bencode? c-sexp? netstrings? The important features I would demand are: able to represent the types you need (e.g. integers instead of strings of digits or floats) and designed to have a canonical presentation from the beginning. There are so many bugs around ASN.1/XML/JSON parsing and "normalisation" that it's simply inexcusable to repeat this mistake. If the designers didn't come up with a really convincing explanation for selecting the wrong serialisation format for a cryptographic protocol how likely is it they read and understood the prior art in other fields like cryptography? Just because a canonical JSON (de-)serialiser is only a npm/pip/gem install is no excuse to burden a specification with the bloat and attack surface. Even fix sized, flat, packed structs are a better solution. K.I.S.S.!

neilalexander 3y ago

> This problem happened with Matrix when they started having to layer JSON around base64-encoded-encrypted-JSON and generally the looseness of the format infecting the entire data model of the protocol. Indeed, this has already created considerable problems for the request and event signing and getting different Matrix implementations written in different languages to interoperate correctly. I now firmly believe that cryptographically-signed JSON is always a disaster waiting to happen, no matter how hard you try to canonicalise it.

gslepak 3y ago

As others have said, there are unfortunately not that many good alternatives for serializing data in web browsers (that are both performant and don't require bulking your app with the serialization implementation). We've done our best to make sure that the way we're serializing objects can either be done in a consistent manner across implementations or, if done inconsistently, it doesn't matter.

treyd 3y ago
I don't really buy the argument that additional serialization logic adds bloat to the application when you compare it to the many thousands of lines of code in modern JS frameworks for manipulating DOM elements. The consistency is really an important part here, are you spending less time doing the testing and bugfixing to ensure that JSON ambiguity is not possible / is harmless than you would writing serialization for cryptographic types in unambiguous byte-oriented structures?
- gslepak 3y ago
  SPMessage serializations are never regenerated after they leave the client, so it's not an issue as far as I'm aware. JSON is supported universally in all languages and all platforms, and natively in the browser. Is there a specific serialization format you're advocating for?
  - Groxx 3y ago
    I suspect it's more that JSON is (rather explicitly) not designed to be stable across all implementations, so there are subtle differences. Numbers don't have a specified precision, strings don't have a specified encoding, maps have no canonical order nor do they require unique keys[1], etc. Nailing all that down so you can e.g. have a truly deterministic signing process often requires custom encoders/decoders and is rather fraught with problems even then, at which point why even bother with JSON. Use something designed to be consistent. [1]: https://stackoverflow.com/a/23195243 https://stackoverflow.com/a/23195243
    - 3y ago