3 ms·

SPMessage serializations are never regenerated after they leave the client, so it's not an issue as far as I'm aware. JSON is supported universally in all langu

SPMessage serializations are never regenerated after they leave the client, so it's not an issue as far as I'm aware. JSON is supported universally in all languages and all platforms, and natively in the browser. Is there a specific serialization format you're advocating for?

Groxx 3y ago
I suspect it's more that JSON is (rather explicitly) not designed to be stable across all implementations, so there are subtle differences. Numbers don't have a specified precision, strings don't have a specified encoding, maps have no canonical order nor do they require unique keys[1], etc. Nailing all that down so you can e.g. have a truly deterministic signing process often requires custom encoders/decoders and is rather fraught with problems even then, at which point why even bother with JSON. Use something designed to be consistent. [1]: https://stackoverflow.com/a/23195243 https://stackoverflow.com/a/23195243
- gslepak 3y ago
  We're aware of these issues with JSON, but they do not affect the integrity of Shelter Protocol, and by using JSON we get a lot of advantages in terms of speed and developer usability (support for tools like `jq`, schema-less serialization, widespread language support, easier debugging, etc.).
  - rpdillon 3y ago
    I was really excited when I tried out the scuttlebutt protocol a few years back. Distributed, peer-to-peer, offline-enabled social networking seemed neat. But a healthy ecosystem of clients never emerged. When I looked into why, I discovered it relied on node's particular method of JSON serialization[0]. Are you confident that the shelter protocol has avoided this pitfall? [0]: https://news.ycombinator.com/item?id=22910752 https://news.ycombinator.com/item?id=22910752
    - gslepak 3y ago
      Yes, I am confident that we have avoided that specific pitfall because as I mentioned we do not ever re-serialize the JSON. However, thinking on it more, there is one thing Groxx mentioned that has given me pause, and that is "Numbers don't have a specified precision". This could be an issue in the future if, for example, computers suddenly use 128-bit numbers and JSON implementations start to actually use such numbers. It's an edge case that might result in incompatibility with clients that use 64-bit numbers. So we'll consider the possibility of changing the serialization format after researching this more.
      - gslepak 3y ago
        > So we'll consider the possibility of changing the serialization format after researching this more. I asked Brendan Eich if JavaScript will ever change the way it represents floating point numbers (from 64-bit IEEE 754 values), and he said no ("don't break the Web"). And to represent larger bit values in JSON you're supposed to use a JSON string and schema to specify the type. I'm not sure exactly what he means by that, but I assume it means something like using JSON keys like "<BigInt>mykey" and string values instead of number values. Then manually parsing the string into whatever native 128-bit type or whatever representation you have. So - JSON still seems to be the favorite here, but I totally get the anxiety around it because of how under-specified it is. If you avoid re-serialization nothing should break.
        Groxx 3y ago
        Avoiding re-serialization does avoid a ton of issues, yea. Though I would suggest building in validation for e.g. no duplicate keys immediately, because unexpected duplicates are a common source of bugs and security exploits (e.g. it's a fairly common issue with http headers when requests cross multiple systems).
        gslepak 3y ago
        After evaluating the various options, we seem to be settling on the idea that the protocol will specify that for duplicate keys, the handling must be the same as in browsers: meaning, the last key is the one that's used.

Groxx 3y ago

I suspect it's more that JSON is (rather explicitly) not designed to be stable across all implementations, so there are subtle differences. Numbers don't have a specified precision, strings don't have a specified encoding, maps have no canonical order nor do they require unique keys[1], etc. Nailing all that down so you can e.g. have a truly deterministic signing process often requires custom encoders/decoders and is rather fraught with problems even then, at which point why even bother with JSON. Use something designed to be consistent. [1]: https://stackoverflow.com/a/23195243 https://stackoverflow.com/a/23195243

gslepak 3y ago
We're aware of these issues with JSON, but they do not affect the integrity of Shelter Protocol, and by using JSON we get a lot of advantages in terms of speed and developer usability (support for tools like `jq`, schema-less serialization, widespread language support, easier debugging, etc.).
- rpdillon 3y ago
  I was really excited when I tried out the scuttlebutt protocol a few years back. Distributed, peer-to-peer, offline-enabled social networking seemed neat. But a healthy ecosystem of clients never emerged. When I looked into why, I discovered it relied on node's particular method of JSON serialization[0]. Are you confident that the shelter protocol has avoided this pitfall? [0]: https://news.ycombinator.com/item?id=22910752 https://news.ycombinator.com/item?id=22910752
  - gslepak 3y ago
    Yes, I am confident that we have avoided that specific pitfall because as I mentioned we do not ever re-serialize the JSON. However, thinking on it more, there is one thing Groxx mentioned that has given me pause, and that is "Numbers don't have a specified precision". This could be an issue in the future if, for example, computers suddenly use 128-bit numbers and JSON implementations start to actually use such numbers. It's an edge case that might result in incompatibility with clients that use 64-bit numbers. So we'll consider the possibility of changing the serialization format after researching this more.
    - gslepak 3y ago
      > So we'll consider the possibility of changing the serialization format after researching this more. I asked Brendan Eich if JavaScript will ever change the way it represents floating point numbers (from 64-bit IEEE 754 values), and he said no ("don't break the Web"). And to represent larger bit values in JSON you're supposed to use a JSON string and schema to specify the type. I'm not sure exactly what he means by that, but I assume it means something like using JSON keys like "<BigInt>mykey" and string values instead of number values. Then manually parsing the string into whatever native 128-bit type or whatever representation you have. So - JSON still seems to be the favorite here, but I totally get the anxiety around it because of how under-specified it is. If you avoid re-serialization nothing should break.
      - Groxx 3y ago
        Avoiding re-serialization does avoid a ton of issues, yea. Though I would suggest building in validation for e.g. no duplicate keys immediately, because unexpected duplicates are a common source of bugs and security exploits (e.g. it's a fairly common issue with http headers when requests cross multiple systems).
        gslepak 3y ago
        After evaluating the various options, we seem to be settling on the idea that the protocol will specify that for duplicate keys, the handling must be the same as in browsers: meaning, the last key is the one that's used.