9 ms·

As others have said, there are unfortunately not that many good alternatives for serializing data in web browsers (that are both performant and don't require bu

As others have said, there are unfortunately not that many good alternatives for serializing data in web browsers (that are both performant and don't require bulking your app with the serialization implementation). We've done our best to make sure that the way we're serializing objects can either be done in a consistent manner across implementations or, if done inconsistently, it doesn't matter.

treyd 3y ago
I don't really buy the argument that additional serialization logic adds bloat to the application when you compare it to the many thousands of lines of code in modern JS frameworks for manipulating DOM elements. The consistency is really an important part here, are you spending less time doing the testing and bugfixing to ensure that JSON ambiguity is not possible / is harmless than you would writing serialization for cryptographic types in unambiguous byte-oriented structures?
- gslepak 3y ago
  SPMessage serializations are never regenerated after they leave the client, so it's not an issue as far as I'm aware. JSON is supported universally in all languages and all platforms, and natively in the browser. Is there a specific serialization format you're advocating for?
  - Groxx 3y ago
    I suspect it's more that JSON is (rather explicitly) not designed to be stable across all implementations, so there are subtle differences. Numbers don't have a specified precision, strings don't have a specified encoding, maps have no canonical order nor do they require unique keys[1], etc. Nailing all that down so you can e.g. have a truly deterministic signing process often requires custom encoders/decoders and is rather fraught with problems even then, at which point why even bother with JSON. Use something designed to be consistent. [1]: https://stackoverflow.com/a/23195243 https://stackoverflow.com/a/23195243
    - gslepak 3y ago
      We're aware of these issues with JSON, but they do not affect the integrity of Shelter Protocol, and by using JSON we get a lot of advantages in terms of speed and developer usability (support for tools like `jq`, schema-less serialization, widespread language support, easier debugging, etc.).
      - rpdillon 3y ago
        I was really excited when I tried out the scuttlebutt protocol a few years back. Distributed, peer-to-peer, offline-enabled social networking seemed neat. But a healthy ecosystem of clients never emerged. When I looked into why, I discovered it relied on node's particular method of JSON serialization[0]. Are you confident that the shelter protocol has avoided this pitfall? [0]: https://news.ycombinator.com/item?id=22910752 https://news.ycombinator.com/item?id=22910752
        gslepak 3y ago
        Yes, I am confident that we have avoided that specific pitfall because as I mentioned we do not ever re-serialize the JSON. However, thinking on it more, there is one thing Groxx mentioned that has given me pause, and that is "Numbers don't have a specified precision". This could be an issue in the future if, for example, computers suddenly use 128-bit numbers and JSON implementations start to actually use such numbers. It's an edge case that might result in incompatibility with clients that use 64-bit numbers. So we'll consider the possibility of changing the serialization format after researching this more.
        gslepak 3y ago
        > So we'll consider the possibility of changing the serialization format after researching this more. I asked Brendan Eich if JavaScript will ever change the way it represents floating point numbers (from 64-bit IEEE 754 values), and he said no ("don't break the Web"). And to represent larger bit values in JSON you're supposed to use a JSON string and schema to specify the type. I'm not sure exactly what he means by that, but I assume it means something like using JSON keys like "<BigInt>mykey" and string values instead of number values. Then manually parsing the string into whatever native 128-bit type or whatever representation you have. So - JSON still seems to be the favorite here, but I totally get the anxiety around it because of how under-specified it is. If you avoid re-serialization nothing should break.

treyd 3y ago

I don't really buy the argument that additional serialization logic adds bloat to the application when you compare it to the many thousands of lines of code in modern JS frameworks for manipulating DOM elements. The consistency is really an important part here, are you spending less time doing the testing and bugfixing to ensure that JSON ambiguity is not possible / is harmless than you would writing serialization for cryptographic types in unambiguous byte-oriented structures?

gslepak 3y ago
SPMessage serializations are never regenerated after they leave the client, so it's not an issue as far as I'm aware. JSON is supported universally in all languages and all platforms, and natively in the browser. Is there a specific serialization format you're advocating for?
- Groxx 3y ago
  I suspect it's more that JSON is (rather explicitly) not designed to be stable across all implementations, so there are subtle differences. Numbers don't have a specified precision, strings don't have a specified encoding, maps have no canonical order nor do they require unique keys[1], etc. Nailing all that down so you can e.g. have a truly deterministic signing process often requires custom encoders/decoders and is rather fraught with problems even then, at which point why even bother with JSON. Use something designed to be consistent. [1]: https://stackoverflow.com/a/23195243 https://stackoverflow.com/a/23195243
  - gslepak 3y ago
    We're aware of these issues with JSON, but they do not affect the integrity of Shelter Protocol, and by using JSON we get a lot of advantages in terms of speed and developer usability (support for tools like `jq`, schema-less serialization, widespread language support, easier debugging, etc.).
    - rpdillon 3y ago
      I was really excited when I tried out the scuttlebutt protocol a few years back. Distributed, peer-to-peer, offline-enabled social networking seemed neat. But a healthy ecosystem of clients never emerged. When I looked into why, I discovered it relied on node's particular method of JSON serialization[0]. Are you confident that the shelter protocol has avoided this pitfall? [0]: https://news.ycombinator.com/item?id=22910752 https://news.ycombinator.com/item?id=22910752
      - gslepak 3y ago
        Yes, I am confident that we have avoided that specific pitfall because as I mentioned we do not ever re-serialize the JSON. However, thinking on it more, there is one thing Groxx mentioned that has given me pause, and that is "Numbers don't have a specified precision". This could be an issue in the future if, for example, computers suddenly use 128-bit numbers and JSON implementations start to actually use such numbers. It's an edge case that might result in incompatibility with clients that use 64-bit numbers. So we'll consider the possibility of changing the serialization format after researching this more.
        gslepak 3y ago
        > So we'll consider the possibility of changing the serialization format after researching this more. I asked Brendan Eich if JavaScript will ever change the way it represents floating point numbers (from 64-bit IEEE 754 values), and he said no ("don't break the Web"). And to represent larger bit values in JSON you're supposed to use a JSON string and schema to specify the type. I'm not sure exactly what he means by that, but I assume it means something like using JSON keys like "<BigInt>mykey" and string values instead of number values. Then manually parsing the string into whatever native 128-bit type or whatever representation you have. So - JSON still seems to be the favorite here, but I totally get the anxiety around it because of how under-specified it is. If you avoid re-serialization nothing should break.