7 ms·

Scuttlebutt is a neat concept, burdened by a bad protocol. Signing a message involves serializing a json object, signing it, adding the signature as a field on

by Vendan 6y ago

Scuttlebutt is a neat concept, burdened by a bad protocol. Signing a message involves serializing a json object, signing it, adding the signature as a field on that json object, and then serializing it again. To verify, you deserialize the message into an object, remove the signature field, and then reserialize it, and verify the signature against that new serialization. This means that all the clients have to have a json serialization that 100% matches the node.js serialization that the majority (totality?) of current clients use. Any other implementation (I wrote one in Go) becomes a nightmare of chasing rarer and rarer differences that prevent verification, and if you miss one before you serialize your own message with it, suddenly you've poisoned your own journal... All in all, it's something to study and learn from, but I strongly recommend not becoming involved unless you are 100% happy with being tied into node.js.

alexh1 6y ago
This is how Cosmos (https://cosmos.network/ https://cosmos.network/) deals with signing transactions, caused _major_ headaches for me trying to figure out why my signatures were invalid when sending them from Node.js
- goliatone 6y ago
  I did something similar for a project that had clients in go and node. I solved this by flattening the object key paths to strings and sorting the keys, basically. You need to build that into the client serialization/deserialization, it feels clunky but I had 0 issues and has been working smoothly for a long while now.
sneak 6y ago
When I was faced with this (signing a structure), I serialized the json into base64, then put that base64 string as a value (along with the MAC) into a new json document. It of course increases deserialization overhead (json, verify, unbase64, inner json) but sidesteps this issue. I thought about sorting keys and other things like that, and the dozen edge cases and potential malleability issues dissuaded me for the compatibility issues mentioned above. How have others solved it?
- flir 6y ago
  Binary protocols. Or out-of-band signing.
  - elviejo 6y ago
    Do you mind going deeper here? Which protocols? Can you point to examples? Because I was about to design a signing Json solution but based on the comments here it is a bad idea.
    - moojd 6y ago
      Protobuf, cap n proto, and messagepack are a few I've seen before
    - zenhack 6y ago
      The signing schemes I've seen used in binary protocols fall into two categories: 1. Canonicalize and sign: the format has a defined canonical form. Convert to that before doing cryptographic operations. If the format is well designed around it, this is doable, whereas JSON doesn't really have this and with many libraries it's hard to control the output to the degree that you'd need. 2. Serialize and sign: serialize the data as flat bytes, and then your signed message just has a "bytes" field that is the signed object. This is conceptually not far off from the base64 solution above, except that there's not extra overhead, since with a binary protocol you'll have a length prefix instead of having to escape stuff.
    - bmm6o 6y ago
      Being able to separate the object and signature saves tons of trouble https://latacora.micro.blog/2019/07/24/how-not-to.html https://latacora.micro.blog/2019/07/24/how-not-to.html
- VWWHFSfQ 6y ago
  I just use JWTs whenever I have to pass signed messages.
- rakoo 6y ago
  Use Bencoding, like bittorrent does: https://en.wikipedia.org/wiki/Bencode https://en.wikipedia.org/wiki/Bencode As I put in another comment, a torrent id is a hash of a map, where one of the keys contains binary data. bencoding solved that decades ago already.
SyrupThinker 6y ago
After reading about the protocol I came to a similar conclusion as you. Although it needs to be noted that the JSON serialization is defined as JSON.stringify as defined in ECMA-262 6th Ed. plus some more. To me it's worse that key order must be preserved, which this standard does not specify the way I understand it. Source: https://ssbc.github.io/scuttlebutt-protocol-guide/#message-format https://ssbc.github.io/scuttlebutt-protocol-guide/#message-f...
- viraptor 6y ago
  This is terrible. It's basically not JSON anymore. It's some custom text protocol with similar value escaping... If they wanted both the descriptions to be visible and the order to be preserved, they could use: [ ["previous", "..."], ["author", "..."], ...
  - tptacek 6y ago
    lvh at Latacora wrote a blog post about this problem. https://latacora.micro.blog/2019/07/24/how-not-to.html https://latacora.micro.blog/2019/07/24/how-not-to.html
- kbumsik 6y ago
  > In brief, the rules are: - Two spaces for indentation. - Dictionary entries and list elements each on their own line. - etc... This is so weird to me. Why a protocol needs a strict, opinionated format of JSON? If they really need a very specific format of JSON, why they even bother JSON? There are better options like protobuf. This seems the worst example of "Use JSON for everything".
  - jitschlit 6y ago
    Just a guess, but maybe so they can reliable hash it?
dboreham 6y ago
Any signing done over structured data has this problem. You always need a canonical representation.
- mouldysammich 6y ago
  could it use a HTTPHeader style system instead to avoid the json serializing back and forth?
  - Vendan 6y ago
    Just serialize, sign, pack the signature and the raw bytes of the serialization. Doesn't matter how, just gotta pack the raw bytes, not futz with it
    - monocasa 6y ago
      You can even have a JSON object containing base64 fields for message and signature if you feel it really needs to be JSON all the way down for whatever reason.
- ben509 6y ago
  It would be simpler to work out a canonical reduction for JSON. This should be reasonably easy since there are so few elements to it. A simple proposal for actual security guys to rip to shreds: Strings are represented as utf-8 blobs, and hashed. Numbers should probably be represented using a proper decimal format and then hashed. If you're reading the JSON in and converting it to floats, you could get slight disagreement in some cases. Arrays are a list of hashes, which itself is hashed. For objects, convert the keys to utf-8 and append the value (which will always be the hashed representation) and then sort these entries bitwise. And then hash all that. Or, better, it'd be great to have an order-independent hash function that's also secure. I doubt xoring all the pairs would be good enough. Update: a possible technique[1] [1]: https://crypto.stackexchange.com/questions/51258/is-there-such-thing-as-an-order-independent-updatable-hash https://crypto.stackexchange.com/questions/51258/is-there-su...
TimJRobinson 6y ago
Once someone has written a library to do this correctly in Go/Rust/Whatever isn't this problem solved? Everyone building scuttlebutt apps with that language can use that library. It didn't seem like this protocol is changing.
- Vendan 6y ago
  I tried, and tried, and tried to make that library for Go. I failed, cause of serialization issues too involved for it to be worth it to me (got to the point where I'd have to write my own json implementation) Just giving people the heads-up that JS seemingly is the only blessed language for scuttlebutt
  - cel 6y ago
    The work was picked up and completed. Rust: https://github.com/sunrise-choir/ssb-legacy-msg/blob/master/src/json/ser.rs https://github.com/sunrise-choir/ssb-legacy-msg/blob/master/... https://github.com/sunrise-choir/ssb-publish/blob/master/src/lib.rs https://github.com/sunrise-choir/ssb-publish/blob/master/src... https://sunrise.social/ https://sunrise.social/ Go: https://github.com/cryptoscope/ssb/blob/master/message/legacy/encode.go https://github.com/cryptoscope/ssb/blob/master/message/legac... https://github.com/cryptoscope/ssb/blob/master/message/legacy/signature.go https://github.com/cryptoscope/ssb/blob/master/message/legac... https://planetary.social https://planetary.social Spec: https://spec.scuttlebutt.nz/feed/messages.html#json-encoding https://spec.scuttlebutt.nz/feed/messages.html#json-encoding https://spec.scuttlebutt.nz/feed/datamodel.html#signing-encoding https://spec.scuttlebutt.nz/feed/datamodel.html#signing-enco...
crispyporkbites 6y ago
I’ve had similar, albeit easy to solve, problems with key formats when signing emails. But that isn’t a reason why scuttlebutt isn’t more popular. It only takes one Go implementation and the problem is solved permanently.
masukomi 6y ago
there is work on C, Go, and Rust versions. It runs in iOS and Android. So, yeah, not so much "tied into node.js". Yes, that's where the majority of the work is, but...
Skinney 6y ago
This might actually improve in the near future: https://github.com/ssbc/ssb-spec-drafts/blob/master/drafts/draft-ssb-core-gabbygrove/00/draft-ssb-core-gabbygrove-00.txt https://github.com/ssbc/ssb-spec-drafts/blob/master/drafts/d...
az77 6y ago
Ju. G
hinkley 6y ago
I spent quite a bit of time with XML digital signatures, which is a similar situation, but with even more surface area to get wrong. Months to implement, over a year to harden.
rakoo 6y ago
Indeed, json maps are not supposed to be ordered so doing anything that depends on the order is bound to fail. This is the exact reason bencode (https://en.wikipedia.org/wiki/Bencode https://en.wikipedia.org/wiki/Bencode) was invented, and I still believe we can replace all uses of json by bencode and be better off it, because it solves all too common issues: - bencoding maps are in lexicographical order of the keys, so no confusion possible for hashing/signing (a torrent id is the hash of a bencoding map) - bencoding is binary friendly, in fact it must be because it stores the pieces hashes of the torrent Why don't we use bencoding everywhere ?
- muldvarp 6y ago
  Interesting that your experience with Bencode was this positive. I've implemented a Bencode serializer/deserializer in Rust and here are a few things I've noticed: * It is very easy to parse/produce Bencode * It's probably fast * The specification is really bad * No float type * No string type, just byte sequences. This is especially bad because most/all dictionary keys will be utf-8 strings in practice but you can't rely on it * Integers are arbitrary length, most implementations just ignore this I think Bencode is an ok format for its use case, but I don't think it should be used instead of json.
deleted 6y ago
[deleted]
antpls 6y ago
This is typically the case that would benefit from a function compiled to webassembly. "Write once, run everywhere." The protocol helper functions would be compiled to a webassembly library, and you would reuse them in Go, Python, the browser, etc Of course, it's not justification for using their protocol (rewriting a protocol in another language is a good test for the protocol specification), but that would be a usecase for webassembly.
black_puppydog 6y ago
As @cel pointed out further down the tree [1] that's not entirely correct any longer. Especially the "100% tied to node.js" part has not been for a while. There are now alternative implementations for Rust and Go. Now, the feed format is indeed a practical problem if you try to make a client from scratch and it is annoying. Luckily, using one of the already existing libraries will handle this for you. Changing that feed format for everyone is not possible, simply because there's already an existing social network built on the old ones that we very much want to preserve since we actually... well... hang out there. Changing feed format thus involves adding a new feed format and making sure other clients can handle and link together both. At the benefit of abstracting away the feed format, and being able to iterate on them. [1]: https://news.ycombinator.com/item?id=22912075 https://news.ycombinator.com/item?id=22912075
ptman 6y ago
https://pypi.org/project/canonicaljson/ https://pypi.org/project/canonicaljson/

by Vendan 6y ago

alexh1 6y ago

This is how Cosmos (https://cosmos.network/ https://cosmos.network/) deals with signing transactions, caused _major_ headaches for me trying to figure out why my signatures were invalid when sending them from Node.js

goliatone 6y ago
I did something similar for a project that had clients in go and node. I solved this by flattening the object key paths to strings and sorting the keys, basically. You need to build that into the client serialization/deserialization, it feels clunky but I had 0 issues and has been working smoothly for a long while now.

sneak 6y ago

When I was faced with this (signing a structure), I serialized the json into base64, then put that base64 string as a value (along with the MAC) into a new json document. It of course increases deserialization overhead (json, verify, unbase64, inner json) but sidesteps this issue. I thought about sorting keys and other things like that, and the dozen edge cases and potential malleability issues dissuaded me for the compatibility issues mentioned above. How have others solved it?

flir 6y ago
Binary protocols. Or out-of-band signing.
- elviejo 6y ago
  Do you mind going deeper here? Which protocols? Can you point to examples? Because I was about to design a signing Json solution but based on the comments here it is a bad idea.
  - moojd 6y ago
    Protobuf, cap n proto, and messagepack are a few I've seen before
  - zenhack 6y ago
    The signing schemes I've seen used in binary protocols fall into two categories: 1. Canonicalize and sign: the format has a defined canonical form. Convert to that before doing cryptographic operations. If the format is well designed around it, this is doable, whereas JSON doesn't really have this and with many libraries it's hard to control the output to the degree that you'd need. 2. Serialize and sign: serialize the data as flat bytes, and then your signed message just has a "bytes" field that is the signed object. This is conceptually not far off from the base64 solution above, except that there's not extra overhead, since with a binary protocol you'll have a length prefix instead of having to escape stuff.
  - bmm6o 6y ago
    Being able to separate the object and signature saves tons of trouble https://latacora.micro.blog/2019/07/24/how-not-to.html https://latacora.micro.blog/2019/07/24/how-not-to.html
VWWHFSfQ 6y ago
I just use JWTs whenever I have to pass signed messages.
rakoo 6y ago
Use Bencoding, like bittorrent does: https://en.wikipedia.org/wiki/Bencode https://en.wikipedia.org/wiki/Bencode As I put in another comment, a torrent id is a hash of a map, where one of the keys contains binary data. bencoding solved that decades ago already.

SyrupThinker 6y ago

After reading about the protocol I came to a similar conclusion as you. Although it needs to be noted that the JSON serialization is defined as JSON.stringify as defined in ECMA-262 6th Ed. plus some more. To me it's worse that key order must be preserved, which this standard does not specify the way I understand it. Source: https://ssbc.github.io/scuttlebutt-protocol-guide/#message-format https://ssbc.github.io/scuttlebutt-protocol-guide/#message-f...

viraptor 6y ago
This is terrible. It's basically not JSON anymore. It's some custom text protocol with similar value escaping... If they wanted both the descriptions to be visible and the order to be preserved, they could use: [ ["previous", "..."], ["author", "..."], ...
- tptacek 6y ago
  lvh at Latacora wrote a blog post about this problem. https://latacora.micro.blog/2019/07/24/how-not-to.html https://latacora.micro.blog/2019/07/24/how-not-to.html
kbumsik 6y ago
> In brief, the rules are: - Two spaces for indentation. - Dictionary entries and list elements each on their own line. - etc... This is so weird to me. Why a protocol needs a strict, opinionated format of JSON? If they really need a very specific format of JSON, why they even bother JSON? There are better options like protobuf. This seems the worst example of "Use JSON for everything".
- jitschlit 6y ago
  Just a guess, but maybe so they can reliable hash it?

dboreham 6y ago

Any signing done over structured data has this problem. You always need a canonical representation.

mouldysammich 6y ago
could it use a HTTPHeader style system instead to avoid the json serializing back and forth?
- Vendan 6y ago
  Just serialize, sign, pack the signature and the raw bytes of the serialization. Doesn't matter how, just gotta pack the raw bytes, not futz with it
  - monocasa 6y ago
    You can even have a JSON object containing base64 fields for message and signature if you feel it really needs to be JSON all the way down for whatever reason.
ben509 6y ago
It would be simpler to work out a canonical reduction for JSON. This should be reasonably easy since there are so few elements to it. A simple proposal for actual security guys to rip to shreds: Strings are represented as utf-8 blobs, and hashed. Numbers should probably be represented using a proper decimal format and then hashed. If you're reading the JSON in and converting it to floats, you could get slight disagreement in some cases. Arrays are a list of hashes, which itself is hashed. For objects, convert the keys to utf-8 and append the value (which will always be the hashed representation) and then sort these entries bitwise. And then hash all that. Or, better, it'd be great to have an order-independent hash function that's also secure. I doubt xoring all the pairs would be good enough. Update: a possible technique[1] [1]: https://crypto.stackexchange.com/questions/51258/is-there-such-thing-as-an-order-independent-updatable-hash https://crypto.stackexchange.com/questions/51258/is-there-su...

TimJRobinson 6y ago

Once someone has written a library to do this correctly in Go/Rust/Whatever isn't this problem solved? Everyone building scuttlebutt apps with that language can use that library. It didn't seem like this protocol is changing.

Vendan 6y ago
I tried, and tried, and tried to make that library for Go. I failed, cause of serialization issues too involved for it to be worth it to me (got to the point where I'd have to write my own json implementation) Just giving people the heads-up that JS seemingly is the only blessed language for scuttlebutt
- cel 6y ago
  The work was picked up and completed. Rust: https://github.com/sunrise-choir/ssb-legacy-msg/blob/master/src/json/ser.rs https://github.com/sunrise-choir/ssb-legacy-msg/blob/master/... https://github.com/sunrise-choir/ssb-publish/blob/master/src/lib.rs https://github.com/sunrise-choir/ssb-publish/blob/master/src... https://sunrise.social/ https://sunrise.social/ Go: https://github.com/cryptoscope/ssb/blob/master/message/legacy/encode.go https://github.com/cryptoscope/ssb/blob/master/message/legac... https://github.com/cryptoscope/ssb/blob/master/message/legacy/signature.go https://github.com/cryptoscope/ssb/blob/master/message/legac... https://planetary.social https://planetary.social Spec: https://spec.scuttlebutt.nz/feed/messages.html#json-encoding https://spec.scuttlebutt.nz/feed/messages.html#json-encoding https://spec.scuttlebutt.nz/feed/datamodel.html#signing-encoding https://spec.scuttlebutt.nz/feed/datamodel.html#signing-enco...

crispyporkbites 6y ago

I’ve had similar, albeit easy to solve, problems with key formats when signing emails. But that isn’t a reason why scuttlebutt isn’t more popular. It only takes one Go implementation and the problem is solved permanently.

masukomi 6y ago

there is work on C, Go, and Rust versions. It runs in iOS and Android. So, yeah, not so much "tied into node.js". Yes, that's where the majority of the work is, but...

Skinney 6y ago

This might actually improve in the near future: https://github.com/ssbc/ssb-spec-drafts/blob/master/drafts/draft-ssb-core-gabbygrove/00/draft-ssb-core-gabbygrove-00.txt https://github.com/ssbc/ssb-spec-drafts/blob/master/drafts/d...

az77 6y ago

Ju. G

hinkley 6y ago

I spent quite a bit of time with XML digital signatures, which is a similar situation, but with even more surface area to get wrong. Months to implement, over a year to harden.

rakoo 6y ago

Indeed, json maps are not supposed to be ordered so doing anything that depends on the order is bound to fail. This is the exact reason bencode (https://en.wikipedia.org/wiki/Bencode https://en.wikipedia.org/wiki/Bencode) was invented, and I still believe we can replace all uses of json by bencode and be better off it, because it solves all too common issues: - bencoding maps are in lexicographical order of the keys, so no confusion possible for hashing/signing (a torrent id is the hash of a bencoding map) - bencoding is binary friendly, in fact it must be because it stores the pieces hashes of the torrent Why don't we use bencoding everywhere ?

muldvarp 6y ago
Interesting that your experience with Bencode was this positive. I've implemented a Bencode serializer/deserializer in Rust and here are a few things I've noticed: * It is very easy to parse/produce Bencode * It's probably fast * The specification is really bad * No float type * No string type, just byte sequences. This is especially bad because most/all dictionary keys will be utf-8 strings in practice but you can't rely on it * Integers are arbitrary length, most implementations just ignore this I think Bencode is an ok format for its use case, but I don't think it should be used instead of json.

deleted 6y ago

[deleted]

antpls 6y ago

This is typically the case that would benefit from a function compiled to webassembly. "Write once, run everywhere." The protocol helper functions would be compiled to a webassembly library, and you would reuse them in Go, Python, the browser, etc Of course, it's not justification for using their protocol (rewriting a protocol in another language is a good test for the protocol specification), but that would be a usecase for webassembly.

black_puppydog 6y ago

As @cel pointed out further down the tree [1] that's not entirely correct any longer. Especially the "100% tied to node.js" part has not been for a while. There are now alternative implementations for Rust and Go. Now, the feed format is indeed a practical problem if you try to make a client from scratch and it is annoying. Luckily, using one of the already existing libraries will handle this for you. Changing that feed format for everyone is not possible, simply because there's already an existing social network built on the old ones that we very much want to preserve since we actually... well... hang out there. Changing feed format thus involves adding a new feed format and making sure other clients can handle and link together both. At the benefit of abstracting away the feed format, and being able to iterate on them. [1]: https://news.ycombinator.com/item?id=22912075 https://news.ycombinator.com/item?id=22912075

ptman 6y ago

https://pypi.org/project/canonicaljson/ https://pypi.org/project/canonicaljson/