3 ms·

> When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in

> When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. What do you mean? Both Protobuf and Cap'n Proto are designed to be extensible. You can add new fields without breaking compatibility. (Disclosure: I wrote Protobuf v2 and Cap'n Proto.)

joshuamorton 7y ago
Further, like, protobuf now supports a Map<str, str>, although IDK about cap'n or flatbuffers. That lets you store the the common things in highly compressed protobuf, and extensions can either be done via extension protos, or with stringly-typed map stuff for testing/weird stuff.
chrismorgan 7y ago
[I am replying to scottlamb and joshuamorton as well as to you in this.] Forgive me; I am not speaking from a strong position of grokking Cap’n Proto or Protobuf (I’m casually familiar with them but have really only once used them, maybe three years ago), and I believe I erred in what I said. For Cap’n Proto, I don’t believe I’m far off the mark: to pick a simple example, extensions may add fields anywhere, and I believe you’re not allowed to skip field numbers, so you can’t just establish a registry of used field numbers and pick and choose the extensions. proto3 looks like it probably can’t handle this either. On proto2, though, it looks like I was approximately wrong, and you could have finite extensibility, and maintain a registry of assigned IDs so that extensions can be compatible. I’m not sure how suitable that would be in practice—the examples I see only allot a hundred numbers for extensions, which seems rather tiny; I’d picture allowing at least a million. Anything like Map<str, str or any or whatever> I’m going to disqualify from consideration because either you use structs in some places and thereby limit the places that extension can apply, or you use maps right through and then get no benefit over JSON. So I think they’d still be mostly technically unsuitable, but maybe Protobuf v2 would work. And maybe I’m completely wrong still—you’re the expert! But really, unless you get a major advantage out of such a thing, an important consideration is social suitability. Protobufs and the likes are not widely-used, require more effort to work with, and require more involved library support than JSON, which is much more universally understood and handled. Just look at how I’ve been fumbling around with them!
- kentonv 7y ago
  I see, the point is that you want to extend someone else's protocol, without their cooperation, and without conflicting with any future protocol changes they make. That is indeed tricky. With protobuf you could claim some high-numbered fields and hope that no one else does the same thing -- with some randomization you can probably be pretty confident. With Cap'n Proto you can't really do it at all since fields are identified by offset, and making structs arbitrarily large is probably not what you want. With JSON, because fields are identified by name, it's possible to choose a name that almost certainly will not collide with anyone else. I'd argue that the right answer here would be to ask the spec owners to add extension points, which hopefully would not be controversial. Conceptually this could take the form of something like map<string, any> (there are cleaner idiomatic ways to do it in each of Protobuf and Cap'n Proto but the details are beside the point of this conversation).

joshuamorton 7y ago

Further, like, protobuf now supports a Map<str, str>, although IDK about cap'n or flatbuffers. That lets you store the the common things in highly compressed protobuf, and extensions can either be done via extension protos, or with stringly-typed map stuff for testing/weird stuff.

chrismorgan 7y ago

[I am replying to scottlamb and joshuamorton as well as to you in this.] Forgive me; I am not speaking from a strong position of grokking Cap’n Proto or Protobuf (I’m casually familiar with them but have really only once used them, maybe three years ago), and I believe I erred in what I said. For Cap’n Proto, I don’t believe I’m far off the mark: to pick a simple example, extensions may add fields anywhere, and I believe you’re not allowed to skip field numbers, so you can’t just establish a registry of used field numbers and pick and choose the extensions. proto3 looks like it probably can’t handle this either. On proto2, though, it looks like I was approximately wrong, and you could have finite extensibility, and maintain a registry of assigned IDs so that extensions can be compatible. I’m not sure how suitable that would be in practice—the examples I see only allot a hundred numbers for extensions, which seems rather tiny; I’d picture allowing at least a million. Anything like Map<str, str or any or whatever> I’m going to disqualify from consideration because either you use structs in some places and thereby limit the places that extension can apply, or you use maps right through and then get no benefit over JSON. So I think they’d still be mostly technically unsuitable, but maybe Protobuf v2 would work. And maybe I’m completely wrong still—you’re the expert! But really, unless you get a major advantage out of such a thing, an important consideration is social suitability. Protobufs and the likes are not widely-used, require more effort to work with, and require more involved library support than JSON, which is much more universally understood and handled. Just look at how I’ve been fumbling around with them!

kentonv 7y ago
I see, the point is that you want to extend someone else's protocol, without their cooperation, and without conflicting with any future protocol changes they make. That is indeed tricky. With protobuf you could claim some high-numbered fields and hope that no one else does the same thing -- with some randomization you can probably be pretty confident. With Cap'n Proto you can't really do it at all since fields are identified by offset, and making structs arbitrarily large is probably not what you want. With JSON, because fields are identified by name, it's possible to choose a name that almost certainly will not collide with anyone else. I'd argue that the right answer here would be to ask the spec owners to add extension points, which hopefully would not be controversial. Conceptually this could take the form of something like map<string, any> (there are cleaner idiomatic ways to do it in each of Protobuf and Cap'n Proto but the details are beside the point of this conversation).