5 ms·
I’m a developer at Fastmail, and I’m going to bite. Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and wri
by chrismorgan 7y ago
I’m a developer at Fastmail, and I’m going to bite.
Personally, I prefer binary formats where feasible, because I like pursuing efficiency to extremes, and write all my own code in Rust.
Professionally, I happen to write mostly JavaScript, and no Rust at all, but we do still focus on performance at Fastmail. (It’s in the name!)
If you’re comparing JSON to s-expressions, it’s a wash in efficiency. They’re both “inefficient”, both representing generic data structures in regular text. They use slightly different punctuation, but that’s about it. The differences are negligible.
But when it comes to support, JSON is well-supported in all mainstream languages, while s-expressions have no central specification on precise syntax, and you’ll need to do your own thing. One of the deliberate design advantages of JMAP over IMAP is doing away with custom parsers, and using the standard tools people are familiar with, HTTPS and JSON.
When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire.
All in all, JSON is a robust choice for just about any API.
Fun fact: at Fastmail we compress both request and response, even using a custom dictionary primed on real JMAP stuff, to improve the compression ratio. I understand standard DEFLATE was about equivalent in size to CBOR (a binary representation of JSON), and the 2KB dictionary brought it down even further, can’t remember how much it was off the top of my head. Neil Jenkins gave a talk about this and related matters last month, https://www.webdirections.org/code/speakers/neil-jenkins.php https://www.webdirections.org/code/speakers/neil-jenkins.php, no public video sadly. I’ve thought about writing a blog post about it all. An unfortunate side-effect of this compression is that you can’t inspect traffic in the network panel of the dev tools; you can add ?compress=0 to disable it, if you’re an interested Fastmail user.
- nerdponx 7y agoIs Messagepack not a useful alternative to JSON in your opinion?
- chrismorgan 7y agoMessagePack and CBOR are equivalent. I haven’t ever had much to do with them (never used them, only known about them), but I’m inclined to think that simple compression achieves between “roughly as much” and “more” in general, and leaves you with the industry standard of JSON underneath. Do not consider me an expert by any means, and I’m fairly ambivalent on your question; but I don’t see a particularly useful niche for them between plain JSON and a binary representation matching a predefined static schema, all potentially with compression.
- deathanatos 7y ago> MessagePack and CBOR are equivalent. MessagePack has an ugly history with text data that's still reflected in some of the libraries interfaces. E.g., the Python library doesn't round-trip strings by default. (They'll turn into bytes objects.) There's a flag to do the right thing, but not everyone is aware of it. Add to that that CBOR has an RFC, and I'll pick it over MessagePack every time. Binary formats are mostly nicer from a parsing perspective: it's much easier to unpack. You're right, I think, that simple compression will mostly close any size gap.
- scottlamb 7y ago> When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. I don't think that's a deal-breaker. protobuf has an extension mechanism. [1] You could have a registry for an org to claim an extension number to a given message (and then use it to hold their own message with as much stuff in it as they like). I'm not as familiar with cap'n proto but assume it has something similar. That said, I think JSON was a fine choice for this purpose. The efficiency is good enough with standard implementations; and there are much faster-than-standard implementations available if you have a special reason to want one. [2] [1] https://developers.google.com/protocol-buffers/docs/proto#extensions https://developers.google.com/protocol-buffers/docs/proto#ex... [2] https://github.com/lemire/simdjson https://github.com/lemire/simdjson
- kentonv 7y ago> When talking efficiency, you might then ask why not to use Cap’n Proto, protobufs or one of those sorts of things. Due to the need for extensibility (e.g. in Fastmail we add a few custom fields to JMAP core and mail data structures), rigid structures are unsuitable; you do need arbitrary key-value arrangement, despite it being larger over the wire. What do you mean? Both Protobuf and Cap'n Proto are designed to be extensible. You can add new fields without breaking compatibility. (Disclosure: I wrote Protobuf v2 and Cap'n Proto.)
- joshuamorton 7y agoFurther, like, protobuf now supports a Map<str, str>, although IDK about cap'n or flatbuffers. That lets you store the the common things in highly compressed protobuf, and extensions can either be done via extension protos, or with stringly-typed map stuff for testing/weird stuff.
- chrismorgan 7y ago[I am replying to scottlamb and joshuamorton as well as to you in this.] Forgive me; I am not speaking from a strong position of grokking Cap’n Proto or Protobuf (I’m casually familiar with them but have really only once used them, maybe three years ago), and I believe I erred in what I said. For Cap’n Proto, I don’t believe I’m far off the mark: to pick a simple example, extensions may add fields anywhere, and I believe you’re not allowed to skip field numbers, so you can’t just establish a registry of used field numbers and pick and choose the extensions. proto3 looks like it probably can’t handle this either. On proto2, though, it looks like I was approximately wrong, and you could have finite extensibility, and maintain a registry of assigned IDs so that extensions can be compatible. I’m not sure how suitable that would be in practice—the examples I see only allot a hundred numbers for extensions, which seems rather tiny; I’d picture allowing at least a million. Anything like Map<str, str or any or whatever> I’m going to disqualify from consideration because either you use structs in some places and thereby limit the places that extension can apply, or you use maps right through and then get no benefit over JSON. So I think they’d still be mostly technically unsuitable, but maybe Protobuf v2 would work. And maybe I’m completely wrong still—you’re the expert! But really, unless you get a major advantage out of such a thing, an important consideration is social suitability. Protobufs and the likes are not widely-used, require more effort to work with, and require more involved library support than JSON, which is much more universally understood and handled. Just look at how I’ve been fumbling around with them!
- zeveb 7y ago> s-expressions have no central specification on precise syntax There is: https://people.csail.mit.edu/rivest/Sexp.txt https://people.csail.mit.edu/rivest/Sexp.txt It's imperfect, but it does get the job done. Among its advantages is that it's both pure text and binary: there's a canonical representation which is binary, and an advanced representation which is textual, and they map 1:1 to one another. Examples: (session (capabilities (urn:ietf:params:jmap:core (max-size-upload 50000000) (max-concurrent-upload 8) (max-size-request 10000000) (max-calls-in-request 32) (max-objects-in-get 256) (max-objects-in-set 128) (collation-algorithms ("i;ascii-numeric" "i;ascii-casemap" "i;unicode-casemap"))) (urn:ietf:params:jmap:mail) (urn:ietf:params:jmap:contacts) (https://example.com/apis/foobar (max-foos-finagled 42))) (accounts (A13824 (name "john@example.com") (is-personal true) (is-read-only false) (account-capabilities (urn:ietf:params:jmap:mail (max-mailboxes-per-email nil) (max-mailbox-depth 10)) (urn:ietf:params:jmap:contacts …))) (A97813 (name "jane@example.com") (is-personal false) (is-read-only true) (account-capabilities (urn:ietf:params:jmap:mail (max-mailboxes-per-email 1) (max-mailbox-depth 10))))) (primary-accounts (urn:ietf:params:jmap:mail A13824) (urn:ietf:params:jmap:contacts A13824)) (username "john@example.com") (api-url "https://jmap.example.com/download/{accountId}/{blobId}/{name}?accept={type}") (upload-url "https://jmap.example.com/upload/{accountId}/") (event-source-url "https://jmap.example.com/eventsource/?types={types}&closeafter={closeafter}&ping={ping}") (state "75128aab4b1b")) Or the binary encoding: {KDc6c2Vzc2lvbigxMjpjYXBhYmlsaXRpZXMoMjU6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29y ZSgxNTptYXgtc2l6ZS11cGxvYWQ4OjUwMDAwMDAwKSgyMTptYXgtY29uY3VycmVudC11cGxv YWQxOjgpKDE2Om1heC1zaXplLXJlcXVlc3Q4OjEwMDAwMDAwKSgyMDptYXgtY2FsbHMtaW4t cmVxdWVzdDI6MzIpKDE4Om1heC1vYmplY3RzLWluLWdldDM6MjU2KSgxODptYXgtb2JqZWN0 cy1pbi1zZXQzOjEyOCkoMjA6Y29sbGF0aW9uLWFsZ29yaXRobXMoMTU6aTthc2NpaS1udW1l cmljMTU6aTthc2NpaS1jYXNlbWFwMTc6aTt1bmljb2RlLWNhc2VtYXApKSkoMjU6dXJuOmll dGY6cGFyYW1zOmptYXA6bWFpbCkoMjk6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29udGFjdHMp KSgzMTpodHRwczovL2V4YW1wbGUuY29tL2FwaXMvZm9vYmFyKDE3Om1heC1mb29zLWZpbmFn bGVkMjo0MikpKDg6YWNjb3VudHMoNjpBMTM4MjQoNDpuYW1lMTY6am9obkBleGFtcGxlLmNv bSkoMTE6aXMtcGVyc29uYWw0OnRydWUpKDEyOmlzLXJlYWQtb25seTU6ZmFsc2UpKDIwOmFj Y291bnQtY2FwYWJpbGl0aWVzKDI1OnVybjppZXRmOnBhcmFtczpqbWFwOm1haWwoMjM6bWF4 LW1haWxib3hlcy1wZXItZW1haWwzOm5pbCkoMTc6bWF4LW1haWxib3gtZGVwdGgyOjEwKSko Mjk6dXJuOmlldGY6cGFyYW1zOmptYXA6Y29udGFjdHMzWiYpKSkoNjpBOTc4MTMoNDpuYW1l MTY6amFuZUBleGFtcGxlLmNvbSkoMTE6aXMtcGVyc29uYWw1OmZhbHNlKSgxMjppcy1yZWFk LW9ubHk0OnRydWUpKDIwOmFjY291bnQtY2FwYWJpbGl0aWVzKDI1dXJuOmlldGY6cGFyYW1z OmptYXA6bWFpbCgyMzptYXgtbWFpbGJveGVzLXBlci1lbWFpbDE6MSkoMTc6bWF4LW1haWxi b3gtZGVwdGgyOjEwKSkpKSkoMTY6cHJpbWFyeS1hY2NvdW50cygyNTp1cm46aWV0ZjpwYXJh bXM6am1hcDptYWlsNjpBMTM4MjQpKDI5OnVybjppZXRmOnBhcmFtczpqbWFwOmNvbnRhY3Rz NjpBMTM4MjQpKSg4OnVzZXJuYW1lMTY6am9obgo=} This is the equivalent of the JSON: { "capabilities": { "urn:ietf:params:jmap:core": { "maxSizeUpload": 50000000, "maxConcurrentUpload": 8, "maxSizeRequest": 10000000, "maxConcurrentRequest": 8, "maxCallsInRequest": 32, "maxObjectsInGet": 256, "maxObjectsInSet": 128, "collationAlgorithms": [ "i;ascii-numeric", "i;ascii-casemap", "i;unicode-casemap" ] }, "urn:ietf:params:jmap:mail": {} "urn:ietf:params:jmap:contacts": {}, "https://example.com/apis/foobar": { "maxFoosFinangled": 42 } }, "accounts": { "A13824": { "name": "john@example.com", "isPersonal": true, "isReadOnly": false, "accountCapabilities": { "urn:ietf:params:jmap:mail": { "maxMailboxesPerEmail": null, "maxMailboxDepth": 10, ... }, "urn:ietf:params:jmap:contacts": { ... } } }, "A97813": { "name": "jane@example.com", "isPersonal": false, "isReadOnly": true, "accountCapabilities": { "urn:ietf:params:jmap:mail": { "maxMailboxesPerEmail": 1, "maxMailboxDepth": 10, ... } } } }, "primaryAccounts": { "urn:ietf:params:jmap:mail": "A13824", "urn:ietf:params:jmap:contacts": "A13824" }, "username": "john@example.com", "apiUrl": "https://jmap.example.com/api/", "downloadUrl": "https://jmap.example.com /download/{accountId}/{blobId}/{name}?accept={type}", "uploadUrl": "https://jmap.example.com/upload/{accountId}/", "eventSourceUrl": "https://jmap.example.com /eventsource/?types={types}&closeafter={closeafter}&ping={ping}", "state": "75128aab4b1b" } I think it's pretty obvious which one is cleaner …