4 ms·

I like the idea that Tim is working on a binary-based protocol interaction with Couch. If the binary protocol integration is generic enough I'd like to discuss

by rkalla 15y ago

I like the idea that Tim is working on a binary-based protocol interaction with Couch. If the binary protocol integration is generic enough I'd like to discuss with them the possibility of supporting the Universal Binary JSON format (http://ubjson.org http://ubjson.org) While the HTTP-only protocol work up until now has been one the nice "Ease of use" standout features for Couch, it also leads to a lot of overhead when you know you absolutely don't benefit from it. All the work on CouchApps (only CouchDB + JavaScript apps) makes a lot of sense to have the rest-only interaction, but if you are trying to scale out a high read/write couch install and are already using tricks like TCP no_delay and server-side batching along with batch-doc operations on the client there is only so much you can do before you slam up against what looks to be a 6-10K docs/sec change rate due in no small part to Couch (even with its new JSON parser) processing data structures into and out of textual JSON.

timanglade 15y ago
To be clear, I'm only working on integrating a binary data format, which is different from a binary protocol. HTTP is baked pretty deep into CouchDB, which makes it hard to remove (but who knows, Couchbase seems to be having some success ripping it out). BigCouch (Cloudant’s distributed version of CouchDB) has the HTTP layer neatly separated through the Fabric library (https://github.com/cloudant/fabric https://github.com/cloudant/fabric), so it'd probably be easier to add a binary protocol there.
- rkalla 15y ago
  Tim, is the Couch server still going from binary disk data to JSON and then you convert that to the binary format or is your work actually allowing Couch to skip any textual JSON generation such that all the REST methods are still there, but their response bodies are just MessagePack binary data instead of text?
  - timanglade 15y ago
    The Couch server goes from binary Erlang terms (our default internal representation) to MessagePack right away, skipping all textual JSON generation. All the REST methods are still there, the body is just sent in MessagePack format with an application/x-msgpack header https://github.com/timanglade/couchdb/commit/b601286dae04bdc2488a0d9bf028c58e6feb3449 https://github.com/timanglade/couchdb/commit/b601286dae04bdc...
    - rkalla 15y ago
      Ah! That is exactly what I was hoping for (skipping the JSON in/out cycle server-side). Keeping the HTTP protocol in this case is fine for me and not require any unexpected changes to Couch. I hope that binary format work can be make configurable in the future with other formats supported. Thanks for the clarifications Tim.
mlmilleratmit 15y ago
Good point. A binary protocol is certainly the ultimate play. Tim has some promising initial estimates, but the gains depend on the document properties (mostly size). In the meantime an alternative solution is to spread the server-side load via BigCouch or some other clustering solution. However, I'd wager that even with slow JSON libs most workflows are still IO limited.

by rkalla 15y ago

timanglade 15y ago

To be clear, I'm only working on integrating a binary data format, which is different from a binary protocol. HTTP is baked pretty deep into CouchDB, which makes it hard to remove (but who knows, Couchbase seems to be having some success ripping it out). BigCouch (Cloudant’s distributed version of CouchDB) has the HTTP layer neatly separated through the Fabric library (https://github.com/cloudant/fabric https://github.com/cloudant/fabric), so it'd probably be easier to add a binary protocol there.

rkalla 15y ago
Tim, is the Couch server still going from binary disk data to JSON and then you convert that to the binary format or is your work actually allowing Couch to skip any textual JSON generation such that all the REST methods are still there, but their response bodies are just MessagePack binary data instead of text?
- timanglade 15y ago
  The Couch server goes from binary Erlang terms (our default internal representation) to MessagePack right away, skipping all textual JSON generation. All the REST methods are still there, the body is just sent in MessagePack format with an application/x-msgpack header https://github.com/timanglade/couchdb/commit/b601286dae04bdc2488a0d9bf028c58e6feb3449 https://github.com/timanglade/couchdb/commit/b601286dae04bdc...
  - rkalla 15y ago
    Ah! That is exactly what I was hoping for (skipping the JSON in/out cycle server-side). Keeping the HTTP protocol in this case is fine for me and not require any unexpected changes to Couch. I hope that binary format work can be make configurable in the future with other formats supported. Thanks for the clarifications Tim.

mlmilleratmit 15y ago

Good point. A binary protocol is certainly the ultimate play. Tim has some promising initial estimates, but the gains depend on the document properties (mostly size). In the meantime an alternative solution is to spread the server-side load via BigCouch or some other clustering solution. However, I'd wager that even with slow JSON libs most workflows are still IO limited.