4 ms·

Call for comments - RethinkDB wire protocol

37 points by coffeemug 14y ago

coffeemug 14y ago
Hey all, slava @ rethink here. As the post mentions, the new protobuf spec is here: https://github.com/rethinkdb/rethinkdb/wiki/protobuf_rfc_raw_spec https://github.com/rethinkdb/rethinkdb/wiki/protobuf_rfc_raw... Would really appreciate anyone's questions or comments!
MichaelGG 14y ago
Why bother trying to encode the query syntax into the wire protocol? Why not just accept queries as a string, then have some simple way to pass parameters? It seems like encoding the whole query language syntax makes the clients far more complex.
- alexpopescu 14y ago
  Transmitting a string over the wire would pose a wide range of problems: 1. it's not efficient 2. we'd need to design a complete new protocol (keep in mind that you'd need to include query parameters) which for advanced query language like RQL is challenging 3. with a new protocol we'd need also versioning (for ensuring backward compatibility) And the list is probably much longer. alex @ rethinkdb
  - MichaelGG 14y ago
    Thanks for entertaining me :). I'm only talking about the actual encoding of the query itself, not parameters, return types, etc. Looking at the github page, it seems all the names of things are still being transmitted as strings (of course), so "r.add(1,2)" is unlikely to be smaller decomposed into discrete items (each with at least a 1-byte pb tag overhead). I'd think it'd be passing a main query string, with placeholders for parameters. "r.add(@a,@b)" or something, then have a parameter block where you can efficiently encode non-string parameters. Basically, keep protocol buffers and the rest, just don't encode the syntax tree - let the server parse it.
    - LeafStorm 14y ago
      So, basically using SQL prepared statements. Which are fine for simple queries where the behavior is always exactly the same, but become very complicated when you outgrow the "one hour blog" demo phase. It's much easier to build queries out of data structures in most languages, and if you're already building a data structure, there's no reason to convert it to a human-editable text form just so it can be parsed again.
      - MichaelGG 14y ago
        Oh, I see. So in the ReQL docs all those examples aren't really a "language"? There's no underlying "RQL", it's just using a language with some nifty client libraries to build up an internal state that gets serialized. That's a neat approach.
        coffeemug 14y ago
        Right. Technically we could write a "language" by just writing a string parser that converts strings to protocol buffers (some day we might do that). But embedding into host languages is so nice, there was no need to do it so far.
- jdoliner 14y ago
  Constructing the query as a string is actually much more difficult for clients. Basically it comes down to the difference between having to splice little snippets of code into a string, being careful that you get it on the right side of the parentheses. And just setting a field on a structure where you have no chance of misplacing the value. Not to mention, assuming results came back in the same format client developers would need to write a string parser and with protocol buffers it's written for them. It also makes it much easier for developers because the server can better detect errors and send more meaningful responses.
- DannoHung 14y ago
  I dunno about you, but in my experience, queries eventually need to be generated dynamically, which means having queries that can be externally represented by data structures. Having a data structure that represents a query in the first place is a pretty good idea. Even better is being able to go back and forth between strings and data structures.
zellyn 14y ago
See the "required is forever" warning here: https://developers.google.com/protocol-buffers/docs/proto https://developers.google.com/protocol-buffers/docs/proto You may want to make even your conceptually required fields optional. As an example, if you tried to proxy a newer version of the protocol through an old proxy (assuming it's deserializing), unknown enumeration values for required fields could cause errors.
- coffeemug 14y ago
  Thanks!!
tedchs 14y ago
Maybe consider re-using an existing wire protocol, such as that for Postgres or MongoDB (whatever is closest conceptually to your database)? Or use HTTP? That way you don't need to re-invent a client for every language.
- coffeemug 14y ago
  We did (even had a few implementations). Nothing could come close to enabling an awesome user experience as a custom one, so we decided it's worth the effort.

37 points by coffeemug 14y ago

Hey all, slava @ rethink here. As the post mentions, the new protobuf spec is here: https://github.com/rethinkdb/rethinkdb/wiki/protobuf_rfc_raw_spec https://github.com/rethinkdb/rethinkdb/wiki/protobuf_rfc_raw... Would really appreciate anyone's questions or comments!

MichaelGG 14y ago

Why bother trying to encode the query syntax into the wire protocol? Why not just accept queries as a string, then have some simple way to pass parameters? It seems like encoding the whole query language syntax makes the clients far more complex.

alexpopescu 14y ago
Transmitting a string over the wire would pose a wide range of problems: 1. it's not efficient 2. we'd need to design a complete new protocol (keep in mind that you'd need to include query parameters) which for advanced query language like RQL is challenging 3. with a new protocol we'd need also versioning (for ensuring backward compatibility) And the list is probably much longer. alex @ rethinkdb
- MichaelGG 14y ago
  Thanks for entertaining me :). I'm only talking about the actual encoding of the query itself, not parameters, return types, etc. Looking at the github page, it seems all the names of things are still being transmitted as strings (of course), so "r.add(1,2)" is unlikely to be smaller decomposed into discrete items (each with at least a 1-byte pb tag overhead). I'd think it'd be passing a main query string, with placeholders for parameters. "r.add(@a,@b)" or something, then have a parameter block where you can efficiently encode non-string parameters. Basically, keep protocol buffers and the rest, just don't encode the syntax tree - let the server parse it.
  - LeafStorm 14y ago
    So, basically using SQL prepared statements. Which are fine for simple queries where the behavior is always exactly the same, but become very complicated when you outgrow the "one hour blog" demo phase. It's much easier to build queries out of data structures in most languages, and if you're already building a data structure, there's no reason to convert it to a human-editable text form just so it can be parsed again.
    - MichaelGG 14y ago
      Oh, I see. So in the ReQL docs all those examples aren't really a "language"? There's no underlying "RQL", it's just using a language with some nifty client libraries to build up an internal state that gets serialized. That's a neat approach.
      - coffeemug 14y ago
        Right. Technically we could write a "language" by just writing a string parser that converts strings to protocol buffers (some day we might do that). But embedding into host languages is so nice, there was no need to do it so far.
jdoliner 14y ago
Constructing the query as a string is actually much more difficult for clients. Basically it comes down to the difference between having to splice little snippets of code into a string, being careful that you get it on the right side of the parentheses. And just setting a field on a structure where you have no chance of misplacing the value. Not to mention, assuming results came back in the same format client developers would need to write a string parser and with protocol buffers it's written for them. It also makes it much easier for developers because the server can better detect errors and send more meaningful responses.
DannoHung 14y ago
I dunno about you, but in my experience, queries eventually need to be generated dynamically, which means having queries that can be externally represented by data structures. Having a data structure that represents a query in the first place is a pretty good idea. Even better is being able to go back and forth between strings and data structures.

zellyn 14y ago

See the "required is forever" warning here: https://developers.google.com/protocol-buffers/docs/proto https://developers.google.com/protocol-buffers/docs/proto You may want to make even your conceptually required fields optional. As an example, if you tried to proxy a newer version of the protocol through an old proxy (assuming it's deserializing), unknown enumeration values for required fields could cause errors.

coffeemug 14y ago
Thanks!!

tedchs 14y ago

Maybe consider re-using an existing wire protocol, such as that for Postgres or MongoDB (whatever is closest conceptually to your database)? Or use HTTP? That way you don't need to re-invent a client for every language.

coffeemug 14y ago
We did (even had a few implementations). Nothing could come close to enabling an awesome user experience as a custom one, so we decided it's worth the effort.