4 ms·

Cap'n Proto v0.3: Python support, better tools, other improvements

101 points by kentonv 13y ago

bsimpson 13y ago
The Python implementation is a wrapper around a C++ module, so it's probably not practical to use on AppEngine.
- kentonv 13y ago
  Yep, that is a big trade-off. There's room for someone to write a pure-Python implementation as well, to fill that niche. You could probably get a lot of the way using Python's `struct` module. But it would be slower.
zheng 13y ago
The claims on this site are pretty impressive, but I have close to zero knowledge of the history here, so can someone comment on how many grains of salt this should be taken with? Otherwise, this looks pretty cool. Something that beats protobufs in overall speed could be really helpful depending on the application.
- plorkyeran 13y ago
  There's a big focus on how it's better than protobufs because it's written by the original creator of protobufs and is his attempt at making a better version based on what he's learned. By the time it hits 1.0 I suspect the only reason to use protobufs rather than it will be for backwards compatibility with existing systems.
- kentonv 13y ago
  So, as the author of Cap'n Proto I'm biased -- though I'm also the author of Protobufs v2, so I'm not completely biased. :) "Infinitely faster" is of course meant more to illustrate how Cap'n Proto works than to be taken as a literal speed measure. Although, if you actually wanted to compare Cap'n Proto to Protobufs, it's unclear what other number you can really come up with. The normal way to compare Protobuf speed vs. anything else is to measure the encode or decode step, but Cap'n Proto has no such step. You can measure an end-to-end system using one vs. the other, but then on the Cap'n Proto side you are basically measuring the speed of everything _except_ the Cap'n Proto code. The git repo includes some contrived benchmarks along those lines which you can try out. I don't post the numbers because I'm not sure they are meaningful (even though they appear very favorable for Cap'n Proto). I'm really hoping to see a few unbiased third parties benchmark Cap'n Proto vs. Protobufs in real-world systems at some point. Of course, the larger point here is that Cap'n Proto allows you to do things that Protobuf simply doesn't support, like mmap()ing in a large file and reading one field out of it in constant time, whereas with Protobuf you have to parse the whole thing making it O(size of file) time.
  - zheng 13y ago
    Thanks a lot for the reply, one of my favorite things about HN is getting questions answered by the authors of the tool in question. After reading more, your decision to not post those benchmarks is a smart one. I get where you're coming from with regards to it being hard to make a performance comparison to protobufs, it makes sense now. If/when I need to reach for some serialization I'll certainly try out Cap'n Proto.
    - kentonv 13y ago
      > If/when I need to reach for some serialization I'll certainly try out Cap'n Proto. If/when you do, remember that the mailing list is friendly and we very much want to hear your feedback and help you with any problems. :)
- munificent 13y ago
  > I have close to zero knowledge of the history here, so can someone comment on how many grains of salt this should be taken with? Not many, as far as I know. Kenton worked on Protobufs at Google for years, so he should know exactly what he's doing here.
- haberman 13y ago
  Just to give a bit of counterpoint, here are some trade-offs that Capn Proto makes compared with protobufs. (Full disclosure: I work at Google and know Kenton from his time here; I have my own protobuf library that I've worked on for several years). I'm sure Kenton will correct me if I get anything wrong. :) Capn Proto's key design characteristic is to use the same encoding on-the-wire as in-memory. Protobufs have a wire format that looks something like: [field number 3][value for field 3] [field number 7][value for field 7] etc. The fieldnum/value pairs can come in any order, and may define as many or as few of the declared fields as are present. This serialization format doesn't work for in-memory usage because for general programming you need O(1) access to each value, so protobufs have a "parse" step that unpacks this into a C++ class where each field has its own member. Protobufs are heavily optimized so this parsing is fast, but it's still a very noticeable cost in high-volume systems. So Capn Proto defines its wire format such that it also has O(1) access to arbitrary fields. This makes it suitable as an in-memory format also. While this avoids a parsing step, it also means that your wire format has to preserve the empty spaces for fields that aren't present. So to get the "infinitely faster" advantage, you have to accept this cost. For dense messages, this can actually be smaller than the comparable protobuf because you don't have to encode the field numbers. But for very sparse messages, this can be arbitrarily larger. As Kenton points out on http://kentonv.github.io/capnproto/encoding.html http://kentonv.github.io/capnproto/encoding.html , lots of zeros compress really well, so even sparse messages can become really small by compressing them. To do this you lose "infinitely faster", but according to Kenton this is still faster than protobufs. In both cases though, the tight coupling between the (uncompressed) wire format and the in-memory format imposes certain things on your application with regards to memory management and the mutation patterns the struct will allow. For example, it appears that the in-memory format was not sufficiently flexible for Python to wrap it directly, so the Python extension does in fact have a parse step. Other cases where you could need a parse/serialize step anyway: if you want to put the wire data into a specialized container like a map or set (or your own custom data classes), or if the supported built-in mutation patterns are not flexible enough for you (for example, the Capn Proto "List" type appears to have limitations on how and when a list can grow in size). It's very cool work, but I don't believe it obsoletes Protocol Buffers. I'm actually interested in making the two interoperate, along with JSON -- these key/value technologies are so similar in concept and usage that I think it's unfortunate they don't interoperate better.
srollyson 13y ago
Hi, Kenton. First: thanks for your work on Protocol Buffers. I've used it fairly extensively for RPC communications between C++/Java clients and a Java service. It made things so much easier to get native objects in each language using a well-defined protocol. One thing that bugged me about Protobuf is that it provided a skeletal mechanism for RPC (e.g. RpcController/RpcChannel) but later deprecated the use of that mechanism in favor of code-generating plugins. Since Cap'n Proto is billed as an "RPC system", do you have plans to include a more fleshed-out version of RPC than was provided in Protocol buffers? Having abstract classes for event-handling and transport mechanisms is a good idea for extensibility but it sure would make it easier for your users if there was at least one default implementation of each. I imagine that Google has standard implementations of these things internally but balked at trying to support them for multiple languages as an open source project.
- kentonv 13y ago
  Yes, in fact, the next release of Cap'n Proto (v0.4) is slated to include RPC support. There are some hints on what it might look like in the docs already: http://kentonv.github.io/capnproto/rpc.html http://kentonv.github.io/capnproto/rpc.html http://kentonv.github.io/capnproto/language.html#interfaces http://kentonv.github.io/capnproto/language.html#interfaces The reason Google never released an RPC system together with protobufs is because Google's RPC implementation simply had too many dependencies on other Google infrastructure, and wasn't appropriate for use outside of Google datacenters. There were a few attempts to untangle the mess and produce something that could be released, but it never happened. The public release had support for generating generic stubs, as you mentioned, but it was later decided that these stubs were actually a poor basis for implementing an RPC system. In their attempt to be generic, their interface ended up being rather awkward. We later decided that it made more sense to support code generator plugins, so that someone implementing an RPC system could provide a plugin that generates code ideal for that particular system. The generic interfaces were then deprecated. Cap'n Proto also supports code generation plugins. But, as I said, we will soon also have an "official" RPC layer as well -- and it will hopefully be somewhat plugable itself, so that you can use a different underlying transport with the same generated interface code. Anyway, this will all become clearer with the next release, so stay tuned!

101 points by kentonv 13y ago

bsimpson 13y ago

The Python implementation is a wrapper around a C++ module, so it's probably not practical to use on AppEngine.

kentonv 13y ago
Yep, that is a big trade-off. There's room for someone to write a pure-Python implementation as well, to fill that niche. You could probably get a lot of the way using Python's `struct` module. But it would be slower.

zheng 13y ago

The claims on this site are pretty impressive, but I have close to zero knowledge of the history here, so can someone comment on how many grains of salt this should be taken with? Otherwise, this looks pretty cool. Something that beats protobufs in overall speed could be really helpful depending on the application.

plorkyeran 13y ago
There's a big focus on how it's better than protobufs because it's written by the original creator of protobufs and is his attempt at making a better version based on what he's learned. By the time it hits 1.0 I suspect the only reason to use protobufs rather than it will be for backwards compatibility with existing systems.
kentonv 13y ago
So, as the author of Cap'n Proto I'm biased -- though I'm also the author of Protobufs v2, so I'm not completely biased. :) "Infinitely faster" is of course meant more to illustrate how Cap'n Proto works than to be taken as a literal speed measure. Although, if you actually wanted to compare Cap'n Proto to Protobufs, it's unclear what other number you can really come up with. The normal way to compare Protobuf speed vs. anything else is to measure the encode or decode step, but Cap'n Proto has no such step. You can measure an end-to-end system using one vs. the other, but then on the Cap'n Proto side you are basically measuring the speed of everything _except_ the Cap'n Proto code. The git repo includes some contrived benchmarks along those lines which you can try out. I don't post the numbers because I'm not sure they are meaningful (even though they appear very favorable for Cap'n Proto). I'm really hoping to see a few unbiased third parties benchmark Cap'n Proto vs. Protobufs in real-world systems at some point. Of course, the larger point here is that Cap'n Proto allows you to do things that Protobuf simply doesn't support, like mmap()ing in a large file and reading one field out of it in constant time, whereas with Protobuf you have to parse the whole thing making it O(size of file) time.
- zheng 13y ago
  Thanks a lot for the reply, one of my favorite things about HN is getting questions answered by the authors of the tool in question. After reading more, your decision to not post those benchmarks is a smart one. I get where you're coming from with regards to it being hard to make a performance comparison to protobufs, it makes sense now. If/when I need to reach for some serialization I'll certainly try out Cap'n Proto.
  - kentonv 13y ago
    > If/when I need to reach for some serialization I'll certainly try out Cap'n Proto. If/when you do, remember that the mailing list is friendly and we very much want to hear your feedback and help you with any problems. :)
munificent 13y ago
> I have close to zero knowledge of the history here, so can someone comment on how many grains of salt this should be taken with? Not many, as far as I know. Kenton worked on Protobufs at Google for years, so he should know exactly what he's doing here.
haberman 13y ago
Just to give a bit of counterpoint, here are some trade-offs that Capn Proto makes compared with protobufs. (Full disclosure: I work at Google and know Kenton from his time here; I have my own protobuf library that I've worked on for several years). I'm sure Kenton will correct me if I get anything wrong. :) Capn Proto's key design characteristic is to use the same encoding on-the-wire as in-memory. Protobufs have a wire format that looks something like: [field number 3][value for field 3] [field number 7][value for field 7] etc. The fieldnum/value pairs can come in any order, and may define as many or as few of the declared fields as are present. This serialization format doesn't work for in-memory usage because for general programming you need O(1) access to each value, so protobufs have a "parse" step that unpacks this into a C++ class where each field has its own member. Protobufs are heavily optimized so this parsing is fast, but it's still a very noticeable cost in high-volume systems. So Capn Proto defines its wire format such that it also has O(1) access to arbitrary fields. This makes it suitable as an in-memory format also. While this avoids a parsing step, it also means that your wire format has to preserve the empty spaces for fields that aren't present. So to get the "infinitely faster" advantage, you have to accept this cost. For dense messages, this can actually be smaller than the comparable protobuf because you don't have to encode the field numbers. But for very sparse messages, this can be arbitrarily larger. As Kenton points out on http://kentonv.github.io/capnproto/encoding.html http://kentonv.github.io/capnproto/encoding.html , lots of zeros compress really well, so even sparse messages can become really small by compressing them. To do this you lose "infinitely faster", but according to Kenton this is still faster than protobufs. In both cases though, the tight coupling between the (uncompressed) wire format and the in-memory format imposes certain things on your application with regards to memory management and the mutation patterns the struct will allow. For example, it appears that the in-memory format was not sufficiently flexible for Python to wrap it directly, so the Python extension does in fact have a parse step. Other cases where you could need a parse/serialize step anyway: if you want to put the wire data into a specialized container like a map or set (or your own custom data classes), or if the supported built-in mutation patterns are not flexible enough for you (for example, the Capn Proto "List" type appears to have limitations on how and when a list can grow in size). It's very cool work, but I don't believe it obsoletes Protocol Buffers. I'm actually interested in making the two interoperate, along with JSON -- these key/value technologies are so similar in concept and usage that I think it's unfortunate they don't interoperate better.

srollyson 13y ago

Hi, Kenton. First: thanks for your work on Protocol Buffers. I've used it fairly extensively for RPC communications between C++/Java clients and a Java service. It made things so much easier to get native objects in each language using a well-defined protocol. One thing that bugged me about Protobuf is that it provided a skeletal mechanism for RPC (e.g. RpcController/RpcChannel) but later deprecated the use of that mechanism in favor of code-generating plugins. Since Cap'n Proto is billed as an "RPC system", do you have plans to include a more fleshed-out version of RPC than was provided in Protocol buffers? Having abstract classes for event-handling and transport mechanisms is a good idea for extensibility but it sure would make it easier for your users if there was at least one default implementation of each. I imagine that Google has standard implementations of these things internally but balked at trying to support them for multiple languages as an open source project.

kentonv 13y ago
Yes, in fact, the next release of Cap'n Proto (v0.4) is slated to include RPC support. There are some hints on what it might look like in the docs already: http://kentonv.github.io/capnproto/rpc.html http://kentonv.github.io/capnproto/rpc.html http://kentonv.github.io/capnproto/language.html#interfaces http://kentonv.github.io/capnproto/language.html#interfaces The reason Google never released an RPC system together with protobufs is because Google's RPC implementation simply had too many dependencies on other Google infrastructure, and wasn't appropriate for use outside of Google datacenters. There were a few attempts to untangle the mess and produce something that could be released, but it never happened. The public release had support for generating generic stubs, as you mentioned, but it was later decided that these stubs were actually a poor basis for implementing an RPC system. In their attempt to be generic, their interface ended up being rather awkward. We later decided that it made more sense to support code generator plugins, so that someone implementing an RPC system could provide a plugin that generates code ideal for that particular system. The generic interfaces were then deprecated. Cap'n Proto also supports code generation plugins. But, as I said, we will soon also have an "official" RPC layer as well -- and it will hopefully be somewhat plugable itself, so that you can use a different underlying transport with the same generated interface code. Anyway, this will all become clearer with the next release, so stay tuned!