4 ms·

Cap'n'Proto: infinitely faster than Protobuf

35 points by jackhalford 3y ago

elromulous 3y ago
Happy to see capnproto on the front page! Fyi - kentonv, its creater, is on here and is usually super responsive.
commandersaki 3y ago
In my testing I’ve found protobuf to be a bit faster.
- bjconlan 3y ago
  I would also be interested in the outcome of these tests. Did you test against other protocols ? I've been meaning to do some tests using the c++ implementations of each used in userver.tech (specifically flatbuffers/protobuffs also unsupported sbe & captnproto) for etl processing "generally" (Again more variables are to be weighed in here for my case but would be interesting to see these things against payload sizes)
  - commandersaki 3y ago
    We were serialising/deserialising a large c++ class that consumes about 100 megabytes of memory. The reason we chose protobuf over capnproto was not performance but because it was already being used in other projects. However I implemented both and didn't really do a deep comparison of performance. Basically protobuf serialisation/deserialisation was a few seconds fast. However, I didn't use capnproto as intended, in that I deserialised from the capnproto objects into an actual object which involved copying; I don't really see using the native format directly all that useful in most applications, but maybe I'm missing the point.
deleted 3y ago
[deleted]
gnabgib 3y ago
Failing both the "misleading or clickbait" and the "please use original title" guidelines[0]. The article admits the image-stamp-ruse in the second paragraph: This benchmark is, of course, unfair. It is only measuring the time to encode and decode a message in memory. Cap’n Proto gets a perfect score because there is no encoding/ decoding step. Title: "Cap'n Proto: Introduction" H1: "Introduction" Submitted many times 2016[1][2][3], 2018[4], 2020[5][6], 2021[7], 2022[8][9], 2023[10][11] [0]: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html [1]: https://news.ycombinator.com/item?id=10825233 https://news.ycombinator.com/item?id=10825233 [2]: https://news.ycombinator.com/item?id=11229153 https://news.ycombinator.com/item?id=11229153 [3]: https://news.ycombinator.com/item?id=12471266 https://news.ycombinator.com/item?id=12471266 [4]: https://news.ycombinator.com/item?id=16397346 https://news.ycombinator.com/item?id=16397346 [5]: https://news.ycombinator.com/item?id=23924978 https://news.ycombinator.com/item?id=23924978 [6]: https://news.ycombinator.com/item?id=24065123 https://news.ycombinator.com/item?id=24065123 [7]: https://news.ycombinator.com/item?id=27061046 https://news.ycombinator.com/item?id=27061046 [8]: https://news.ycombinator.com/item?id=31189502 https://news.ycombinator.com/item?id=31189502 [9]: https://news.ycombinator.com/item?id=32468581 https://news.ycombinator.com/item?id=32468581 [10]: https://news.ycombinator.com/item?id=35128998 https://news.ycombinator.com/item?id=35128998 [11]: https://news.ycombinator.com/item?id=36913680 https://news.ycombinator.com/item?id=36913680
bb88 3y ago
This has been around since 2013. It's good, but it's not new.
gvkhna 3y ago
To be fair protobufs are not purely focused on speed. And what speed exactly? Speed to parse, read, transfer? Flatbuffers are purely fast read/write by storing buffers that can be mmap’d right into memory. At the cost of some size bloat (slower to transfer over the wire). Protobufs are a mix of speed but also size over the wire being reduced which does reduce speed of parsing but increases speed transferring over the wire. I think capnproto is a fantastic project, but considering google is built on top of grpc/protobufs, they will be around for effectively ever which gives it a lot of reliability in my book.
- sshine 3y ago
  > capnproto is a fantastic project, but considering google is built on top of grpc/protobufs, they will be around for effectively ever When choosing an RPC interface, we went with gRPC because of maturity and familiarity, not lifetime support specifically. Building RPC on Cap'n Proto is a bit more barebones: https://capnproto.org/rpc.html https://capnproto.org/rpc.html But the library support is getting there (converts a description into code at build): https://crates.io/crates/capnp-rpc https://crates.io/crates/capnp-rpc I look forward to trying out Cap'n Proto for RPC. But I also look forward to try Selium as a message system: https://selium.com/ https://selium.com/ Maybe in combination.
moribvndvs 3y ago
I had a shitty manager 20 years ago that constantly used “infinitely” literally and unironically (and also without qualification or very “gracious” caveats in his favor), thanks for the PTSD!
- jackhalford 3y ago
  I’m sure cap n proto understands that infinity doesn’t exist in the real world
  - frizlab 3y ago
    Infinity does exist though. The benchmark is specifically done on the encoding/decoding steps on the formats. But Cap'n Proto does not have that, so it is, in effect infinitely faster for this step.
    - germandiago 3y ago
      I found an instance of infinity in real life. Weird as it sounds, put a mirror in front of another, same size and shape better. The reflection will vound "infinitely" from one mirror to the other creating a "never-ending" effect of mirrors inside other mirrors. In my university gym changing rooms there were two mirrors in front of each other like that. First time I saw it it kept me thinking about infinity :D
      - frizlab 3y ago
        True, but the light reflects less and less each time and in the end the reflection stops.
        germandiago 3y ago
        I was also thought that. At some scale it must stop... so not real infinite? :D

35 points by jackhalford 3y ago

elromulous 3y ago

Happy to see capnproto on the front page! Fyi - kentonv, its creater, is on here and is usually super responsive.

commandersaki 3y ago

In my testing I’ve found protobuf to be a bit faster.

bjconlan 3y ago
I would also be interested in the outcome of these tests. Did you test against other protocols ? I've been meaning to do some tests using the c++ implementations of each used in userver.tech (specifically flatbuffers/protobuffs also unsupported sbe & captnproto) for etl processing "generally" (Again more variables are to be weighed in here for my case but would be interesting to see these things against payload sizes)
- commandersaki 3y ago
  We were serialising/deserialising a large c++ class that consumes about 100 megabytes of memory. The reason we chose protobuf over capnproto was not performance but because it was already being used in other projects. However I implemented both and didn't really do a deep comparison of performance. Basically protobuf serialisation/deserialisation was a few seconds fast. However, I didn't use capnproto as intended, in that I deserialised from the capnproto objects into an actual object which involved copying; I don't really see using the native format directly all that useful in most applications, but maybe I'm missing the point.

deleted 3y ago

[deleted]

gnabgib 3y ago

Failing both the "misleading or clickbait" and the "please use original title" guidelines[0]. The article admits the image-stamp-ruse in the second paragraph: This benchmark is, of course, unfair. It is only measuring the time to encode and decode a message in memory. Cap’n Proto gets a perfect score because there is no encoding/ decoding step. Title: "Cap'n Proto: Introduction" H1: "Introduction" Submitted many times 2016[1][2][3], 2018[4], 2020[5][6], 2021[7], 2022[8][9], 2023[10][11] [0]: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html [1]: https://news.ycombinator.com/item?id=10825233 https://news.ycombinator.com/item?id=10825233 [2]: https://news.ycombinator.com/item?id=11229153 https://news.ycombinator.com/item?id=11229153 [3]: https://news.ycombinator.com/item?id=12471266 https://news.ycombinator.com/item?id=12471266 [4]: https://news.ycombinator.com/item?id=16397346 https://news.ycombinator.com/item?id=16397346 [5]: https://news.ycombinator.com/item?id=23924978 https://news.ycombinator.com/item?id=23924978 [6]: https://news.ycombinator.com/item?id=24065123 https://news.ycombinator.com/item?id=24065123 [7]: https://news.ycombinator.com/item?id=27061046 https://news.ycombinator.com/item?id=27061046 [8]: https://news.ycombinator.com/item?id=31189502 https://news.ycombinator.com/item?id=31189502 [9]: https://news.ycombinator.com/item?id=32468581 https://news.ycombinator.com/item?id=32468581 [10]: https://news.ycombinator.com/item?id=35128998 https://news.ycombinator.com/item?id=35128998 [11]: https://news.ycombinator.com/item?id=36913680 https://news.ycombinator.com/item?id=36913680

bb88 3y ago

This has been around since 2013. It's good, but it's not new.

gvkhna 3y ago

To be fair protobufs are not purely focused on speed. And what speed exactly? Speed to parse, read, transfer? Flatbuffers are purely fast read/write by storing buffers that can be mmap’d right into memory. At the cost of some size bloat (slower to transfer over the wire). Protobufs are a mix of speed but also size over the wire being reduced which does reduce speed of parsing but increases speed transferring over the wire. I think capnproto is a fantastic project, but considering google is built on top of grpc/protobufs, they will be around for effectively ever which gives it a lot of reliability in my book.

sshine 3y ago
> capnproto is a fantastic project, but considering google is built on top of grpc/protobufs, they will be around for effectively ever When choosing an RPC interface, we went with gRPC because of maturity and familiarity, not lifetime support specifically. Building RPC on Cap'n Proto is a bit more barebones: https://capnproto.org/rpc.html https://capnproto.org/rpc.html But the library support is getting there (converts a description into code at build): https://crates.io/crates/capnp-rpc https://crates.io/crates/capnp-rpc I look forward to trying out Cap'n Proto for RPC. But I also look forward to try Selium as a message system: https://selium.com/ https://selium.com/ Maybe in combination.

moribvndvs 3y ago

I had a shitty manager 20 years ago that constantly used “infinitely” literally and unironically (and also without qualification or very “gracious” caveats in his favor), thanks for the PTSD!

jackhalford 3y ago
I’m sure cap n proto understands that infinity doesn’t exist in the real world
- frizlab 3y ago
  Infinity does exist though. The benchmark is specifically done on the encoding/decoding steps on the formats. But Cap'n Proto does not have that, so it is, in effect infinitely faster for this step.
  - germandiago 3y ago
    I found an instance of infinity in real life. Weird as it sounds, put a mirror in front of another, same size and shape better. The reflection will vound "infinitely" from one mirror to the other creating a "never-ending" effect of mirrors inside other mirrors. In my university gym changing rooms there were two mirrors in front of each other like that. First time I saw it it kept me thinking about infinity :D
    - frizlab 3y ago
      True, but the light reflects less and less each time and in the end the reflection stops.
      - germandiago 3y ago
        I was also thought that. At some scale it must stop... so not real infinite? :D