7 ms·

Cap'n Proto 0.9

144 points by Shelnutt2 5y ago

appleflaxen 5y ago
I am a huge Kenton Varda fan. his sandcats platform was genius. I'm happy to see it open sourced, but sad to see the development slow down. great to see he is still pushing cap'n proto forward!
- studentrob 5y ago
  You mean sandstorm? https://sandstorm.io/ https://sandstorm.io/ Edit: oh you do mean sandcats.. https://docs.sandstorm.io/en/latest/administering/sandcats/ https://docs.sandstorm.io/en/latest/administering/sandcats/
hamandcheese 5y ago
Aside from the speed benefits, are there any other major advantages of Cap’n Proto? I’ve been using grpc/protobuf with Java and Scala. Overall I like it, but the grpc Java library does feel like it comes with a lot of opinions baked in. As an example, I was looking in to implementing automatic retries on the client side for certain classes of error. Grpc Java has some experimental retry support, but it seemed tightly coupled to their also-opinionated way of doing service discovery. I can see why you might want the server to inform clients how to do retries, keepalives, etc. But for less sophisticated operations (read: me) it is frustrating and over complicated. Why can’t I just specify a retry policy when creating a client stub? Still, it’s better than anything else I’ve seen. But I do long for a simpler alternative. Is that Cap’n proto? Seems like for Java there is only 3rd party support for serialization only, and not RPC.
- richieartoul 5y ago
  I also agree that the gRPCs libraries (in all languages) tend to be opinionated, and most annoyingly, opinionated about the wrong things. That said I also haven’t found anything better either and the technology is popular enough that I stick with it and just work around the limitations. For your specific case I highly recommend ignoring the built in GRPC retry nonsense and using the Java failsafe library retry policies instead. That has worked well for my team at least.
  - chii 5y ago
    > annoyingly, opinionated about the wrong things. what's what opinions are - they necessarily must be wrong to some people. Otherwise, it'd just be called facts!
    - setr 5y ago
      Having an opinion is not the same as having a good opinion. There’s a range of quality to the matter
      - edoceo 5y ago
        The point is that "good" and "quality" is subjective and highly dependent on use-case.
        setr 5y ago
        And my point is that it’s not — the starting pieces may differ, and thus the conclusions, but there’s clearly a qualitative difference between the opinion of an expert, and the opinion of a novice. If only that the expert brings far more knowledge into his opinion than the novice. But also in that his opinion is of better foundation, of more supporting infrastructure, of more stable reasoning, and can in general be both better explained and better defended. Opinions don’t exist in a void — they’re still the outcome of argument, and can easily be compared by that ruler. And in this case, it’s trivial to imagine utterly terrible opinionated outcomes — use hyphen to represent addition, having functions like c’s get() (impossible to use correctly, and having defaults that are backwards in the common case. It’s also trivial to imagine what a poorly reasoned opinion might look like: “ what's what opinions are - they necessarily must be wrong to some people. Otherwise, it'd just be called facts!”
    - ithkuil 5y ago
      Some try to confuse this aspect by calling opinions "best practices"
- mixedCase 5y ago
  >are there any other major advantages of Cap’n Proto? Parametric type polymorphism AKA generics. I regularly think of Cap'n Proto when working with gRPC or GraphQL, despite not ever having had the chance to use it in a real project.
- asimpletune 5y ago
  The actual language itself is very nice https://capnproto.org/language.html https://capnproto.org/language.html, and it has a bunch of really great security features. I actually think of it the other way around and don't understand why it's not used more.
  - jchw 5y ago
    On paper I agree. However it does have a couple of problems: - It has yet to escape the ZeroVer versioning scheme and produce a “stable” API. - Maturity for languages other than C++ can be a bit spotty. - The RPC protocol is cool, but apparently most of it was never implemented. The C++ implementation only implements “level 1.” Cap’n Proto is undeniably cooler than Protobuf, although in practice I worry primarily about maturity and support.
    - aseipp 5y ago
      It's fine if most implementations of RPC are Level 1, even the main C++ bindings; due to the design of the protocol there only needs to exist a single Level 4 implementation written in any language, and all other implementations can then piggyback on it for their own needs beyond Level 1 (so cap proxying, etc. all come "for free" once someone does that.) I've mentioned Level 4 support to Kenton a while back and the basic answer was "I haven't needed it so it's not done." It's not intended for there to be a Level 4 implementation in every language or anything. The biggest issue is undeniably that the language maturity isn't quite as solid as the alterntaives. You've got C++, Rust, Python, C#, maybe one or two others. Only C++ is first class. So gRPC and Protobuf win here. But the actual RPC design and IDL design of Cap'n is solid and way more well engineered than the alternatives, IMO. I have no opinions on the ZeroVer thing. Cap'n Proto is very stable in my experience but I see why others would raise their eyebrows.
      - heavenlyblue 5y ago
        Python implementation is super low quality. A lot of their tests were checking for Exception raises and were basically testing AttributeError due to a change in interface instead of the initial error they were testing for.
- tyingq 5y ago
  >are there any other major advantages of Cap’n Proto There's a list at https://capnproto.org/ https://capnproto.org/ under "other advantages".
- caturopath 5y ago
  The 'cap' in Cap'n Proto is for https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security -- Cap'n Proto isn't actually an RPC system, it's a capabilities framework. You can work with results without actually having the results, e.g. before the results exist.
  - dpwm 5y ago
    > You can work with results without actually having the results, e.g. before the results exist. This results in cool things like promise pipelining. It is very interesting how it can simplify interface design, and there are examples on the RPC page[0]. [0] https://capnproto.org/rpc.html https://capnproto.org/rpc.html
  - hamandcheese 5y ago
    > The 'cap' in Cap'n Proto is for https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security This seems to go completely unmentioned on the home page, other than the word “capability system” in the very first sentence. What does capability based security mean for a data interchange format? The notion hardly makes any sense to me. From the homepage: “Think JSON, except binary.” What would JSON look like with capability-based security? > You can work with results without actually having the results, e.g. before the results exist This sounds exactly like promises, or perhaps a functional effect system. What does this have to do with data serialization?
    - lucozade 5y ago
      Cap'n Proto has a remote object system using interface references. When a reference is constructed, only the creator has the capability of invoking the interface. If the reference is then passed to another server, they and only they have the capability of invoking the interface. That's what they mean by capability.
    - c-cube 5y ago
      It's a promise you can send to other processes, even to the machine that is supposed to fulfill that promise. That allows batching of calls that depend on one another.
- PhilippGille 5y ago
  > But I do long for a simpler alternative If you're happy with protobuf but just not with gRPC, you can check out Twirp [1] and DRPC [2]. Both aim to be simpler alternatives while keeping protobuf for serialization. Their development is Go focused though. [1] https://github.com/twitchtv/twirp https://github.com/twitchtv/twirp [2] https://github.com/storj/drpc/ https://github.com/storj/drpc/
- moonchrome 5y ago
  If you're looking for a binary serialisation library that generates simpler/more readable code and doesn't do parsing look into flatbuffers. Going through generated protobuff code in . NET last time I used it was vomit inducing. Flatbuffers don't provide RPC AFAIK so not comparable with gRPC Capnproto tried to do too much with distributed objects and RPC, I was looking into it years ago but it still seems like the C++ impl is the only one that supports those advanced concepts.
  - jiggawatts 5y ago
    Apparently at C# implementation supports promise pipelining: https://github.com/c80k/capnproto-dotnetcore#features https://github.com/c80k/capnproto-dotnetcore#features It is missing many nice-to-have features however and is also apparently unmaintained, with the last release 16 months ago...
    - naasking 5y ago
      No activity doesn't necessarily mean unmaintained, it could just mean usable and stable/bug-free.
  - Communitivity 5y ago
    For your own sanity, please stay away from Flatbuffers. The key here is 'doesn't do parsing'. I used it, and wound up writing a ton of extract code to unpack things in the way I wanted. Behind the scenes everything in FlatBuffers is stored in a byte array or byte buffer. It can be made to work, but protobuffs worked better for me. Caveats: 1) I needed to work in C++ and Java 2) I was transporting serialized messages over ZeroMQ
    - kortex 5y ago
      At one point I dug a bit into the python impl of Flatbuffers, and while I forget the exact details, I was struck by how inefficient it looked. Basically every accessor call requires parsing/casting (into python) bytes into objects, without any awareness of whether it had been parsed before.
lilyball 5y ago
What will it take before it’s declared as 1.0?
- Tempest1981 5y ago
  Maybe never, if ZeroVer is the new hotness... https://news.ycombinator.com/item?id=28154187 https://news.ycombinator.com/item?id=28154187
- kentonv 5y ago
  There's a change I want to make to the kj::AsyncOutputStream interface, to make it have an explicit `end()` method rather than assuming EOF on destruction. That's backwards-incompatible, so I don't want to declare 1.0 until I get around to doing it... Also, I need to make a web site for KJ.
PostThisTooFast 5y ago
Why is this better than flatbuffers?
rawoke083600 5y ago
Very good. Where does the "speed increase" comes from ? Better encoding scheme or better decoding implementation ?
- the_biot 5y ago
  It's right there on the front page.
SV_BubbleTime 5y ago
Is anyone using a non-self describing schema format like this in C? I used Protobufs for awhile and really didn’t like the C options.

144 points by Shelnutt2 5y ago

I am a huge Kenton Varda fan. his sandcats platform was genius. I'm happy to see it open sourced, but sad to see the development slow down. great to see he is still pushing cap'n proto forward!

studentrob 5y ago
You mean sandstorm? https://sandstorm.io/ https://sandstorm.io/ Edit: oh you do mean sandcats.. https://docs.sandstorm.io/en/latest/administering/sandcats/ https://docs.sandstorm.io/en/latest/administering/sandcats/

hamandcheese 5y ago

Aside from the speed benefits, are there any other major advantages of Cap’n Proto? I’ve been using grpc/protobuf with Java and Scala. Overall I like it, but the grpc Java library does feel like it comes with a lot of opinions baked in. As an example, I was looking in to implementing automatic retries on the client side for certain classes of error. Grpc Java has some experimental retry support, but it seemed tightly coupled to their also-opinionated way of doing service discovery. I can see why you might want the server to inform clients how to do retries, keepalives, etc. But for less sophisticated operations (read: me) it is frustrating and over complicated. Why can’t I just specify a retry policy when creating a client stub? Still, it’s better than anything else I’ve seen. But I do long for a simpler alternative. Is that Cap’n proto? Seems like for Java there is only 3rd party support for serialization only, and not RPC.

richieartoul 5y ago
I also agree that the gRPCs libraries (in all languages) tend to be opinionated, and most annoyingly, opinionated about the wrong things. That said I also haven’t found anything better either and the technology is popular enough that I stick with it and just work around the limitations. For your specific case I highly recommend ignoring the built in GRPC retry nonsense and using the Java failsafe library retry policies instead. That has worked well for my team at least.
- chii 5y ago
  > annoyingly, opinionated about the wrong things. what's what opinions are - they necessarily must be wrong to some people. Otherwise, it'd just be called facts!
  - setr 5y ago
    Having an opinion is not the same as having a good opinion. There’s a range of quality to the matter
    - edoceo 5y ago
      The point is that "good" and "quality" is subjective and highly dependent on use-case.
      - setr 5y ago
        And my point is that it’s not — the starting pieces may differ, and thus the conclusions, but there’s clearly a qualitative difference between the opinion of an expert, and the opinion of a novice. If only that the expert brings far more knowledge into his opinion than the novice. But also in that his opinion is of better foundation, of more supporting infrastructure, of more stable reasoning, and can in general be both better explained and better defended. Opinions don’t exist in a void — they’re still the outcome of argument, and can easily be compared by that ruler. And in this case, it’s trivial to imagine utterly terrible opinionated outcomes — use hyphen to represent addition, having functions like c’s get() (impossible to use correctly, and having defaults that are backwards in the common case. It’s also trivial to imagine what a poorly reasoned opinion might look like: “ what's what opinions are - they necessarily must be wrong to some people. Otherwise, it'd just be called facts!”
  - ithkuil 5y ago
    Some try to confuse this aspect by calling opinions "best practices"
mixedCase 5y ago
>are there any other major advantages of Cap’n Proto? Parametric type polymorphism AKA generics. I regularly think of Cap'n Proto when working with gRPC or GraphQL, despite not ever having had the chance to use it in a real project.
asimpletune 5y ago
The actual language itself is very nice https://capnproto.org/language.html https://capnproto.org/language.html, and it has a bunch of really great security features. I actually think of it the other way around and don't understand why it's not used more.
- jchw 5y ago
  On paper I agree. However it does have a couple of problems: - It has yet to escape the ZeroVer versioning scheme and produce a “stable” API. - Maturity for languages other than C++ can be a bit spotty. - The RPC protocol is cool, but apparently most of it was never implemented. The C++ implementation only implements “level 1.” Cap’n Proto is undeniably cooler than Protobuf, although in practice I worry primarily about maturity and support.
  - aseipp 5y ago
    It's fine if most implementations of RPC are Level 1, even the main C++ bindings; due to the design of the protocol there only needs to exist a single Level 4 implementation written in any language, and all other implementations can then piggyback on it for their own needs beyond Level 1 (so cap proxying, etc. all come "for free" once someone does that.) I've mentioned Level 4 support to Kenton a while back and the basic answer was "I haven't needed it so it's not done." It's not intended for there to be a Level 4 implementation in every language or anything. The biggest issue is undeniably that the language maturity isn't quite as solid as the alterntaives. You've got C++, Rust, Python, C#, maybe one or two others. Only C++ is first class. So gRPC and Protobuf win here. But the actual RPC design and IDL design of Cap'n is solid and way more well engineered than the alternatives, IMO. I have no opinions on the ZeroVer thing. Cap'n Proto is very stable in my experience but I see why others would raise their eyebrows.
    - heavenlyblue 5y ago
      Python implementation is super low quality. A lot of their tests were checking for Exception raises and were basically testing AttributeError due to a change in interface instead of the initial error they were testing for.
tyingq 5y ago
>are there any other major advantages of Cap’n Proto There's a list at https://capnproto.org/ https://capnproto.org/ under "other advantages".
caturopath 5y ago
The 'cap' in Cap'n Proto is for https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security -- Cap'n Proto isn't actually an RPC system, it's a capabilities framework. You can work with results without actually having the results, e.g. before the results exist.
- dpwm 5y ago
  > You can work with results without actually having the results, e.g. before the results exist. This results in cool things like promise pipelining. It is very interesting how it can simplify interface design, and there are examples on the RPC page[0]. [0] https://capnproto.org/rpc.html https://capnproto.org/rpc.html
- hamandcheese 5y ago
  > The 'cap' in Cap'n Proto is for https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security This seems to go completely unmentioned on the home page, other than the word “capability system” in the very first sentence. What does capability based security mean for a data interchange format? The notion hardly makes any sense to me. From the homepage: “Think JSON, except binary.” What would JSON look like with capability-based security? > You can work with results without actually having the results, e.g. before the results exist This sounds exactly like promises, or perhaps a functional effect system. What does this have to do with data serialization?
  - lucozade 5y ago
    Cap'n Proto has a remote object system using interface references. When a reference is constructed, only the creator has the capability of invoking the interface. If the reference is then passed to another server, they and only they have the capability of invoking the interface. That's what they mean by capability.
  - c-cube 5y ago
    It's a promise you can send to other processes, even to the machine that is supposed to fulfill that promise. That allows batching of calls that depend on one another.
PhilippGille 5y ago
> But I do long for a simpler alternative If you're happy with protobuf but just not with gRPC, you can check out Twirp [1] and DRPC [2]. Both aim to be simpler alternatives while keeping protobuf for serialization. Their development is Go focused though. [1] https://github.com/twitchtv/twirp https://github.com/twitchtv/twirp [2] https://github.com/storj/drpc/ https://github.com/storj/drpc/
moonchrome 5y ago
If you're looking for a binary serialisation library that generates simpler/more readable code and doesn't do parsing look into flatbuffers. Going through generated protobuff code in . NET last time I used it was vomit inducing. Flatbuffers don't provide RPC AFAIK so not comparable with gRPC Capnproto tried to do too much with distributed objects and RPC, I was looking into it years ago but it still seems like the C++ impl is the only one that supports those advanced concepts.
- jiggawatts 5y ago
  Apparently at C# implementation supports promise pipelining: https://github.com/c80k/capnproto-dotnetcore#features https://github.com/c80k/capnproto-dotnetcore#features It is missing many nice-to-have features however and is also apparently unmaintained, with the last release 16 months ago...
  - naasking 5y ago
    No activity doesn't necessarily mean unmaintained, it could just mean usable and stable/bug-free.
- Communitivity 5y ago
  For your own sanity, please stay away from Flatbuffers. The key here is 'doesn't do parsing'. I used it, and wound up writing a ton of extract code to unpack things in the way I wanted. Behind the scenes everything in FlatBuffers is stored in a byte array or byte buffer. It can be made to work, but protobuffs worked better for me. Caveats: 1) I needed to work in C++ and Java 2) I was transporting serialized messages over ZeroMQ
  - kortex 5y ago
    At one point I dug a bit into the python impl of Flatbuffers, and while I forget the exact details, I was struck by how inefficient it looked. Basically every accessor call requires parsing/casting (into python) bytes into objects, without any awareness of whether it had been parsed before.

lilyball 5y ago

What will it take before it’s declared as 1.0?

Tempest1981 5y ago
Maybe never, if ZeroVer is the new hotness... https://news.ycombinator.com/item?id=28154187 https://news.ycombinator.com/item?id=28154187
kentonv 5y ago
There's a change I want to make to the kj::AsyncOutputStream interface, to make it have an explicit `end()` method rather than assuming EOF on destruction. That's backwards-incompatible, so I don't want to declare 1.0 until I get around to doing it... Also, I need to make a web site for KJ.

PostThisTooFast 5y ago

Why is this better than flatbuffers?

rawoke083600 5y ago

Very good. Where does the "speed increase" comes from ? Better encoding scheme or better decoding implementation ?

the_biot 5y ago
It's right there on the front page.

SV_BubbleTime 5y ago

Is anyone using a non-self describing schema format like this in C? I used Protobufs for awhile and really didn’t like the C options.