3 ms·

I always liked the idea of capnp, but it bothers me that what is ultimately a message encoding protocol has an opinion on how I should architect my server. FWI

I always liked the idea of capnp, but it bothers me that what is ultimately a message encoding protocol has an opinion on how I should architect my server. FWIW, gRPC certainly has this problem too, but it’s very clearly distinct from protobuf, although pb has gRPC-related features. That entanglement makes me lean towards flatbuffers or even protobuf every time I weigh them against capnp, especially since it means that fb and pb have much simpler implementations, and I place great value on simplicity for both security and maintenance reasons. I think the lack of good third-party language implementations speaks directly to the reasonability of that assessment. It also makes the bus factor and longevity story very poor. Simplicity rules.

insanitybit 3y ago
How does the serialization layer impact your rpc choice?
- cmrdporcupine 3y ago
  Cap'N'Proto comes with a (quite good) RPC facility. Based on asynchronous promises and grounded in capabilities. You don't have to use it. You could just use it just as a 'serialization' layer but if you're writing services you could be missing half the advantage, really. And if you're writing in C++ you'll end up having to use their KJ library anyways. If you take the whole package the zero copy, capability-security, and asynchrouny (a word I just coined!) all fit together nicely.
  - insanitybit 3y ago
    Yeah I'm aware of all of that. What I'm saying is that I don't see what about the Schema Definition Language pushes you towards the RPC other than that they obviously go well together, just like gRPC is almost always used with protobuf, or http with JSON. > but it bothers me that what is ultimately a message encoding protocol has an opinion on how I should architect my server. To me, this is like saying "Using JSON is unfortunate because it has an opinion that I should use HTTP" when I don't think anyone would argue that at all, and I don't see the argument for capnp much either.
    - kentonv 3y ago
      The main thing that Cap'n Proto PRC really requires about the serialization is that object references are a first-class type. That is, when you make an RPC, the parameters or results can contain references to new, remote RPC objects. Upon receiving such a reference, that object is now callable. Making this work nicely requires some integration between the serialization layer and the RPC layer, though it's certainly possible to imagine Protobuf being extended with some sort of hooks for this.
  - ajkjk 3y ago
    Asynchrony?
cmrdporcupine 3y ago
Part of the problem with cap'n'proto whenever I've approached it is that not only does it have an opinion on how to architect your server (fine, whatever) but in C++ it ends up shipping with its own very opinionated alternative to the STL ("KJ") and when I played with it some years ago it really ended up getting its fingers everywhere and was hard to work into an existing codebase. The Rust version also comes with its own normative lifestyle assumptions; many of which make sense in the context of its zero-copy world but still make a lot of things hard to express, and the documentation was hard to parse. I tend to reach for flatbuffers instead, for this reason alone. Still I think someday I hope to have need and use for cap'n'proto; or at least finish one of several hobby projects I've forked off to try to use it over the years. There's some high quality engineering there.
- kentonv 3y ago
  Yes, it's true, the C++ implementation has become extremely opinionated. I didn't initially intend for KJ to become as all-encompassing as it has. I guess I kept running into things that didn't work well about the standard library, so I'd make an alternative that worked well, but then other parts of the standard library would not play nicely with my alternative, so it snowballed a bit. At the time the project started, C++11 -- which completely changed the language -- was brand new, and the standard library hadn't been updated to really work well with the new features. The KJ Promise library in particular, which made asynchronous programming much nicer using the newly-introduced lambdas, predated any equivalent landing in the standard library by quite a bit. This is probably the most opinionated part of KJ, hardest to integrate with other systems. (Though KJ's even loop does actually have the ability to sit on top of other event loops, with some effort.) And then I ended up with a complete ecosystem of libraries on top of Promises, like KJ HTTP. With the Workers Runtime being built entirely in that ecosystem, it ends up making sense for me to keep improving that ecosystem, rather than try to make things work better across ecosystems... so here we are.
  - cmrdporcupine 3y ago
    Oh I understand completely how that would happen. I believe the first time I played with your work was not long after the C++11 transition, and so I could see why it happened. This is why these days I just work in Rust :-) Less heterogenous of an environment (so far).

insanitybit 3y ago

How does the serialization layer impact your rpc choice?

cmrdporcupine 3y ago
Cap'N'Proto comes with a (quite good) RPC facility. Based on asynchronous promises and grounded in capabilities. You don't have to use it. You could just use it just as a 'serialization' layer but if you're writing services you could be missing half the advantage, really. And if you're writing in C++ you'll end up having to use their KJ library anyways. If you take the whole package the zero copy, capability-security, and asynchrouny (a word I just coined!) all fit together nicely.
- insanitybit 3y ago
  Yeah I'm aware of all of that. What I'm saying is that I don't see what about the Schema Definition Language pushes you towards the RPC other than that they obviously go well together, just like gRPC is almost always used with protobuf, or http with JSON. > but it bothers me that what is ultimately a message encoding protocol has an opinion on how I should architect my server. To me, this is like saying "Using JSON is unfortunate because it has an opinion that I should use HTTP" when I don't think anyone would argue that at all, and I don't see the argument for capnp much either.
  - kentonv 3y ago
    The main thing that Cap'n Proto PRC really requires about the serialization is that object references are a first-class type. That is, when you make an RPC, the parameters or results can contain references to new, remote RPC objects. Upon receiving such a reference, that object is now callable. Making this work nicely requires some integration between the serialization layer and the RPC layer, though it's certainly possible to imagine Protobuf being extended with some sort of hooks for this.
- ajkjk 3y ago
  Asynchrony?

cmrdporcupine 3y ago

Part of the problem with cap'n'proto whenever I've approached it is that not only does it have an opinion on how to architect your server (fine, whatever) but in C++ it ends up shipping with its own very opinionated alternative to the STL ("KJ") and when I played with it some years ago it really ended up getting its fingers everywhere and was hard to work into an existing codebase. The Rust version also comes with its own normative lifestyle assumptions; many of which make sense in the context of its zero-copy world but still make a lot of things hard to express, and the documentation was hard to parse. I tend to reach for flatbuffers instead, for this reason alone. Still I think someday I hope to have need and use for cap'n'proto; or at least finish one of several hobby projects I've forked off to try to use it over the years. There's some high quality engineering there.

kentonv 3y ago
Yes, it's true, the C++ implementation has become extremely opinionated. I didn't initially intend for KJ to become as all-encompassing as it has. I guess I kept running into things that didn't work well about the standard library, so I'd make an alternative that worked well, but then other parts of the standard library would not play nicely with my alternative, so it snowballed a bit. At the time the project started, C++11 -- which completely changed the language -- was brand new, and the standard library hadn't been updated to really work well with the new features. The KJ Promise library in particular, which made asynchronous programming much nicer using the newly-introduced lambdas, predated any equivalent landing in the standard library by quite a bit. This is probably the most opinionated part of KJ, hardest to integrate with other systems. (Though KJ's even loop does actually have the ability to sit on top of other event loops, with some effort.) And then I ended up with a complete ecosystem of libraries on top of Promises, like KJ HTTP. With the Workers Runtime being built entirely in that ecosystem, it ends up making sense for me to keep improving that ecosystem, rather than try to make things work better across ecosystems... so here we are.
- cmrdporcupine 3y ago
  Oh I understand completely how that would happen. I believe the first time I played with your work was not long after the C++11 transition, and so I could see why it happened. This is why these days I just work in Rust :-) Less heterogenous of an environment (so far).