5 ms·

Damned if you're not the most patient self-advocate for an open source project I've ever seen. I feel like I can always count on finding The Kenton Varda in the

Damned if you're not the most patient self-advocate for an open source project I've ever seen. I feel like I can always count on finding The Kenton Varda in the comments whenever a protobuf discussion comes up, graciously discussing design trade-offs and dispelling misconceptions. Did you ever consider looking for official / commercial backing for Cap N Proto? It seems demonstrably better than all the alternatives, but I get the sense that it hasn't taken over because "nobody ever got fired for buying IBM". I know for certain I'd have an easier time pushing capnp in my own organization if it was getting that type of backing.

kentonv 6y ago
Hah, thanks. > Did you ever consider looking for official / commercial backing for Cap N Proto? Well, I don't think Cap'n Proto is marketable (for any kind of profit) on its own. It's one of those things that has to be free and open source to get any adoption. So a corporate backer would need to get some indirect benefit from its development and adoption. Also, at this point my goal in building and maintaining Cap'n Proto is primarily to benefit my other projects that use it -- if other people benefit too, great, but wide adoption of Cap'n Proto is not an intrinsic goal of mine. Right now, Cap'n Proto development is de facto backed by Cloudflare, as we are using it heavily in Cloudflare Workers and that is driving all the recent work on the C++ implementation. Currently, this use is internal-only and so we have little reason to build out support for multiple languages. But, as the Workers platform grows the ability to support increasingly complex apps and distributed systems (especially with Durable Objects), limiting applications to communicating with HTTP only is getting awkward. One idea that has been tossed around is to expose Cap'n Proto directly to apps. It makes a lot of sense: it would be very easy for us to support since we already use it under the hood, and zero-copy communications make a ton of sense especially for worker-to-worker comms happening within a single process. If we decided to do this, Cloudflare would become the commercial backer. But, there's also a strong argument to support gRPC directly in Workers, given the existing ecosystem. Would we want to support both? Maybe, maybe not. There's a lot of trade-offs still to think about here, and my goal would be to make the best possible product decision for Cloudflare Workers -- not necessarily for Cap'n Proto.
- sebastos 6y ago
  Thanks for the insight, that makes sense! Perhaps this is too unhip, but I wonder - have you ever dealt with the OMG's Data Distribution Service (DDS)? I used to think it was too fuddy-duddy to even glance at, but the recent adoption of DDS by ROS2 made me take another look. DDS seems to have a lot of interesting properties, particularly the QoS and discovery mechanisms. What made me think of it is that DDS is really a free and open standard, but there are a number of commercial entities making money off of it. They provide DDS implementations for free, and then they make money by offering professional consulting and support. Maybe a Cloudflare-backed capnp could end up looking the same way. Anyway, I'm sure you have your head around the possibilities and tradeoffs, not sure why I'm rambling.
  - kentonv 6y ago
    Eh, I'm not a big fan of the "provide paid support for open tech" model... it doesn't scale well, and it seems like it creates a perverse incentive to make the tech hard to use, to generate contracts.
    - sebastos 6y ago
      Yeah, I see your point, and I have to admit that it's been kind of a turnoff as I've been considering DDS for my own company. The other thing it does is create incentives to create optional paid add-ons. Then you find the support company talking out of both sides of their mouth, because they're trying to sell you on how great it is that the underlying standard is free and open, but simultaneously trying to sell you on how their paid, proprietary add-on is absolutely crucial. It creates an unease, where you're never sure where the border between paid and free will lie.
- mintyc 6y ago
  Cap'n'proto has a number of drawbacks both from the 'one man band' pov. (Witness the two year lull when Kenton went sandstorming), an awkward API and limited language support. Personally I don't think there is a perfect protocol because different people want different things whether self describing, easy/optimised memory management, zero copy, partial decode. The list goes on.... At a pinch, flatbuffers with flexbuffer evolution would be close to my goals but I'd much prefer having a meta description of messages and perhaps access ,authentication, transport security and use that (e.g. an OpenAPI v3.1 spec) to generate an implementation whether in protobuf, msgpack, JSON, ASN1 etc. whichever is suitable for a use case and using an appropriate transport whether quic, TCP, UDP. Some of the high performance work I've seen uses ASN.1 on a very large virtual server at 100Gb line rates because the messages lend themselves to parallel decode. I think Mike Acton had it right by suggesting things are tailored to the data needs and not overgeneralised.

kentonv 6y ago

Hah, thanks. > Did you ever consider looking for official / commercial backing for Cap N Proto? Well, I don't think Cap'n Proto is marketable (for any kind of profit) on its own. It's one of those things that has to be free and open source to get any adoption. So a corporate backer would need to get some indirect benefit from its development and adoption. Also, at this point my goal in building and maintaining Cap'n Proto is primarily to benefit my other projects that use it -- if other people benefit too, great, but wide adoption of Cap'n Proto is not an intrinsic goal of mine. Right now, Cap'n Proto development is de facto backed by Cloudflare, as we are using it heavily in Cloudflare Workers and that is driving all the recent work on the C++ implementation. Currently, this use is internal-only and so we have little reason to build out support for multiple languages. But, as the Workers platform grows the ability to support increasingly complex apps and distributed systems (especially with Durable Objects), limiting applications to communicating with HTTP only is getting awkward. One idea that has been tossed around is to expose Cap'n Proto directly to apps. It makes a lot of sense: it would be very easy for us to support since we already use it under the hood, and zero-copy communications make a ton of sense especially for worker-to-worker comms happening within a single process. If we decided to do this, Cloudflare would become the commercial backer. But, there's also a strong argument to support gRPC directly in Workers, given the existing ecosystem. Would we want to support both? Maybe, maybe not. There's a lot of trade-offs still to think about here, and my goal would be to make the best possible product decision for Cloudflare Workers -- not necessarily for Cap'n Proto.

sebastos 6y ago
Thanks for the insight, that makes sense! Perhaps this is too unhip, but I wonder - have you ever dealt with the OMG's Data Distribution Service (DDS)? I used to think it was too fuddy-duddy to even glance at, but the recent adoption of DDS by ROS2 made me take another look. DDS seems to have a lot of interesting properties, particularly the QoS and discovery mechanisms. What made me think of it is that DDS is really a free and open standard, but there are a number of commercial entities making money off of it. They provide DDS implementations for free, and then they make money by offering professional consulting and support. Maybe a Cloudflare-backed capnp could end up looking the same way. Anyway, I'm sure you have your head around the possibilities and tradeoffs, not sure why I'm rambling.
- kentonv 6y ago
  Eh, I'm not a big fan of the "provide paid support for open tech" model... it doesn't scale well, and it seems like it creates a perverse incentive to make the tech hard to use, to generate contracts.
  - sebastos 6y ago
    Yeah, I see your point, and I have to admit that it's been kind of a turnoff as I've been considering DDS for my own company. The other thing it does is create incentives to create optional paid add-ons. Then you find the support company talking out of both sides of their mouth, because they're trying to sell you on how great it is that the underlying standard is free and open, but simultaneously trying to sell you on how their paid, proprietary add-on is absolutely crucial. It creates an unease, where you're never sure where the border between paid and free will lie.
mintyc 6y ago
Cap'n'proto has a number of drawbacks both from the 'one man band' pov. (Witness the two year lull when Kenton went sandstorming), an awkward API and limited language support. Personally I don't think there is a perfect protocol because different people want different things whether self describing, easy/optimised memory management, zero copy, partial decode. The list goes on.... At a pinch, flatbuffers with flexbuffer evolution would be close to my goals but I'd much prefer having a meta description of messages and perhaps access ,authentication, transport security and use that (e.g. an OpenAPI v3.1 spec) to generate an implementation whether in protobuf, msgpack, JSON, ASN1 etc. whichever is suitable for a use case and using an appropriate transport whether quic, TCP, UDP. Some of the high performance work I've seen uses ASN.1 on a very large virtual server at 100Gb line rates because the messages lend themselves to parallel decode. I think Mike Acton had it right by suggesting things are tailored to the data needs and not overgeneralised.