14 ms·

Cap'n Proto 1.0

720 points by kentonv 3y ago

hgsgm 3y ago
Tell me about your uses of capn proto.
- bkiran 3y ago
  I'm using Cap'N Proto in a message broker application(LcuidMQ) I'm building for serialization. It has allowed me to created client applications rather quickly. There are some quirks can be difficult to wrap your head around, but once you understand it is really solid. There are some difference between the language libraries and documentation can be lacking around those language specific solutions. I'm hoping to add blog articles and or contribute back to the example of these repositories to help future users who want to dabble. Check out my repo here for how I use it across Rust and Python, with Golang coming soon: https://github.com/lucidmq/lucidmq https://github.com/lucidmq/lucidmq
maccam912 3y ago
If any cloudflare employees end up here who helped decide on Capn Proto over other stuff (e.g. protobuf), what considerations went into that choice? I'm curious if the reasons will be things important to me, or things that you don't need to worry about unless you deal with huge scale.
- matlin 3y ago
  The lead dev of Cloudflare workers is the creator of Cap'n Proto so that likely made it an easy choice
  - mikesurowiec 3y ago
    The post states "they used Cap’n Proto before they hired me"
    - oxygen_crisis 3y ago
      He helped build the Workers platform after they hired him.
  - saghm 3y ago
    The article says they were using it before hiring him though, so there must have been some prior motivation: > In fact, you are using Cap’n Proto right now, to view this site, which is served by Cloudflare, which uses Cap’n Proto extensively (and is also my employer, although they used Cap’n Proto before they hired me)
- coolsunglasses 3y ago
  I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects. If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like. So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short. https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here. Cap'n Proto was originally made for https://sandstorm.io/ https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers. Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-215203333 https://github.com/google/flatbuffers/issues/2#issuecomment-...
  - hgsgm 3y ago
    Aside from CF Workers using capn proto, how is it related to capn proto or sandstorm?
    - kentonv 3y ago
      They are all projects I started. But other than who worked on them, and sharing some technology choices under the hood, there's mostly no relationship between Workers and Sandstorm.
- kentonv 3y ago
  Here's a blog post about Cloudflare's use of Cap'n Proto in 2014, three years before I joined: https://blog.cloudflare.com/introducing-lua-capnproto-better-serialization-in-lua/ https://blog.cloudflare.com/introducing-lua-capnproto-better... To this day, Cloudflare's data pipeline (which produces logs and analytics from the edge) is largely based on Cap'n Proto serialization. I haven't personally been much involved with that project. As for Cloudflare Workers, of course, I started the project, so I used my stuff. Probably not the justification you're looking for. :) That said, I would argue the extreme expressiveness of Cap'n Proto's RPC protocol compared to alternatives has been a big help in implementing sandboxing in the Workers Runtime, as well as distributed systems features like Durable Objects. https://blog.cloudflare.com/introducing-workers-durable-objects/ https://blog.cloudflare.com/introducing-workers-durable-obje...
- hblanks 3y ago
  To summarize something from a little over a year after I joined there: Cloudflare was building out a way to ship logs from its edge to a central point for customer analytics and serving logs to enterprise customers. As I understood it, the primary engineer who built all of that out, Albert Strasheim, benchmarked the most likely serialization options available and found Cap'n Proto to be appreciably faster than protobuf. It had a great C++ implementation (which we could use from nginx, IIRC with some lua involved) and while the Go implementation, which we used on the consuming side, had its warts, folks were able to fix the key parts that needed attention. Anyway. Cloudflare's always been pretty cost efficient machine wise, so it was a natural choice given the performance needs we had. In my time in the data team there, Cap'n Proto was always pretty easy to work with, and sharing proto definitions from a central schema repo worked pretty well, too. Thanks for your work, Kenton!
  - fullung 3y ago
    Albert here :) Decision basically came down to the following: we needed to build a 1 million events/sec log processing pipeline, and had only 5 servers (more were on the way, but would take many months to arrive at the data center). So v1 of this pipeline was 5 servers, each running Kafka 0.8, a service to receive events from the edge and put it into Kafka, and a consumer to aggregate the data. To squeeze all of this onto this hardware footprint, I spent about a week looking for a format that optimized for deserialization speed, since we had a few thousand edge servers serializing data, but only 5 deserializing. Capnproto was a good fit :)
    - kentonv 3y ago
      And that's how I first learned about Cloudflare. Which I eventually joined, and built Cloudflare Workers. :)
Timon3 3y ago
Congrats on the release! It must be very exciting after 10 years :) If you don't mind the question: will there be more work on implementations for other languages in the future? I really like the idea of the format, but the main languages in our stack aren't supported in a way I'd use in a product.
- kentonv 3y ago
  This is indeed the main weakness of Cap'n Proto. I only really maintain the C++ implementation. Other implementations come from various contributors which can lead to varying levels of completeness and quality. Unfortunately I can't really promise anything new here. My work on Cap'n Proto is driven by the needs of my main project, the Cloudflare Workers runtime, which is primarily C++. We do interact with Go and Rust services, and the respective implementations seem to get the job done there. Put another way, Cap'n Proto is an open source project, and I hope it is useful to people, but it is not a product I'm trying to sell, so I am not particularly focused on trying to get everyone to adopt it. As always, contributions are welcome. The one case where I might foresee a big change is if we (Cloudflare) decided to make Cap'n Proto be a public-facing feature of the Workers platform. Then we'd have a direct need to really polish it in many languages. That is certainly something we discuss from time to time but there are no plans at present.
  - Timon3 3y ago
    That's completely understandable, thank you for the answer! I'd love to try and help with at least one implementation for those languages, but there's a good chance that it would end up like the existing implementations due to lack of time. Anyway, thank you for making it open source and for working on it all this time!
  - thegagne 3y ago
    > if we (Cloudflare) decided to make Cap'n Proto be a public-facing feature of the Workers platform. How likely is this? What would be the benefits and use-cases of doing this? Would it be a standardized JS offering, or something specific to Workers that is deserialized before it hits the runtime?
CodesInChaos 3y ago
I find it surprising how few protocols (besides Cap'n Proto) have promise pipelining. The only other example I can think of is 9p, but that's not a general purpose protocol. https://capnproto.org/news/2013-12-13-promise-pipelining-capnproto-vs-ice.html https://capnproto.org/news/2013-12-13-promise-pipelining-cap...
- giovannibonetti 3y ago
  Redis transactions [1] also apply pipelining, but AFAICT there is no practical way to use them for implementing generic RPC. [1] https://redis.com/ebook/part-2-core-concepts/chapter-4-keeping-data-safe-and-ensuring-performance/4-4-redis-transactions/ https://redis.com/ebook/part-2-core-concepts/chapter-4-keepi...
  - ackfoobar 3y ago
    Does the pipelining in Redis allow you to have the second command depend on the result of the first command?
    - byroot 3y ago
      No, but for that use case you have EVAL which execute an arbitrary lua script on the server. https://redis.io/commands/eval/ https://redis.io/commands/eval/
      - jauntywundrkind 3y ago
        That assumes you know & can generate the complete pipeline ahead of time. The elegance of promise pipelining is that your pipeline can also be asynchronously grown.
- jayd16 3y ago
  As neat as it is I guess it's hard optimize the backend for it compared to explicitly grouping the queries. I imagine a bespoke RPC call that results in a single SQL query is better than several pipelined but separate RPC calls, for example. But even still, you would think it would be more popular.
dannyobrien 3y ago
I'm excited by Cap'n Proto's participation in the OCAPN standardization effort. Can you speak to if that's going to be part of the Cap'n Proto 2.0 work? https://github.com/ocapn/ocapn https://github.com/ocapn/ocapn
- kentonv 3y ago
  Sadly, the person leading that participation, Ian "zenhack" Denhardt, recently and unexpectedly passed away. For my part, I'm a fan of OCapN, but I am not sure how much time I can personally commit to it, with everything on my plate. I wish I had better news here. This was a tragic loss for all of us.
  - the_common_man 3y ago
    That's tragic. Was he also not helping maintain sandstorm? Would appreciate a blog post or a note about him there
    - kentonv 3y ago
      Yes, he was the most active developer over the last few years, although it wasn't a huge amount of activity. And that activity had dropped off this year as Ian shifted his focus to Tempest, a mostly-from-scratch rewrite. https://github.com/zenhack/tempest https://github.com/zenhack/tempest For my part I stopped pushing monthly Sandstorm updates this year as there hasn't really been anything to push. Unfortunately Sandstorm's biggest dependencies can't even be updated anymore because of breaking changes that would take significant effort to work around. I agree a blog post is probably in order.
      - ocdtrekkie 3y ago
        I have thought a bit on this but I have also been pretty much just been recovering lately. We will probably need to assemble a blog post at some point soonish, but we need to talk to a few people first.
batch12 3y ago
I know this isn't new, but I wonder if the name is an intentional nod to Star Trek Voyager or is there another reference I'm not aware of. https://memory-alpha.fandom.com/wiki/Captain_Proton https://memory-alpha.fandom.com/wiki/Captain_Proton
- deleted 3y ago
  [deleted]
- azornathogron 3y ago
  Given that it's billed as a "cerealization protocol", I always assumed it was a reference to Cap'n Crunch cereal.
- kentonv 3y ago
  Huh, that reference actually never occurred to me. The name Cap'n Proto actually originally meant "Capabilities and Protobufs" -- it was a capability-based RPC protocol based on Protocol Buffers. However, early on I decided I wanted to try a whole different serialization format instead. "Proto" still makes sense, since it is a protocol, so I kept the name. The pun "cerealization protocol" is actually something someone else had to point out to me, but I promptly added it to the logo. :)
  - mi_lk 3y ago
    What does capacity-based mean in this context?
    - kentonv 3y ago
      Capability, not capacity. https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security https://capnproto.org/rpc.html#distributed-objects https://capnproto.org/rpc.html#distributed-objects The idea really goes way beyond security and RPC. It's hard to explain concisely but it's sort of a way of thinking about software architecture.
IshKebab 3y ago
Great achievement. To be honest I wouldn't recommend Capnp. The C++ API is very awkward. The zero copy parsing is less of a benefit than you'd expect - pretty unlikely you're going to want to keep your data as a Capnp data structure because of how awkward it is to use. 99% of the time you'll just copy it into your own data structures anyway. There's also more friction with the rest of the world which has more or less settled on Protobuf as the most popular binary implementation of this sort of idea. I only used it for serialisation. Maybe the RPC stuff is more compelling. I really wish Thrift had taken off instead of Protobuf/gRPC. It was so much better designed and more flexible than anything I've seen before or since. I think it died mainly due to terrible documentation. I guess it also didn't have a big name behind it.
- mgaunard 3y ago
  You mean flatbuffers, not protobuf. It has established itself as the de-facto standard, with a few other places using SBE instead. In any case the main problems with binary serialization are: - schemas and message version management - delta-encoding If you ignore these, flat binary serialization is trivial. No library provides a good solution that covers the two points above.
  - kentonv 3y ago
    What part of the industry are you in where flatbuffers is seen as the de facto standard? Personally I've never randomly encountered a project using flatbuffers. I see protobuf all the time. (I've randomly run into Cap'n Proto maybe 2-3 times but to be fair I'm probably more likely to notice that.)
    - ynx 3y ago
      As of the last time I was close, flatbuffer usage is or was near ubiquitous for use in FB's (ha ha, go figure) mobile apps, across Android and iOS at least.
    - cmrdporcupine 3y ago
      Flatbuffers seems to have penetration in the games industry. And it sounds like from other posters that Facebook uses it. I recently started a job doing work on autonomy systems that run in tractors, and was surprised to see we use it (flatbuffers) in the messaging layer (in both C++ and Rust)
s17n 3y ago
It’s a testament to the subtlety of software engineering that even after four tries (protobuf 1-3, capn proto 1) there are still breaking changes that need to be made to the solution of what on the surface appears to be a relatively constrained problem.
- kentonv 3y ago
  Of course, nothing is ever "solved". :) I assume you are talking about the cancellation change. This is interesting, actually. When originally designing Cap'n Proto, I was convinced by a capabilities expert I talked to that cancellation should be considered dangerous, because software that isn't expecting it might be vulnerable to attacks if cancellation occurs at an unexpected place. Especially in a language like C++, which lacks garbage collection or borrow checking, you might expect use-after-free to be a big issue. I found the argument compelling. In practice, though, I've found the opposite: In a language with explicit lifetimes, and with KJ's particular approach to Promises (used to handle async tasks in Cap'n Proto's C++ implementation), cancellation safety is a natural side-effect of writing code to have correct lifetimes. You have to make cancellation safe because you have to cancel tasks all the time when the objects they depend on are going to be destroyed. Moreover, in a fault-tolerant distributed system, you have to assume any code might not complete, e.g. due to a power outage or maybe just throwing an unexpected exception in the middle, and you have to program defensively for that anyway. This all becomes second-nature pretty quick. So all our code ends up cancellation-safe by default. We end up with way more problems from cancellation unexpectedly being prevented when we need it, than happening when we didn't expect it. EDIT: Re-reading, maybe you were referring to the breaking changes slated for 2.0. But those are primarily changes to the KJ toolkit library, not Cap'n Proto, and is all about API design... I'd say API design is not a constrained problem.
binary132 3y ago
I always liked the idea of capnp, but it bothers me that what is ultimately a message encoding protocol has an opinion on how I should architect my server. FWIW, gRPC certainly has this problem too, but it’s very clearly distinct from protobuf, although pb has gRPC-related features. That entanglement makes me lean towards flatbuffers or even protobuf every time I weigh them against capnp, especially since it means that fb and pb have much simpler implementations, and I place great value on simplicity for both security and maintenance reasons. I think the lack of good third-party language implementations speaks directly to the reasonability of that assessment. It also makes the bus factor and longevity story very poor. Simplicity rules.
- insanitybit 3y ago
  How does the serialization layer impact your rpc choice?
  - cmrdporcupine 3y ago
    Cap'N'Proto comes with a (quite good) RPC facility. Based on asynchronous promises and grounded in capabilities. You don't have to use it. You could just use it just as a 'serialization' layer but if you're writing services you could be missing half the advantage, really. And if you're writing in C++ you'll end up having to use their KJ library anyways. If you take the whole package the zero copy, capability-security, and asynchrouny (a word I just coined!) all fit together nicely.
    - insanitybit 3y ago
      Yeah I'm aware of all of that. What I'm saying is that I don't see what about the Schema Definition Language pushes you towards the RPC other than that they obviously go well together, just like gRPC is almost always used with protobuf, or http with JSON. > but it bothers me that what is ultimately a message encoding protocol has an opinion on how I should architect my server. To me, this is like saying "Using JSON is unfortunate because it has an opinion that I should use HTTP" when I don't think anyone would argue that at all, and I don't see the argument for capnp much either.
dtech 3y ago
While I never used Cap'n Proto, I want to thank kentonv for the extremely informative FAQ answer [1] on why required fields are problematic in a protocol I link it to people all the time, especially when they ask why protobuf 3 doesn't have required fields. [1] https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require...
- kccqzy 3y ago
  This is some very valuable perspective. Personally, I previously also struggled to understand why. For me, the thing that clicked was to understand protobuf and Cap'n proto as serialization formats that need to work across API boundaries and need to work with different versions of their schema in a backwards- and forwards-compatible way; do not treat them as in-memory data structures that represent the world from the perspective of a single process running a single version without no compatibility concerns. Thus, the widely repeated mantra of "making illegal states unrepresentable" does not apply.
  - chubot 3y ago
    Rich Hickey (creator of the Clojure language) has a good talk "Maybe Not" that touches on these issues, with a nice way of explaining it https://www.youtube.com/watch?v=YR5WdGrpoug https://www.youtube.com/watch?v=YR5WdGrpoug The capnproto link explains it concretely in terms of a message bus example, which is useful. But more abstractly you can think of the shape of data (aka schema, names and types) and field presence (optional/required) as separate things https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey First, being valid or invalid with respect to a static type system is a GLOBAL property of program -- writing a type checker will convince you of that. And big distributed systems don't have such global properties: https://news.ycombinator.com/item?id=36590799 https://news.ycombinator.com/item?id=36590799 If they did, they'd be small :) Namely you could just reboot the whole thing at once. You can't reboot say the entire Internet at once, and this also holds for smaller systems, like the ones at say Google (and I'm sure Cloudflare, etc.). So the idea is that the shape/schema is a GLOBAL property -- you never want two messages called foo.RequestX or two fields called "num_bar" with different types -- ever, anywhere. But optional/required is LOCAL property. It depends on what version of a schema is deployed in a particular binary. Inherently, you need to be able to handle a mix of inconsistent versions running simultaneously. --- To be pedantic, I woulds say "making illegal states unrepresentable" DOES apply, but you can't do it in a STATIC type system. [1] Your Maybe<T> type is not useful for data that crosses process boundaries. A distributed system isn't a state machine. 1. Lamport showed us one important reason why: the relative order of messages means that there is no globally coherent state. You need something like Paxos to turn a distributed system back into a state machine (and this is very expensive in general) 2. The second reason is probably a consequence of the first. You can think of deploying a binary to a node as a message to that node. So you don't have a completely consistent state -- you always have an in-between state, a mix of versions. And presumably you want your system to keep working during this time period :) And that coarse-grained problem (code versioning and deployment) implies the fine-grained problem (whether a specific message in a field is present). This is because protobufs generate parsers with validation for you -- or they used to! --- tl;dr Think of the shape of data (aka schema) and field presence (optional/required) as different dimensions of data modeling. Maybe<T> mixes those up, which is fine in a single process, but doesn't work across processes. --- [1] A very specific example of making illegal states unrepresentable without static types - my Oils project uses a DSL for algebraic data types, borrowed from CPython. The funny thing is that in CPython, it generates C code, which doesn't have any static notion of Maybe<T>. It has tagged unions. And in Oils we first generated dynamically typed Python at first. Somewhat surprisingly, algebraic data types are STILL useful there. Now the generated code is statically typed with MyPy (and with C++), and we do pleasant type-driven refactorings. But using algebraic data types were still extremely useful before static typing. They made illegal states unrepresentable -- but you would get the error at runtime.
insanitybit 3y ago
Any plans to improve the Rust side of things? The API could definitely use some more work/ docs around it.
- dwrensha 3y ago
  I intend to continue work on capnproto-rust, at my own pace and according to my own priorities. Are there any particular pain points that you want to call attention to?
shdh 3y ago
We have a great plethora of binary serialization libraries now, but I've noticed none of them offer the following: * Specification of the number of bits I want to cap out a field at during serialization, ie: `int` that only uses 3 bits. * Delta encoding for serialization and deserialization, this would further decrease the size of each message if there is an older message that I can use as the initial message to delta encode/decode from.
- jeffbee 3y ago
  > `int` that only uses 3 bits. CBOR approximates this, since it has several different widths for integers. > an older message that I can use as the initial message to delta encode/decode from. General-purpose compression on the encoded stream would do something toward this goal, but some protocol buffers library implementations offer merge functions. The question is what semantics of "merge" you expect. For repeated fields do you want to append or clobber?
  - Dylan16807 3y ago
    CBOR has different integer widths, yes, but 1/2/3/5/9 bytes is not particularly flexible. And if someone is asking for exact bit sizes and deltas, they probably don't want a format that embeds every key in every message.
- no_circuit 3y ago
  Take a look at FAST protocol [1]. It has been around for a while. Was created for market/trading data. There appears to be some open source implementations, but I don't think in general they'd be maintained well since trading is, well, secretive. [1] https://en.wikipedia.org/wiki/FAST_protocol https://en.wikipedia.org/wiki/FAST_protocol
- IshKebab 3y ago
  Most formats use varints, so you can't have a 3-bit int but they will store a 64-bit int in one byte if it fits. Going to smaller than a byte isn't worth the extra complexity and slowness. If you're that space sensitive you need to add proper compression. By delta compression you mean across messages? Yeah I've never seen that but it's hard to imagine a scenario where it would be useful and worth the insane complexity.
synthetigram 3y ago
After exploring a few constant access serialization formats, I had to pass on Capn Proto in favor of Apache Avro. Capn has a great experience for C++ users, but Java codegen ended up being too annoying to get started with. If Capn Proto improved the developer experience for the other languages people write, I think it would really help a lot.
unixhero 3y ago
Congratulations to kentonv and the team
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
vicaya 3y ago
Worker should really adopt Apache Arrow, which has a much bigger ecosystem. https://github.com/apache/arrow https://github.com/apache/arrow
Timothycquinn 3y ago
Congrats in 10 years! Question: Can Cap'n Proto be used as an alternative to Python Pickle library for serializing and de-serializing python object structures?
- kentonv 3y ago
  If your goal is to serialize an arbitrary Python object, Pickle is the way to go. Cap'n Proto requires you to define a schema, in Cap'n Proto schema language, for whatever you wan to serialize. It can't just take an arbitrary Python value.
emtel 3y ago
I used capnproto years ago as the network serialization format for a multiplayer RTS game. Although the API can be quite awkward, it was overall a joy to use and I wish I was able to use it in more projects.
richardfey 3y ago
Amazing to see Cap'n Proto come this far! I wonder how easy it would be to swap it for gRPC, and still have advanced load balancing support for it.
hiddencost 3y ago
For context: Kenton ran Google's in house proto system for many years, before leaving and building his own open source version.
- AceJohnny2 3y ago
  I believe he was the one who open-sourced protobufs.
omginternets 3y ago
I have some very unfortunate news to share with the Cap'n Proto and Sandstorm communities. Ian Denhardt (zenhack on HN), a lead contributor to the Go implementation, suddenly and unexpectedly passed away a few weeks ago. Before making a request to the community, I want to express how deeply saddened I am by this loss. Ian and I collaborated extensively over the past three years, and we had become friends. As the de facto project lead, it now befalls me to fill Ian's very big shoes. Please, if you're able to contribute to the project, I could really use the help. And if you're a contributor or maintainer of some other implementation (C++, Rust, etc.), I would *REALLY* appreciate it if we could connect. I'm going to need to surround myself with very smart people if I am to continue Ian's work. RIP Ian, and thank you. I learned so much working with you. ------ P.S: I can be reached in the following places - https://github.com/lthibault https://github.com/lthibault - https://matrix.to/#/#go-capnp:matrix.org https://matrix.to/#/#go-capnp:matrix.org - Telegram: @lthibault - gmail: louist87
- pja 3y ago
  It seems @zenhack maintained the Haskell bindings as well.
  - omginternets 3y ago
    I think the Haskell project was a complete implementation, not just bindings. But yes, Ian was truly prolific.
- jcalabro 3y ago
  Oh gosh, I didn't know that. Thank you for sharing :( I really loved his blog. That's awful.
  - omginternets 3y ago
    Indeed. His blog was outstanding. It might be a good idea to mirror his blog before the domain expires, as it's a real treasure-trove.
jFriedensreich 3y ago
i love how the main reference for workerd can be just one capnp file. https://github.com/cloudflare/workerd/blob/main/src/workerd/server/workerd.capnp https://github.com/cloudflare/workerd/blob/main/src/workerd/... this changed my world how i think about computing on the web. if there was just a good enough js library as for lua and you could directly send capnp messages to workerd instead of always going through files. I guess one day i have to relearn c++ and understand how the internals actually work.
deleted 3y ago
[deleted]
zX41ZdbW 3y ago
We support CapNProto as an import/export format in ClickHouse. It was contributed by Marek Vavruša from CloudFlare in 2017: https://github.com/ClickHouse/ClickHouse/pull/1387 https://github.com/ClickHouse/ClickHouse/pull/1387 Since then, we extended and improved the support: added it for export (initially, it was only for import) and improved the performance. About strange stuff in the library - it uses a non-obvious approach for exception handling, and their C++ code feels like they too much focusing on some non-orthodox approaches.
Xeoncross 3y ago
[2013] Show HN: Cap'n Proto, by the ex-maintainer of Protocol Buffers: https://news.ycombinator.com/item?id=5482081 https://news.ycombinator.com/item?id=5482081
xvilka 3y ago
I am trying to revive C implementation [1]. Any help is welcome. [1] https://github.com/opensourcerouting/c-capnproto https://github.com/opensourcerouting/c-capnproto
up2isomorphism 3y ago
Some of these “high perf” RPC libraries never get one key point, if I really need something to be fast, the most important aspect is that it must be simple.
jviotti 3y ago
If you are interested in learning more about binary serialization (for Cap'n Proto and others), I wrote two free-to-read papers extensively looking at the scene from a space-efficiency point of view that people here might be interested in: - https://arxiv.org/abs/2201.02089 https://arxiv.org/abs/2201.02089 - https://arxiv.org/abs/2201.03051 https://arxiv.org/abs/2201.03051 Not something that aims to compete with Cap'n Proto (different use cases), but I've been also working on a binary serialization format that is pure JSON-compatible with a focus on space-efficiency: https://www.jsonbinpack.org https://www.jsonbinpack.org. Benchmarks here: https://arxiv.org/abs/2211.12799 https://arxiv.org/abs/2211.12799

Cap'n Proto 1.0

720 points by kentonv 3y ago

hgsgm 3y ago

Tell me about your uses of capn proto.

bkiran 3y ago
I'm using Cap'N Proto in a message broker application(LcuidMQ) I'm building for serialization. It has allowed me to created client applications rather quickly. There are some quirks can be difficult to wrap your head around, but once you understand it is really solid. There are some difference between the language libraries and documentation can be lacking around those language specific solutions. I'm hoping to add blog articles and or contribute back to the example of these repositories to help future users who want to dabble. Check out my repo here for how I use it across Rust and Python, with Golang coming soon: https://github.com/lucidmq/lucidmq https://github.com/lucidmq/lucidmq

maccam912 3y ago

If any cloudflare employees end up here who helped decide on Capn Proto over other stuff (e.g. protobuf), what considerations went into that choice? I'm curious if the reasons will be things important to me, or things that you don't need to worry about unless you deal with huge scale.

matlin 3y ago
The lead dev of Cloudflare workers is the creator of Cap'n Proto so that likely made it an easy choice
- mikesurowiec 3y ago
  The post states "they used Cap’n Proto before they hired me"
  - oxygen_crisis 3y ago
    He helped build the Workers platform after they hired him.
- saghm 3y ago
  The article says they were using it before hiring him though, so there must have been some prior motivation: > In fact, you are using Cap’n Proto right now, to view this site, which is served by Cloudflare, which uses Cap’n Proto extensively (and is also my employer, although they used Cap’n Proto before they hired me)
coolsunglasses 3y ago
I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects. If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like. So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short. https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here. Cap'n Proto was originally made for https://sandstorm.io/ https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers. Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-215203333 https://github.com/google/flatbuffers/issues/2#issuecomment-...
- hgsgm 3y ago
  Aside from CF Workers using capn proto, how is it related to capn proto or sandstorm?
  - kentonv 3y ago
    They are all projects I started. But other than who worked on them, and sharing some technology choices under the hood, there's mostly no relationship between Workers and Sandstorm.
kentonv 3y ago
Here's a blog post about Cloudflare's use of Cap'n Proto in 2014, three years before I joined: https://blog.cloudflare.com/introducing-lua-capnproto-better-serialization-in-lua/ https://blog.cloudflare.com/introducing-lua-capnproto-better... To this day, Cloudflare's data pipeline (which produces logs and analytics from the edge) is largely based on Cap'n Proto serialization. I haven't personally been much involved with that project. As for Cloudflare Workers, of course, I started the project, so I used my stuff. Probably not the justification you're looking for. :) That said, I would argue the extreme expressiveness of Cap'n Proto's RPC protocol compared to alternatives has been a big help in implementing sandboxing in the Workers Runtime, as well as distributed systems features like Durable Objects. https://blog.cloudflare.com/introducing-workers-durable-objects/ https://blog.cloudflare.com/introducing-workers-durable-obje...
hblanks 3y ago
To summarize something from a little over a year after I joined there: Cloudflare was building out a way to ship logs from its edge to a central point for customer analytics and serving logs to enterprise customers. As I understood it, the primary engineer who built all of that out, Albert Strasheim, benchmarked the most likely serialization options available and found Cap'n Proto to be appreciably faster than protobuf. It had a great C++ implementation (which we could use from nginx, IIRC with some lua involved) and while the Go implementation, which we used on the consuming side, had its warts, folks were able to fix the key parts that needed attention. Anyway. Cloudflare's always been pretty cost efficient machine wise, so it was a natural choice given the performance needs we had. In my time in the data team there, Cap'n Proto was always pretty easy to work with, and sharing proto definitions from a central schema repo worked pretty well, too. Thanks for your work, Kenton!
- fullung 3y ago
  Albert here :) Decision basically came down to the following: we needed to build a 1 million events/sec log processing pipeline, and had only 5 servers (more were on the way, but would take many months to arrive at the data center). So v1 of this pipeline was 5 servers, each running Kafka 0.8, a service to receive events from the edge and put it into Kafka, and a consumer to aggregate the data. To squeeze all of this onto this hardware footprint, I spent about a week looking for a format that optimized for deserialization speed, since we had a few thousand edge servers serializing data, but only 5 deserializing. Capnproto was a good fit :)
  - kentonv 3y ago
    And that's how I first learned about Cloudflare. Which I eventually joined, and built Cloudflare Workers. :)

Timon3 3y ago

Congrats on the release! It must be very exciting after 10 years :) If you don't mind the question: will there be more work on implementations for other languages in the future? I really like the idea of the format, but the main languages in our stack aren't supported in a way I'd use in a product.

kentonv 3y ago
This is indeed the main weakness of Cap'n Proto. I only really maintain the C++ implementation. Other implementations come from various contributors which can lead to varying levels of completeness and quality. Unfortunately I can't really promise anything new here. My work on Cap'n Proto is driven by the needs of my main project, the Cloudflare Workers runtime, which is primarily C++. We do interact with Go and Rust services, and the respective implementations seem to get the job done there. Put another way, Cap'n Proto is an open source project, and I hope it is useful to people, but it is not a product I'm trying to sell, so I am not particularly focused on trying to get everyone to adopt it. As always, contributions are welcome. The one case where I might foresee a big change is if we (Cloudflare) decided to make Cap'n Proto be a public-facing feature of the Workers platform. Then we'd have a direct need to really polish it in many languages. That is certainly something we discuss from time to time but there are no plans at present.
- Timon3 3y ago
  That's completely understandable, thank you for the answer! I'd love to try and help with at least one implementation for those languages, but there's a good chance that it would end up like the existing implementations due to lack of time. Anyway, thank you for making it open source and for working on it all this time!
- thegagne 3y ago
  > if we (Cloudflare) decided to make Cap'n Proto be a public-facing feature of the Workers platform. How likely is this? What would be the benefits and use-cases of doing this? Would it be a standardized JS offering, or something specific to Workers that is deserialized before it hits the runtime?

CodesInChaos 3y ago

I find it surprising how few protocols (besides Cap'n Proto) have promise pipelining. The only other example I can think of is 9p, but that's not a general purpose protocol. https://capnproto.org/news/2013-12-13-promise-pipelining-capnproto-vs-ice.html https://capnproto.org/news/2013-12-13-promise-pipelining-cap...

giovannibonetti 3y ago
Redis transactions [1] also apply pipelining, but AFAICT there is no practical way to use them for implementing generic RPC. [1] https://redis.com/ebook/part-2-core-concepts/chapter-4-keeping-data-safe-and-ensuring-performance/4-4-redis-transactions/ https://redis.com/ebook/part-2-core-concepts/chapter-4-keepi...
- ackfoobar 3y ago
  Does the pipelining in Redis allow you to have the second command depend on the result of the first command?
  - byroot 3y ago
    No, but for that use case you have EVAL which execute an arbitrary lua script on the server. https://redis.io/commands/eval/ https://redis.io/commands/eval/
    - jauntywundrkind 3y ago
      That assumes you know & can generate the complete pipeline ahead of time. The elegance of promise pipelining is that your pipeline can also be asynchronously grown.
jayd16 3y ago
As neat as it is I guess it's hard optimize the backend for it compared to explicitly grouping the queries. I imagine a bespoke RPC call that results in a single SQL query is better than several pipelined but separate RPC calls, for example. But even still, you would think it would be more popular.

dannyobrien 3y ago

I'm excited by Cap'n Proto's participation in the OCAPN standardization effort. Can you speak to if that's going to be part of the Cap'n Proto 2.0 work? https://github.com/ocapn/ocapn https://github.com/ocapn/ocapn

kentonv 3y ago
Sadly, the person leading that participation, Ian "zenhack" Denhardt, recently and unexpectedly passed away. For my part, I'm a fan of OCapN, but I am not sure how much time I can personally commit to it, with everything on my plate. I wish I had better news here. This was a tragic loss for all of us.
- the_common_man 3y ago
  That's tragic. Was he also not helping maintain sandstorm? Would appreciate a blog post or a note about him there
  - kentonv 3y ago
    Yes, he was the most active developer over the last few years, although it wasn't a huge amount of activity. And that activity had dropped off this year as Ian shifted his focus to Tempest, a mostly-from-scratch rewrite. https://github.com/zenhack/tempest https://github.com/zenhack/tempest For my part I stopped pushing monthly Sandstorm updates this year as there hasn't really been anything to push. Unfortunately Sandstorm's biggest dependencies can't even be updated anymore because of breaking changes that would take significant effort to work around. I agree a blog post is probably in order.
    - ocdtrekkie 3y ago
      I have thought a bit on this but I have also been pretty much just been recovering lately. We will probably need to assemble a blog post at some point soonish, but we need to talk to a few people first.

batch12 3y ago

I know this isn't new, but I wonder if the name is an intentional nod to Star Trek Voyager or is there another reference I'm not aware of. https://memory-alpha.fandom.com/wiki/Captain_Proton https://memory-alpha.fandom.com/wiki/Captain_Proton

deleted 3y ago
[deleted]
azornathogron 3y ago
Given that it's billed as a "cerealization protocol", I always assumed it was a reference to Cap'n Crunch cereal.
kentonv 3y ago
Huh, that reference actually never occurred to me. The name Cap'n Proto actually originally meant "Capabilities and Protobufs" -- it was a capability-based RPC protocol based on Protocol Buffers. However, early on I decided I wanted to try a whole different serialization format instead. "Proto" still makes sense, since it is a protocol, so I kept the name. The pun "cerealization protocol" is actually something someone else had to point out to me, but I promptly added it to the logo. :)
- mi_lk 3y ago
  What does capacity-based mean in this context?
  - kentonv 3y ago
    Capability, not capacity. https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security https://capnproto.org/rpc.html#distributed-objects https://capnproto.org/rpc.html#distributed-objects The idea really goes way beyond security and RPC. It's hard to explain concisely but it's sort of a way of thinking about software architecture.

IshKebab 3y ago

Great achievement. To be honest I wouldn't recommend Capnp. The C++ API is very awkward. The zero copy parsing is less of a benefit than you'd expect - pretty unlikely you're going to want to keep your data as a Capnp data structure because of how awkward it is to use. 99% of the time you'll just copy it into your own data structures anyway. There's also more friction with the rest of the world which has more or less settled on Protobuf as the most popular binary implementation of this sort of idea. I only used it for serialisation. Maybe the RPC stuff is more compelling. I really wish Thrift had taken off instead of Protobuf/gRPC. It was so much better designed and more flexible than anything I've seen before or since. I think it died mainly due to terrible documentation. I guess it also didn't have a big name behind it.

mgaunard 3y ago
You mean flatbuffers, not protobuf. It has established itself as the de-facto standard, with a few other places using SBE instead. In any case the main problems with binary serialization are: - schemas and message version management - delta-encoding If you ignore these, flat binary serialization is trivial. No library provides a good solution that covers the two points above.
- kentonv 3y ago
  What part of the industry are you in where flatbuffers is seen as the de facto standard? Personally I've never randomly encountered a project using flatbuffers. I see protobuf all the time. (I've randomly run into Cap'n Proto maybe 2-3 times but to be fair I'm probably more likely to notice that.)
  - ynx 3y ago
    As of the last time I was close, flatbuffer usage is or was near ubiquitous for use in FB's (ha ha, go figure) mobile apps, across Android and iOS at least.
  - cmrdporcupine 3y ago
    Flatbuffers seems to have penetration in the games industry. And it sounds like from other posters that Facebook uses it. I recently started a job doing work on autonomy systems that run in tractors, and was surprised to see we use it (flatbuffers) in the messaging layer (in both C++ and Rust)

s17n 3y ago

It’s a testament to the subtlety of software engineering that even after four tries (protobuf 1-3, capn proto 1) there are still breaking changes that need to be made to the solution of what on the surface appears to be a relatively constrained problem.

kentonv 3y ago
Of course, nothing is ever "solved". :) I assume you are talking about the cancellation change. This is interesting, actually. When originally designing Cap'n Proto, I was convinced by a capabilities expert I talked to that cancellation should be considered dangerous, because software that isn't expecting it might be vulnerable to attacks if cancellation occurs at an unexpected place. Especially in a language like C++, which lacks garbage collection or borrow checking, you might expect use-after-free to be a big issue. I found the argument compelling. In practice, though, I've found the opposite: In a language with explicit lifetimes, and with KJ's particular approach to Promises (used to handle async tasks in Cap'n Proto's C++ implementation), cancellation safety is a natural side-effect of writing code to have correct lifetimes. You have to make cancellation safe because you have to cancel tasks all the time when the objects they depend on are going to be destroyed. Moreover, in a fault-tolerant distributed system, you have to assume any code might not complete, e.g. due to a power outage or maybe just throwing an unexpected exception in the middle, and you have to program defensively for that anyway. This all becomes second-nature pretty quick. So all our code ends up cancellation-safe by default. We end up with way more problems from cancellation unexpectedly being prevented when we need it, than happening when we didn't expect it. EDIT: Re-reading, maybe you were referring to the breaking changes slated for 2.0. But those are primarily changes to the KJ toolkit library, not Cap'n Proto, and is all about API design... I'd say API design is not a constrained problem.

binary132 3y ago

I always liked the idea of capnp, but it bothers me that what is ultimately a message encoding protocol has an opinion on how I should architect my server. FWIW, gRPC certainly has this problem too, but it’s very clearly distinct from protobuf, although pb has gRPC-related features. That entanglement makes me lean towards flatbuffers or even protobuf every time I weigh them against capnp, especially since it means that fb and pb have much simpler implementations, and I place great value on simplicity for both security and maintenance reasons. I think the lack of good third-party language implementations speaks directly to the reasonability of that assessment. It also makes the bus factor and longevity story very poor. Simplicity rules.

insanitybit 3y ago
How does the serialization layer impact your rpc choice?
- cmrdporcupine 3y ago
  Cap'N'Proto comes with a (quite good) RPC facility. Based on asynchronous promises and grounded in capabilities. You don't have to use it. You could just use it just as a 'serialization' layer but if you're writing services you could be missing half the advantage, really. And if you're writing in C++ you'll end up having to use their KJ library anyways. If you take the whole package the zero copy, capability-security, and asynchrouny (a word I just coined!) all fit together nicely.
  - insanitybit 3y ago
    Yeah I'm aware of all of that. What I'm saying is that I don't see what about the Schema Definition Language pushes you towards the RPC other than that they obviously go well together, just like gRPC is almost always used with protobuf, or http with JSON. > but it bothers me that what is ultimately a message encoding protocol has an opinion on how I should architect my server. To me, this is like saying "Using JSON is unfortunate because it has an opinion that I should use HTTP" when I don't think anyone would argue that at all, and I don't see the argument for capnp much either.

dtech 3y ago

While I never used Cap'n Proto, I want to thank kentonv for the extremely informative FAQ answer [1] on why required fields are problematic in a protocol I link it to people all the time, especially when they ask why protobuf 3 doesn't have required fields. [1] https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require...

kccqzy 3y ago
This is some very valuable perspective. Personally, I previously also struggled to understand why. For me, the thing that clicked was to understand protobuf and Cap'n proto as serialization formats that need to work across API boundaries and need to work with different versions of their schema in a backwards- and forwards-compatible way; do not treat them as in-memory data structures that represent the world from the perspective of a single process running a single version without no compatibility concerns. Thus, the widely repeated mantra of "making illegal states unrepresentable" does not apply.
- chubot 3y ago
  Rich Hickey (creator of the Clojure language) has a good talk "Maybe Not" that touches on these issues, with a nice way of explaining it https://www.youtube.com/watch?v=YR5WdGrpoug https://www.youtube.com/watch?v=YR5WdGrpoug The capnproto link explains it concretely in terms of a message bus example, which is useful. But more abstractly you can think of the shape of data (aka schema, names and types) and field presence (optional/required) as separate things https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey First, being valid or invalid with respect to a static type system is a GLOBAL property of program -- writing a type checker will convince you of that. And big distributed systems don't have such global properties: https://news.ycombinator.com/item?id=36590799 https://news.ycombinator.com/item?id=36590799 If they did, they'd be small :) Namely you could just reboot the whole thing at once. You can't reboot say the entire Internet at once, and this also holds for smaller systems, like the ones at say Google (and I'm sure Cloudflare, etc.). So the idea is that the shape/schema is a GLOBAL property -- you never want two messages called foo.RequestX or two fields called "num_bar" with different types -- ever, anywhere. But optional/required is LOCAL property. It depends on what version of a schema is deployed in a particular binary. Inherently, you need to be able to handle a mix of inconsistent versions running simultaneously. --- To be pedantic, I woulds say "making illegal states unrepresentable" DOES apply, but you can't do it in a STATIC type system. [1] Your Maybe<T> type is not useful for data that crosses process boundaries. A distributed system isn't a state machine. 1. Lamport showed us one important reason why: the relative order of messages means that there is no globally coherent state. You need something like Paxos to turn a distributed system back into a state machine (and this is very expensive in general) 2. The second reason is probably a consequence of the first. You can think of deploying a binary to a node as a message to that node. So you don't have a completely consistent state -- you always have an in-between state, a mix of versions. And presumably you want your system to keep working during this time period :) And that coarse-grained problem (code versioning and deployment) implies the fine-grained problem (whether a specific message in a field is present). This is because protobufs generate parsers with validation for you -- or they used to! --- tl;dr Think of the shape of data (aka schema) and field presence (optional/required) as different dimensions of data modeling. Maybe<T> mixes those up, which is fine in a single process, but doesn't work across processes. --- [1] A very specific example of making illegal states unrepresentable without static types - my Oils project uses a DSL for algebraic data types, borrowed from CPython. The funny thing is that in CPython, it generates C code, which doesn't have any static notion of Maybe<T>. It has tagged unions. And in Oils we first generated dynamically typed Python at first. Somewhat surprisingly, algebraic data types are STILL useful there. Now the generated code is statically typed with MyPy (and with C++), and we do pleasant type-driven refactorings. But using algebraic data types were still extremely useful before static typing. They made illegal states unrepresentable -- but you would get the error at runtime.

insanitybit 3y ago

Any plans to improve the Rust side of things? The API could definitely use some more work/ docs around it.

dwrensha 3y ago
I intend to continue work on capnproto-rust, at my own pace and according to my own priorities. Are there any particular pain points that you want to call attention to?

shdh 3y ago

We have a great plethora of binary serialization libraries now, but I've noticed none of them offer the following: * Specification of the number of bits I want to cap out a field at during serialization, ie: `int` that only uses 3 bits. * Delta encoding for serialization and deserialization, this would further decrease the size of each message if there is an older message that I can use as the initial message to delta encode/decode from.

jeffbee 3y ago
> `int` that only uses 3 bits. CBOR approximates this, since it has several different widths for integers. > an older message that I can use as the initial message to delta encode/decode from. General-purpose compression on the encoded stream would do something toward this goal, but some protocol buffers library implementations offer merge functions. The question is what semantics of "merge" you expect. For repeated fields do you want to append or clobber?
- Dylan16807 3y ago
  CBOR has different integer widths, yes, but 1/2/3/5/9 bytes is not particularly flexible. And if someone is asking for exact bit sizes and deltas, they probably don't want a format that embeds every key in every message.
no_circuit 3y ago
Take a look at FAST protocol [1]. It has been around for a while. Was created for market/trading data. There appears to be some open source implementations, but I don't think in general they'd be maintained well since trading is, well, secretive. [1] https://en.wikipedia.org/wiki/FAST_protocol https://en.wikipedia.org/wiki/FAST_protocol
IshKebab 3y ago
Most formats use varints, so you can't have a 3-bit int but they will store a 64-bit int in one byte if it fits. Going to smaller than a byte isn't worth the extra complexity and slowness. If you're that space sensitive you need to add proper compression. By delta compression you mean across messages? Yeah I've never seen that but it's hard to imagine a scenario where it would be useful and worth the insane complexity.

synthetigram 3y ago

After exploring a few constant access serialization formats, I had to pass on Capn Proto in favor of Apache Avro. Capn has a great experience for C++ users, but Java codegen ended up being too annoying to get started with. If Capn Proto improved the developer experience for the other languages people write, I think it would really help a lot.

unixhero 3y ago

Congratulations to kentonv and the team

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

vicaya 3y ago

Worker should really adopt Apache Arrow, which has a much bigger ecosystem. https://github.com/apache/arrow https://github.com/apache/arrow

Timothycquinn 3y ago

Congrats in 10 years! Question: Can Cap'n Proto be used as an alternative to Python Pickle library for serializing and de-serializing python object structures?

kentonv 3y ago
If your goal is to serialize an arbitrary Python object, Pickle is the way to go. Cap'n Proto requires you to define a schema, in Cap'n Proto schema language, for whatever you wan to serialize. It can't just take an arbitrary Python value.

emtel 3y ago

I used capnproto years ago as the network serialization format for a multiplayer RTS game. Although the API can be quite awkward, it was overall a joy to use and I wish I was able to use it in more projects.

richardfey 3y ago

Amazing to see Cap'n Proto come this far! I wonder how easy it would be to swap it for gRPC, and still have advanced load balancing support for it.

hiddencost 3y ago

For context: Kenton ran Google's in house proto system for many years, before leaving and building his own open source version.

AceJohnny2 3y ago
I believe he was the one who open-sourced protobufs.

omginternets 3y ago

I have some very unfortunate news to share with the Cap'n Proto and Sandstorm communities. Ian Denhardt (zenhack on HN), a lead contributor to the Go implementation, suddenly and unexpectedly passed away a few weeks ago. Before making a request to the community, I want to express how deeply saddened I am by this loss. Ian and I collaborated extensively over the past three years, and we had become friends. As the de facto project lead, it now befalls me to fill Ian's very big shoes. Please, if you're able to contribute to the project, I could really use the help. And if you're a contributor or maintainer of some other implementation (C++, Rust, etc.), I would *REALLY* appreciate it if we could connect. I'm going to need to surround myself with very smart people if I am to continue Ian's work. RIP Ian, and thank you. I learned so much working with you. ------ P.S: I can be reached in the following places - https://github.com/lthibault https://github.com/lthibault - https://matrix.to/#/#go-capnp:matrix.org https://matrix.to/#/#go-capnp:matrix.org - Telegram: @lthibault - gmail: louist87

pja 3y ago
It seems @zenhack maintained the Haskell bindings as well.
- omginternets 3y ago
  I think the Haskell project was a complete implementation, not just bindings. But yes, Ian was truly prolific.
jcalabro 3y ago
Oh gosh, I didn't know that. Thank you for sharing :( I really loved his blog. That's awful.
- omginternets 3y ago
  Indeed. His blog was outstanding. It might be a good idea to mirror his blog before the domain expires, as it's a real treasure-trove.

jFriedensreich 3y ago

i love how the main reference for workerd can be just one capnp file. https://github.com/cloudflare/workerd/blob/main/src/workerd/server/workerd.capnp https://github.com/cloudflare/workerd/blob/main/src/workerd/... this changed my world how i think about computing on the web. if there was just a good enough js library as for lua and you could directly send capnp messages to workerd instead of always going through files. I guess one day i have to relearn c++ and understand how the internals actually work.

deleted 3y ago

[deleted]

zX41ZdbW 3y ago

We support CapNProto as an import/export format in ClickHouse. It was contributed by Marek Vavruša from CloudFlare in 2017: https://github.com/ClickHouse/ClickHouse/pull/1387 https://github.com/ClickHouse/ClickHouse/pull/1387 Since then, we extended and improved the support: added it for export (initially, it was only for import) and improved the performance. About strange stuff in the library - it uses a non-obvious approach for exception handling, and their C++ code feels like they too much focusing on some non-orthodox approaches.

Xeoncross 3y ago

[2013] Show HN: Cap'n Proto, by the ex-maintainer of Protocol Buffers: https://news.ycombinator.com/item?id=5482081 https://news.ycombinator.com/item?id=5482081

xvilka 3y ago

I am trying to revive C implementation [1]. Any help is welcome. [1] https://github.com/opensourcerouting/c-capnproto https://github.com/opensourcerouting/c-capnproto

up2isomorphism 3y ago

Some of these “high perf” RPC libraries never get one key point, if I really need something to be fast, the most important aspect is that it must be simple.

jviotti 3y ago

If you are interested in learning more about binary serialization (for Cap'n Proto and others), I wrote two free-to-read papers extensively looking at the scene from a space-efficiency point of view that people here might be interested in: - https://arxiv.org/abs/2201.02089 https://arxiv.org/abs/2201.02089 - https://arxiv.org/abs/2201.03051 https://arxiv.org/abs/2201.03051 Not something that aims to compete with Cap'n Proto (different use cases), but I've been also working on a binary serialization format that is pure JSON-compatible with a focus on space-efficiency: https://www.jsonbinpack.org https://www.jsonbinpack.org. Benchmarks here: https://arxiv.org/abs/2211.12799 https://arxiv.org/abs/2211.12799