8 ms·

Show HN: Comprehensive inter-process communication (IPC) toolkit in modern C++

88 points by ygoldfeld 2y ago

If you work in C++, and you would like 2+ programs to share data structures (and/or native I/O handles a.k.a. FDs) among each other, there is a good chance Flow-IPC - will make it considerably less annoying to code than typical approaches; and - may massively reduce the latency involved. Those sharing Cap'n Proto-encoded data may have particular interest. Cap'n Proto (https://capnproto.org https://capnproto.org) is fantastic at its core task - in-place serialization with zero-copy - and we wanted to make the IPC (inter-process communication) involving capnp-serialized messages be zero-copy, end-to-end. That said, we paid equal attention to other varieties of payload; it's not limited to capnp-encoded messages. For example there is painless (<-- I hope!) zero-copy transmission of arbitrary combinations of STL-compliant native C++ data structures. To help determine whether Flow-IPC is relevant to you we wrote an intro blog post. It works through an example, summarizes the available features, and has some performance results. https://www.linode.com/blog/open-source/flow-ipc-introduction-low-latency-cpp-toolkit https://www.linode.com/blog/open-source/flow-ipc-introductio... Of course there's nothing wrong with going straight to the GitHub link and getting into the README and docs. Currently Flow-IPC is for Linux. (macOS/ARM64 and Windows support could follow soon, depending on demand/contributions.)

ygoldfeld 2y ago
It's so hard to communicate this stuff in writing! There are several angles of potential interest; I wish I could simply chat in-person with anyone curious, you know? Of course that is impossible. (I'll do my best here at HN and the Flow-IPC Discussions board at GitHub.) I hope the above 2 links get the job done in communicating the key points. There is certainly no shortage of documentation! Still: If you'll indulge me, I do want to share how this project got started and became open-source. I actually do suspect this might help one get a feeling of what this thing is, and is not. My name is Yuri Goldfeld. I have worked at Akamai since 2005 (with a break for startup shenanigans, and VMware, in the middle). I designed or co-designed Flow-IPC and wrote about 75% of it (by lines of code ignoring comments); my colleague Eddy Chan wrote the rest, including the bulk of the SHM-jemalloc component (which is really cool IMO). Akamai in certain core parts is a C++/Linux shop, with dogged scrutiny to latency. Every millisecond along the request path is scrutinized. A few years ago I was asked to do a couple things: - Determine the best serializer to use, in general, but especially for IPC protocols. The answer there was easy IMO: Cap'n Proto. - Split-up a certain important C++ service into several parts, for various reasons, without adding latency to the request path. The latter task meant, among other things, communicating large amounts of user data from server application to server application. capnp-encoded structures (sometimes big - but not necessarily) would also need to be transmitted; as would FDs. The technical answers to these challenges are not necessarily rocket science. FDs can be transmitted via Unix domain socket as "ancillary data"; the POSIX `sendmsg()` API is hairy but usable. Small messages can be transmitted via Unix domain socket, or pipe, or POSIX MQ (etc.). Large blobs of data it would not be okay to transmit via those transports, as too much copying into and out of kernel buffers is involved and would add major latency, so we'd have to use shared memory (SHM). Certainly a hairy technology... but again, doable. And as for capnp - well - you "just" code a `MessageBuilder` implementation that allocates segments in SHM instead of regular heap like `capnp::MallocMessageBuilder` does. Thing is, I noticed that various parts of the company had similar needs. I've observed some variation of each of the aforementioned tasks custom-implemented - again, and again, and again. None of these implementations could really be reused anywhere else. Most of them ran into the same problems - none of which is that big a deal on its own, but together (and across projects) it more than adds up. To coders it's annoying. And to the business, it's expensive! Plus, at least one thing actually proved to be technically quite hard. Sharing (via SHM) a native C++ structure involving STL containers and/or raw pointers: downright tough to achieve in a general way. At least with Boost.interprocess (https://www.boost.org/doc/libs/1_84_0/doc/html/interprocess.html https://www.boost.org/doc/libs/1_84_0/doc/html/interprocess....) - which is really quite thoughtful - one can accomplish a lot... but even then, there are key limitations, in terms of safety and ease of use/reusability. (I'm being a bit vague here... trying to keep the length under control.) So, I decided to not just design/code an "IPC thing" for that original key C++ service I was being asked to split... but rather one that could be used as a general toolkit, for any C++ applications. Originally we named it Akamai-IPC, then renamed it Flow-IPC. As a result of that origin story, Flow-IPC is... hmmm... meat-and-potatoes, pragmatic. It is not a "framework." It does not replace or compete with gRPC. (It can, instead, speed RPC frameworks up by providing the zero-copy transmission substrate.) I hope that it is neither niche nor high-maintenance. To wit: If you merely want to send some binary-blob messages and/or FDs, it'll do that - and make it easier by letting you set-up a single session between the 2 processes, instead of making you worry about socket names and cleanup. (But, that's optional! If you simply want to set up a Unix domain socket yourself, you can.) If you want to add structured messaging, it supports Cap'n Proto - as noted - and right out of the box it'll be zero-copy end-to-end. That is, it'll do all the SHM stuff without a single `shm_open()` or `mmap()` or `ftruncate()` on your part. And if you want to customize how that all works, those layers and concepts are formally available to you. (No need to modify Flow-IPC yourself: just implement certain concepts and plug them in, at compile-time.) Lastly, for those who want to work with native C++ data directly in SHM, it'll simplify setup/cleanup considerably compared to what's typical. For the original Akamai service in question, we needed to use SHM as intensively as one typically uses the regular heap. So in particular Boost.interprocess's built-in 2 SHM-allocation algorithms were not sufficient. We needed something more industrial-strength. So we adapted jemalloc (https://jemalloc.net/ https://jemalloc.net/) to work in SHM, and worked that into Flow-IPC as a standard available feature. (jemalloc powers FreeBSD and big parts of Meta.) So jemalloc's anti-fragmentation algorithms, thread caching - all that stuff - will work for our SHM allocations. Having accepted this basic plan - develop a reusable IPC library that handled the above oft-repeated needs - Eddy Chan joined and especially heavily contributed on the jemalloc aspects. A couple years later we had it ready for internal Akamai use. All throughout we kept it general - not Akamai-specific (and certainly not specific to that original C++ service that started it all off) - and personally I felt it was a very natural candidate for open-source. To my delight, once I announced it internally, the immediate reaction from higher-up was, "you should open-source it." Not only that, we were given the resources and goodwill to actually do it. I have learned that it's not easy to make something like this presentable publicly, even having developed it with that in mind. (BTW it is about 69k lines of code, 92k lines of comments, excluding the Manual.) So, that's what happened. We wrote a thing useful for various teams internally at Akamai - and then Akamai decided we should share it with the world. That's how open-source thrives, we figured. On a personal level, of course it would be gratifying if others found it useful and/or themselves contributed. What a cool feeling that would be! After working with exemplary open-source stuff like capnp, it'd be amazing to offer even a fraction of that usefulness. But, we don't gain from "market share." It really is just there to be useful. So we hope it is!
- OnlyMortal 2y ago
  I’ve spent a lot of time with boost asio and serialisation of objects into a boost variant to send that across the wire. The server vists the variant to process the message. Including boost shared memory for file data. Both for unix domain sockets and TCP. There’re plenty of boost examples around so, I’d suggest, you take their examples and work them for your framework. As I’m sure you’re aware, a clean and easy to read example will make a difference. It’s great that you’re open source and I hope you get some traction.
  - ygoldfeld 2y ago
    Indeed, examples from every angle are probably the one deficit of the existing documentation. There are a couple, such as the perf_demo described in the blog post. I’d like to add ones showing integration with - epoll based event loop - boost.asio based event loop (Boost.interprocess and boost.asio are huge inspirations and are both used inside!) As for traction: it’s tough! Have to get eyeballs; and then have to convey a sense of being worth one’s trust. Thank you for your time.
    - OnlyMortal 2y ago
      Integration with boost asio would be of interest to many - myself included. It is the defacto for anyone who’s got past Stephen’s Unix Network Programming. It would gain a level of trust with developers.
      - ygoldfeld 2y ago
        Roger dodger. For what it is worth at this time - obviously acting on the following statement will require some level of trust - It is very much ready to use with boost.asio. (I know that, because I myself use boost.asio religiously. If it were not compatible with it, I'd pretty much have to not use Flow-IPC myself.) Though, it could (fairly easily) gain a number of wrapper classes that would turn our stuff into actual boost.asio I/O objects; then it'd be even more straightforward. Topic is covered here: https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/async_loop.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera... There's even the little section entitled, "I'm a boost.asio user. Can't I just give your constructor my io_context, and then you'll place the completion handler directly onto it?" To summarize, though... -1- You can have Flow-IPC create background threads as-needed and ping your completion handler (e.g., "message received") from such threads. -2- You can have it not create any background threads, instead asking you to .async_wait() (via boost.asio, most easily; but also manually with poll() or whatever you want) whenever it needs internally to async-await something. Your own completion handler (e.g., handle just-received message M) shall execute synchronously at only predictable points, in non-blocking fashion. -3- Direct integration with boost.asio - meaning ipc::transport::Channel (e.g.) would take an io_context/executor/whatever in its ctor, and .async_X(F) would indeed post F onto that io_context/executor/whatever = essentially syntactic sugar = a TODO. (I'd best file an Issue, I just remembered.) The perf_demo (partially recreated in the blog-post) integrates into a single-threaded boost.asio io_context, using technique #2 above. In the source code snippets in the blog, we avoided anything asynchronous just to keep it focused for the max # of readers (hopefully).
ygoldfeld 2y ago
Whoa. I’m the lead developer on this - I got to this post totally by accident: was googling for my own Show HN post about this from a couple days ago - and it took me here without my noticing. There’s some discussion on it in Show HN, and of course I can answer anything here that people might be interested in too. I’m very proud of it and very grateful Akamai gave the resources to open-source it. I’d like to have a flashier friendlier site with a slick intro video - haven’t had the time to do that stuff - but the substance and API documentation + Manual are very serious and complete, I hope. All linked off the blog-post!
- dang 2y ago
  That's great! Show HNs are preferred so I've re-upped your original post and have merged the other thread (https://news.ycombinator.com/item?id=40000104 https://news.ycombinator.com/item?id=40000104) hither. But readers will probably want to look at the other article as well: https://www.linode.com/blog/open-source/flow-ipc-introduction-low-latency-cpp-toolkit/ https://www.linode.com/blog/open-source/flow-ipc-introductio....
  - ygoldfeld 2y ago
    Whoa (again), super-cool, appreciate that.
- emmelaich 2y ago
  Your submission is [dead], dunno why. https://news.ycombinator.com/item?id=39987732 https://news.ycombinator.com/item?id=39987732
  - ygoldfeld 2y ago
    It lives! https://news.ycombinator.com/item?id=40028118 https://news.ycombinator.com/item?id=40028118
ygoldfeld 2y ago
(Akamai owns Linode and uses the blog on Linode.com as a developer-oriented blog. So that’s why the link is to there.)
signa11 2y ago
i have done something exactly identical at my current place of employment, and am always inquisitive to see how others have 'stacked-da-cat'. we _unfortunately_ gravitated towards protobuf's despite my fervent appeal to go with capn-proto. that has caused a cascade of troubles / missed opportunities for optimizations etc. etc.
- fwsgonzo 2y ago
  I tried to migrate to capn-proto but it just doesn't build on MinGW so I have no choice but to wait. Like you say, it gets worse the more I wait. But, if the APIs are somewhat sane, they should hopefully also be somewhat similar: Able to switch case on oneofs, movable data structures etc. I don't like that protobuf has recently started linking with abseil, which despite being a good framework, I can't use it if it doesn't build absolutely everywhere I need it to. So, maybe I'll be forced over to CapnProto one of these days?
  - Arelius 2y ago
    Honestly I like capn-proto. But it's community is a bit light. Even Windows which is "supported" has been a bit fix-it-yourself. If MinGW isn't already supported, if probably expect to have to take that over yourself.
- sgtnoodle 2y ago
  I've also developed a strikingly similar low latency real-time IPC message bus for work. It also uses sockets with transparent shared memory optimization. In my case, it's the backbone for an autonomous aircraft's avionics. I made everything agnostic to the message scheme, though, and most of the tooling supports an in-house schema, protobuf, JSON, YAML, etc. There's also clients implemented in C++, Rust, Python and Julia. What troubles has protobuf caused you?
  - signa11 2y ago
    > What troubles has protobuf caused you? we are primarily a c++ shop, so all comments are from that vantage point. in no particular order following: 1. extremely bloated generated code. 2. wire format is not optimal, specifically for large nested messages which trashes the cache on a already severely constrained h/w. 3. not easy to perform shared-memory exchange for large messages, for example, if you have 'string' types in your messages, those cannot be allocated on arenas. 4. <this space is for rent>
nonane 2y ago
Cool stuff! Does Flow-IPC protect against malformed messages? For example a client sending malformed messages to a server process
- sgtnoodle 2y ago
  Given that it's shared memory based, it seems like there has to be some degree of trust that the participants are well behaved. What do you mean by a malformed message, though? If you're talking about the payload of the message, that seems like a matter of the message scheme you're using. If you're talking about correctness of the IPC protocol itself, integrity checking is unfortunately at odds with latency.
  - ygoldfeld 2y ago
    Reply co-signed! That said, I'll add that my Manual (https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/pages.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera...) gets into the topic of safety (with a few words on how this differs from security; this indeed involves trust). So indeed Flow-IPC does not attempt to provide security for malicious/untrusted conversation-partner -- but it does provide a few enabled-by-default safety mechanisms. The Manual page in question: https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/safety_perms.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera... Among other things, the capnp-layer (which, as I noted in a recent reply earlier, is optional to use -- you can and sometimes certainly should "just" go native, and/or combine the two approaches) uses an internally-generated token to ensure some kind of terrible bug/thing/problem/act-of-god isn't mis-routing messages. It's a safety mechanism. But in terms of, say, incompatible schemas or mis-matching native `struct`s -- what if you used different compilers with slightly different STLs on the 2 sides?! -- it is indeed on the developer to not make that error/not converse with untrusted code. Flow-IPC will guard against misuses to the extent it's reasonable though, IMO. P.S. [Internal impl note] Oh! And, although at the moment it is the protocol version v1 (for all internally-involved protocols at all layers), I did build-in a protocol-version-checking system from the start, so as to avoid shooting ourselves in the foot, in case Flow-IPC needs to expand some of its internal protocol(s) in later versions. At the very worst, Flow-IPC would refuse to establish a channel/session upon encountering a partner with Flow-IPC version such that their protocols are incompatible. (Again -- academic at the moment, since there is only v1 -- but might be different in the future. At that point a new protocol might be developed to be backward-compatible with earlier-Flow-IPCs and just still work; or worst case throw the aforementioned error if not.)
jeffreygoesto 2y ago
Does the schema help a lot? For C++ you can get very fast without, for example with IceOryx https://github.com/eclipse-iceoryx/iceoryx https://github.com/eclipse-iceoryx/iceoryx In contrast to Cap'n'Proto you get compiler optimized struct layout as benefit from using raw structs. Benchmarks are here https://iceoryx.io/v2.0.2/examples/iceperf/ https://iceoryx.io/v2.0.2/examples/iceperf/
- ygoldfeld 2y ago
  I think that (whether a native-struct versus a capnp schema-based struct = helps/how much) is a general question of what kind of serialization is best for a particular use-case. I wouldn't want to litigate that here fully. Personally though I've found capnp-based IPC protocols to be neat and helpful, across versions and protocol changes (where e.g. there are well-defined rules of forward-compatibility; and Flow-IPC gives you niceties including request-response and message-type demultiplexing to a particular handler). [footnote 1 below] BUT!!! Some algorithms don't require an "IPC protocol" per se, necessarily, but more like = 2+ applications collaborating on a data structure. In that case native structures are for sure superior, or at times even essentially required. (E.g., if you have some custom optimized hash-table -- you're not going to want to express it as a capnp structure probably.) So, more to the point: - Flow-IPC 100% supports transmitting/sharing (and constructing, and auto-destroyting) native C++ structures. Compared to iceoryx, on this point, it appears to have some extra capabilities, namely full support for structures with pointers/references and/or STL-compliant containers. (This example https://iceoryx.io/latest/examples/complexdata/ https://iceoryx.io/latest/examples/complexdata/ and other pages say things like, "To implement zero-copy data transfer we use a shared memory approach. This requires that every data structure needs to be entirely contained in the shared memory and must not internally use pointers or references. The complete list of restrictions can be found...".) Flow-IPC, in this context, means no need to write custom containers sans heap-use, or eliminate pointers in an existing structure. [footnote 2 below] - Indeed, the capnp framing (only if you choose to use the Flow-IPC capnp-protocol feature in question!) adds processing and thus some computational and RAM-use overhead. For many applications, the 10s of microseconds added there don't matter much -- as long as they are constant regardless of structure size, and as long as they are 10s of microseconds. So a 100usec (modulo processor model of course!) RTT (size-independent) is pretty good still. Of course I would never claim this overhead doesn't matter to anyone, and iceoryx's results here are straight-up admirable. [footnote 1] The request/response/demultiplexing/etc. niceties added by Flow-IPC's capnp-protocol feature-in-question work well IMO, but one might prefer the sweet RPC-semantics + promise pipelining of capnp-RPC. Kenton V (capnp inventor/owner) and I have spoken recently about using Flow-IPC to zero-copy-ify capnp-RPC. I'm looking into it! (He suspects it is pretty simple/natural, given that we handle the capnp-serialization layer already, and capnp-RPC is built on that.) This wouldn't change Flow-IPC's existing features but rather exercise another way of using them. In a way Flow-IPC provides a simple-but-effective-out-of-the-box schema-based conversation protocol via capnp-serialization, and capnp-RPC would provide an alternate (to that out-of-the-box guy) conversation protocol option. I tried pretty hard to design Flow-IPC in a grounded and layered way, so such work would be natural as opposed to daunting. [footnote 2] In fact the Flow-IPC capnp-based structured-channel feature (internally) itself uses Flow-IPC's own native-structure-transmission feature in its implementation (eat our own dog-food). Since a capnp serialization = sequence of buffers (a.k.a. segments), for us it is (internally) represented as essentially an STL list<vector<uint8_t>>. So we construct/build one of those in SHM (internally); then only a small SHM-handle is (internally) transmitted over the IPC-transport [footnote 3]; and the receiver then obtains the in-place list<vector<uint8_t>> (essentially) which is then treated as the capnp-encoding it really is. This would all happen (internally) when executing the quite-short example in the blog (https://www.linode.com/blog/open-source/flow-ipc-introduction-low-latency-cpp-toolkit/ https://www.linode.com/blog/open-source/flow-ipc-introductio...). As you can see there, to the Flow-IPC-using developer, it's just -- like -- "create a message with this schema here, call some mutators, send"; and conversely "receive a message expected to have that (same) schema, OK -- got it; call some accessors." [footnote 3] IPC-transport = Unix domain socket or one 2 MQ types -- you can choose via template arg (or add your own IPC-transport by implementing a certain pair of simple concepts).
rurban 2y ago
I also went this route and came to the very same conclusions. Cap'n proto for fast reading, SHM for shared data, simple short messaging, just everything in C. My only problem is MacOS with its too small default SHM buffers, you need to enhance them. Most solutions need a reboot, but a simple setter is enough. Like sudo sysctl -w kern.sysv.shmmax=16777216
- ygoldfeld 2y ago
  Interesting! I'd best write this down. Current notes on macOS and Windows port work: https://github.com/Flow-IPC/ipc/issues/101 https://github.com/Flow-IPC/ipc/issues/101 (<= https://github.com/orgs/Flow-IPC/discussions/98 https://github.com/orgs/Flow-IPC/discussions/98) For macOS/ARM64, currently it looks to me like the apparent lack of /dev/shm equivalent (unless I messed up in searching for it) means the most significant amount of new work necessary to port it ... but you just mentioned a thing I did not know about. (SHM size/count limits definitely were a thing in Linux, though, indeed.) TY
  - rurban 2y ago
    I wrote some shm API wrappers here: https://github.com/rurban/smart/blob/new/source/algos/include/shmids.h https://github.com/rurban/smart/blob/new/source/algos/includ... Never use /dev/shm directly unless Linux only.
    - ygoldfeld 2y ago
      Oh… Sys V SHM… not POSIX. Old-school.
- abcd_f 2y ago
  There's no reason to use (the very ancient) SHM API over mmap, not today. You can literally do everything with mmap that you can do with shm, without hitting OS caps, no performance penalty and with a code that's simpler.
  - monocasa 2y ago
seego 2y ago
Do you have any concrete plans about a potential network extension yet?
- ygoldfeld 2y ago
  A couple - 1. The obvious one is “just” extending stuff internally working via Unix domain sockets to TCP sockets. Various internal code is written with an eye to that, including anticipating that certain operations (such as connect) that are instant locally can would-block in a network. If people enjoy the API, this would be a no-brainer value-add, even if lots of people would scoff and use actual dedicated networking techniques (HTTP, whatever) directly instead. 2. The much more fun and unique idea is using RDMA, “sort of” a networked-SHM type of setup (internally). Hope to get a go-ahead (or contribution, of course) on this. I mention these in the intro page of the Manual, I think.
forrestthewoods 2y ago
> Currently Flow-IPC is for Linux Dang. I was excited for a brief moment, but support for macOS + Windows is mandatory for all of my use cases. To be honest what I actually want is NOT "the fastest possible thing". All I actually care about is "easy advertisement, discovery, and message sending". I use localhost TCP way more than I want because it "just works". Maybe someday I'll stumble across my dream IPC library.
- ygoldfeld 2y ago
  Oooh, so close. We’ve got the advertisement/discovery and messaging for sure. Concretely what it would take to port it to those OS: https://github.com/Flow-IPC/ipc/issues/101 https://github.com/Flow-IPC/ipc/issues/101 Given a couple weeks to work on it, this thing would be on macOS no problem. With Windows I personally need to understand its FD-passing and native handle concepts first, but I’m guessing it’d be a similar amount of effort in the end.
  - elfenpiff 2y ago
    I thought the same when we ported iceoryx to Mac OS and Windows. Mac OS is pretty straightforward, except it does not support unnamed semaphores. But Windows is an entirely different story. For once, it supports only streaming unix domain sockets, and it didn't support the transfer of FDs when we ported iceoryx to Windows back then - maybe it supports it now. Also, when you want to perform some access control with access rights, you have to face sid— and ace-strings - oh they are fun. And, of course, there are all the nasty details; for instance, Windows defines macros that lead to compilation failures since they collide with internal naming. Take a look at this here, maybe it makes your efforts less painless: https://github.com/eclipse-iceoryx/iceoryx/blob/master/iceoryx_platform/win/include/iceoryx_platform/platform_correction.hpp https://github.com/eclipse-iceoryx/iceoryx/blob/master/iceor... You could reuse the iceoryx platform layer that enables iceoryx to run on every platform from qnx, linux, freertos, mac, windows. Maybe it can help you as well: https://github.com/eclipse-iceoryx/iceoryx/blob/master/doc/website/advanced/custom-iceoryx-platform.md https://github.com/eclipse-iceoryx/iceoryx/blob/master/doc/w...
mgaunard 2y ago
Serialization is the trivial part; the hard part is building a lockfree mpmc queue or message bus (depending on what you want) on top of fixed-size pre-allocated memory segments. I can't tell what this library does; the blog articles and readme all talk about stuff that isn't close to any of the challenges that I see.
- ygoldfeld 2y ago
  While I wouldn't dream of claiming Flow-IPC will fit every IPCer's priorities, nor of trying to change yours or anyone's, nor of debating about what is trivial versus hard -- it should at least be easily possible to know what's in Flow-IPC. I'm here to help; this is the API overview with various code-snippet synopses, etc.: https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/api_overview.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera... I should also note that Flow-IPC does not provide "serialization"; it does however enable the use of an existing/best serializer (capnp) for zero-copy messaging. This is only one feature -- albeit oft requested, hence my decision to base the blog/README example on it. (I'm currently also looking into extending this to capnp-RPC.) But, of course, we don't expect it to match what everyone is looking for; in your case IceOryx might be more your speed -- have a look.
ygoldfeld 2y ago
At the risk of almost-spamming -- this post has taken off, which is sweet, and I have noticed some trends about what seems to interest readers which may not be placed as prominently up-top as would fit this audience. To wit: the API Overview from the Manual covers the available stuff, with code snippets (and some words). Could save people some time: https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/api_overview.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera...

88 points by ygoldfeld 2y ago

ygoldfeld 2y ago

It's so hard to communicate this stuff in writing! There are several angles of potential interest; I wish I could simply chat in-person with anyone curious, you know? Of course that is impossible. (I'll do my best here at HN and the Flow-IPC Discussions board at GitHub.) I hope the above 2 links get the job done in communicating the key points. There is certainly no shortage of documentation! Still: If you'll indulge me, I do want to share how this project got started and became open-source. I actually do suspect this might help one get a feeling of what this thing is, and is not. My name is Yuri Goldfeld. I have worked at Akamai since 2005 (with a break for startup shenanigans, and VMware, in the middle). I designed or co-designed Flow-IPC and wrote about 75% of it (by lines of code ignoring comments); my colleague Eddy Chan wrote the rest, including the bulk of the SHM-jemalloc component (which is really cool IMO). Akamai in certain core parts is a C++/Linux shop, with dogged scrutiny to latency. Every millisecond along the request path is scrutinized. A few years ago I was asked to do a couple things: - Determine the best serializer to use, in general, but especially for IPC protocols. The answer there was easy IMO: Cap'n Proto. - Split-up a certain important C++ service into several parts, for various reasons, without adding latency to the request path. The latter task meant, among other things, communicating large amounts of user data from server application to server application. capnp-encoded structures (sometimes big - but not necessarily) would also need to be transmitted; as would FDs. The technical answers to these challenges are not necessarily rocket science. FDs can be transmitted via Unix domain socket as "ancillary data"; the POSIX `sendmsg()` API is hairy but usable. Small messages can be transmitted via Unix domain socket, or pipe, or POSIX MQ (etc.). Large blobs of data it would not be okay to transmit via those transports, as too much copying into and out of kernel buffers is involved and would add major latency, so we'd have to use shared memory (SHM). Certainly a hairy technology... but again, doable. And as for capnp - well - you "just" code a `MessageBuilder` implementation that allocates segments in SHM instead of regular heap like `capnp::MallocMessageBuilder` does. Thing is, I noticed that various parts of the company had similar needs. I've observed some variation of each of the aforementioned tasks custom-implemented - again, and again, and again. None of these implementations could really be reused anywhere else. Most of them ran into the same problems - none of which is that big a deal on its own, but together (and across projects) it more than adds up. To coders it's annoying. And to the business, it's expensive! Plus, at least one thing actually proved to be technically quite hard. Sharing (via SHM) a native C++ structure involving STL containers and/or raw pointers: downright tough to achieve in a general way. At least with Boost.interprocess (https://www.boost.org/doc/libs/1_84_0/doc/html/interprocess.html https://www.boost.org/doc/libs/1_84_0/doc/html/interprocess....) - which is really quite thoughtful - one can accomplish a lot... but even then, there are key limitations, in terms of safety and ease of use/reusability. (I'm being a bit vague here... trying to keep the length under control.) So, I decided to not just design/code an "IPC thing" for that original key C++ service I was being asked to split... but rather one that could be used as a general toolkit, for any C++ applications. Originally we named it Akamai-IPC, then renamed it Flow-IPC. As a result of that origin story, Flow-IPC is... hmmm... meat-and-potatoes, pragmatic. It is not a "framework." It does not replace or compete with gRPC. (It can, instead, speed RPC frameworks up by providing the zero-copy transmission substrate.) I hope that it is neither niche nor high-maintenance. To wit: If you merely want to send some binary-blob messages and/or FDs, it'll do that - and make it easier by letting you set-up a single session between the 2 processes, instead of making you worry about socket names and cleanup. (But, that's optional! If you simply want to set up a Unix domain socket yourself, you can.) If you want to add structured messaging, it supports Cap'n Proto - as noted - and right out of the box it'll be zero-copy end-to-end. That is, it'll do all the SHM stuff without a single `shm_open()` or `mmap()` or `ftruncate()` on your part. And if you want to customize how that all works, those layers and concepts are formally available to you. (No need to modify Flow-IPC yourself: just implement certain concepts and plug them in, at compile-time.) Lastly, for those who want to work with native C++ data directly in SHM, it'll simplify setup/cleanup considerably compared to what's typical. For the original Akamai service in question, we needed to use SHM as intensively as one typically uses the regular heap. So in particular Boost.interprocess's built-in 2 SHM-allocation algorithms were not sufficient. We needed something more industrial-strength. So we adapted jemalloc (https://jemalloc.net/ https://jemalloc.net/) to work in SHM, and worked that into Flow-IPC as a standard available feature. (jemalloc powers FreeBSD and big parts of Meta.) So jemalloc's anti-fragmentation algorithms, thread caching - all that stuff - will work for our SHM allocations. Having accepted this basic plan - develop a reusable IPC library that handled the above oft-repeated needs - Eddy Chan joined and especially heavily contributed on the jemalloc aspects. A couple years later we had it ready for internal Akamai use. All throughout we kept it general - not Akamai-specific (and certainly not specific to that original C++ service that started it all off) - and personally I felt it was a very natural candidate for open-source. To my delight, once I announced it internally, the immediate reaction from higher-up was, "you should open-source it." Not only that, we were given the resources and goodwill to actually do it. I have learned that it's not easy to make something like this presentable publicly, even having developed it with that in mind. (BTW it is about 69k lines of code, 92k lines of comments, excluding the Manual.) So, that's what happened. We wrote a thing useful for various teams internally at Akamai - and then Akamai decided we should share it with the world. That's how open-source thrives, we figured. On a personal level, of course it would be gratifying if others found it useful and/or themselves contributed. What a cool feeling that would be! After working with exemplary open-source stuff like capnp, it'd be amazing to offer even a fraction of that usefulness. But, we don't gain from "market share." It really is just there to be useful. So we hope it is!

OnlyMortal 2y ago
I’ve spent a lot of time with boost asio and serialisation of objects into a boost variant to send that across the wire. The server vists the variant to process the message. Including boost shared memory for file data. Both for unix domain sockets and TCP. There’re plenty of boost examples around so, I’d suggest, you take their examples and work them for your framework. As I’m sure you’re aware, a clean and easy to read example will make a difference. It’s great that you’re open source and I hope you get some traction.
- ygoldfeld 2y ago
  Indeed, examples from every angle are probably the one deficit of the existing documentation. There are a couple, such as the perf_demo described in the blog post. I’d like to add ones showing integration with - epoll based event loop - boost.asio based event loop (Boost.interprocess and boost.asio are huge inspirations and are both used inside!) As for traction: it’s tough! Have to get eyeballs; and then have to convey a sense of being worth one’s trust. Thank you for your time.
  - OnlyMortal 2y ago
    Integration with boost asio would be of interest to many - myself included. It is the defacto for anyone who’s got past Stephen’s Unix Network Programming. It would gain a level of trust with developers.
    - ygoldfeld 2y ago
      Roger dodger. For what it is worth at this time - obviously acting on the following statement will require some level of trust - It is very much ready to use with boost.asio. (I know that, because I myself use boost.asio religiously. If it were not compatible with it, I'd pretty much have to not use Flow-IPC myself.) Though, it could (fairly easily) gain a number of wrapper classes that would turn our stuff into actual boost.asio I/O objects; then it'd be even more straightforward. Topic is covered here: https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/async_loop.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera... There's even the little section entitled, "I'm a boost.asio user. Can't I just give your constructor my io_context, and then you'll place the completion handler directly onto it?" To summarize, though... -1- You can have Flow-IPC create background threads as-needed and ping your completion handler (e.g., "message received") from such threads. -2- You can have it not create any background threads, instead asking you to .async_wait() (via boost.asio, most easily; but also manually with poll() or whatever you want) whenever it needs internally to async-await something. Your own completion handler (e.g., handle just-received message M) shall execute synchronously at only predictable points, in non-blocking fashion. -3- Direct integration with boost.asio - meaning ipc::transport::Channel (e.g.) would take an io_context/executor/whatever in its ctor, and .async_X(F) would indeed post F onto that io_context/executor/whatever = essentially syntactic sugar = a TODO. (I'd best file an Issue, I just remembered.) The perf_demo (partially recreated in the blog-post) integrates into a single-threaded boost.asio io_context, using technique #2 above. In the source code snippets in the blog, we avoided anything asynchronous just to keep it focused for the max # of readers (hopefully).

ygoldfeld 2y ago

Whoa. I’m the lead developer on this - I got to this post totally by accident: was googling for my own Show HN post about this from a couple days ago - and it took me here without my noticing. There’s some discussion on it in Show HN, and of course I can answer anything here that people might be interested in too. I’m very proud of it and very grateful Akamai gave the resources to open-source it. I’d like to have a flashier friendlier site with a slick intro video - haven’t had the time to do that stuff - but the substance and API documentation + Manual are very serious and complete, I hope. All linked off the blog-post!

dang 2y ago
That's great! Show HNs are preferred so I've re-upped your original post and have merged the other thread (https://news.ycombinator.com/item?id=40000104 https://news.ycombinator.com/item?id=40000104) hither. But readers will probably want to look at the other article as well: https://www.linode.com/blog/open-source/flow-ipc-introduction-low-latency-cpp-toolkit/ https://www.linode.com/blog/open-source/flow-ipc-introductio....
- ygoldfeld 2y ago
  Whoa (again), super-cool, appreciate that.
emmelaich 2y ago
Your submission is [dead], dunno why. https://news.ycombinator.com/item?id=39987732 https://news.ycombinator.com/item?id=39987732
- ygoldfeld 2y ago
  It lives! https://news.ycombinator.com/item?id=40028118 https://news.ycombinator.com/item?id=40028118

ygoldfeld 2y ago

(Akamai owns Linode and uses the blog on Linode.com as a developer-oriented blog. So that’s why the link is to there.)

signa11 2y ago

i have done something exactly identical at my current place of employment, and am always inquisitive to see how others have 'stacked-da-cat'. we _unfortunately_ gravitated towards protobuf's despite my fervent appeal to go with capn-proto. that has caused a cascade of troubles / missed opportunities for optimizations etc. etc.

fwsgonzo 2y ago
I tried to migrate to capn-proto but it just doesn't build on MinGW so I have no choice but to wait. Like you say, it gets worse the more I wait. But, if the APIs are somewhat sane, they should hopefully also be somewhat similar: Able to switch case on oneofs, movable data structures etc. I don't like that protobuf has recently started linking with abseil, which despite being a good framework, I can't use it if it doesn't build absolutely everywhere I need it to. So, maybe I'll be forced over to CapnProto one of these days?
- Arelius 2y ago
  Honestly I like capn-proto. But it's community is a bit light. Even Windows which is "supported" has been a bit fix-it-yourself. If MinGW isn't already supported, if probably expect to have to take that over yourself.
sgtnoodle 2y ago
I've also developed a strikingly similar low latency real-time IPC message bus for work. It also uses sockets with transparent shared memory optimization. In my case, it's the backbone for an autonomous aircraft's avionics. I made everything agnostic to the message scheme, though, and most of the tooling supports an in-house schema, protobuf, JSON, YAML, etc. There's also clients implemented in C++, Rust, Python and Julia. What troubles has protobuf caused you?
- signa11 2y ago
  > What troubles has protobuf caused you? we are primarily a c++ shop, so all comments are from that vantage point. in no particular order following: 1. extremely bloated generated code. 2. wire format is not optimal, specifically for large nested messages which trashes the cache on a already severely constrained h/w. 3. not easy to perform shared-memory exchange for large messages, for example, if you have 'string' types in your messages, those cannot be allocated on arenas. 4. <this space is for rent>

nonane 2y ago

Cool stuff! Does Flow-IPC protect against malformed messages? For example a client sending malformed messages to a server process

sgtnoodle 2y ago
Given that it's shared memory based, it seems like there has to be some degree of trust that the participants are well behaved. What do you mean by a malformed message, though? If you're talking about the payload of the message, that seems like a matter of the message scheme you're using. If you're talking about correctness of the IPC protocol itself, integrity checking is unfortunately at odds with latency.
- ygoldfeld 2y ago
  Reply co-signed! That said, I'll add that my Manual (https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/pages.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera...) gets into the topic of safety (with a few words on how this differs from security; this indeed involves trust). So indeed Flow-IPC does not attempt to provide security for malicious/untrusted conversation-partner -- but it does provide a few enabled-by-default safety mechanisms. The Manual page in question: https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/safety_perms.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera... Among other things, the capnp-layer (which, as I noted in a recent reply earlier, is optional to use -- you can and sometimes certainly should "just" go native, and/or combine the two approaches) uses an internally-generated token to ensure some kind of terrible bug/thing/problem/act-of-god isn't mis-routing messages. It's a safety mechanism. But in terms of, say, incompatible schemas or mis-matching native `struct`s -- what if you used different compilers with slightly different STLs on the 2 sides?! -- it is indeed on the developer to not make that error/not converse with untrusted code. Flow-IPC will guard against misuses to the extent it's reasonable though, IMO. P.S. [Internal impl note] Oh! And, although at the moment it is the protocol version v1 (for all internally-involved protocols at all layers), I did build-in a protocol-version-checking system from the start, so as to avoid shooting ourselves in the foot, in case Flow-IPC needs to expand some of its internal protocol(s) in later versions. At the very worst, Flow-IPC would refuse to establish a channel/session upon encountering a partner with Flow-IPC version such that their protocols are incompatible. (Again -- academic at the moment, since there is only v1 -- but might be different in the future. At that point a new protocol might be developed to be backward-compatible with earlier-Flow-IPCs and just still work; or worst case throw the aforementioned error if not.)

jeffreygoesto 2y ago

Does the schema help a lot? For C++ you can get very fast without, for example with IceOryx https://github.com/eclipse-iceoryx/iceoryx https://github.com/eclipse-iceoryx/iceoryx In contrast to Cap'n'Proto you get compiler optimized struct layout as benefit from using raw structs. Benchmarks are here https://iceoryx.io/v2.0.2/examples/iceperf/ https://iceoryx.io/v2.0.2/examples/iceperf/

ygoldfeld 2y ago
I think that (whether a native-struct versus a capnp schema-based struct = helps/how much) is a general question of what kind of serialization is best for a particular use-case. I wouldn't want to litigate that here fully. Personally though I've found capnp-based IPC protocols to be neat and helpful, across versions and protocol changes (where e.g. there are well-defined rules of forward-compatibility; and Flow-IPC gives you niceties including request-response and message-type demultiplexing to a particular handler). [footnote 1 below] BUT!!! Some algorithms don't require an "IPC protocol" per se, necessarily, but more like = 2+ applications collaborating on a data structure. In that case native structures are for sure superior, or at times even essentially required. (E.g., if you have some custom optimized hash-table -- you're not going to want to express it as a capnp structure probably.) So, more to the point: - Flow-IPC 100% supports transmitting/sharing (and constructing, and auto-destroyting) native C++ structures. Compared to iceoryx, on this point, it appears to have some extra capabilities, namely full support for structures with pointers/references and/or STL-compliant containers. (This example https://iceoryx.io/latest/examples/complexdata/ https://iceoryx.io/latest/examples/complexdata/ and other pages say things like, "To implement zero-copy data transfer we use a shared memory approach. This requires that every data structure needs to be entirely contained in the shared memory and must not internally use pointers or references. The complete list of restrictions can be found...".) Flow-IPC, in this context, means no need to write custom containers sans heap-use, or eliminate pointers in an existing structure. [footnote 2 below] - Indeed, the capnp framing (only if you choose to use the Flow-IPC capnp-protocol feature in question!) adds processing and thus some computational and RAM-use overhead. For many applications, the 10s of microseconds added there don't matter much -- as long as they are constant regardless of structure size, and as long as they are 10s of microseconds. So a 100usec (modulo processor model of course!) RTT (size-independent) is pretty good still. Of course I would never claim this overhead doesn't matter to anyone, and iceoryx's results here are straight-up admirable. [footnote 1] The request/response/demultiplexing/etc. niceties added by Flow-IPC's capnp-protocol feature-in-question work well IMO, but one might prefer the sweet RPC-semantics + promise pipelining of capnp-RPC. Kenton V (capnp inventor/owner) and I have spoken recently about using Flow-IPC to zero-copy-ify capnp-RPC. I'm looking into it! (He suspects it is pretty simple/natural, given that we handle the capnp-serialization layer already, and capnp-RPC is built on that.) This wouldn't change Flow-IPC's existing features but rather exercise another way of using them. In a way Flow-IPC provides a simple-but-effective-out-of-the-box schema-based conversation protocol via capnp-serialization, and capnp-RPC would provide an alternate (to that out-of-the-box guy) conversation protocol option. I tried pretty hard to design Flow-IPC in a grounded and layered way, so such work would be natural as opposed to daunting. [footnote 2] In fact the Flow-IPC capnp-based structured-channel feature (internally) itself uses Flow-IPC's own native-structure-transmission feature in its implementation (eat our own dog-food). Since a capnp serialization = sequence of buffers (a.k.a. segments), for us it is (internally) represented as essentially an STL list<vector<uint8_t>>. So we construct/build one of those in SHM (internally); then only a small SHM-handle is (internally) transmitted over the IPC-transport [footnote 3]; and the receiver then obtains the in-place list<vector<uint8_t>> (essentially) which is then treated as the capnp-encoding it really is. This would all happen (internally) when executing the quite-short example in the blog (https://www.linode.com/blog/open-source/flow-ipc-introduction-low-latency-cpp-toolkit/ https://www.linode.com/blog/open-source/flow-ipc-introductio...). As you can see there, to the Flow-IPC-using developer, it's just -- like -- "create a message with this schema here, call some mutators, send"; and conversely "receive a message expected to have that (same) schema, OK -- got it; call some accessors." [footnote 3] IPC-transport = Unix domain socket or one 2 MQ types -- you can choose via template arg (or add your own IPC-transport by implementing a certain pair of simple concepts).

rurban 2y ago

I also went this route and came to the very same conclusions. Cap'n proto for fast reading, SHM for shared data, simple short messaging, just everything in C. My only problem is MacOS with its too small default SHM buffers, you need to enhance them. Most solutions need a reboot, but a simple setter is enough. Like sudo sysctl -w kern.sysv.shmmax=16777216

ygoldfeld 2y ago
Interesting! I'd best write this down. Current notes on macOS and Windows port work: https://github.com/Flow-IPC/ipc/issues/101 https://github.com/Flow-IPC/ipc/issues/101 (<= https://github.com/orgs/Flow-IPC/discussions/98 https://github.com/orgs/Flow-IPC/discussions/98) For macOS/ARM64, currently it looks to me like the apparent lack of /dev/shm equivalent (unless I messed up in searching for it) means the most significant amount of new work necessary to port it ... but you just mentioned a thing I did not know about. (SHM size/count limits definitely were a thing in Linux, though, indeed.) TY
- rurban 2y ago
  I wrote some shm API wrappers here: https://github.com/rurban/smart/blob/new/source/algos/include/shmids.h https://github.com/rurban/smart/blob/new/source/algos/includ... Never use /dev/shm directly unless Linux only.
  - ygoldfeld 2y ago
    Oh… Sys V SHM… not POSIX. Old-school.
abcd_f 2y ago
There's no reason to use (the very ancient) SHM API over mmap, not today. You can literally do everything with mmap that you can do with shm, without hitting OS caps, no performance penalty and with a code that's simpler.
- monocasa 2y ago

seego 2y ago

Do you have any concrete plans about a potential network extension yet?

ygoldfeld 2y ago
A couple - 1. The obvious one is “just” extending stuff internally working via Unix domain sockets to TCP sockets. Various internal code is written with an eye to that, including anticipating that certain operations (such as connect) that are instant locally can would-block in a network. If people enjoy the API, this would be a no-brainer value-add, even if lots of people would scoff and use actual dedicated networking techniques (HTTP, whatever) directly instead. 2. The much more fun and unique idea is using RDMA, “sort of” a networked-SHM type of setup (internally). Hope to get a go-ahead (or contribution, of course) on this. I mention these in the intro page of the Manual, I think.

forrestthewoods 2y ago

> Currently Flow-IPC is for Linux Dang. I was excited for a brief moment, but support for macOS + Windows is mandatory for all of my use cases. To be honest what I actually want is NOT "the fastest possible thing". All I actually care about is "easy advertisement, discovery, and message sending". I use localhost TCP way more than I want because it "just works". Maybe someday I'll stumble across my dream IPC library.

ygoldfeld 2y ago
Oooh, so close. We’ve got the advertisement/discovery and messaging for sure. Concretely what it would take to port it to those OS: https://github.com/Flow-IPC/ipc/issues/101 https://github.com/Flow-IPC/ipc/issues/101 Given a couple weeks to work on it, this thing would be on macOS no problem. With Windows I personally need to understand its FD-passing and native handle concepts first, but I’m guessing it’d be a similar amount of effort in the end.
- elfenpiff 2y ago
  I thought the same when we ported iceoryx to Mac OS and Windows. Mac OS is pretty straightforward, except it does not support unnamed semaphores. But Windows is an entirely different story. For once, it supports only streaming unix domain sockets, and it didn't support the transfer of FDs when we ported iceoryx to Windows back then - maybe it supports it now. Also, when you want to perform some access control with access rights, you have to face sid— and ace-strings - oh they are fun. And, of course, there are all the nasty details; for instance, Windows defines macros that lead to compilation failures since they collide with internal naming. Take a look at this here, maybe it makes your efforts less painless: https://github.com/eclipse-iceoryx/iceoryx/blob/master/iceoryx_platform/win/include/iceoryx_platform/platform_correction.hpp https://github.com/eclipse-iceoryx/iceoryx/blob/master/iceor... You could reuse the iceoryx platform layer that enables iceoryx to run on every platform from qnx, linux, freertos, mac, windows. Maybe it can help you as well: https://github.com/eclipse-iceoryx/iceoryx/blob/master/doc/website/advanced/custom-iceoryx-platform.md https://github.com/eclipse-iceoryx/iceoryx/blob/master/doc/w...

mgaunard 2y ago

Serialization is the trivial part; the hard part is building a lockfree mpmc queue or message bus (depending on what you want) on top of fixed-size pre-allocated memory segments. I can't tell what this library does; the blog articles and readme all talk about stuff that isn't close to any of the challenges that I see.

ygoldfeld 2y ago
While I wouldn't dream of claiming Flow-IPC will fit every IPCer's priorities, nor of trying to change yours or anyone's, nor of debating about what is trivial versus hard -- it should at least be easily possible to know what's in Flow-IPC. I'm here to help; this is the API overview with various code-snippet synopses, etc.: https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/api_overview.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera... I should also note that Flow-IPC does not provide "serialization"; it does however enable the use of an existing/best serializer (capnp) for zero-copy messaging. This is only one feature -- albeit oft requested, hence my decision to base the blog/README example on it. (I'm currently also looking into extending this to capnp-RPC.) But, of course, we don't expect it to match what everyone is looking for; in your case IceOryx might be more your speed -- have a look.

ygoldfeld 2y ago

At the risk of almost-spamming -- this post has taken off, which is sweet, and I have noticed some trends about what seems to interest readers which may not be placed as prominently up-top as would fit this audience. To wit: the API Overview from the Manual covers the available stuff, with code snippets (and some words). Could save people some time: https://flow-ipc.github.io/doc/flow-ipc/versions/main/generated/html_public/api_overview.html https://flow-ipc.github.io/doc/flow-ipc/versions/main/genera...