5 ms·
As someone having to debug issues on OpenWRT derivatives, I wish I had a time machine to tell the inventors of Unix to never add other forms of IPC than pipes.
by PhilipRoman 3y ago
As someone having to debug issues on OpenWRT derivatives, I wish I had a time machine to tell the inventors of Unix to never add other forms of IPC than pipes.
It's all a big pile of stateful daemons notifying other daemons with a billion race conditions and zero debugging capabilities, like a parody of how not to create reliable systems. When you have shell scripts parsing JSON messages, you know it's over.
- westurner 3y agoBut then how to add datatypes - schema - to pipes? How do we add a column to the messages on the pipe? Newline-delimited != SOTA https://en.wikipedia.org/wiki/Apache_Arrow https://en.wikipedia.org/wiki/Apache_Arrow : > Arrow allows for zero-copy reads and fast data access and interchange without serialization overhead between these languages and systems JSON lines formatted messages can be UTF-8 JSON-LD: https://jsonlines.org/ https://jsonlines.org/ Linux networking is now all built on eBPF.
- packetlost 3y agoThe same way you add it to any other SOCK_STREAM? How do we add types and schemas to TCP connections?
- westurner 3y agoBut then when I need to scale to more than one node with multiple cores, do pipes scale?
- packetlost 3y agoYes, with some work: https://mazzo.li/posts/fast-pipes.html https://mazzo.li/posts/fast-pipes.html
- westurner 3y agohttps://mazzo.li/posts/fast-pipes.html#splicing https://mazzo.li/posts/fast-pipes.html#splicing : > In general when we write to a socket, or a file, or in our case a pipe, we’re first writing to a buffer somewhere in the kernel, and then let the kernel do its work. In the case of pipes, the pipe is a series of buffers in the kernel. All this copying is undesirable if we’re in the business of performance. > Luckily, Linux includes system calls to speed things up when we want to move data to and from pipes, without copying. Specifically: > - splice moves data from a pipe to a file descriptor, and vice-versa. > - vmsplice moves data from user memory into a pipe. > Crucially, both operations work without copying anything. But then to scale to more than one node, everything has to be copied from the pipe buffer to the network socket buffer; unless splice()'ing from a pipe to a socket is Zero-copy like sendfile() (which doesn't work with pipes IIRC). "SOCKMAP - TCP splicing of the future" (2019) https://blog.cloudflare.com/sockmap-tcp-splicing-of-the-future/ https://blog.cloudflare.com/sockmap-tcp-splicing-of-the-futu...
- packetlost 3y agosendfile is implemented with splice, but the scenario in which splice is actually zero-copy is not well defined (man pages just say "it's generally avoided"), so you'd have to dig into your kernel source to know for sure. That being said, chances are the cost of copying the data would be far outpaced by the cost of serializing over the network anyways, so you would want to architect your application in a way that you're sending as little data over the network as possible if performance is of maximum concern.
- westurner 3y agoI haven't checked, but by the end of the day, I doubt eBPF is much slower than select() on a pipe()?
- hydroid7 3y agoUnfortunately pipes don't cover all kind of IPC we use today.
- the8472 3y agoUsing SOCK_SEQPACKET would be saner than pipes or SOCK_STREAM.
- codehero 3y agoPipes and named FIFOs are easy and great. I say this after implementing various IPC methods (unix domain sockets with fd passing, POSIX message queues, 0MQ, XML RPC, local TCP sockets, just to name a few). Use a simple line oriented protocol. If you are passing complex data through your IPC, you know it's time for files. Shared memory is another way to do IPC but hopefully you have robust method of detecting the liveliness of your local processes and you want to give up the Unix file paradigm.