6 ms·
How fast are Linux pipes anyway? (2022)
- aeonik 1y agoI feel bad that this doesn't have any comments, the article was really great. I'd like to use splice more, but the end of the article talked about the security implications and some ABI breaking. I'm curious to know if long term plans are to keep splice around? I'd also be curious how hard it would be to patch the default pipe to always use splice for performance improvements.
- amelius 1y agoFor more comments, see: https://news.ycombinator.com/item?id=44347412 https://news.ycombinator.com/item?id=44347412
- gkfasdfasdf 1y agoGreat article, discussed previously on HN: https://news.ycombinator.com/item?id=31592934 https://news.ycombinator.com/item?id=31592934 (200 comments) https://news.ycombinator.com/item?id=37782493 https://news.ycombinator.com/item?id=37782493 (105 comments)
- gigatexal 1y agoThis is such a dope article. I love that it comes from time to time.
- gigatexal 1y agos/comes/comes up
- layer8 1y ago(2022)
- zh3 1y agoSeared into my soul is the experience porting a linux pipe-based application to Windows, thinking it's all posix and given it's all in memory the performance will be more or less the same. The performance was hideous, even after we found that having pipes waiting for a connection more or less ground windows to a halt. Some years later this got revisited due to needing to use the same thing under C# on Win10 and while it was better it was still a major embarrassment how big the performance gap was.
- andrewmcwatters 1y agoDid you find that you needed interprocess communication to replace the gap?
- spacechild1 1y agopipes are a form of interprocess communication :) I guess you meant shared memory?
- andrewmcwatters 1y agoYes. Yeah, you're right. Sockets could also be used, but I guess when I think of IPC, I generally think of shared memory.
- asveikau 1y agoSome years back Windows added AF_UNIX sockets, I wonder how those would perform relative to Win32 pipes. My guess is better.
- manwe150 1y agoSeems to reportedly be slightly faster in a few cases, but nothing particularly dramatic https://www.yanxurui.cc/posts/server/2023-11-28-benchmark-tcp-uds-namedpipe/ https://www.yanxurui.cc/posts/server/2023-11-28-benchmark-tc...
- 1y ago
- johnisgood 1y agoFWIW there is readv() / writev(), splice(), sendfile(), funopen(), and io_buffer() as well. splice() is great when transferring data between pipes and UNIX sockets with zero-copy, but it is Linux-only. splice() is the fastest and most efficient way to transfer data through pipes (on Linux), especially for large volumes. It bypasses memory allocations in userspace (as opposed to read(v)/write(v)), there is no extra buffer management logic, there is no memcpy() or iovec traversal. Sadly on BSDs, for pipes, readv() / writev() is the most performant way to achieve the same if I am not mistaken. Please correct me if I am wrong. At any rate, this is a great article.
- messe 1y ago> sendfile() is file-to-socket (zero-copy as well), and has very high performance as well, for both Linux and BSDs. It only supports file-to-socket, however, and well, to stay relevant, sendmsg() can't be used with pipes in the general case, it is for UNIX domain sockets, INET sockets, and other socket types. On Linux, sendfile supports more than just file to socket, as it's implemented using splice. I've used it for file-to-block-device in the past.
- johnisgood 1y agoOn BSDs probably not, as they don't have splice, but that is good to know. I wonder if on BSDs it really is readv() and writev() that are the fastest way to achieve the same thing as has been done in the article. Maybe I am missing something. I would like to be corrected.
- lukeh 1y agoDoes modern Linux have anything close to Doors? I’ve an embedded application where two processes exchange small amounts of data which are latency sensitive, and I’m wondering if there’s anything better than AF_UNIX.
- the8472 1y agoshared memory provides the lowest latency, but you still need to deal with task wakeup, which is usually done via futexs. Google was working on a FUTEX_SWAP call for linux which would have allowed direct handover from one task to another, not sure what happened to that.
- Galanwe 1y agoIf you really want low latency, then you should be OK to trade power/CPU for it, and you can just spin instead of being woken up.
- mort96 1y agoWould be helpful to know what your problem is with AF_UNIX at the moment. Is it lacking in features you want? Is it higher latency than you'd want? Is the server/client socket API style not appropriate for your use-case?
- lukeh 1y agoWell, it’s probably fine but, it’s an audio application where metering (not audio) is delivered from a control plane process to a UI process. Lower latency is better. But haven’t measured it.
- themerone 1y agoWhat are Doors, it's too common a word to Google.
- kjellsbells 1y agoLightweight IPC invented by Sun. https://en.m.wikipedia.org/wiki/Doors_(computing) https://en.m.wikipedia.org/wiki/Doors_(computing) Look for Doors solaris and there are quite a few articles.
- deleted 1y ago[deleted]