7 ms·
Unix Domain Sockets vs Loopback TCP Sockets (2014)
- c7DJTLrn 3y agoA lot of modern software disregards the existence of unix sockets, probably because TCP sockets are an OS agnostic concept and perform well enough. You'd need to write Windows-specific code to handle named pipes if you didn't want to use TCP sockets.
- giovannibonetti 3y agoI imagine there should be some OS-agnostic libraries somewhere that handle it and provide the developer a unified interface.
- nsteel 3y agoGoing forward, hopefully modern software will use the modern approach of AF_UNIX sockets in Windows 10 and above: https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ https://devblogs.microsoft.com/commandline/af_unix-comes-to-... EDIT: And it would be interesting for someone to reproduce a benchmark like this on Windows to compare TCP loopback and the new(ish) unix socket support.
- deleted 3y ago[deleted]
- zamadatix 3y agoA couple of years after this article came out Windows added support for SOCK_STREM Unix sockets.
- rnmmrnm 3y agowindows is exactly the reason they didn't prevail imo. Windows named pipes have weird security caveats and are not really supported in high level languages. I think this lead everyone to just using loopback TCP as the portable IPC communication API instead of going with unix sockets.
- duped 3y agoIME a lot of developers have never even heard of address families and treat "socket" as synonymous with TCP (or possibly, but rarely, UDP).
- deleted 3y ago[deleted]
- jjice 3y agoWindows actually added Unix sockets about six years ago, and with how aggressive Microsoft EOLs older versions of their OS (relative to something like enterprise linux at least), it's probably a pretty safe bet to use at this point. https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ https://devblogs.microsoft.com/commandline/af_unix-comes-to-...
- Aachen 3y agoWith how aggressively Microsoft EOLs older versions of their OS, we're still finding decades-old server and client systems at clients. While Server 2003 is getting more rare and the last sighting of Windows 98/2000 has been a while, they're all running at the very least a few months after the last free security support is gone. But whether that's something you want to support as a developer is your choice to make.
- marcosdumay 3y agoThat's not very relevant. If you start developing a new software today, it won't need to run on those computers. And if it's old enough that it need to, you can bet all of those architectural decisions were already made and written into stone all over the place.
- radlad 3y ago> If you start developing a new software today, it won't need to run on those computers. This is a weird argument to make. For context, I work on mesh overlay VPNs at Defined.net. We initially used Unix domain sockets for our daemon-client control model. This supported Windows 10 / Server 2019+. We very quickly found our users needed support for Server 2016. Some are even still running 2012. Ultimately, as a software vendor, we can't just force customers to upgrade their datacenters.
- foobiekr 3y agoIt’s actually the opposite of Microsoft quickly eoling on the server side. Server 2012 was EVERYWHERE as late as 2018-2019. They were still issuing service packs in 2018.
- eptcyka 3y agoYes, but there's NamedPipes and they can be used the same way on Windows. And Windows also supports UDS as well today. It's no excuse.
- duped 3y agoWhat's in the way of TCP hitting the same performance as unix sockets, is it just netfilter?
- woodruffw 3y agoI believe the conventional wisdom here is that UDS performs better because of fewer context switches and copies between userspace and kernelspace.
- foobiekr 3y agoNo. This is exactly the same. Think about life of a data gram or stream bytes on the syscall edge for each.
- woodruffw 3y agoI’m not sure I understand. This isn’t something I haven’t thought about in a while, but it’s pretty intuitive to me that a loopback TCP connection would pretty much always be slower: each transmission unit goes through the entire TCP stack, feeds into the TCP state machine, etc. Thats more time spent in the kernel.
- foobiekr 3y agoYea but those aren’t context switches.
- noselasd 3y agoTCP has a lot of rules nailed down in numerous RFCs - everything from how to handle sequence numbers, the 3-way handshake, congestion control, and much more. That translates into a whole lot of code that needs to run, while unix sockets are not that much more than a kernel buffer and code to copy data back and forth in that buffer - which doesn't need a lot of code to make happen.
- foobiekr 3y agoThe ip stack.
- rollcat 3y agoI'd be more interested in the security and usability aspect. Loopback sockets (assuming you don't accidentally bind to 0.0.0.0, which would make it even worse) are effectively rwx to any process on the same machine that has the permission to open network connections, unless you bother with setting up a local firewall (which requires admin privileges). On top of that you need to figure out which port is free to bind to, and have a backup plan in case the port isn't free. Domain sockets are simpler in both aspects: you can create one in any suitable directory, give it an arbitrary name, chmod it to control access, etc.
- majke 3y agoAlways use Unix Domain sockets if you can. There are at least three concerns with TCP. First, local port numbers are a limited resource. https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/ https://blog.cloudflare.com/how-to-stop-running-out-of-ephem... https://blog.cloudflare.com/the-quantum-state-of-a-tcp-port/ https://blog.cloudflare.com/the-quantum-state-of-a-tcp-port/ https://blog.cloudflare.com/this-is-strictly-a-violation-of-the-tcp-specification/ https://blog.cloudflare.com/this-is-strictly-a-violation-of-... Then the TCP buffer autotune can go berserk: https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-fo... https://blog.cloudflare.com/when-the-window-is-not-fully-open-your-tcp-stack-is-doing-more-than-you-think/ https://blog.cloudflare.com/when-the-window-is-not-fully-ope... Finally, conntrack. https://blog.cloudflare.com/conntrack-tales-one-thousand-and-one-flows/ https://blog.cloudflare.com/conntrack-tales-one-thousand-and... https://blog.cloudflare.com/conntrack-turns-a-blind-eye-to-dropped-syns/ https://blog.cloudflare.com/conntrack-turns-a-blind-eye-to-d... These issues don't exist in Unix Sockets land.
- rwmj 3y agoAF_VSOCK is another one to consider these days. It's a kind of hybrid of loopback and Unix. Although they are designed for communicating between virtual machines, vsock sockets work just as well between regular processes. Also supported on Windows. https://www.man7.org/linux/man-pages/man7/vsock.7.html https://www.man7.org/linux/man-pages/man7/vsock.7.html https://wiki.qemu.org/Features/VirtioVsock https://wiki.qemu.org/Features/VirtioVsock
- cout 3y agoI did not know about this. Thanks for the tip!
- touisteur 3y agoWith some luck and love in the future hopefully we'll also be able to use them in containers https://patchwork.kernel.org/project/kvm/cover/20200116172428.311437-1-sgarzare@redhat.com/ https://patchwork.kernel.org/project/kvm/cover/2020011617242... which would simplify a lot of little things.
- tptacek 3y agoWhat's the advantage to vsocks over Unix domain sockets? UDS's are very fast, and much easier to use.
- rwmj 3y agoI didn't mean to imply any advantage, just that they are another socket-based method for two processes to communicate. Since vsocks use a distinct implementation they should probably be benchmarked alongside Unix domain sockets and loopback sockets in any comparisons. My expectation is they would be somewhere in the middle - not as well optimized as Unix domain sockets, but with less general overhead than TCP loopback. If you are using vsocks between two VMs as intended then they have the advantage that they allow communication without involving the network stack. This is used by VMs to implement guest agent communications (screen resizing, copy and paste and so on) where the comms don't require the network to have been set up at all or be routable to the host.
- 3y ago
- _8j50 3y agoWhy not UDP? Less overhead and you can use multicast to expand messaging to machines in a lan. TCP on localhost makes little sense, especially when simple ack's can be implemented in UDP. But even then, I wonder how the segmentation in TCP is affecting performance in addition to windowing. Another thing I always wanted to try was using raw IP packets, why not? Just sequence requests and let the sender close a send transaction only when it gets an ack packet with the sequence # for each send. Even better, a raw AF_PACKET socket on the loopback interface! That might beat UDS!
- sophacles 3y agoGive it a try and find out! I'd give that blog post a read. I suspect you'd run into all sorts of interesting issues... particularly if the server is one process but there are N>1 clients and you're using AF_PACKET.
- _8j50 3y agoWhy not, I will try to not let the downvotes discourage me lol
- spacechild1 3y ago> Two communicating processes on a single machine have a few options Curiously, the article does not even mention pipes, which I would assume to be the most obvious solution for this task (but not necessarily the best, of course!) In particular, I am wondering how Unix domain sockets compare to (a pair of) pipes. At first glance, they appear to be very similar. What are the trade-offs?
- tptacek 3y agoThe pipe vs. socket perf debate is a very old one. Sockets are more flexible and tunable, which may net you better performance (for instance, by tweaking buffer sizes), but my guess is that the high order bit of how a UDS and a pipe perform are the same. Using pipes instead of a UDS: * Requires managing an extra set of file descriptors to get bidirectionality * Requires processes to be related * Surrenders socket features like file descriptor passing * Is more fiddly than the socket code, which can often be interchangeable with TCP sockets (see, for instant, the Go standard library) If you're sticking with Linux, I can't personally see a reason ever to prefer pipes. A UDS is probably the best default answer for generic IPC on Linux.
- deleted 3y ago[deleted]
- xuhu 3y agoWith pipes, the sender has to add a SIGPIPE handler which is not trivial to do if it's a library doing the send/recv. With sockets it can use send(fd, buf, MSG_NOSIGNAL) instead.
- svanwaa 3y agoWould TCP_NODELAY make any difference (good or bad)?
- deleted 3y ago[deleted]
- inv2004 3y agoWould be better to retest If I remember correct, we had the same results described in article in 2014, but also I remember that linux loopback was optimized after it and different was much smaller if visible
- dugite-code 3y agoOne of the best things I did for my old home-server was to switch to Unix sockets for the majority of my selfhosted services' databases, the performance difference on that very low end hardware was monumental. Now I'm on NVME with a modern CPU the difference isn't as marked but why leave any performance on the table?