6 ms·

Always use Unix Domain sockets if you can. There are at least three concerns with TCP. First, local port numbers are a limited resource. https://blog.cloudfla

by majke 3y ago

Always use Unix Domain sockets if you can. There are at least three concerns with TCP. First, local port numbers are a limited resource. https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/ https://blog.cloudflare.com/how-to-stop-running-out-of-ephem... https://blog.cloudflare.com/the-quantum-state-of-a-tcp-port/ https://blog.cloudflare.com/the-quantum-state-of-a-tcp-port/ https://blog.cloudflare.com/this-is-strictly-a-violation-of-the-tcp-specification/ https://blog.cloudflare.com/this-is-strictly-a-violation-of-... Then the TCP buffer autotune can go berserk: https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-for-receive-buffers-and-how-we-fixed-it/ https://blog.cloudflare.com/unbounded-memory-usage-by-tcp-fo... https://blog.cloudflare.com/when-the-window-is-not-fully-open-your-tcp-stack-is-doing-more-than-you-think/ https://blog.cloudflare.com/when-the-window-is-not-fully-ope... Finally, conntrack. https://blog.cloudflare.com/conntrack-tales-one-thousand-and-one-flows/ https://blog.cloudflare.com/conntrack-tales-one-thousand-and... https://blog.cloudflare.com/conntrack-turns-a-blind-eye-to-dropped-syns/ https://blog.cloudflare.com/conntrack-turns-a-blind-eye-to-d... These issues don't exist in Unix Sockets land.

lokar 3y ago
Also UDS have more features, for example you can get the remote peer UID and pass FDs
- the8472 3y ago
  And SOCK_SEQPACKET which greatly simplifies fd-passing
  - etaham 3y ago
    +1
    - mike_hock 3y ago
      If only there was a button for that.
  - chadaustin 3y ago
    How does SOCK_SEQPACKET simplify fd-passing? Writing a streaming IPC crate as we speak and wondering if there are land mines beyond https://gist.github.com/kentonv/bc7592af98c68ba2738f4436920868dc https://gist.github.com/kentonv/bc7592af98c68ba2738f44369208...
    - the8472 3y ago
      Well, the kernel does create implicit packetization boundary when you attach FDs to a byte-stream... but this is underdocumented and there's an impedance mismatch between byte streams and discrete application-level messages. You can also send zero-sized messages to pass an FD. with byte streams you must send at least one byte. Which means you can send the FDs separately after sending the bytes which makes it easier to notify the application that it should expect FDs (in case it's not always using recvmsg with an cmsg allocation prepared). SEQPACKET just makes it more straight-forward because 1 message (+ancillary data) is always one sendmsg/recvmsg pair.
      - chadaustin 3y ago
        I appreciate your reply! My approach has been to send a header with the number of fds and bytes the next packet will contain, and the number of payload bytes is naturally never 0 in my case.
kevincox 3y ago
The biggest reason for me is that you can use filesystem permissions to control access. Often I want to run a service locally and do auth at the reverse proxy, but if the service binds to localhost then all local processes can access without auth. If I only grant the reverse proxy permissions on the filesystem socket then you can't access without going through the auth.
- piperswe 3y ago
  And with `SO_PEERCRED`, you can even implement more complex transparent authorization & logging based on the uid of the connecting process.
  - kevincox 3y ago
    This is true but to me mostly negates the benefit for this use case. The goal is to offload the auth work to the reverse proxy not to add more rules. Although I guess you could have the reverse proxy listen both on IP and UNIX sockets. It can then do different auth depending on how the connection came in. So you could auth with TLS Cert or Password over IP or using your PID/UNIX account over the UNIX socket.
cout 3y ago
I agree. Always choose unix domain sockets over local TCP if it is an option. There are some valid reasons though to choose TCP. In the past, I've chosen local TCP sockets because I can configure the receive buffer size to avoid burdening the sender (ideally both TCP and unix domain sockets should correctly handle EAGAIN, but I haven't always had control over the code that does the write). IIRC the max buffer size for unix domain sockets is lower than for TCP. Another limitation of unix domain sockets is that the size of the path string must be less than PATH_MAX. I've run into this when the only directory I had write access to was already close to the limit. Local TCP sockets obviously do not have this limitation. Local TCP sockets can also bypass the kernel if you have a user-space TCP stack. I don't know if you can do this with unix domain sockets (I've never tried). I can also use local tcp for websockets. I have no idea if that's possible with unix domain sockets. In general, I choose a shared memory queue for local-only inter-process communication.
- duped 3y ago
  > Local TCP sockets can also bypass the kernel if you have a user-space TCP stack. I don't know if you can do this with unix domain sockets (I've never tried). Kernel bypass exists because hardware can handle more packets than the kernel can read or write, and all the tricks employed are clever workarounds (read: kinda hacks) to get the packets managed in user space. This is kind of an orthogonal problem to IPC, and there's already a well defined interface for multiple processes to communicate without buffering through the kernel - and that's shared memory. You could employ some of the tricks (like LD_PRELOAD to hijack socket/accept/bind/send/recv) and implement it in terms of shared memory, but at that point why not just use it directly? If speed is your concern, shared memory is always the fastest IPC. The tradeoff is that you now have to manage the messaging across that channel.
  - bheadmaster 3y ago
    In my experience, for small unbatchable messages, UNIX sockets are fast enough not to warrant the complexity of dealing with shared memory. However, for bigger and/or batchable messages, shared memory ringbuffer + UNIX socket for synchronization is the most convenient yet fast IPC I've used.
alexvitkov 3y ago
If different components of your system are talking over a pretend network you've already architectured yourself face first into a pile of shit. There's no argument for quality either way so I'll just use TCP sockets and save myself 2 hours when I inevitably have to get it running on Windows.
- radlad 3y ago
  FYI, Windows supports Unix domain sockets since Windows 10 / Server 2019.
  - alexvitkov 3y ago
    Good thing to mention, thanks. That's mostly why I said 2 hours and not a day, as you still have to deal with paths (there's no /run) and you may have to fickle with UAC or god save us NTFS permissions
  - adzm 3y ago
    I had not head of this! Long story short, AF_UNIX now exists for Windows development. https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ https://devblogs.microsoft.com/commandline/af_unix-comes-to-... https://visualrecode.com/blog/unix-sockets/#:~:text=Unix%20domain%20sockets%20are%20an,work%20like%20regular%20network%20sockets https://visualrecode.com/blog/unix-sockets/#:~:text=Unix%20d....
- mhuffman 3y ago
  >If different components of your system are talking over a pretend network you've already architectured yourself face first into a pile of shit. How do you have your file delivery, database, and business logic "talk" to each other? Everything on the same computer is a "pretend network" to some extent, right? Do you always architect your own database right into your business logic along with a web-server as a single monolith? One off SPAs must take 2-3 months!
  - alexvitkov 3y ago
    Yes, replacing a full duty database with an in-process SQLite generally simplifies things if you can afford it. Even if not that's a bad example, since in prod your fat database will be om another computer for real, so you'd never use a Unix socket when developing locally.
booleanbetrayal 3y ago
We've seen observable performance increases in migrating to unix domain sockets wherever possible, as some TCP stack overhead is bypassed.
Karrot_Kream 3y ago
These matter if you have need to bind to multiple ports, but if you're only running a handful of services that need to bind a socket, then port number allocation isn't a big issue. TCP Buffer autotune having problems also matters at certain scale, but in my experience requires a tipping point. TCP sockets also have configurable buffer sizes while Unix sockets have a fixed buffer size, so TCP socket buffers can get much deeper. At my last role we benchmarked TCP sockets vs Unix sockets in a variety of scenarios. In our benchmarks, only certain cases benefited from Unix sockets and generally the complexity of using them in containerized environments made them less attractive than TCP unless we needed to talk to a high throughput cache or we were doing things like farming requests out to a FastCGI process manager. Generally speaking, using less chatty protocols than REST (involving a lot less serde overhead and making it easier to allocate ingest structures) made a much bigger difference. I was actually a huge believer in deferring to Unix sockets where possible, due to blog posts like these and my understanding of the implementation details (I've implemented toy IPC in a toy kernel before), but a coworker challenged me to benchmark my belief. Sure enough on benchmark it turned out that in most cases TCP sockets were fine and simplified a containerized architecture enough that Unix sockets just weren't worth it.
- kelnos 3y ago
  > the complexity of using [UNIX sockets] in containerized environments made them less attractive than TCP Huh, I would think UNIX sockets would be easier; since sharing the socket between the host and a container (or between containers) is as simple as mounting a volume in the container and setting permissions on the socket appropriately. Using TCP means dealing with iptables and seems... less fun. I easily run into cases where the host's iptables firewall interferes with what Docker wants to do with iptables such that it takes hours just to get simple things working properly.
  - Galanwe 3y ago
    This. Especially, docker does a lot of magic dynamically adding/removing iptables rules, which is already a nightmare to manage, so you really want to avoid dealing with more.
  - 3y ago
LinAGKar 3y ago
One problem I've run into when trying to use Unix sockets though is that it can only buffer fairly few messages at once, so if you have a lot of messages in flight at once you can easily end up with sends failing. TCP sockets can handle a lot more messages.
- count 3y ago
  Can't you tune this with sysctl?
  - LinAGKar 3y ago
    You can set net.core.wmem_default, though that's a system-wide setting you have to override. And then you can end up with large message queues instead, if you have a lot of small messages (which is a concern in the embedded systems I work on). The problem is really the large overhead of the messages, of close to a kilobyte per message. TCP sockets have just a fraction of the overhead.
o11c 3y ago
Adjacently, remember that with TCP sockets you can vary the address anywhere within 127.0.0.0/8
- majke 3y ago
  However this is not the case for ipv6. Technically you can use only ::1, unless you do Ipv6 FREEBIND
  - nine_k 3y ago
    You usually have a whole bunch of link-local IPv6 addresses. Can't you use them?
gte525u 3y ago
It's a bit obscure but 127.x.x.x is a /8. So you have quite a few loopback IPs/port combos. I've tested it and it works with Windows, Linux, GHS integrity.
- dathinab 3y ago
  on some systems it's /8 but you still can only bind to 127.0.0.x we ran into that during testing before

by majke 3y ago

lokar 3y ago

Also UDS have more features, for example you can get the remote peer UID and pass FDs

the8472 3y ago
And SOCK_SEQPACKET which greatly simplifies fd-passing
- etaham 3y ago
  +1
  - mike_hock 3y ago
    If only there was a button for that.
- chadaustin 3y ago
  How does SOCK_SEQPACKET simplify fd-passing? Writing a streaming IPC crate as we speak and wondering if there are land mines beyond https://gist.github.com/kentonv/bc7592af98c68ba2738f4436920868dc https://gist.github.com/kentonv/bc7592af98c68ba2738f44369208...
  - the8472 3y ago
    Well, the kernel does create implicit packetization boundary when you attach FDs to a byte-stream... but this is underdocumented and there's an impedance mismatch between byte streams and discrete application-level messages. You can also send zero-sized messages to pass an FD. with byte streams you must send at least one byte. Which means you can send the FDs separately after sending the bytes which makes it easier to notify the application that it should expect FDs (in case it's not always using recvmsg with an cmsg allocation prepared). SEQPACKET just makes it more straight-forward because 1 message (+ancillary data) is always one sendmsg/recvmsg pair.
    - chadaustin 3y ago
      I appreciate your reply! My approach has been to send a header with the number of fds and bytes the next packet will contain, and the number of payload bytes is naturally never 0 in my case.

kevincox 3y ago

The biggest reason for me is that you can use filesystem permissions to control access. Often I want to run a service locally and do auth at the reverse proxy, but if the service binds to localhost then all local processes can access without auth. If I only grant the reverse proxy permissions on the filesystem socket then you can't access without going through the auth.

piperswe 3y ago
And with `SO_PEERCRED`, you can even implement more complex transparent authorization & logging based on the uid of the connecting process.
- kevincox 3y ago
  This is true but to me mostly negates the benefit for this use case. The goal is to offload the auth work to the reverse proxy not to add more rules. Although I guess you could have the reverse proxy listen both on IP and UNIX sockets. It can then do different auth depending on how the connection came in. So you could auth with TLS Cert or Password over IP or using your PID/UNIX account over the UNIX socket.

cout 3y ago

I agree. Always choose unix domain sockets over local TCP if it is an option. There are some valid reasons though to choose TCP. In the past, I've chosen local TCP sockets because I can configure the receive buffer size to avoid burdening the sender (ideally both TCP and unix domain sockets should correctly handle EAGAIN, but I haven't always had control over the code that does the write). IIRC the max buffer size for unix domain sockets is lower than for TCP. Another limitation of unix domain sockets is that the size of the path string must be less than PATH_MAX. I've run into this when the only directory I had write access to was already close to the limit. Local TCP sockets obviously do not have this limitation. Local TCP sockets can also bypass the kernel if you have a user-space TCP stack. I don't know if you can do this with unix domain sockets (I've never tried). I can also use local tcp for websockets. I have no idea if that's possible with unix domain sockets. In general, I choose a shared memory queue for local-only inter-process communication.

duped 3y ago
> Local TCP sockets can also bypass the kernel if you have a user-space TCP stack. I don't know if you can do this with unix domain sockets (I've never tried). Kernel bypass exists because hardware can handle more packets than the kernel can read or write, and all the tricks employed are clever workarounds (read: kinda hacks) to get the packets managed in user space. This is kind of an orthogonal problem to IPC, and there's already a well defined interface for multiple processes to communicate without buffering through the kernel - and that's shared memory. You could employ some of the tricks (like LD_PRELOAD to hijack socket/accept/bind/send/recv) and implement it in terms of shared memory, but at that point why not just use it directly? If speed is your concern, shared memory is always the fastest IPC. The tradeoff is that you now have to manage the messaging across that channel.
- bheadmaster 3y ago
  In my experience, for small unbatchable messages, UNIX sockets are fast enough not to warrant the complexity of dealing with shared memory. However, for bigger and/or batchable messages, shared memory ringbuffer + UNIX socket for synchronization is the most convenient yet fast IPC I've used.

alexvitkov 3y ago

If different components of your system are talking over a pretend network you've already architectured yourself face first into a pile of shit. There's no argument for quality either way so I'll just use TCP sockets and save myself 2 hours when I inevitably have to get it running on Windows.

radlad 3y ago
FYI, Windows supports Unix domain sockets since Windows 10 / Server 2019.
- alexvitkov 3y ago
  Good thing to mention, thanks. That's mostly why I said 2 hours and not a day, as you still have to deal with paths (there's no /run) and you may have to fickle with UAC or god save us NTFS permissions
- adzm 3y ago
  I had not head of this! Long story short, AF_UNIX now exists for Windows development. https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ https://devblogs.microsoft.com/commandline/af_unix-comes-to-... https://visualrecode.com/blog/unix-sockets/#:~:text=Unix%20domain%20sockets%20are%20an,work%20like%20regular%20network%20sockets https://visualrecode.com/blog/unix-sockets/#:~:text=Unix%20d....
mhuffman 3y ago
>If different components of your system are talking over a pretend network you've already architectured yourself face first into a pile of shit. How do you have your file delivery, database, and business logic "talk" to each other? Everything on the same computer is a "pretend network" to some extent, right? Do you always architect your own database right into your business logic along with a web-server as a single monolith? One off SPAs must take 2-3 months!
- alexvitkov 3y ago
  Yes, replacing a full duty database with an in-process SQLite generally simplifies things if you can afford it. Even if not that's a bad example, since in prod your fat database will be om another computer for real, so you'd never use a Unix socket when developing locally.

booleanbetrayal 3y ago

We've seen observable performance increases in migrating to unix domain sockets wherever possible, as some TCP stack overhead is bypassed.

Karrot_Kream 3y ago

These matter if you have need to bind to multiple ports, but if you're only running a handful of services that need to bind a socket, then port number allocation isn't a big issue. TCP Buffer autotune having problems also matters at certain scale, but in my experience requires a tipping point. TCP sockets also have configurable buffer sizes while Unix sockets have a fixed buffer size, so TCP socket buffers can get much deeper. At my last role we benchmarked TCP sockets vs Unix sockets in a variety of scenarios. In our benchmarks, only certain cases benefited from Unix sockets and generally the complexity of using them in containerized environments made them less attractive than TCP unless we needed to talk to a high throughput cache or we were doing things like farming requests out to a FastCGI process manager. Generally speaking, using less chatty protocols than REST (involving a lot less serde overhead and making it easier to allocate ingest structures) made a much bigger difference. I was actually a huge believer in deferring to Unix sockets where possible, due to blog posts like these and my understanding of the implementation details (I've implemented toy IPC in a toy kernel before), but a coworker challenged me to benchmark my belief. Sure enough on benchmark it turned out that in most cases TCP sockets were fine and simplified a containerized architecture enough that Unix sockets just weren't worth it.

kelnos 3y ago
> the complexity of using [UNIX sockets] in containerized environments made them less attractive than TCP Huh, I would think UNIX sockets would be easier; since sharing the socket between the host and a container (or between containers) is as simple as mounting a volume in the container and setting permissions on the socket appropriately. Using TCP means dealing with iptables and seems... less fun. I easily run into cases where the host's iptables firewall interferes with what Docker wants to do with iptables such that it takes hours just to get simple things working properly.
- Galanwe 3y ago
  This. Especially, docker does a lot of magic dynamically adding/removing iptables rules, which is already a nightmare to manage, so you really want to avoid dealing with more.
- 3y ago

LinAGKar 3y ago

One problem I've run into when trying to use Unix sockets though is that it can only buffer fairly few messages at once, so if you have a lot of messages in flight at once you can easily end up with sends failing. TCP sockets can handle a lot more messages.

count 3y ago
Can't you tune this with sysctl?
- LinAGKar 3y ago
  You can set net.core.wmem_default, though that's a system-wide setting you have to override. And then you can end up with large message queues instead, if you have a lot of small messages (which is a concern in the embedded systems I work on). The problem is really the large overhead of the messages, of close to a kilobyte per message. TCP sockets have just a fraction of the overhead.

o11c 3y ago

Adjacently, remember that with TCP sockets you can vary the address anywhere within 127.0.0.0/8

majke 3y ago
However this is not the case for ipv6. Technically you can use only ::1, unless you do Ipv6 FREEBIND
- nine_k 3y ago
  You usually have a whole bunch of link-local IPv6 addresses. Can't you use them?

gte525u 3y ago

It's a bit obscure but 127.x.x.x is a /8. So you have quite a few loopback IPs/port combos. I've tested it and it works with Windows, Linux, GHS integrity.

dathinab 3y ago
on some systems it's /8 but you still can only bind to 127.0.0.x we ran into that during testing before