3 ms·

These matter if you have need to bind to multiple ports, but if you're only running a handful of services that need to bind a socket, then port number allocatio

These matter if you have need to bind to multiple ports, but if you're only running a handful of services that need to bind a socket, then port number allocation isn't a big issue. TCP Buffer autotune having problems also matters at certain scale, but in my experience requires a tipping point. TCP sockets also have configurable buffer sizes while Unix sockets have a fixed buffer size, so TCP socket buffers can get much deeper. At my last role we benchmarked TCP sockets vs Unix sockets in a variety of scenarios. In our benchmarks, only certain cases benefited from Unix sockets and generally the complexity of using them in containerized environments made them less attractive than TCP unless we needed to talk to a high throughput cache or we were doing things like farming requests out to a FastCGI process manager. Generally speaking, using less chatty protocols than REST (involving a lot less serde overhead and making it easier to allocate ingest structures) made a much bigger difference. I was actually a huge believer in deferring to Unix sockets where possible, due to blog posts like these and my understanding of the implementation details (I've implemented toy IPC in a toy kernel before), but a coworker challenged me to benchmark my belief. Sure enough on benchmark it turned out that in most cases TCP sockets were fine and simplified a containerized architecture enough that Unix sockets just weren't worth it.

kelnos 3y ago
> the complexity of using [UNIX sockets] in containerized environments made them less attractive than TCP Huh, I would think UNIX sockets would be easier; since sharing the socket between the host and a container (or between containers) is as simple as mounting a volume in the container and setting permissions on the socket appropriately. Using TCP means dealing with iptables and seems... less fun. I easily run into cases where the host's iptables firewall interferes with what Docker wants to do with iptables such that it takes hours just to get simple things working properly.
- Galanwe 3y ago
  This. Especially, docker does a lot of magic dynamically adding/removing iptables rules, which is already a nightmare to manage, so you really want to avoid dealing with more.
- dathinab 3y ago
  it's an issue of tooling I thing, through dependent on what containerized runtimes e.g. in docker you can use -p to publish ports of containers on the host, this trends to get much more messy less ad-hoc usage where you want to publish them between containers, but docker-compose and similar handle all that for you the benefit of that is this works with the container rubbing using a vm or a namespace created by you or root and it even can work if the container is run somewhere else with pipes you have to volume mount them and do so in a way which works with whatever docker uses to do so, which if you then also mix in docker on windows or Mac can get a bit annoying through of we speak about containerization for apps e.g. using snap/flatpack pipes should work just fine and in the end they are the most common used for cross process communication on the same system, i.e. use case wher you don't have to worry about vms and cross os communication

kelnos 3y ago

> the complexity of using [UNIX sockets] in containerized environments made them less attractive than TCP Huh, I would think UNIX sockets would be easier; since sharing the socket between the host and a container (or between containers) is as simple as mounting a volume in the container and setting permissions on the socket appropriately. Using TCP means dealing with iptables and seems... less fun. I easily run into cases where the host's iptables firewall interferes with what Docker wants to do with iptables such that it takes hours just to get simple things working properly.

Galanwe 3y ago
This. Especially, docker does a lot of magic dynamically adding/removing iptables rules, which is already a nightmare to manage, so you really want to avoid dealing with more.
dathinab 3y ago
it's an issue of tooling I thing, through dependent on what containerized runtimes e.g. in docker you can use -p to publish ports of containers on the host, this trends to get much more messy less ad-hoc usage where you want to publish them between containers, but docker-compose and similar handle all that for you the benefit of that is this works with the container rubbing using a vm or a namespace created by you or root and it even can work if the container is run somewhere else with pipes you have to volume mount them and do so in a way which works with whatever docker uses to do so, which if you then also mix in docker on windows or Mac can get a bit annoying through of we speak about containerization for apps e.g. using snap/flatpack pipes should work just fine and in the end they are the most common used for cross process communication on the same system, i.e. use case wher you don't have to worry about vms and cross os communication