5 ms·

For me SO_REUSEPORT was only a solution to be able to immediately reuse a port when a process crashed in some odd way and the OS was not releasing the port imme

For me SO_REUSEPORT was only a solution to be able to immediately reuse a port when a process crashed in some odd way and the OS was not releasing the port immediately. When I read that multiple processes can use the port it finally made sense to me how nginx is able to spawn so many workers. If they are listening to port 80 or 443 with SO_REUSEPORT, each worker would get a new client in a round robin fashion, effectively spreading the workload without depending on the main parent process to pipe the data to them (and back to the client), which is how I thought that it was working. The funny thing about this submission was that when I read the title I somehow thought I was reading it on Stack Overflow and was expecting a noob asking this question and didn't know what kind of answers I was about to be confronted with and if I should get excited to see a really good one. Then, while in the article, I liked how it started to raise so many issues, fanning out into these special cases which really made it a valid question to ask what it does mean to listen to a port, how complicated it can become. Sadly the article does not go into any detail, like showing snippets of C code from the networking stack and explaining how it is really working, how each requirement (or kluge, for that matter) is being satisfied, but it's OK, it's an article which touches the subject in an enticing enough way while having an excuse to not deep-dive into the details by keeping it in the form of a dialog between two students. I liked it.

y04nn 5y ago
There is data piping needed between the main thread and workers, each new connection is handled to a worker as a socket descriptor, like a pointer.
moltenguardian 5y ago
Are you referring to SO_REUSEADDR?
- osrec 5y ago
  Perhaps, but see https://stackoverflow.com/questions/14388706/how-do-so-reuseaddr-and-so-reuseport-differ https://stackoverflow.com/questions/14388706/how-do-so-reuse...
- qwertox 5y ago
  Thanks for pointing this out. Indeed it was SO_REUSEADDR.
arkj 5y ago
> When I read that multiple processes can use the port it finally made sense to me how nginx is able to spawn so many workers. It’s the socket descriptor that matters. When the listen call unblocks and the thread accepts the connection it gets a new socket descriptor, it’s through this socket the subsequent read/write happens for the connection.
toast0 5y ago
I'm not sure about nginx, but the traditional way to share a listen port is to listen on the port, and then fork. This shares the socket with the child, and if you fork several workers, they can all accept on that socket, and the OS will figure out which one wins (which may or may not be round robin). You can select/kqueue/epoll/whatever if you prefer, as well.
- jhugo 5y ago
  Yes, this is how nginx does it; whether it sets SO_REUSEPORT can be configured.
srgpqt 5y ago
Note that the round robin behavior is platform specific, and AFAIK only happens on linux.
- kevans91 5y ago
  Right, platform specific. On FreeBSD, we have SO_REUSEPORT_LB for load-balancing.
randomswede 5y ago
Typically, the IP stack places the port in TIMEWAIT after the listening process has exited, in order to ensure that packets that were in flight on the network at the time of exit has a chance to dissipate before the next program starts. But, when you say "I am OK with this co-existing", you can listen to a port that is in TIMEWAIT, and quick restart is then possible.
roenxi 5y ago
> ... like showing snippets of C code from the networking stack ... I think this point deserves extra attention. I've been finding that in situations like this it is often much faster to read the implementation source code than try to work it out from documentation and trial-and-error. Something in this (quite realistic) vignette is that there is a lot of potential to mis-learn when the Linux source code for this is probably quite short and readable. This is a particularly useful skill for terse languages where the source code is easy to read. And a good time to feel a bit of empathy towards the poor souls who are forced to use closed source software. In fact, I now know what I'll be skimming through tonight ... [0]. Not short, but a lot more useful than guesswork. [0] https://github.com/torvalds/linux/blob/master/include/net/sock.h#L118 https://github.com/torvalds/linux/blob/master/include/net/so...
- carlmr 5y ago
  Aside from the code I was wondering how much of this is the spec and how much is the specific implementation of the network stack they're using. Is this the same on MacOS? On Windows?

y04nn 5y ago

There is data piping needed between the main thread and workers, each new connection is handled to a worker as a socket descriptor, like a pointer.

moltenguardian 5y ago

Are you referring to SO_REUSEADDR?

osrec 5y ago
Perhaps, but see https://stackoverflow.com/questions/14388706/how-do-so-reuseaddr-and-so-reuseport-differ https://stackoverflow.com/questions/14388706/how-do-so-reuse...
qwertox 5y ago
Thanks for pointing this out. Indeed it was SO_REUSEADDR.

arkj 5y ago

> When I read that multiple processes can use the port it finally made sense to me how nginx is able to spawn so many workers. It’s the socket descriptor that matters. When the listen call unblocks and the thread accepts the connection it gets a new socket descriptor, it’s through this socket the subsequent read/write happens for the connection.

toast0 5y ago

I'm not sure about nginx, but the traditional way to share a listen port is to listen on the port, and then fork. This shares the socket with the child, and if you fork several workers, they can all accept on that socket, and the OS will figure out which one wins (which may or may not be round robin). You can select/kqueue/epoll/whatever if you prefer, as well.

jhugo 5y ago
Yes, this is how nginx does it; whether it sets SO_REUSEPORT can be configured.

srgpqt 5y ago

Note that the round robin behavior is platform specific, and AFAIK only happens on linux.

kevans91 5y ago
Right, platform specific. On FreeBSD, we have SO_REUSEPORT_LB for load-balancing.

randomswede 5y ago

Typically, the IP stack places the port in TIMEWAIT after the listening process has exited, in order to ensure that packets that were in flight on the network at the time of exit has a chance to dissipate before the next program starts. But, when you say "I am OK with this co-existing", you can listen to a port that is in TIMEWAIT, and quick restart is then possible.

roenxi 5y ago

> ... like showing snippets of C code from the networking stack ... I think this point deserves extra attention. I've been finding that in situations like this it is often much faster to read the implementation source code than try to work it out from documentation and trial-and-error. Something in this (quite realistic) vignette is that there is a lot of potential to mis-learn when the Linux source code for this is probably quite short and readable. This is a particularly useful skill for terse languages where the source code is easy to read. And a good time to feel a bit of empathy towards the poor souls who are forced to use closed source software. In fact, I now know what I'll be skimming through tonight ... [0]. Not short, but a lot more useful than guesswork. [0] https://github.com/torvalds/linux/blob/master/include/net/sock.h#L118 https://github.com/torvalds/linux/blob/master/include/net/so...

carlmr 5y ago
Aside from the code I was wondering how much of this is the spec and how much is the specific implementation of the network stack they're using. Is this the same on MacOS? On Windows?