4 ms·

Maintaining 65k open connections in a single Ruby process

58 points by WJW 8y ago

verdverm 8y ago
Reminds me of https://medium.freecodecamp.org/million-websockets-and-go-cc58418460bb https://medium.freecodecamp.org/million-websockets-and-go-cc...
romed 8y ago
As they say: is that a lot?
- kbenson 8y ago
  Well, if it's TCP/UDP connections, there's only 65k ports available to assign, so it would be all the ports. That's a lot of ports, but I'm not sure why it's supposed to be particularly hard. Off to read the article now... Edit: As masklin referenced in reply to me[1] that jsnell pointed out, that's the ports for a single IP. So yeah, it's not all the ports, but it is all the ports for a single source IP, unless I'm missing something. You can of course assign multiple source IPs to extend this, but it is a limit to be dealt with. 1: And then deleted it, at least as of now, even though I think it was a good point.
  - deleted 8y ago
    [deleted]
  - anothergoogler 8y ago
    I think you're confused about network programming in general. In a post like this, the assumption is that the concurrency is happening over a single network interface. It doesn't matter really. The most conspicuous limiting factor on a POSIX-compliant system is the number of available file descriptors. When you listen() on an address and accept() a connection, the networking stack allocates a file descriptor for that connection. Then you can handle more connections. http://pubs.opengroup.org/onlinepubs/9699919799/functions/accept.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/ac... http://pubs.opengroup.org/onlinepubs/9699919799/functions/listen.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/li... Beej's guide is a popular intro, but I'd bet any text on network programming covers sockets. https://beej.us/guide/bgnet/ https://beej.us/guide/bgnet/
    - kbenson 8y ago
      Yes, I was actually mistaken in assuming a server would use a port only once per IP for all connections, and not just once per client IP (and even farther, client IP and client port combo). It makes much more sense, and in fact I'd been pondering that since I commented as how that could work in practice with servers that actually keep connections open but can handle a lot of requests wasn't obvious in my prior mental model. > Beej's guide is a popular intro, but I'd bet any text on network programming covers sockets. I highly suspect I knew this at some point, but forgot it over the past 15-20 years since it didn't have direct relevance to most of my projects since that time. :/
      - deleted 8y ago
        [deleted]
- PopeDotNinja 8y ago
  As they say: it depends :) . Practically speaking, think of it this way... Total computer power - Cost of maintaining sockets = Available computer power to use sockets. If total computer power - cost of maintaining sockets is close to 0, there's nothing one can really do with those sockets. If the system has 65K sockets open is it's barely breaking a sweat, you can still do plenty of work!
edoo 8y ago
The issue was solved with epoll back in 2004. All modern socket processing should be using async IO. This is just an example of how to hold the sockets open and use it more like a TCP server. Ruby might be the good choice for this if you only know Ruby.
- WJW 8y ago
  Hey, author here! This thing came out one of the internal hackathons in our company and I decided to make a blog post about because it gives some interesting insights into Ruby internals. It's not meant for production and if you need something like this "for real", I would always advise Go, Elang/Elixir or Haskell as a solid base to build on. With MRI Ruby you'll still effectively be limited to one core due to the GIL, so if you need to do any compute intensive work for each socket instead of just sending a string you'll rapidly run into that barrier. As I state in the post, memory usage is not all that great either.
- protomyth 8y ago
  https://illumos.org/man/5/epoll https://illumos.org/man/5/epoll While a best effort has been made to mimic the Linux semantics, there are some semantics that are too peculiar or ill-conceived to merit accommodation. In particular, the Linux epoll facility will -- by design -- continue to generate events for closed file descriptors where/when the underlying file description remains open. For example, if one were to fork(2) and subsequently close an actively epoll'd file descriptor in the parent, any events generated in the child on the implicitly duplicated file descriptor will continue to be delivered to the parent -- despite the fact that the parent itself no longer has any notion of the file description! This epoll facility refuses to honor these semantics; closing the EPOLL_CTL_ADD'd file descriptor will always result in no further events being generated for that event description. Brian Cantrill on epoll https://youtu.be/l6XQUciI-Sc?t=3424 https://youtu.be/l6XQUciI-Sc?t=3424
- masklinn 8y ago
  > The issue was solved with epoll back in 2004. You mean kqueue, and that was in 2000. Epoll came later, and is bad.
jsnell 8y ago
> The maximum amount of connections achieved was 65523, just 13 short of 2^16. This is the theoretical maximum number of connections for a single IP address on a single port, That's not right. Connections are identified by the 4-tuple of (source ip, source port, destination ip, destination port). The server ip and port are fixed in this example, but there's still 2^48 combinations of client ip/port. > It is amazing that a problem which took huge engineering efforts to solve back in the early 2000s can now be solved easily by anyone in just a handful lines of code. But you didn't solve the problem! The whole point of C10K was the need for scalable ways to figure out which connections could be operated on (i.e. had something to receive, or could be written to after having previously been blocked for writes). Keeping an array of sockets and just repeatedly iterating over it with the assumption that all those sockets are active is the opposite of scalable.
- john37386 8y ago
  I'm sure that your ruby can hold +100k connctions with enough RAM.
  - WJW 8y ago
    I ran 4 `ab` processes against it, each with 20k concurrent connections, so the maximum amount of connections "should" have been 80k. The fact that the actual number was so close to 65536 makes me very suspicious that there is some kernel level thing at play here. I agree that the ruby program can probably scale up much more with enough RAM.
    - john37386 8y ago
      Was iptables running? Maybe they hit a conntrack limit
    - tyingq 8y ago
      What does ulimit -a say?
    - andrewmackrodt 8y ago
      Would this have been affected by the number of available ports on your single loopback address, assuming 127.0.0.1. You'd have to connect via multiple network devices. Edit: comment below clarifies servers.
jashmatthews 8y ago
Neat idea! If you're interested in production quality high concurrency Ruby check out https://github.com/postrank-labs/goliath/ https://github.com/postrank-labs/goliath/ which can support hundreds of thousands of concurrent connections.
- WJW 8y ago
  That looks really interesting! I built this during a hackathon and ran out of time before managing to get beyond the 2^16 barrier. I'll definitely take a look at Goliath to see if I can borrow some tricks to push it even further. :)
salim_semaoune 8y ago
i think that the `conn_waiting_area` variable is undefined in the source code

58 points by WJW 8y ago

Reminds me of https://medium.freecodecamp.org/million-websockets-and-go-cc58418460bb https://medium.freecodecamp.org/million-websockets-and-go-cc...

romed 8y ago

As they say: is that a lot?

kbenson 8y ago
Well, if it's TCP/UDP connections, there's only 65k ports available to assign, so it would be all the ports. That's a lot of ports, but I'm not sure why it's supposed to be particularly hard. Off to read the article now... Edit: As masklin referenced in reply to me[1] that jsnell pointed out, that's the ports for a single IP. So yeah, it's not all the ports, but it is all the ports for a single source IP, unless I'm missing something. You can of course assign multiple source IPs to extend this, but it is a limit to be dealt with. 1: And then deleted it, at least as of now, even though I think it was a good point.
- deleted 8y ago
  [deleted]
- anothergoogler 8y ago
  I think you're confused about network programming in general. In a post like this, the assumption is that the concurrency is happening over a single network interface. It doesn't matter really. The most conspicuous limiting factor on a POSIX-compliant system is the number of available file descriptors. When you listen() on an address and accept() a connection, the networking stack allocates a file descriptor for that connection. Then you can handle more connections. http://pubs.opengroup.org/onlinepubs/9699919799/functions/accept.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/ac... http://pubs.opengroup.org/onlinepubs/9699919799/functions/listen.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/li... Beej's guide is a popular intro, but I'd bet any text on network programming covers sockets. https://beej.us/guide/bgnet/ https://beej.us/guide/bgnet/
  - kbenson 8y ago
    Yes, I was actually mistaken in assuming a server would use a port only once per IP for all connections, and not just once per client IP (and even farther, client IP and client port combo). It makes much more sense, and in fact I'd been pondering that since I commented as how that could work in practice with servers that actually keep connections open but can handle a lot of requests wasn't obvious in my prior mental model. > Beej's guide is a popular intro, but I'd bet any text on network programming covers sockets. I highly suspect I knew this at some point, but forgot it over the past 15-20 years since it didn't have direct relevance to most of my projects since that time. :/
    - deleted 8y ago
      [deleted]
PopeDotNinja 8y ago
As they say: it depends :) . Practically speaking, think of it this way... Total computer power - Cost of maintaining sockets = Available computer power to use sockets. If total computer power - cost of maintaining sockets is close to 0, there's nothing one can really do with those sockets. If the system has 65K sockets open is it's barely breaking a sweat, you can still do plenty of work!

edoo 8y ago

The issue was solved with epoll back in 2004. All modern socket processing should be using async IO. This is just an example of how to hold the sockets open and use it more like a TCP server. Ruby might be the good choice for this if you only know Ruby.

WJW 8y ago
Hey, author here! This thing came out one of the internal hackathons in our company and I decided to make a blog post about because it gives some interesting insights into Ruby internals. It's not meant for production and if you need something like this "for real", I would always advise Go, Elang/Elixir or Haskell as a solid base to build on. With MRI Ruby you'll still effectively be limited to one core due to the GIL, so if you need to do any compute intensive work for each socket instead of just sending a string you'll rapidly run into that barrier. As I state in the post, memory usage is not all that great either.
protomyth 8y ago
https://illumos.org/man/5/epoll https://illumos.org/man/5/epoll While a best effort has been made to mimic the Linux semantics, there are some semantics that are too peculiar or ill-conceived to merit accommodation. In particular, the Linux epoll facility will -- by design -- continue to generate events for closed file descriptors where/when the underlying file description remains open. For example, if one were to fork(2) and subsequently close an actively epoll'd file descriptor in the parent, any events generated in the child on the implicitly duplicated file descriptor will continue to be delivered to the parent -- despite the fact that the parent itself no longer has any notion of the file description! This epoll facility refuses to honor these semantics; closing the EPOLL_CTL_ADD'd file descriptor will always result in no further events being generated for that event description. Brian Cantrill on epoll https://youtu.be/l6XQUciI-Sc?t=3424 https://youtu.be/l6XQUciI-Sc?t=3424
masklinn 8y ago
> The issue was solved with epoll back in 2004. You mean kqueue, and that was in 2000. Epoll came later, and is bad.

jsnell 8y ago

> The maximum amount of connections achieved was 65523, just 13 short of 2^16. This is the theoretical maximum number of connections for a single IP address on a single port, That's not right. Connections are identified by the 4-tuple of (source ip, source port, destination ip, destination port). The server ip and port are fixed in this example, but there's still 2^48 combinations of client ip/port. > It is amazing that a problem which took huge engineering efforts to solve back in the early 2000s can now be solved easily by anyone in just a handful lines of code. But you didn't solve the problem! The whole point of C10K was the need for scalable ways to figure out which connections could be operated on (i.e. had something to receive, or could be written to after having previously been blocked for writes). Keeping an array of sockets and just repeatedly iterating over it with the assumption that all those sockets are active is the opposite of scalable.

john37386 8y ago
I'm sure that your ruby can hold +100k connctions with enough RAM.
- WJW 8y ago
  I ran 4 `ab` processes against it, each with 20k concurrent connections, so the maximum amount of connections "should" have been 80k. The fact that the actual number was so close to 65536 makes me very suspicious that there is some kernel level thing at play here. I agree that the ruby program can probably scale up much more with enough RAM.
  - john37386 8y ago
    Was iptables running? Maybe they hit a conntrack limit
  - tyingq 8y ago
    What does ulimit -a say?
  - andrewmackrodt 8y ago
    Would this have been affected by the number of available ports on your single loopback address, assuming 127.0.0.1. You'd have to connect via multiple network devices. Edit: comment below clarifies servers.

jashmatthews 8y ago

Neat idea! If you're interested in production quality high concurrency Ruby check out https://github.com/postrank-labs/goliath/ https://github.com/postrank-labs/goliath/ which can support hundreds of thousands of concurrent connections.

WJW 8y ago
That looks really interesting! I built this during a hackathon and ran out of time before managing to get beyond the 2^16 barrier. I'll definitely take a look at Goliath to see if I can borrow some tricks to push it even further. :)

salim_semaoune 8y ago

i think that the `conn_waiting_area` variable is undefined in the source code