4 ms·

> The maximum amount of connections achieved was 65523, just 13 short of 2^16. This is the theoretical maximum number of connections for a single IP address on

by jsnell 8y ago

> The maximum amount of connections achieved was 65523, just 13 short of 2^16. This is the theoretical maximum number of connections for a single IP address on a single port, That's not right. Connections are identified by the 4-tuple of (source ip, source port, destination ip, destination port). The server ip and port are fixed in this example, but there's still 2^48 combinations of client ip/port. > It is amazing that a problem which took huge engineering efforts to solve back in the early 2000s can now be solved easily by anyone in just a handful lines of code. But you didn't solve the problem! The whole point of C10K was the need for scalable ways to figure out which connections could be operated on (i.e. had something to receive, or could be written to after having previously been blocked for writes). Keeping an array of sockets and just repeatedly iterating over it with the assumption that all those sockets are active is the opposite of scalable.

john37386 8y ago
I'm sure that your ruby can hold +100k connctions with enough RAM.
- WJW 8y ago
  I ran 4 `ab` processes against it, each with 20k concurrent connections, so the maximum amount of connections "should" have been 80k. The fact that the actual number was so close to 65536 makes me very suspicious that there is some kernel level thing at play here. I agree that the ruby program can probably scale up much more with enough RAM.
  - john37386 8y ago
    Was iptables running? Maybe they hit a conntrack limit
  - tyingq 8y ago
    What does ulimit -a say?
  - andrewmackrodt 8y ago
    Would this have been affected by the number of available ports on your single loopback address, assuming 127.0.0.1. You'd have to connect via multiple network devices. Edit: comment below clarifies servers.
WJW 8y ago
Can you explain more how you get to 2^48? I thought the protocol specification only reserved two bytes for the port and so getting over 65536 connections would either require using more IP adresses or more ports. This seemed supported by answers like https://www.quora.com/What-is-the-maximum-number-of-concurrent-tcp-connections-system-can-support https://www.quora.com/What-is-the-maximum-number-of-concurre.... If you can enlighten me, I'd be much obliged. You are definitely right that this is not a "full" solution for the c10k problem, but it is an interesting start IMO. Maybe for next month I'll implement basic epoll functionality to make an echo server or something :)
- masklinn 8y ago
  > Can you explain more how you get to 2^48? I thought the protocol specification only reserved two bytes for the port and so getting over 65536 connections would either require using more IP adresses or more ports. 32b IP * 16 bit port? > You are definitely right that this is not a "full" solution for the c10k problem, but it is an interesting start IMO. It's not a solution to the C10K at all, because the C10K problem is outdated and was trivialised a decade back. Whatsapp was doing 2 million connections on a single box (with resources to spare) back in 2012: https://blog.whatsapp.com/196/1-million-is-so-2011 https://blog.whatsapp.com/196/1-million-is-so-2011 and folks were working on C10M: http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html http://highscalability.com/blog/2013/5/13/the-secret-to-10-m... > Maybe for next month I'll implement basic epoll functionality to make an echo server or something :) Epoll is bad, why would you want to do that?
  - meguest 8y ago
    > epoll is bad Source please! I'm genuine - what is bad about it?
    - kjeetgill 8y ago
      It comes up all the time but I'm not qualified to speak to it off the top of my head. Google comparisons with kqueue from BSD and iocp on Windows.
tyingq 8y ago
It does make you wonder what was throttling it to 65k. I didn't see any mention of tweaking kernel/ulimit/systemd max open file settings.
- masklinn 8y ago
  Since client connecting to a single port? So you'd get 1 client IP, 65k client ports, connecting to 1 destination IP on 1 port.
  - tyingq 8y ago
    "Eventually we got it working by using four servers each running a single instance of ab in addition to the one running our Ruby process" So 4 source ips. Guessing they were just bumping into a (configurable) max open file setting.
xmichael999 8y ago
haha, ouch that is embarrassing! Write an entire blog post about how you did something awesome, but ........

by jsnell 8y ago

john37386 8y ago

I'm sure that your ruby can hold +100k connctions with enough RAM.

WJW 8y ago
I ran 4 `ab` processes against it, each with 20k concurrent connections, so the maximum amount of connections "should" have been 80k. The fact that the actual number was so close to 65536 makes me very suspicious that there is some kernel level thing at play here. I agree that the ruby program can probably scale up much more with enough RAM.
- john37386 8y ago
  Was iptables running? Maybe they hit a conntrack limit
- tyingq 8y ago
  What does ulimit -a say?
- andrewmackrodt 8y ago
  Would this have been affected by the number of available ports on your single loopback address, assuming 127.0.0.1. You'd have to connect via multiple network devices. Edit: comment below clarifies servers.

WJW 8y ago

Can you explain more how you get to 2^48? I thought the protocol specification only reserved two bytes for the port and so getting over 65536 connections would either require using more IP adresses or more ports. This seemed supported by answers like https://www.quora.com/What-is-the-maximum-number-of-concurrent-tcp-connections-system-can-support https://www.quora.com/What-is-the-maximum-number-of-concurre.... If you can enlighten me, I'd be much obliged. You are definitely right that this is not a "full" solution for the c10k problem, but it is an interesting start IMO. Maybe for next month I'll implement basic epoll functionality to make an echo server or something :)

masklinn 8y ago
> Can you explain more how you get to 2^48? I thought the protocol specification only reserved two bytes for the port and so getting over 65536 connections would either require using more IP adresses or more ports. 32b IP * 16 bit port? > You are definitely right that this is not a "full" solution for the c10k problem, but it is an interesting start IMO. It's not a solution to the C10K at all, because the C10K problem is outdated and was trivialised a decade back. Whatsapp was doing 2 million connections on a single box (with resources to spare) back in 2012: https://blog.whatsapp.com/196/1-million-is-so-2011 https://blog.whatsapp.com/196/1-million-is-so-2011 and folks were working on C10M: http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html http://highscalability.com/blog/2013/5/13/the-secret-to-10-m... > Maybe for next month I'll implement basic epoll functionality to make an echo server or something :) Epoll is bad, why would you want to do that?
- meguest 8y ago
  > epoll is bad Source please! I'm genuine - what is bad about it?
  - kjeetgill 8y ago
    It comes up all the time but I'm not qualified to speak to it off the top of my head. Google comparisons with kqueue from BSD and iocp on Windows.

tyingq 8y ago

It does make you wonder what was throttling it to 65k. I didn't see any mention of tweaking kernel/ulimit/systemd max open file settings.

masklinn 8y ago
Since client connecting to a single port? So you'd get 1 client IP, 65k client ports, connecting to 1 destination IP on 1 port.
- tyingq 8y ago
  "Eventually we got it working by using four servers each running a single instance of ab in addition to the one running our Ruby process" So 4 source ips. Guessing they were just bumping into a (configurable) max open file setting.

xmichael999 8y ago

haha, ouch that is embarrassing! Write an entire blog post about how you did something awesome, but ........