4 ms·

Speed Limit of PaaS - 64K TCP Ports

20 points by kowsik 15y ago

MostAwesomeDude 15y ago
There are several assumptions in this article. The biggest is that, at scale, the same NIC will be used for both front-end and back-end stuff. What if it's not? Then you aren't penalized for your back-end ports on the front-end because they're on different routes. Also, why is Node special here? Did people not notice Comet, Athena, Flash, Java, etc.? We've been doing long-polling and extended socket usage for a long time, not to mention browser connection reuse. This port count problem hasn't been a problem at all yet. Finally, scaling up to 60k useful active concurrent connections on a single box is pretty damn tricky. Usually, you'll be splitting that kind of load, and doing NAT, and all of a sudden you no longer have to worry about port exhaustion.
- kowsik 15y ago
  My observation/assumption is that your app is running in a multi-tenant PaaS environment, where mysteriously you start getting bind errors because the ephemeral port space is shared with other tenants.
  - MostAwesomeDude 15y ago
    You have observed this? There's a trick to the way sockets work: They track a bunch of data about each connection, not just the local port number and IP. You're talking about exhausting a space of 60k+ connections/user, not connections/server. If you've actually seen this in the wild, then I'd like to know what you're running, what you're selling, how you're doing so remarkably well, etc. This sounds like the best problem to have. :3
    - nknight 15y ago
      No, it's connections/server, unless the provider has separate back-end IPs for each user (not a bad idea, but I'll bet they're not doing that). The tuple consists of the source IP, destination IP, source port, and destination port. The first, second, and last are, in a naive implementation, potentially universal to all applications/instances/sessions on the server, so your number space is reduced to the ~60k source ports. This is an incredibly easy problem to solve, but it does need a small bit of attention by PaaS providers to ensure they're not going to hit it.
      - MostAwesomeDude 15y ago
        I am confused as to how the source IP can be the same for every connection, as even the naivest socket server will be accepting connections from many different users, each with their own source IP, which makes this 60kconns/port/user.
        nknight 15y ago
        You and several other people are completely failing to read and comprehend the blog post in its entirety. This has nothing whatsoever to do with where USERS are connecting from. If you have X sessions on a single front-end server opening one or more individual connections to a single back-end server (e.g. CouchDB), at least X ephemeral ports are used on the front-end server to connect to the back-end server. As X increases, you can reach a point where you have a serious problem. If X gets anywhere near ~60k, it's game over.
        kowsik 15y ago
        thank you, thank you! whew Hopefully the new picture on the blog clarifies which ports I'm talking about.
        MostAwesomeDude 15y ago
        Oh. Well, looks like I was stupid on the Internet again.
superjared 15y ago
This is bollocks. The argument is that you can only have ~60k connections due to the limit of ephemeral ports. This applies on a per-client basis, so you can have nearly limitless connections. I wrote about this here under "The 64k connection myth": http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-for-c500k/ http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-...
- kowsik 15y ago
  Read the blog again. This is bollocks if you are running your own server where you can be multi-homed, add loopback aliases etc and crank up the memory to support 100,000+ sockets. I'm talking PaaS where your app is an multi-tenant environment with no root access.
  - superjared 15y ago
    You don't need loopback aliases for > 64k sockets. You can have 64k connections to your server per ip address that connects. EDIT: Formatting
    - kowsik 15y ago
      Okay, we are talking past each other. If you have a server on an EC2 instance, with enough RAM you can support more than 64K inbound connections for exactly the reasons you mention. Try generating more than 64K connections outbound and this is exactly what I'm talking about.
      - superjared 15y ago
        You don't need that either. The limitation is based on a tuple like this: (incoming ip, outgoing ip, standard port, ephemeral port) So yes, you're limited to 64k connections to a single MySQL port, for example.
        bigiain 15y ago
        Yeah, inbound connection to your service are not the issue the article is discussing. It's the outbound connection (for example, to memcached or mongodb) that your service makes in response to those potentially-greater-than-64k in bound connections that might trip you up. The site I'm working on right now makes 4 memcached requests for a logged in user hitting the homepage - that design may well fall apart at ~16k simultaneous connections if it's running on a shared host paas type host (or well before that if I'm sharing resources with another customer on the same paas hardware as my app)
ericflo 15y ago
I wasn't able to understand what the author was trying to say under "Connection Pools"--is he suggesting that the database should be written using node.js so that, presumably, it can "scale out" and not require a connection pool?
- superjared 15y ago
  Either that, or allow for >64k simultaneous connections from node so that each of his requests has its own connection. The author doesn't seem to understand that more connections to a database does not equal higher throughput.
  - kowsik 15y ago
    We had a node.js app where each inbound request would go through a set of HTTP connections to the back-end couchDB. When the concurrency reached around 1,000 then there were 1,000 requests made through 10 couchDB pipelined connections. The result was the web requests were starting to get slower and slower. Think of a 6-lane road merging into a 1-lane road. Congestion.
    - wmf 15y ago
      Sure, if you're trying to shove 1,000 things through 10 connections you're going to get contention. But you could open 10K pooled connections to several different backends and still stay under the 64K limit; this should be OK up to concurrency of ~100K. Beyond that, there's always SCTP or SPDY...
    - ericflo 15y ago
      So your assertion is that if you bumped up the CouchDB connection limit to 1,000 (believe it or not, Erlang--in addition to Node--is capable of this), then this slowdown would not have occurred?
      - kowsik 15y ago
        My point is the thing you are connecting in the back-end has to scale at the same level as node - if not better. I'm sure making independent non-pipelined requests to CouchDB will help, but if CouchDB views are slow or it can't handle the concurrency, then the web requests to node.js will start slowing down resulting in a pipeline stall.
prodigal_erik 15y ago
Strange to warn of a problem having already dismissed the solution (pooling). Unless your backend actually has fifty thousand cores, making fifty thousand concurrent connections to it accomplishes nothing more than moving the bottleneck around while defeating TCP flow control. Capacity is always finite, some part of the system is inevitably going to play the role of a bounded queue, and you want that visible and close to (preferably in) the frontend so you can shed load through degraded service.
DasIch 15y ago
Before you even come close to that limit any decent PaaS solution should move your application to a less-used server or spawn your application on several servers by load alone. (Assuming you invested enough money.)
kowsik 15y ago
Talking about port exhaustion is, I realize, a little confusing because there are the web requests coming in and the web-tier is making outbound requests to other services. Adding an image to the blog to hopefully clarify things.
salem 15y ago
Also, if there is high churn on those outgoing connections, then the problem is a little worse. The connections per second will be much lower than 60k, because of the timeout before the port number is reused for a new connection. Although, this timeout can be tuned.

20 points by kowsik 15y ago

There are several assumptions in this article. The biggest is that, at scale, the same NIC will be used for both front-end and back-end stuff. What if it's not? Then you aren't penalized for your back-end ports on the front-end because they're on different routes. Also, why is Node special here? Did people not notice Comet, Athena, Flash, Java, etc.? We've been doing long-polling and extended socket usage for a long time, not to mention browser connection reuse. This port count problem hasn't been a problem at all yet. Finally, scaling up to 60k useful active concurrent connections on a single box is pretty damn tricky. Usually, you'll be splitting that kind of load, and doing NAT, and all of a sudden you no longer have to worry about port exhaustion.

kowsik 15y ago
My observation/assumption is that your app is running in a multi-tenant PaaS environment, where mysteriously you start getting bind errors because the ephemeral port space is shared with other tenants.
- MostAwesomeDude 15y ago
  You have observed this? There's a trick to the way sockets work: They track a bunch of data about each connection, not just the local port number and IP. You're talking about exhausting a space of 60k+ connections/user, not connections/server. If you've actually seen this in the wild, then I'd like to know what you're running, what you're selling, how you're doing so remarkably well, etc. This sounds like the best problem to have. :3
  - nknight 15y ago
    No, it's connections/server, unless the provider has separate back-end IPs for each user (not a bad idea, but I'll bet they're not doing that). The tuple consists of the source IP, destination IP, source port, and destination port. The first, second, and last are, in a naive implementation, potentially universal to all applications/instances/sessions on the server, so your number space is reduced to the ~60k source ports. This is an incredibly easy problem to solve, but it does need a small bit of attention by PaaS providers to ensure they're not going to hit it.
    - MostAwesomeDude 15y ago
      I am confused as to how the source IP can be the same for every connection, as even the naivest socket server will be accepting connections from many different users, each with their own source IP, which makes this 60kconns/port/user.
      - nknight 15y ago
        You and several other people are completely failing to read and comprehend the blog post in its entirety. This has nothing whatsoever to do with where USERS are connecting from. If you have X sessions on a single front-end server opening one or more individual connections to a single back-end server (e.g. CouchDB), at least X ephemeral ports are used on the front-end server to connect to the back-end server. As X increases, you can reach a point where you have a serious problem. If X gets anywhere near ~60k, it's game over.
        kowsik 15y ago
        thank you, thank you! whew Hopefully the new picture on the blog clarifies which ports I'm talking about.
        MostAwesomeDude 15y ago
        Oh. Well, looks like I was stupid on the Internet again.

superjared 15y ago

This is bollocks. The argument is that you can only have ~60k connections due to the limit of ephemeral ports. This applies on a per-client basis, so you can have nearly limitless connections. I wrote about this here under "The 64k connection myth": http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-for-c500k/ http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-...

kowsik 15y ago
Read the blog again. This is bollocks if you are running your own server where you can be multi-homed, add loopback aliases etc and crank up the memory to support 100,000+ sockets. I'm talking PaaS where your app is an multi-tenant environment with no root access.
- superjared 15y ago
  You don't need loopback aliases for > 64k sockets. You can have 64k connections to your server per ip address that connects. EDIT: Formatting
  - kowsik 15y ago
    Okay, we are talking past each other. If you have a server on an EC2 instance, with enough RAM you can support more than 64K inbound connections for exactly the reasons you mention. Try generating more than 64K connections outbound and this is exactly what I'm talking about.
    - superjared 15y ago
      You don't need that either. The limitation is based on a tuple like this: (incoming ip, outgoing ip, standard port, ephemeral port) So yes, you're limited to 64k connections to a single MySQL port, for example.
      - bigiain 15y ago
        Yeah, inbound connection to your service are not the issue the article is discussing. It's the outbound connection (for example, to memcached or mongodb) that your service makes in response to those potentially-greater-than-64k in bound connections that might trip you up. The site I'm working on right now makes 4 memcached requests for a logged in user hitting the homepage - that design may well fall apart at ~16k simultaneous connections if it's running on a shared host paas type host (or well before that if I'm sharing resources with another customer on the same paas hardware as my app)

ericflo 15y ago

I wasn't able to understand what the author was trying to say under "Connection Pools"--is he suggesting that the database should be written using node.js so that, presumably, it can "scale out" and not require a connection pool?

superjared 15y ago
Either that, or allow for >64k simultaneous connections from node so that each of his requests has its own connection. The author doesn't seem to understand that more connections to a database does not equal higher throughput.
- kowsik 15y ago
  We had a node.js app where each inbound request would go through a set of HTTP connections to the back-end couchDB. When the concurrency reached around 1,000 then there were 1,000 requests made through 10 couchDB pipelined connections. The result was the web requests were starting to get slower and slower. Think of a 6-lane road merging into a 1-lane road. Congestion.
  - wmf 15y ago
    Sure, if you're trying to shove 1,000 things through 10 connections you're going to get contention. But you could open 10K pooled connections to several different backends and still stay under the 64K limit; this should be OK up to concurrency of ~100K. Beyond that, there's always SCTP or SPDY...
  - ericflo 15y ago
    So your assertion is that if you bumped up the CouchDB connection limit to 1,000 (believe it or not, Erlang--in addition to Node--is capable of this), then this slowdown would not have occurred?
    - kowsik 15y ago
      My point is the thing you are connecting in the back-end has to scale at the same level as node - if not better. I'm sure making independent non-pipelined requests to CouchDB will help, but if CouchDB views are slow or it can't handle the concurrency, then the web requests to node.js will start slowing down resulting in a pipeline stall.

prodigal_erik 15y ago

Strange to warn of a problem having already dismissed the solution (pooling). Unless your backend actually has fifty thousand cores, making fifty thousand concurrent connections to it accomplishes nothing more than moving the bottleneck around while defeating TCP flow control. Capacity is always finite, some part of the system is inevitably going to play the role of a bounded queue, and you want that visible and close to (preferably in) the frontend so you can shed load through degraded service.

DasIch 15y ago

Before you even come close to that limit any decent PaaS solution should move your application to a less-used server or spawn your application on several servers by load alone. (Assuming you invested enough money.)

kowsik 15y ago

Talking about port exhaustion is, I realize, a little confusing because there are the web requests coming in and the web-tier is making outbound requests to other services. Adding an image to the blog to hopefully clarify things.

salem 15y ago

Also, if there is high churn on those outgoing connections, then the problem is a little worse. The connections per second will be much lower than 60k, because of the timeout before the port number is reused for a new connection. Although, this timeout can be tuned.