5 ms·

"imagine you have a fleet of 50,000 servers..." It's interesting to consider when you would ever need so many servers, from first principles, considering that a

"imagine you have a fleet of 50,000 servers..." It's interesting to consider when you would ever need so many servers, from first principles, considering that a single server can easily handle a million simultaneous connections, and very few applications are used by every man woman and child on Earth and even fewer are used by multiples of that.

tilolebo 4y ago
> considering that a single server can easily handle a million simultaneous connections Can you elaborate on that part?
- joisig 4y ago
  Not the GP but they might be referring to [0] or one of several other articles you will find if you Google "handle a million connections in *". Realistically you also usually need to perform some non-trivial work from time to time for some non-trivial portion of those connections, which will further load your server, but still. [0] https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://phoenixframework.org/blog/the-road-to-2-million-webs...
  - crgwbr 4y ago
    Genuinely curious on this—with port numbers only being 16 bits, how is it possible for one machine to ever handle more than 65k concurrent connections?
    - dxhdr 4y ago
      A connection is the tuple {source_ip, source_port, dest_ip, dest_port}
    - pipe_connector 4y ago
      Connections must have unique IP:Port pairs between client and server. You're limited to 65K concurrent connections for the same client. In practice, no one is opening that many connections from a single client.
      - dtjohnnymonkey 4y ago
        You might run into this limitation more quickly if you are receiving connections via a load balancer.
        adra 4y ago
        As such, the load balancer itself can probably hold a group of source IPs to use as second-hop solution to this problem as well if we're sincerely talking about load balancers holding a ton of largely idle connections simultaneously. The more likely load balancer outcome would be DNS split on inbound client IPs, and scaling out until each load balancer handles the appropriate amount of traffic (by some measure and scale out if exceeded).
        NavinF 4y ago
        Only for L4 load balancers. L3 (switches doing ECMP) won’t modify source IP and L7 (e.g. nginx) will reuse TCP connections to the backend servers.
    - alduin32 4y ago
      For outbound connections, it can be done using multiple IP addresses.
unboxingelf 4y ago
Not every problem is network or io bound. Imagine a system that ingests data from clients. Only a subset of the capacity may service client connections, while the rest may be running computations over data.
- javajosh 4y ago
  Yes, of course you might be CPU bound, in which case even a single user might take up 50k servers (think of a scientist doing a climate simulation, or an AI researcher building a very large model). All things being equal, CPU-bound is the exception, not the rule. Most every program we think of as an "application" is chat with some structure, persistence, access control added, and are indeed IO bound.
- foobiekr 4y ago
  Basically in an ideal system you are only bound by hardware limits. memory bandwidth bound memory size bound cache hit rate and bandwidth bound TLB size bound CPU decode and issue logic bound CPU renaming / OOO buffer bound kernel implementation bound (esp. locks, interrupts,...) network physical layer bound GPU/TPU bound (which are variations of the above, though mostly memory) power and cooling bound ... ... so basically unless you're running out of functional units, you're hopefully I/O bound depending on where you consider i/o. But in reality, most SW is just "crappy SW making poor use of resources bound." That's where we mostly are now as an industry. Bad language choices, terrible design, no cross-layer comprehension.
  - javajosh 4y ago
    >But in reality, most SW is just "crappy SW making poor use of resources bound." I agree, but the number one culprit is premature distribution, thanks to the widely pervasive cargo culting Amazon-style microservices.
    - aserafini 4y ago
      I often think about this. We have 100s of services running on 1000s of servers and 10,000s of containers but I wonder how much of that is ‘waste’ in the sense that we are running 10,000x Operating Systems, 100s of load balancers, database per service/region etc. Seems possible that horizontal scaling at every layer can be wasteful, but the alternatives are hard for me to even conceive.
SnowHill9902 4y ago
I guess it’s just an example. It can be sensors, cars, apps, …
dchuk 4y ago
I work at a telematics company. Technically, every black box in a vehicle is a server, and we have much more on the road than that figure
- deleted 4y ago
  [deleted]
Cthulhu_ 4y ago
Connections is fine, but applications do more than just accept connections, generally speaking.
foobiekr 4y ago
Think differently about this problem. Not all applications are web pages where the lifecycle of a given connection is "establish TCP, establish TLS, receive series of requests and produce series of responses mostly by hitting external caches or DBs" [or the QUIC variant of this]. That problem space was one of the very first scale challenges to arrive in 1998 and was one of the very first that actually got addressed. It's not the challenge space now and has not been, basically, for 20+ years, except perhaps the whole scaling-of-database problem, which has been dealt with by sharding, and the distribution of flows problem, which has mostly been solved with clever applications of intensely performant _scale-up_ hardware in the form of modern switch NPUs running variations of ECMP and clever uses of anycast, DNS load balancing, routing, etc. All of this stuff was pretty common by the mid-2000s. But bottlenecks in systems always exist. They move around. They can be anywhere in the stack. Connections is the simplest one that got a lot of attention for twenty years - real pre-emptive threads (in Linux, and Solaris's weird diversion into m:n), select() scalability (both in terms of bookkeeping and in terms of basic limits - that is, the lack thereof) giving rise to kqueue on freebsd, WFMO() on NT and years of attempts on Linux to get something that actually worked, which took awhile, _then_ the c10k problem, and so on. After connectivity you have issues in layer 3 - TCP offload, cost of TLS, etc. Userland to kernel copies (basic stuff like sendfile() to different userland driver schemes). And so on. Physical layer - servers move from NICs with lots of copies, to ring based with scatter-gather, to TSO, crypto offload, to ... while going from 10mb to 100, 1000, 2.5g, 10g, 40g, 100G and sooner or later 400G on server will not be as uncommon as it is now. But as your networking capacity and throughput scale, you start bumping into other things. Once you're talking high speed links and various schemes to get the kernel out of the way, you are mostly - not always, but mostly - talking about data movement problems. Elephant flows have their own system level problems in networks, and for data moving and staging you actually don't want the host _cpu_ involved if you can avoid it, let alone the kernel. Now you are in the area of doing (remote)->NIC----PCIe---->NVME (or --->GPU) directly, if you can. Now your NVME storage device becomes the bottleneck. 90s era supercomputing clusters had all of these problems with slightly different technologies, AI clusters have them today. These are not connection limited, they do not scale with people. Their primary scale challenge is utilization/CAPEX, but that's a longer discussion.
- deleted 4y ago
yashap 4y ago
In 2016, Gartner estimated that Google ran ~2.5 million servers (https://www.datacenterknowledge.com/archives/2017/03/16/google-data-center-faq https://www.datacenterknowledge.com/archives/2017/03/16/goog...), and it’s probably even more today. Obviously Google are an extreme outlier, but that’s 50x 50,000. Most servers do far, far more compute intensive things than handling connections - that’s a pretty meaningless number. I work in the mobility space, we’ve got servers that solve large/complex vehicle routing problems, and ideally they’re performing computation for just ONE user at a time. Certainly 50,000 servers is a lot, but tonnes of large tech companies run 100s to 1000s of servers at a time.
- javajosh 4y ago
  Sure, but the simple Postgres soln the author mentions works for 1000 servers. Assuming a power law distribution, technology like this is useful for a vanishingly small number of organizations. What's even more interesting is that it's pure admin overhead for those looking to centralize control over vast numbers of systems. So often technologists are asked to consider the moral or political implications of their work, and I think this is a good opportunity to do so.
hinkley 4y ago
My first scaled project, we were aiming at 30req/s per server. On UltraSPARC boxes, so maybe 8 cores? At a time when memcached was brand spanking new and thus not trustworthy, and F5 had just got their traffic shaping logic to actually work. We did it, but the architecture was terrible and we should have been able to do 50-100 req/s/s if we had taken certain principles more seriously. My current project is doing 5x the traffic but with 20x the cores, it’s embarrassing. And that’s just counting “our” servers, which are only about 1/3 of the whole enterprise (heading toward 50% if I wasn’t on the scene). I look at all the waste in our project and then I think about why I would ever need 50k cores, let alone servers and I just can’t fathom it. Who is handling a tens of millions of requests per second? And what on earth are you fucking up so badly that you need 2 million servers? Is Google doing 4 billion requests per second? At some point I have to ask myself if making it easy to manage more servers was really their best strategy. Often friction and constraints are where innovation comes from. When things are easy people don’t think about them until they are gone.
- morelisp 4y ago
  > Who is handling a tens of millions of requests per second? Speaking as someone less than one order of magnitude below that, it takes us about ~100-200 cores. (Depending on how you amortize shared infrastructure like our Kafka brokers, etc.) So even 10x'ing our infrastructure I can't imagine 50k servers.
  - hinkley 4y ago
    It's very reminiscent of that feeling when you were a child that adults have everything figured out, and so you're in a big hurry to get there because then everything will make sense. Then that moment of dawning horror when you see that nobody knows what the fuck they're doing and everyone is faking it at best, and just a child playing dress-up at worst. Google has certainly figured out a number of things, but anyone using 2.5 million servers in 2016 on a planet of only 7 billion people is playing dress-up.
    - hazz99 4y ago
maccard 4y ago
I work on multiplayer video games, and our game servers are often stateful programs that persist for 10-60m. If you have 5 players per session and 250,000 peak concurrent users (remember that steam is only one platform, consoles exist too, as do mobile platforms), you can have 50k servers. Sure they're not necessarily 50k unique machines, but they might be container instances or just processes on a single box, but theres 50k individual processes doing "stuff"
- deleted 4y ago
  [deleted]
- ClumsyPilot 4y ago
  > you can have 50k servers. Sure they're not necessarily 50k unique machines Props to game servers- they atlre more complex than much of eterprise software. However this is a big distinction - depending on a game, 10 - 100 game istances could run on one physical machine
  - maccard 4y ago
    From the receiving end it doesn't matter whether they're unique machines, instances packed on one machine, containers on a machine or serverless nodes, it's still receiving updates from 50k processess though! You also need to manage the complexity of packing those game instances onto the same machine for your game which is a whole other fish.
Johnny555 4y ago
A single server can only handle a million connections if each "connection" is doing a trivial amount of work. There are lots of compute bound services that need more power than a single server (or even 50,000 servers) can provide.

tilolebo 4y ago

> considering that a single server can easily handle a million simultaneous connections Can you elaborate on that part?

joisig 4y ago
Not the GP but they might be referring to [0] or one of several other articles you will find if you Google "handle a million connections in *". Realistically you also usually need to perform some non-trivial work from time to time for some non-trivial portion of those connections, which will further load your server, but still. [0] https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://phoenixframework.org/blog/the-road-to-2-million-webs...
- crgwbr 4y ago
  Genuinely curious on this—with port numbers only being 16 bits, how is it possible for one machine to ever handle more than 65k concurrent connections?
  - dxhdr 4y ago
    A connection is the tuple {source_ip, source_port, dest_ip, dest_port}
  - pipe_connector 4y ago
    Connections must have unique IP:Port pairs between client and server. You're limited to 65K concurrent connections for the same client. In practice, no one is opening that many connections from a single client.
    - dtjohnnymonkey 4y ago
      You might run into this limitation more quickly if you are receiving connections via a load balancer.
      - adra 4y ago
        As such, the load balancer itself can probably hold a group of source IPs to use as second-hop solution to this problem as well if we're sincerely talking about load balancers holding a ton of largely idle connections simultaneously. The more likely load balancer outcome would be DNS split on inbound client IPs, and scaling out until each load balancer handles the appropriate amount of traffic (by some measure and scale out if exceeded).
      - NavinF 4y ago
        Only for L4 load balancers. L3 (switches doing ECMP) won’t modify source IP and L7 (e.g. nginx) will reuse TCP connections to the backend servers.
  - alduin32 4y ago
    For outbound connections, it can be done using multiple IP addresses.

unboxingelf 4y ago

Not every problem is network or io bound. Imagine a system that ingests data from clients. Only a subset of the capacity may service client connections, while the rest may be running computations over data.

javajosh 4y ago
Yes, of course you might be CPU bound, in which case even a single user might take up 50k servers (think of a scientist doing a climate simulation, or an AI researcher building a very large model). All things being equal, CPU-bound is the exception, not the rule. Most every program we think of as an "application" is chat with some structure, persistence, access control added, and are indeed IO bound.
foobiekr 4y ago
Basically in an ideal system you are only bound by hardware limits. memory bandwidth bound memory size bound cache hit rate and bandwidth bound TLB size bound CPU decode and issue logic bound CPU renaming / OOO buffer bound kernel implementation bound (esp. locks, interrupts,...) network physical layer bound GPU/TPU bound (which are variations of the above, though mostly memory) power and cooling bound ... ... so basically unless you're running out of functional units, you're hopefully I/O bound depending on where you consider i/o. But in reality, most SW is just "crappy SW making poor use of resources bound." That's where we mostly are now as an industry. Bad language choices, terrible design, no cross-layer comprehension.
- javajosh 4y ago
  >But in reality, most SW is just "crappy SW making poor use of resources bound." I agree, but the number one culprit is premature distribution, thanks to the widely pervasive cargo culting Amazon-style microservices.
  - aserafini 4y ago
    I often think about this. We have 100s of services running on 1000s of servers and 10,000s of containers but I wonder how much of that is ‘waste’ in the sense that we are running 10,000x Operating Systems, 100s of load balancers, database per service/region etc. Seems possible that horizontal scaling at every layer can be wasteful, but the alternatives are hard for me to even conceive.

SnowHill9902 4y ago

I guess it’s just an example. It can be sensors, cars, apps, …

dchuk 4y ago

I work at a telematics company. Technically, every black box in a vehicle is a server, and we have much more on the road than that figure

deleted 4y ago
[deleted]

Cthulhu_ 4y ago

Connections is fine, but applications do more than just accept connections, generally speaking.

foobiekr 4y ago

Think differently about this problem. Not all applications are web pages where the lifecycle of a given connection is "establish TCP, establish TLS, receive series of requests and produce series of responses mostly by hitting external caches or DBs" [or the QUIC variant of this]. That problem space was one of the very first scale challenges to arrive in 1998 and was one of the very first that actually got addressed. It's not the challenge space now and has not been, basically, for 20+ years, except perhaps the whole scaling-of-database problem, which has been dealt with by sharding, and the distribution of flows problem, which has mostly been solved with clever applications of intensely performant _scale-up_ hardware in the form of modern switch NPUs running variations of ECMP and clever uses of anycast, DNS load balancing, routing, etc. All of this stuff was pretty common by the mid-2000s. But bottlenecks in systems always exist. They move around. They can be anywhere in the stack. Connections is the simplest one that got a lot of attention for twenty years - real pre-emptive threads (in Linux, and Solaris's weird diversion into m:n), select() scalability (both in terms of bookkeeping and in terms of basic limits - that is, the lack thereof) giving rise to kqueue on freebsd, WFMO() on NT and years of attempts on Linux to get something that actually worked, which took awhile, _then_ the c10k problem, and so on. After connectivity you have issues in layer 3 - TCP offload, cost of TLS, etc. Userland to kernel copies (basic stuff like sendfile() to different userland driver schemes). And so on. Physical layer - servers move from NICs with lots of copies, to ring based with scatter-gather, to TSO, crypto offload, to ... while going from 10mb to 100, 1000, 2.5g, 10g, 40g, 100G and sooner or later 400G on server will not be as uncommon as it is now. But as your networking capacity and throughput scale, you start bumping into other things. Once you're talking high speed links and various schemes to get the kernel out of the way, you are mostly - not always, but mostly - talking about data movement problems. Elephant flows have their own system level problems in networks, and for data moving and staging you actually don't want the host _cpu_ involved if you can avoid it, let alone the kernel. Now you are in the area of doing (remote)->NIC----PCIe---->NVME (or --->GPU) directly, if you can. Now your NVME storage device becomes the bottleneck. 90s era supercomputing clusters had all of these problems with slightly different technologies, AI clusters have them today. These are not connection limited, they do not scale with people. Their primary scale challenge is utilization/CAPEX, but that's a longer discussion.

deleted 4y ago

yashap 4y ago

In 2016, Gartner estimated that Google ran ~2.5 million servers (https://www.datacenterknowledge.com/archives/2017/03/16/google-data-center-faq https://www.datacenterknowledge.com/archives/2017/03/16/goog...), and it’s probably even more today. Obviously Google are an extreme outlier, but that’s 50x 50,000. Most servers do far, far more compute intensive things than handling connections - that’s a pretty meaningless number. I work in the mobility space, we’ve got servers that solve large/complex vehicle routing problems, and ideally they’re performing computation for just ONE user at a time. Certainly 50,000 servers is a lot, but tonnes of large tech companies run 100s to 1000s of servers at a time.

javajosh 4y ago
Sure, but the simple Postgres soln the author mentions works for 1000 servers. Assuming a power law distribution, technology like this is useful for a vanishingly small number of organizations. What's even more interesting is that it's pure admin overhead for those looking to centralize control over vast numbers of systems. So often technologists are asked to consider the moral or political implications of their work, and I think this is a good opportunity to do so.

hinkley 4y ago

My first scaled project, we were aiming at 30req/s per server. On UltraSPARC boxes, so maybe 8 cores? At a time when memcached was brand spanking new and thus not trustworthy, and F5 had just got their traffic shaping logic to actually work. We did it, but the architecture was terrible and we should have been able to do 50-100 req/s/s if we had taken certain principles more seriously. My current project is doing 5x the traffic but with 20x the cores, it’s embarrassing. And that’s just counting “our” servers, which are only about 1/3 of the whole enterprise (heading toward 50% if I wasn’t on the scene). I look at all the waste in our project and then I think about why I would ever need 50k cores, let alone servers and I just can’t fathom it. Who is handling a tens of millions of requests per second? And what on earth are you fucking up so badly that you need 2 million servers? Is Google doing 4 billion requests per second? At some point I have to ask myself if making it easy to manage more servers was really their best strategy. Often friction and constraints are where innovation comes from. When things are easy people don’t think about them until they are gone.

morelisp 4y ago
> Who is handling a tens of millions of requests per second? Speaking as someone less than one order of magnitude below that, it takes us about ~100-200 cores. (Depending on how you amortize shared infrastructure like our Kafka brokers, etc.) So even 10x'ing our infrastructure I can't imagine 50k servers.
- hinkley 4y ago
  It's very reminiscent of that feeling when you were a child that adults have everything figured out, and so you're in a big hurry to get there because then everything will make sense. Then that moment of dawning horror when you see that nobody knows what the fuck they're doing and everyone is faking it at best, and just a child playing dress-up at worst. Google has certainly figured out a number of things, but anyone using 2.5 million servers in 2016 on a planet of only 7 billion people is playing dress-up.
  - hazz99 4y ago

maccard 4y ago

I work on multiplayer video games, and our game servers are often stateful programs that persist for 10-60m. If you have 5 players per session and 250,000 peak concurrent users (remember that steam is only one platform, consoles exist too, as do mobile platforms), you can have 50k servers. Sure they're not necessarily 50k unique machines, but they might be container instances or just processes on a single box, but theres 50k individual processes doing "stuff"

deleted 4y ago
[deleted]
ClumsyPilot 4y ago
> you can have 50k servers. Sure they're not necessarily 50k unique machines Props to game servers- they atlre more complex than much of eterprise software. However this is a big distinction - depending on a game, 10 - 100 game istances could run on one physical machine
- maccard 4y ago
  From the receiving end it doesn't matter whether they're unique machines, instances packed on one machine, containers on a machine or serverless nodes, it's still receiving updates from 50k processess though! You also need to manage the complexity of packing those game instances onto the same machine for your game which is a whole other fish.

Johnny555 4y ago

A single server can only handle a million connections if each "connection" is doing a trivial amount of work. There are lots of compute bound services that need more power than a single server (or even 50,000 servers) can provide.