4 ms·

Ok, so how is thousands of servers 0.1%? That implies they have millions of servers or one for every 9000 people on earth - are companies this size really that

by inglor 5y ago

Ok, so how is thousands of servers 0.1%? That implies they have millions of servers or one for every 9000 people on earth - are companies this size really that wasteful in terms of servers needed?

akiselev 5y ago
If they have two million servers that would mean about 1000 daily active users per server. Assuming the average user makes 2000 requests (API calls, images, videos, etc.) a day mindlessly browsing the infinite feed, that works out to about 1 request per second. Facebook makes its money from advertisers so that's likely where most of the compute resources are going - users just see the ads at the end of all that computation. Combined with the mandatory over provisioning, the overhead of a massive distributed systems, tracing, etc, I'm not surprised those are the numbers. Assuming each server cost an average of $20k, that's $40 billion which is two quarters worth of revenue but amortized over 5+ years. It's really not all that much.
packetslave 5y ago
The official public answer is "millions of servers" (see, for example, https://engineering.fb.com/2018/07/19/data-infrastructure/location-aware-distribution-configuring-servers-at-scale/ https://engineering.fb.com/2018/07/19/data-infrastructure/lo...). Keep in mind that this includes Instagram and Whatsapp too, as far as I know. As for "wasteful", well.. 1.88 billion DAU (Q1 earning report) / 86400 = 21759 "users per second" (note: I made this up) Multiply that by N where N is the number of frontend and backend queries it takes to service one user, and you have a lot of QPS. Now add in headroom, redundancy, edge POPs and CDN to put servers as close to users as possible, etc. It's hard to fathom just how big traffic to FAANG services can be, until you see what it takes to serve it. Is there some waste? Sure, probably, but not as much as you'd think.

by inglor 5y ago

Ok, so how is thousands of servers 0.1%? That implies they have millions of servers or one for every 9000 people on earth - are companies this size really that wasteful in terms of servers needed?

akiselev 5y ago

If they have two million servers that would mean about 1000 daily active users per server. Assuming the average user makes 2000 requests (API calls, images, videos, etc.) a day mindlessly browsing the infinite feed, that works out to about 1 request per second. Facebook makes its money from advertisers so that's likely where most of the compute resources are going - users just see the ads at the end of all that computation. Combined with the mandatory over provisioning, the overhead of a massive distributed systems, tracing, etc, I'm not surprised those are the numbers. Assuming each server cost an average of $20k, that's $40 billion which is two quarters worth of revenue but amortized over 5+ years. It's really not all that much.

packetslave 5y ago

The official public answer is "millions of servers" (see, for example, https://engineering.fb.com/2018/07/19/data-infrastructure/location-aware-distribution-configuring-servers-at-scale/ https://engineering.fb.com/2018/07/19/data-infrastructure/lo...). Keep in mind that this includes Instagram and Whatsapp too, as far as I know. As for "wasteful", well.. 1.88 billion DAU (Q1 earning report) / 86400 = 21759 "users per second" (note: I made this up) Multiply that by N where N is the number of frontend and backend queries it takes to service one user, and you have a lot of QPS. Now add in headroom, redundancy, edge POPs and CDN to put servers as close to users as possible, etc. It's hard to fathom just how big traffic to FAANG services can be, until you see what it takes to serve it. Is there some waste? Sure, probably, but not as much as you'd think.