4 ms·

Their scale is enormous. I first heard the phrase "thundering herds" in a Facebook MySQL talk. (Thundering herds refers to what happens to your infra when somet

Their scale is enormous. I first heard the phrase "thundering herds" in a Facebook MySQL talk. (Thundering herds refers to what happens to your infra when something big happens in real life that lights up social networking/search/etc.)

MuffinFlavored 5y ago
Let's try to put a guess in numbers. > With roughly 2.85 billion monthly active users as of the first quarter of 2021 > 1.8 billion of Facebook users (66%) use the app on a daily basis Say 1 database could store the data + handle the reads/writes for 100,000 users (literally just guessing) 2.85 billion monthly active users / 100k users per server = 28,500 servers/"instances" for MySQL That's not even counting Kubernetes pods (if they use that) running Docker containers (if they use that) for their API / content serving. I wonder what their average transactions per second looks like. For Twitter: > Every second, on average, around 6,000 tweets are tweeted on Twitter, which corresponds to over 350,000 tweets sent per minute, 500 million tweets per day and around 200 billion tweets per year. > As of the first quarter of 2019, Twitter averaged 330 million monthly active users Twitter has 11.5% the MAU of Facebook (330m of 2.85b)? I don't know if 1 tweet can be extrapolated to 1 Facebook like/comment/share/post but if it did, 6k tweets per sec * 8.7x the user base = 52.2k writes per second? Found this: > Facebook now has about 30,000 servers supporting its operations, hosts 80 billion photos, and serves up more than 600,000 photos to its users every second.
- packetslave 5y ago
  Kubernetes, no. Twine (aka "Don't call it Tupperware"), yes https://engineering.fb.com/2019/06/06/data-center-engineering/twine/ https://engineering.fb.com/2019/06/06/data-center-engineerin... Also, that "30,000 servers" number is off by... a lot. No comment on by how many orders of magnitude. :-)
  - dweekly 5y ago
    Yep. Here's how I'd napkin math it. Facebook has purchased 6GW of renewable energy to power its datacenters [1]. Its datacenters have a PUE of about 1.06 last they published (they unfortunately seem to have taken their dashboards down) so that's about 5.6GW of electricity actually powering real load. Some of that will power the networking fabric so we could conservatively estimate 5GW of compute load. The OCP Yosemite v3 platform power budget is a max of 1.5kW [2]. Also conservatively, let's assume a real average 1kW load. I'll leave the division as an exercise for the reader to estimate the number of servers they operate. 1 https://www.google.com/amp/s/www.euronews.com/green/amp/2021/04/15/facebook-announces-global-operations-now-supported-by-100-renewable-energy https://www.google.com/amp/s/www.euronews.com/green/amp/2021... 2 https://www.opencompute.org/documents/ocp-yosemite-v3-platform-design-specification-1v16-pdf https://www.opencompute.org/documents/ocp-yosemite-v3-platfo...
    - MuffinFlavored 5y ago
      5 Gigawatts to Kilowatts = 5000000 5000000 kilowatts / 1kW per machine = 5,000,000 machines internationally?
      - dweekly 5y ago
        Yes, and that's a conservative lower bound (their web application servers likely draw much less than a kilowatt each and constitute a healthy portion of the fleet).
        MuffinFlavored 5y ago
        Global warming wise, should society really be focusing on using paper straws or should Facebook aim to make their server code 10% more efficient?
        dweekly 5y ago
        The good news is that electricity still costs a lot, even at scale, so there is lots of financial incentive for hyperscalars to make their datacenters and computation maximally efficient. To wit: there are a lot of people at Facebook (and Google, and Amazon, and Microsoft, and Baidu....) who work on making their server code more efficient. Small changes can result in millions of dollars a year in savings (in electricity cost, infrastructure cost, and opportunity cost in not needing to procure more servers).

MuffinFlavored 5y ago

Let's try to put a guess in numbers. > With roughly 2.85 billion monthly active users as of the first quarter of 2021 > 1.8 billion of Facebook users (66%) use the app on a daily basis Say 1 database could store the data + handle the reads/writes for 100,000 users (literally just guessing) 2.85 billion monthly active users / 100k users per server = 28,500 servers/"instances" for MySQL That's not even counting Kubernetes pods (if they use that) running Docker containers (if they use that) for their API / content serving. I wonder what their average transactions per second looks like. For Twitter: > Every second, on average, around 6,000 tweets are tweeted on Twitter, which corresponds to over 350,000 tweets sent per minute, 500 million tweets per day and around 200 billion tweets per year. > As of the first quarter of 2019, Twitter averaged 330 million monthly active users Twitter has 11.5% the MAU of Facebook (330m of 2.85b)? I don't know if 1 tweet can be extrapolated to 1 Facebook like/comment/share/post but if it did, 6k tweets per sec * 8.7x the user base = 52.2k writes per second? Found this: > Facebook now has about 30,000 servers supporting its operations, hosts 80 billion photos, and serves up more than 600,000 photos to its users every second.

packetslave 5y ago
Kubernetes, no. Twine (aka "Don't call it Tupperware"), yes https://engineering.fb.com/2019/06/06/data-center-engineering/twine/ https://engineering.fb.com/2019/06/06/data-center-engineerin... Also, that "30,000 servers" number is off by... a lot. No comment on by how many orders of magnitude. :-)
- dweekly 5y ago
  Yep. Here's how I'd napkin math it. Facebook has purchased 6GW of renewable energy to power its datacenters [1]. Its datacenters have a PUE of about 1.06 last they published (they unfortunately seem to have taken their dashboards down) so that's about 5.6GW of electricity actually powering real load. Some of that will power the networking fabric so we could conservatively estimate 5GW of compute load. The OCP Yosemite v3 platform power budget is a max of 1.5kW [2]. Also conservatively, let's assume a real average 1kW load. I'll leave the division as an exercise for the reader to estimate the number of servers they operate. 1 https://www.google.com/amp/s/www.euronews.com/green/amp/2021/04/15/facebook-announces-global-operations-now-supported-by-100-renewable-energy https://www.google.com/amp/s/www.euronews.com/green/amp/2021... 2 https://www.opencompute.org/documents/ocp-yosemite-v3-platform-design-specification-1v16-pdf https://www.opencompute.org/documents/ocp-yosemite-v3-platfo...
  - MuffinFlavored 5y ago
    5 Gigawatts to Kilowatts = 5000000 5000000 kilowatts / 1kW per machine = 5,000,000 machines internationally?
    - dweekly 5y ago
      Yes, and that's a conservative lower bound (their web application servers likely draw much less than a kilowatt each and constitute a healthy portion of the fleet).
      - MuffinFlavored 5y ago
        Global warming wise, should society really be focusing on using paper straws or should Facebook aim to make their server code 10% more efficient?
        dweekly 5y ago
        The good news is that electricity still costs a lot, even at scale, so there is lots of financial incentive for hyperscalars to make their datacenters and computation maximally efficient. To wit: there are a lot of people at Facebook (and Google, and Amazon, and Microsoft, and Baidu....) who work on making their server code more efficient. Small changes can result in millions of dollars a year in savings (in electricity cost, infrastructure cost, and opportunity cost in not needing to procure more servers).