3 ms·

Kubernetes, no. Twine (aka "Don't call it Tupperware"), yes https://engineering.fb.com/2019/06/06/data-center-engineering/twine/ https://engineering.fb.com/20

Kubernetes, no. Twine (aka "Don't call it Tupperware"), yes https://engineering.fb.com/2019/06/06/data-center-engineering/twine/ https://engineering.fb.com/2019/06/06/data-center-engineerin... Also, that "30,000 servers" number is off by... a lot. No comment on by how many orders of magnitude. :-)

dweekly 5y ago
Yep. Here's how I'd napkin math it. Facebook has purchased 6GW of renewable energy to power its datacenters [1]. Its datacenters have a PUE of about 1.06 last they published (they unfortunately seem to have taken their dashboards down) so that's about 5.6GW of electricity actually powering real load. Some of that will power the networking fabric so we could conservatively estimate 5GW of compute load. The OCP Yosemite v3 platform power budget is a max of 1.5kW [2]. Also conservatively, let's assume a real average 1kW load. I'll leave the division as an exercise for the reader to estimate the number of servers they operate. 1 https://www.google.com/amp/s/www.euronews.com/green/amp/2021/04/15/facebook-announces-global-operations-now-supported-by-100-renewable-energy https://www.google.com/amp/s/www.euronews.com/green/amp/2021... 2 https://www.opencompute.org/documents/ocp-yosemite-v3-platform-design-specification-1v16-pdf https://www.opencompute.org/documents/ocp-yosemite-v3-platfo...
- MuffinFlavored 5y ago
  5 Gigawatts to Kilowatts = 5000000 5000000 kilowatts / 1kW per machine = 5,000,000 machines internationally?
  - dweekly 5y ago
    Yes, and that's a conservative lower bound (their web application servers likely draw much less than a kilowatt each and constitute a healthy portion of the fleet).
    - MuffinFlavored 5y ago
      Global warming wise, should society really be focusing on using paper straws or should Facebook aim to make their server code 10% more efficient?
      - dweekly 5y ago
        The good news is that electricity still costs a lot, even at scale, so there is lots of financial incentive for hyperscalars to make their datacenters and computation maximally efficient. To wit: there are a lot of people at Facebook (and Google, and Amazon, and Microsoft, and Baidu....) who work on making their server code more efficient. Small changes can result in millions of dollars a year in savings (in electricity cost, infrastructure cost, and opportunity cost in not needing to procure more servers).

dweekly 5y ago

Yep. Here's how I'd napkin math it. Facebook has purchased 6GW of renewable energy to power its datacenters [1]. Its datacenters have a PUE of about 1.06 last they published (they unfortunately seem to have taken their dashboards down) so that's about 5.6GW of electricity actually powering real load. Some of that will power the networking fabric so we could conservatively estimate 5GW of compute load. The OCP Yosemite v3 platform power budget is a max of 1.5kW [2]. Also conservatively, let's assume a real average 1kW load. I'll leave the division as an exercise for the reader to estimate the number of servers they operate. 1 https://www.google.com/amp/s/www.euronews.com/green/amp/2021/04/15/facebook-announces-global-operations-now-supported-by-100-renewable-energy https://www.google.com/amp/s/www.euronews.com/green/amp/2021... 2 https://www.opencompute.org/documents/ocp-yosemite-v3-platform-design-specification-1v16-pdf https://www.opencompute.org/documents/ocp-yosemite-v3-platfo...

MuffinFlavored 5y ago
5 Gigawatts to Kilowatts = 5000000 5000000 kilowatts / 1kW per machine = 5,000,000 machines internationally?
- dweekly 5y ago
  Yes, and that's a conservative lower bound (their web application servers likely draw much less than a kilowatt each and constitute a healthy portion of the fleet).
  - MuffinFlavored 5y ago
    Global warming wise, should society really be focusing on using paper straws or should Facebook aim to make their server code 10% more efficient?
    - dweekly 5y ago
      The good news is that electricity still costs a lot, even at scale, so there is lots of financial incentive for hyperscalars to make their datacenters and computation maximally efficient. To wit: there are a lot of people at Facebook (and Google, and Amazon, and Microsoft, and Baidu....) who work on making their server code more efficient. Small changes can result in millions of dollars a year in savings (in electricity cost, infrastructure cost, and opportunity cost in not needing to procure more servers).