4 ms·

You don't need people in a datacenter. You only need a skeleton crew for hands when things physically break, everything else should be done remotely

cvwright 4y ago
So what then is the argument about them needing more employees to serve more users? The frontend apps are all the same, whether they have 3 users or 3 billion.
- mapme 4y ago
  The front end apps are not all the same. If you built an app that handles 3 users well do you think it would easily handle 3 billion with no tweaks at any layer (app, framework, OS, network, hardware, edge traffic routing, internal dc network fabric, caching layer, storage for binary/ graph/ KV/ relational, cluster scheduler, security )? Another example, ML backed products (eg. Search, any RecSys) requires tuning features based on product specific needs. You also end up leaving massive efficiencies on the floor by not hiring more specialists as you scale. This is a good read which explains it in detail https://danluu.com/in-house/ https://danluu.com/in-house/ I would also suggest reading their published papers on in house systems to understand the challenges that come with operating at a large scale. https://news.ycombinator.com/item?id=30940595 https://news.ycombinator.com/item?id=30940595 I’m sure FB could operate with a range of employees 40k-100k(?), but that number scaling with users is far from surprising.
  - milosmns 4y ago
    You've mostly explained backend-related issues, no?
    - mapme 4y ago
      Yeah, most of that is backend issues. Part of them dictated by product requirements eg. We require serving 5 year old, <4MB photos with 99th percentile latency of 1 second. Product and UI has its own complexities, but it seems clear each product is unique and would require different logic.
      - MichaelZuo 4y ago
        Then at most 3 people per product frontend would be enough? But the real number is more like 300+, so I can see the parent's point.
UncleMeat 4y ago
When your datacenters are huge, things physically break constantly.
- grepfru_it 4y ago
  What size crew do you infer when thinking of a “huge data center” I’m still thinking it’s just enough to repair failures in a timely manner (and assuming there is spare capacity so you are not blocked if something breaks). Which means only a few hands on deck and the rest being triaged remotely. Or not maybe we are still in the 90s when you need someone sitting at a desk in a server room to handle password resets :P

cvwright 4y ago

So what then is the argument about them needing more employees to serve more users? The frontend apps are all the same, whether they have 3 users or 3 billion.

mapme 4y ago
The front end apps are not all the same. If you built an app that handles 3 users well do you think it would easily handle 3 billion with no tweaks at any layer (app, framework, OS, network, hardware, edge traffic routing, internal dc network fabric, caching layer, storage for binary/ graph/ KV/ relational, cluster scheduler, security )? Another example, ML backed products (eg. Search, any RecSys) requires tuning features based on product specific needs. You also end up leaving massive efficiencies on the floor by not hiring more specialists as you scale. This is a good read which explains it in detail https://danluu.com/in-house/ https://danluu.com/in-house/ I would also suggest reading their published papers on in house systems to understand the challenges that come with operating at a large scale. https://news.ycombinator.com/item?id=30940595 https://news.ycombinator.com/item?id=30940595 I’m sure FB could operate with a range of employees 40k-100k(?), but that number scaling with users is far from surprising.
- milosmns 4y ago
  You've mostly explained backend-related issues, no?
  - mapme 4y ago
    Yeah, most of that is backend issues. Part of them dictated by product requirements eg. We require serving 5 year old, <4MB photos with 99th percentile latency of 1 second. Product and UI has its own complexities, but it seems clear each product is unique and would require different logic.
    - MichaelZuo 4y ago
      Then at most 3 people per product frontend would be enough? But the real number is more like 300+, so I can see the parent's point.

UncleMeat 4y ago

When your datacenters are huge, things physically break constantly.

grepfru_it 4y ago
What size crew do you infer when thinking of a “huge data center” I’m still thinking it’s just enough to repair failures in a timely manner (and assuming there is spare capacity so you are not blocked if something breaks). Which means only a few hands on deck and the rest being triaged remotely. Or not maybe we are still in the 90s when you need someone sitting at a desk in a server room to handle password resets :P