5 ms·

I live really close to their largest data center and only about 500 people work there. It’s 4x larger now than when they initially broke ground and even then it

I live really close to their largest data center and only about 500 people work there. It’s 4x larger now than when they initially broke ground and even then it was insanely huge. Edit. I just saw the photo for their Nebraska data center, so I guess each of their locations are about the same size and shape.

grepfru_it 4y ago
You don't need people in a datacenter. You only need a skeleton crew for hands when things physically break, everything else should be done remotely
- cvwright 4y ago
  So what then is the argument about them needing more employees to serve more users? The frontend apps are all the same, whether they have 3 users or 3 billion.
  - mapme 4y ago
    The front end apps are not all the same. If you built an app that handles 3 users well do you think it would easily handle 3 billion with no tweaks at any layer (app, framework, OS, network, hardware, edge traffic routing, internal dc network fabric, caching layer, storage for binary/ graph/ KV/ relational, cluster scheduler, security )? Another example, ML backed products (eg. Search, any RecSys) requires tuning features based on product specific needs. You also end up leaving massive efficiencies on the floor by not hiring more specialists as you scale. This is a good read which explains it in detail https://danluu.com/in-house/ https://danluu.com/in-house/ I would also suggest reading their published papers on in house systems to understand the challenges that come with operating at a large scale. https://news.ycombinator.com/item?id=30940595 https://news.ycombinator.com/item?id=30940595 I’m sure FB could operate with a range of employees 40k-100k(?), but that number scaling with users is far from surprising.
    - milosmns 4y ago
      You've mostly explained backend-related issues, no?
      - mapme 4y ago
        Yeah, most of that is backend issues. Part of them dictated by product requirements eg. We require serving 5 year old, <4MB photos with 99th percentile latency of 1 second. Product and UI has its own complexities, but it seems clear each product is unique and would require different logic.
        MichaelZuo 4y ago
        Then at most 3 people per product frontend would be enough? But the real number is more like 300+, so I can see the parent's point.
- UncleMeat 4y ago
  When your datacenters are huge, things physically break constantly.
  - grepfru_it 4y ago
    What size crew do you infer when thinking of a “huge data center” I’m still thinking it’s just enough to repair failures in a timely manner (and assuming there is spare capacity so you are not blocked if something breaks). Which means only a few hands on deck and the rest being triaged remotely. Or not maybe we are still in the 90s when you need someone sitting at a desk in a server room to handle password resets :P
DavidPeiffer 4y ago
I'm in the same area, and agree the scale is dramatic. One land surveyor a friend ran into has been working there for 7 consecutive years. I think if I were in that role, I would want to see different environments from time to time.
ghaff 4y ago
>only about 500 people work there I'm actually surprised there would be that many. This report [1] from the US chamber of commerce says 157 after initial construction and I've seen numbers lower than that. (Some may be a function of presumably outsourced workers like security guards and janitorial services.) Datacenters bring money to an area while they're being built but there are very few local jobs once they're operational. (Obviously engineers do work connected to datacenters from other locations.) [1] https://www.uschamber.com/sites/default/files/ctec_datacenterrpt_lowres.pdf https://www.uschamber.com/sites/default/files/ctec_datacente...
- magicalhippo 4y ago
  > Datacenters bring money to an area while they're being built but there are very few local jobs once they're operational. I really wish local politicians would get this. Here in Norway, several small communities got wooed by businesses wanting free land and cheap electricity for data centers, in return for promises of hundreds if not thousands of jobs from all those supposedly using the data center. One even ended up having to buy back land they gave for free[1] for ~$700k. [1]: https://www.nrk.no/nordland/ballangen-ga-bort-tomt-for-a-fa-verdens-storste-datasenter-_-de-den-tilbake-1.15456985 https://www.nrk.no/nordland/ballangen-ga-bort-tomt-for-a-fa-...

grepfru_it 4y ago

You don't need people in a datacenter. You only need a skeleton crew for hands when things physically break, everything else should be done remotely

cvwright 4y ago
So what then is the argument about them needing more employees to serve more users? The frontend apps are all the same, whether they have 3 users or 3 billion.
- mapme 4y ago
  The front end apps are not all the same. If you built an app that handles 3 users well do you think it would easily handle 3 billion with no tweaks at any layer (app, framework, OS, network, hardware, edge traffic routing, internal dc network fabric, caching layer, storage for binary/ graph/ KV/ relational, cluster scheduler, security )? Another example, ML backed products (eg. Search, any RecSys) requires tuning features based on product specific needs. You also end up leaving massive efficiencies on the floor by not hiring more specialists as you scale. This is a good read which explains it in detail https://danluu.com/in-house/ https://danluu.com/in-house/ I would also suggest reading their published papers on in house systems to understand the challenges that come with operating at a large scale. https://news.ycombinator.com/item?id=30940595 https://news.ycombinator.com/item?id=30940595 I’m sure FB could operate with a range of employees 40k-100k(?), but that number scaling with users is far from surprising.
  - milosmns 4y ago
    You've mostly explained backend-related issues, no?
    - mapme 4y ago
      Yeah, most of that is backend issues. Part of them dictated by product requirements eg. We require serving 5 year old, <4MB photos with 99th percentile latency of 1 second. Product and UI has its own complexities, but it seems clear each product is unique and would require different logic.
      - MichaelZuo 4y ago
        Then at most 3 people per product frontend would be enough? But the real number is more like 300+, so I can see the parent's point.
UncleMeat 4y ago
When your datacenters are huge, things physically break constantly.
- grepfru_it 4y ago
  What size crew do you infer when thinking of a “huge data center” I’m still thinking it’s just enough to repair failures in a timely manner (and assuming there is spare capacity so you are not blocked if something breaks). Which means only a few hands on deck and the rest being triaged remotely. Or not maybe we are still in the 90s when you need someone sitting at a desk in a server room to handle password resets :P

DavidPeiffer 4y ago

I'm in the same area, and agree the scale is dramatic. One land surveyor a friend ran into has been working there for 7 consecutive years. I think if I were in that role, I would want to see different environments from time to time.

ghaff 4y ago

>only about 500 people work there I'm actually surprised there would be that many. This report [1] from the US chamber of commerce says 157 after initial construction and I've seen numbers lower than that. (Some may be a function of presumably outsourced workers like security guards and janitorial services.) Datacenters bring money to an area while they're being built but there are very few local jobs once they're operational. (Obviously engineers do work connected to datacenters from other locations.) [1] https://www.uschamber.com/sites/default/files/ctec_datacenterrpt_lowres.pdf https://www.uschamber.com/sites/default/files/ctec_datacente...

magicalhippo 4y ago
> Datacenters bring money to an area while they're being built but there are very few local jobs once they're operational. I really wish local politicians would get this. Here in Norway, several small communities got wooed by businesses wanting free land and cheap electricity for data centers, in return for promises of hundreds if not thousands of jobs from all those supposedly using the data center. One even ended up having to buy back land they gave for free[1] for ~$700k. [1]: https://www.nrk.no/nordland/ballangen-ga-bort-tomt-for-a-fa-verdens-storste-datasenter-_-de-den-tilbake-1.15456985 https://www.nrk.no/nordland/ballangen-ga-bort-tomt-for-a-fa-...