5 ms·

An Inside Look at Facebook’s Approach to Automation and Human Work

93 points by riqbal 11y ago

s_kilk 11y ago
The paywall is popping up with a message "You've hit your limit of 5 free articles as an anonymous user this month.", despite the fact I haven't been reading anything on HBR for as long as I can remember.
- Dewie3 11y ago
  Ditto.
- i0nutzb 11y ago
  Incognito mode does the job for me. Have you tried?
dingdingdang 11y ago
Someone, OP maybe, post a mirror please.
drussell 11y ago
Those having trouble reading, try copying the title into a web search and going direct from there and/or using incognito mode.
fishnchips 11y ago
I would not say that FB compares favourably to Google in terms of infra, and I worked for both, in infra. Culture is way more relaxed though, especially if you're not into cults.
- ProAm 11y ago
  Can you speak more about this? Id be interested in some of the details (you dont have to throw either under the bus, just curious about the nitty gritty stuff)
  - nickpsecurity 11y ago
    I don't work for either but I do follow their projects a bit. I'd say you can start by comparing things like Hadoop or HDFS to things like Google File System and their Spanner-based F1 RDBMS (strong consistency + NoSQL scale). There's clearly a difference in talent or applying it to solve toughest problems at least. Both companies try to work around problems too hard to solve but Google seems to solve the hard one's more often.
    - fishnchips 11y ago
      Pretty much that. FB has a strong culture of trying to use FOSS 1st, no matter what. The problem is that at the scale FB operates this simply produces 2nd rate solutions with a ton of duct tape. If you're into solving hard problems Google is a way better place to be. In FB it feels like you're solving a lot of problems so it kind of feels way more productive but you quickly realise that most of these are obvious self-inflicted wounds. There just isn't much political will to invest in large non-customer-facing projects (like Google's GFS and its successor, BigTable, Spanner etc.) and perhaps talent is spread a bit too thin. That being said Google inside too often feels like a cult and I don't miss hearing the word 'googliness' at least 10 times a day.
      - walterbell 11y ago
        > There just isn't much political will to invest in large non-customer-facing projects How about OpenCompute? It was promoted by FB (even if inspired by Google) and had so much impact on the industry that even HP is now having FoxConn build HP-branded OpenCompute-style servers.
        eitally 11y ago
        My company (Newisys, under Sanmina) helped with hardware refinements because they were having cooling/airflow problems. They then contracted with Compal to manufacture the servers and then us to do system assembly. Compal's QC was horrifically bad -- we had about a 10% failure rate of servers they sent us. I'm happy OpenCompute is starting to receive traction, but it's not because FB is awesome at hardware. They've gotten a ton of both raw engineering and also DFM assistance from the pros.
- deleted 11y ago
  [deleted]
r721 11y ago
Mirror: https://archive.is/57N99 https://archive.is/57N99
gumby 11y ago
Unusually for the HBR there is a lot of interesting stuff in there. For example having the compiler team mixed in with the front end development team.
- tomnipotent 11y ago
  The compiler team wasn't mixed in. The front end team developed a compiler within its own team over a multi-year period - this was in response to the question "Do you still separate the teams working on innovations from the teams maintaining today’s core business?"
  - gumby 11y ago
    That's what I meant. Sorry I wasn't clear.
nickpsecurity 11y ago
The one technician for 25,000 servers part stands out the most to me. That's quite an achievement. Seems like they're ahead of supercomputers in management efficiency. Another along these lines is AOL's "lights out" datacenter [1] that they ignore for long periods of time except for efficient, batch jobs of maintenance. [1] http://www.zdnet.com/article/aol-shrinks-its-lights-out-data-center/?_escaped_fragment_=# http://www.zdnet.com/article/aol-shrinks-its-lights-out-data...!
lnkmails 11y ago
This blog post is interesting and it's good to see some numbers (one technician per 25,000 servers). Also, it touches upon one interesting aspect - "When do you automate a thing?". It would be interesting to know the actual metrics they use. Shameless plug: http://stackstorm.com/ http://stackstorm.com/ is very similar to FBAR and is helping customers reap the benefits of automation. We share the vision and we are completely open source. Feedback is appreciated. (I am a full time developer with StackStorm.)
- epowell2015 11y ago
  Folks interested in this kind of approach may want to join the Event Driven Automation meet-up in the Bay Area. Facebook is discussing FBAR and hosting an upcoming one (already "sold out"). LinkedIn is speaking at the next one (in SF 6/18) http://www.meetup.com/Auto-Remediation-and-Event-Driven-Automation/ http://www.meetup.com/Auto-Remediation-and-Event-Driven-Auto...
akeck 11y ago
My intuition is that there's a Moore's Law-like labor rule for IT labor due to its high cost and the explosion of computing power: "Every 18-24, an organization must be able to manage an order of magnitude more servers (virtual and physical) with the same number of staff."

93 points by riqbal 11y ago

The paywall is popping up with a message "You've hit your limit of 5 free articles as an anonymous user this month.", despite the fact I haven't been reading anything on HBR for as long as I can remember.

Dewie3 11y ago
Ditto.
i0nutzb 11y ago
Incognito mode does the job for me. Have you tried?

dingdingdang 11y ago

Someone, OP maybe, post a mirror please.

drussell 11y ago

Those having trouble reading, try copying the title into a web search and going direct from there and/or using incognito mode.

fishnchips 11y ago

I would not say that FB compares favourably to Google in terms of infra, and I worked for both, in infra. Culture is way more relaxed though, especially if you're not into cults.

ProAm 11y ago
Can you speak more about this? Id be interested in some of the details (you dont have to throw either under the bus, just curious about the nitty gritty stuff)
- nickpsecurity 11y ago
  I don't work for either but I do follow their projects a bit. I'd say you can start by comparing things like Hadoop or HDFS to things like Google File System and their Spanner-based F1 RDBMS (strong consistency + NoSQL scale). There's clearly a difference in talent or applying it to solve toughest problems at least. Both companies try to work around problems too hard to solve but Google seems to solve the hard one's more often.
  - fishnchips 11y ago
    Pretty much that. FB has a strong culture of trying to use FOSS 1st, no matter what. The problem is that at the scale FB operates this simply produces 2nd rate solutions with a ton of duct tape. If you're into solving hard problems Google is a way better place to be. In FB it feels like you're solving a lot of problems so it kind of feels way more productive but you quickly realise that most of these are obvious self-inflicted wounds. There just isn't much political will to invest in large non-customer-facing projects (like Google's GFS and its successor, BigTable, Spanner etc.) and perhaps talent is spread a bit too thin. That being said Google inside too often feels like a cult and I don't miss hearing the word 'googliness' at least 10 times a day.
    - walterbell 11y ago
      > There just isn't much political will to invest in large non-customer-facing projects How about OpenCompute? It was promoted by FB (even if inspired by Google) and had so much impact on the industry that even HP is now having FoxConn build HP-branded OpenCompute-style servers.
      - eitally 11y ago
        My company (Newisys, under Sanmina) helped with hardware refinements because they were having cooling/airflow problems. They then contracted with Compal to manufacture the servers and then us to do system assembly. Compal's QC was horrifically bad -- we had about a 10% failure rate of servers they sent us. I'm happy OpenCompute is starting to receive traction, but it's not because FB is awesome at hardware. They've gotten a ton of both raw engineering and also DFM assistance from the pros.
deleted 11y ago
[deleted]

r721 11y ago

Mirror: https://archive.is/57N99 https://archive.is/57N99

gumby 11y ago

Unusually for the HBR there is a lot of interesting stuff in there. For example having the compiler team mixed in with the front end development team.

tomnipotent 11y ago
The compiler team wasn't mixed in. The front end team developed a compiler within its own team over a multi-year period - this was in response to the question "Do you still separate the teams working on innovations from the teams maintaining today’s core business?"
- gumby 11y ago
  That's what I meant. Sorry I wasn't clear.

nickpsecurity 11y ago

The one technician for 25,000 servers part stands out the most to me. That's quite an achievement. Seems like they're ahead of supercomputers in management efficiency. Another along these lines is AOL's "lights out" datacenter [1] that they ignore for long periods of time except for efficient, batch jobs of maintenance. [1] http://www.zdnet.com/article/aol-shrinks-its-lights-out-data-center/?_escaped_fragment_=# http://www.zdnet.com/article/aol-shrinks-its-lights-out-data...!

lnkmails 11y ago

This blog post is interesting and it's good to see some numbers (one technician per 25,000 servers). Also, it touches upon one interesting aspect - "When do you automate a thing?". It would be interesting to know the actual metrics they use. Shameless plug: http://stackstorm.com/ http://stackstorm.com/ is very similar to FBAR and is helping customers reap the benefits of automation. We share the vision and we are completely open source. Feedback is appreciated. (I am a full time developer with StackStorm.)

epowell2015 11y ago
Folks interested in this kind of approach may want to join the Event Driven Automation meet-up in the Bay Area. Facebook is discussing FBAR and hosting an upcoming one (already "sold out"). LinkedIn is speaking at the next one (in SF 6/18) http://www.meetup.com/Auto-Remediation-and-Event-Driven-Automation/ http://www.meetup.com/Auto-Remediation-and-Event-Driven-Auto...

akeck 11y ago

My intuition is that there's a Moore's Law-like labor rule for IT labor due to its high cost and the explosion of computing power: "Every 18-24, an organization must be able to manage an order of magnitude more servers (virtual and physical) with the same number of staff."