9 ms·

Notes from Facebook's Developer Infrastructure at Scale F8 Talk

162 points by cpeterso 12y ago

raverbashing 12y ago
Interesting info about XCode and the FB app. I wonder if Apple is collaborating with FB on this or if it's mostly deaf ears... "So they built their own IDE. Set of plugins on top of ATOM. Not a fork. They like hackable and web-y nature of ATOM. The demo showing iOS development looks very nice! " Seems that they have the right idea about "optimizing for Developers"
- coldcode 12y ago
  Although I don't really like XCode but use it all the time, what the hell takes 5 minutes to open a project????? The FB app isn't that big. Maybe it includes downloading an entire repo or something? I've worked on big iOS apps and this was never an issue.
  - insaneirish 12y ago
    Facebook has a 54 GB monolithic repo (as of almost a year ago, April 2014). Primary source: https://twitter.com/feross/status/459259593630433280 https://twitter.com/feross/status/459259593630433280 HN article: https://news.ycombinator.com/item?id=7648237 https://news.ycombinator.com/item?id=7648237
brazzlemobile 12y ago
Anyone have a mirror? Getting gateway timeouts.
- indygreg2 12y ago
  The site is hosted via GitHub pages. Maybe it's not available to you due to the lingering GitHub DoS. If you care to read HTML and can manage to get through to GitHub: https://github.com/indygreg/indygreg.github.com/blob/master/blog/2015/03/28/notes-from-facebook%27s-developer-infrastructure-at-scale-f8-talk/index.html https://github.com/indygreg/indygreg.github.com/blob/master/...
Bahamut 12y ago
A lot of nice summaries and points, but one caught my eye - the mention about building Nuclide on top of Atom. My experience with Atom is that it gets painfully slow quickly, with me experiencing a crash every few days. I would like to hear more thoughts from FB folks on their experiences.
- SwellJoe 12y ago
  Java desktop applications have shown that with enough resources thrown at the problem, slow and flaky things can become fast and stable. JavaScript desktop applications seem to be at the "slow and flaky but has tons of resources being thrown at them" stage. The VM, as with Java ten years ago, is also seeing remarkable improvements.
beliu 12y ago
On the topic of having one IDE that just works for multiple languages/platforms, you should check out srclib (https://srclib.org https://srclib.org), an open-source language analysis tool (I'm one of the creators) that abstracts away the language-specific parts of IDE support so that IDEs/editor plugins can hook into one API and automatically get support for a bunch of different languages. This way, you don't have to build MxN plugins (M editors, N languages) to support all combos of editors/languages people might want to use. The currently supported languages are Go, Java, Python, JavaScript, and Ruby, so this wouldn't work yet for iOS, but Objective-C and Swift are high on our priority list. Editor plugins exist for Emacs, Sublime, and Atom. Would love to hear anyone's feedback from trying it out.
- wting 12y ago
  This sounds a lot like Steve Yegge's Grok: http://bsumm.net/2012/08/11/steve-yegge-and-grok.html http://bsumm.net/2012/08/11/steve-yegge-and-grok.html It's unlikely that his version will ever be open sourced or usable outside of Google though.
  - durin42 12y ago
    IIRC http://kythe.io http://kythe.io is at least related by heritage to grok.
    - beliu 12y ago
      Yep, Kythe is the open-source version of grok. It's awesome, and the srclib team is in touch with the Kythe team at Google to build on top of their analysis libs (for C++ and Java). Long term vision is to get rid of the "my editor X doesn't support language Y" problem as well as build a lot of other multi-language analysis tools on top of it. Shameless plug: if you care about developer productivity, join us!
- fallat 12y ago
  >On the topic of having one IDE that just works for multiple languages/platforms, I've recently given Eclipse another chance after 4 years. It has impressed me with its C, PHP and Haskell support so far. I really recommend people give it a shot. Of course I installed a half decent GTK theme to make it look nice :)
131hn 12y ago
* Yep, another "better than ever"/"last one you'll ever need" IDE. Thank you facebook. (see you soon for the next one) * A DVCS that rely on a central server for merging (sandcastle) is no longer.. distributed... (and you cannot have distributed team work here, this is wrong in multiple way) * I think i'll never let a centralized, monolithic repository to be set up in my company. All great stuffs/talent i ever learned came from differents sources, differents independant projets (from git, hg or SVN). Loose that and i think i'll get narrow minded. * All those fancy stuffs make facebookers better "facebook developers" but less prone to share with others (we don't share the same language, culture or tools anymore, even the design of the monolithic repo cannot help here) * This is more a lesson on "how we made X thousands lambda devs work together" than actual tools i might need to use someday. * I'll bet that facebook "infra" developers are less likely to use the tools they describe, and that is ironic. Those tools mostly apply to some hidden mass (100k commit / week.. ) of uniform developers that I'll never meet.
- DannyBee 12y ago
  "A DVCS that rely on a central server for merging (sandcastle) is no longer.. distributed... (and you cannot have distributed team work here, this is wrong in multiple way)" Yet doing things like rebasing or merging pull requests properly, requires you have an up-to-date master, making it a centralized operation. People seem quite happy with this.
  - hueving 12y ago
    >Yet doing things like rebasing or merging pull requests properly Rebasing is a smell of bad source code management. Merging pull requests is fine whether or not you are up to date. I don't know what you mean by 'properly'.
    - indygreg2 12y ago
      It is much easier to bisect linear history than history with merges.
      - 12y ago
larsberg 12y ago
This talk and my experiences at Microsoft are why I tend to try to steer our projects at Mozilla Research to partner with external offerings rather than doing too much building of our own. It's hard to imagine a world where we build and continue to improve a source control / issue tracking system better than GitHub; a continuous integration solution better than Travis CI; a build system; etc. Even if we can imagine building a better product today, the ongoing maintenance and extension of those services quickly become full-time multi-person investments with many hundreds of dedicated machines, neither of which even our core infra teams have, much less the little tiny research org.
- indygreg2 12y ago
  Except it isn't internal only. Many of Facebook's tools are open sourced. And not in the "throw it over the wall" sense. Their open sourced projects tend to gain traction and get a significant amount of community contributors. By turning a number of their core tools into successful open source projects, they are leveraging the community effect to lesson the ongoing maintenance burden for these projects. Yes, they are paying a high initial cost to develop these offerings. But since they tend to produce high quality products that attract nearly-free-to-Facebook labor via open source, the investment tends to pays off in the long run. It's a savvy business move. And one that can arguably only be pulled off by a talented engineer organization.
  - larsberg 12y ago
    > Yes, they are paying a high initial cost to develop these offerings. I certainly agree that for organizations who can afford these high initial costs internally this is a great development model. My challenge (as a manager at Mozilla) is that, given we don't have such resources available, how do we build partnerships with groups or other projects that do in order to ensure such offerings are built? Of course, another interesting approach would be to say that GitHub, Travis, Heroku, etc. show that DevOps-as-company is a viable business model and really what we should be doing is spinning up companies that raise funding and build products around each of the areas that are still lacking products that serve enterprises (read: customers with money) today.
- 12y ago
necubi 12y ago
The most interesting part for me was > They appear to use code coverage to determine what tests > to run. "We're not going to run a test unless your diff > might actually have broken it." This seems like such an obvious optimization in hindsight, I'm surprised it's not more common. Does anybody know of other CIs that can do this?
- TheLoneWolfling 12y ago
  This seems like a good optimization in theory. In practice? I think it may end up selectively pruning for hard-to-find bugs, which is not a desirable feature of a test suite.
- dperfect 12y ago
  I'd be hesitant to rely 100% on this approach (and from the notes it sounds like FB still runs the other tests - just less frequently in some cases), especially where the code coverage tools aren't guaranteed to account for unintended interactions between seemingly unrelated pieces of code. In theory, well-written code/tests shouldn't have those kinds of problems in the first place, but larger projects often don't deserve that level of trust (in my opinion). Still, I think it's a great idea for giving quick, "preliminary" feedback to developers - as long as the full test suite is still run periodically.
  - necubi 12y ago
    Yeah, absolutely. You'd want to run your full test suite (which probably includes heavier integration tests as well) before deploying, but I think in a lot of cases it's valuable to provide, say, 95% accurate feedback quickly.
    - icefox 12y ago
      Another way to put it would be I will happily only take 95% if the other option is we don't do anything and people can even commit stuff that doesn't even build let alone pass tests.
- 12y ago
rurban 12y ago
"We made mercurial up to 50x faster than git, more than 2000 improvements" on big repos with lot of history. This would be their blog post about this: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ https://code.facebook.com/posts/218678814984400/scaling-merc... and the code at http://selenic.com/repo/hg/shortlog/ http://selenic.com/repo/hg/shortlog/
ghuntley 12y ago
Does anyone have a .torrent of all the videos from F8? It's a shame that they are self-hosting otherwise it would be possible to youtube-dl it :-)
- ghuntley 12y ago
  Here's a bash script that will download all of the videos: $ cat download-facebook-f8-2015-videos.sh #!/usr/bin/env bash youtube-dl "https://www.facebook.com/video.php?v=10152795258318553" youtube-dl "https://www.facebook.com/video.php?v=10152795258318553" youtube-dl "https://www.facebook.com/video.php?v=10152795404193553" youtube-dl "https://www.facebook.com/video.php?v=10152795420103553" youtube-dl "https://www.facebook.com/video.php?v=10152795617423553" youtube-dl "https://www.facebook.com/video.php?v=10152795634488553" youtube-dl "https://www.facebook.com/video.php?v=10152795636318553" youtube-dl "https://www.facebook.com/video.php?v=10152795737378553" youtube-dl "https://www.facebook.com/video.php?v=10152795739268553" youtube-dl "https://www.facebook.com/video.php?v=10152795771278553" youtube-dl "https://www.facebook.com/video.php?v=10152795771278553" youtube-dl "https://www.facebook.com/video.php?v=10152795779043553" youtube-dl "https://www.facebook.com/video.php?v=10152795794003553" youtube-dl "https://www.facebook.com/video.php?v=10152797088108553" youtube-dl "https://www.facebook.com/video.php?v=10152797098293553" youtube-dl "https://www.facebook.com/video.php?v=10152797107658553" youtube-dl "https://www.facebook.com/video.php?v=10152797148263553" youtube-dl "https://www.facebook.com/video.php?v=10152797156538553" youtube-dl "https://www.facebook.com/video.php?v=10152797345373553" youtube-dl "https://www.facebook.com/video.php?v=10152797350653553" youtube-dl "https://www.facebook.com/video.php?v=10152797720553553" youtube-dl "https://www.facebook.com/video.php?v=10152797736763553" youtube-dl "https://www.facebook.com/video.php?v=10152800459113553" youtube-dl "https://www.facebook.com/video.php?v=10152800485138553" youtube-dl "https://www.facebook.com/video.php?v=10152800517193553" youtube-dl "https://www.facebook.com/video.php?v=10152800539128553" youtube-dl "https://www.facebook.com/video.php?v=10152800550043553" youtube-dl "https://www.facebook.com/video.php?v=10152800554083553" youtube-dl "https://www.facebook.com/video.php?v=10152800569888553" youtube-dl "https://www.facebook.com/video.php?v=10152800582978553" youtube-dl "https://www.facebook.com/video.php?v=10152800594638553" youtube-dl "https://www.facebook.com/video.php?v=10152800597428553" youtube-dl "https://www.facebook.com/video.php?v=10152800611948553" youtube-dl "https://www.facebook.com/video.php?v=10152800614133553" youtube-dl "https://www.facebook.com/video.php?v=10152800617653553" youtube-dl "https://www.facebook.com/video.php?v=10152800624928553" youtube-dl "https://www.facebook.com/video.php?v=10152800744793553" youtube-dl "https://www.facebook.com/video.php?v=10152800779103553" youtube-dl "https://www.facebook.com/video.php?v=10152800781003553" youtube-dl "https://www.facebook.com/video.php?v=10152800789943553" youtube-dl "https://www.facebook.com/video.php?v=10152800795178553" youtube-dl "https://www.facebook.com/video.php?v=10152800797663553"
freedrull 12y ago
"Push-pull-rebase bottleneck: if you rebase and push and someone beats you to it, you have to pull, rebase, and try again. This gets worse as commit rate increases and people do needless legwork. Facebook has moved to server-side rebasing on push to mostly eliminate this pain point. (This is part of a still-experimental feature in Mercurial, which should hopefully lose its experimental flag soon.)" What if there are merge conflicts? I don't know about Mercurial, but in Git there are tons of cases where rebasing cannot happen automatically.
- evgen 12y ago
  The process is optimistic. You submit your commit and it runs async in the background with success or failure sent to you via email and SMS. You still run into merge conflicts, especially if you are touching a frequently tweaked bit of core, but not having to babysit the process is almost always a win.
coldcode 12y ago
Although I love reading how big companies solve developmental problems I am also glad I don't work there. Programming at FB seems more like working in a salt mine.

162 points by cpeterso 12y ago

Interesting info about XCode and the FB app. I wonder if Apple is collaborating with FB on this or if it's mostly deaf ears... "So they built their own IDE. Set of plugins on top of ATOM. Not a fork. They like hackable and web-y nature of ATOM. The demo showing iOS development looks very nice! " Seems that they have the right idea about "optimizing for Developers"

coldcode 12y ago
Although I don't really like XCode but use it all the time, what the hell takes 5 minutes to open a project????? The FB app isn't that big. Maybe it includes downloading an entire repo or something? I've worked on big iOS apps and this was never an issue.
- insaneirish 12y ago
  Facebook has a 54 GB monolithic repo (as of almost a year ago, April 2014). Primary source: https://twitter.com/feross/status/459259593630433280 https://twitter.com/feross/status/459259593630433280 HN article: https://news.ycombinator.com/item?id=7648237 https://news.ycombinator.com/item?id=7648237

brazzlemobile 12y ago

Anyone have a mirror? Getting gateway timeouts.

indygreg2 12y ago
The site is hosted via GitHub pages. Maybe it's not available to you due to the lingering GitHub DoS. If you care to read HTML and can manage to get through to GitHub: https://github.com/indygreg/indygreg.github.com/blob/master/blog/2015/03/28/notes-from-facebook%27s-developer-infrastructure-at-scale-f8-talk/index.html https://github.com/indygreg/indygreg.github.com/blob/master/...

Bahamut 12y ago

A lot of nice summaries and points, but one caught my eye - the mention about building Nuclide on top of Atom. My experience with Atom is that it gets painfully slow quickly, with me experiencing a crash every few days. I would like to hear more thoughts from FB folks on their experiences.

SwellJoe 12y ago
Java desktop applications have shown that with enough resources thrown at the problem, slow and flaky things can become fast and stable. JavaScript desktop applications seem to be at the "slow and flaky but has tons of resources being thrown at them" stage. The VM, as with Java ten years ago, is also seeing remarkable improvements.

beliu 12y ago

On the topic of having one IDE that just works for multiple languages/platforms, you should check out srclib (https://srclib.org https://srclib.org), an open-source language analysis tool (I'm one of the creators) that abstracts away the language-specific parts of IDE support so that IDEs/editor plugins can hook into one API and automatically get support for a bunch of different languages. This way, you don't have to build MxN plugins (M editors, N languages) to support all combos of editors/languages people might want to use. The currently supported languages are Go, Java, Python, JavaScript, and Ruby, so this wouldn't work yet for iOS, but Objective-C and Swift are high on our priority list. Editor plugins exist for Emacs, Sublime, and Atom. Would love to hear anyone's feedback from trying it out.

wting 12y ago
This sounds a lot like Steve Yegge's Grok: http://bsumm.net/2012/08/11/steve-yegge-and-grok.html http://bsumm.net/2012/08/11/steve-yegge-and-grok.html It's unlikely that his version will ever be open sourced or usable outside of Google though.
- durin42 12y ago
  IIRC http://kythe.io http://kythe.io is at least related by heritage to grok.
  - beliu 12y ago
    Yep, Kythe is the open-source version of grok. It's awesome, and the srclib team is in touch with the Kythe team at Google to build on top of their analysis libs (for C++ and Java). Long term vision is to get rid of the "my editor X doesn't support language Y" problem as well as build a lot of other multi-language analysis tools on top of it. Shameless plug: if you care about developer productivity, join us!
fallat 12y ago
>On the topic of having one IDE that just works for multiple languages/platforms, I've recently given Eclipse another chance after 4 years. It has impressed me with its C, PHP and Haskell support so far. I really recommend people give it a shot. Of course I installed a half decent GTK theme to make it look nice :)

131hn 12y ago

* Yep, another "better than ever"/"last one you'll ever need" IDE. Thank you facebook. (see you soon for the next one) * A DVCS that rely on a central server for merging (sandcastle) is no longer.. distributed... (and you cannot have distributed team work here, this is wrong in multiple way) * I think i'll never let a centralized, monolithic repository to be set up in my company. All great stuffs/talent i ever learned came from differents sources, differents independant projets (from git, hg or SVN). Loose that and i think i'll get narrow minded. * All those fancy stuffs make facebookers better "facebook developers" but less prone to share with others (we don't share the same language, culture or tools anymore, even the design of the monolithic repo cannot help here) * This is more a lesson on "how we made X thousands lambda devs work together" than actual tools i might need to use someday. * I'll bet that facebook "infra" developers are less likely to use the tools they describe, and that is ironic. Those tools mostly apply to some hidden mass (100k commit / week.. ) of uniform developers that I'll never meet.

DannyBee 12y ago
"A DVCS that rely on a central server for merging (sandcastle) is no longer.. distributed... (and you cannot have distributed team work here, this is wrong in multiple way)" Yet doing things like rebasing or merging pull requests properly, requires you have an up-to-date master, making it a centralized operation. People seem quite happy with this.
- hueving 12y ago
  >Yet doing things like rebasing or merging pull requests properly Rebasing is a smell of bad source code management. Merging pull requests is fine whether or not you are up to date. I don't know what you mean by 'properly'.
  - indygreg2 12y ago
    It is much easier to bisect linear history than history with merges.
    - 12y ago

larsberg 12y ago

This talk and my experiences at Microsoft are why I tend to try to steer our projects at Mozilla Research to partner with external offerings rather than doing too much building of our own. It's hard to imagine a world where we build and continue to improve a source control / issue tracking system better than GitHub; a continuous integration solution better than Travis CI; a build system; etc. Even if we can imagine building a better product today, the ongoing maintenance and extension of those services quickly become full-time multi-person investments with many hundreds of dedicated machines, neither of which even our core infra teams have, much less the little tiny research org.

indygreg2 12y ago
Except it isn't internal only. Many of Facebook's tools are open sourced. And not in the "throw it over the wall" sense. Their open sourced projects tend to gain traction and get a significant amount of community contributors. By turning a number of their core tools into successful open source projects, they are leveraging the community effect to lesson the ongoing maintenance burden for these projects. Yes, they are paying a high initial cost to develop these offerings. But since they tend to produce high quality products that attract nearly-free-to-Facebook labor via open source, the investment tends to pays off in the long run. It's a savvy business move. And one that can arguably only be pulled off by a talented engineer organization.
- larsberg 12y ago
  > Yes, they are paying a high initial cost to develop these offerings. I certainly agree that for organizations who can afford these high initial costs internally this is a great development model. My challenge (as a manager at Mozilla) is that, given we don't have such resources available, how do we build partnerships with groups or other projects that do in order to ensure such offerings are built? Of course, another interesting approach would be to say that GitHub, Travis, Heroku, etc. show that DevOps-as-company is a viable business model and really what we should be doing is spinning up companies that raise funding and build products around each of the areas that are still lacking products that serve enterprises (read: customers with money) today.
12y ago

necubi 12y ago

The most interesting part for me was > They appear to use code coverage to determine what tests > to run. "We're not going to run a test unless your diff > might actually have broken it." This seems like such an obvious optimization in hindsight, I'm surprised it's not more common. Does anybody know of other CIs that can do this?

TheLoneWolfling 12y ago
This seems like a good optimization in theory. In practice? I think it may end up selectively pruning for hard-to-find bugs, which is not a desirable feature of a test suite.
dperfect 12y ago
I'd be hesitant to rely 100% on this approach (and from the notes it sounds like FB still runs the other tests - just less frequently in some cases), especially where the code coverage tools aren't guaranteed to account for unintended interactions between seemingly unrelated pieces of code. In theory, well-written code/tests shouldn't have those kinds of problems in the first place, but larger projects often don't deserve that level of trust (in my opinion). Still, I think it's a great idea for giving quick, "preliminary" feedback to developers - as long as the full test suite is still run periodically.
- necubi 12y ago
  Yeah, absolutely. You'd want to run your full test suite (which probably includes heavier integration tests as well) before deploying, but I think in a lot of cases it's valuable to provide, say, 95% accurate feedback quickly.
  - icefox 12y ago
    Another way to put it would be I will happily only take 95% if the other option is we don't do anything and people can even commit stuff that doesn't even build let alone pass tests.
12y ago

rurban 12y ago

"We made mercurial up to 50x faster than git, more than 2000 improvements" on big repos with lot of history. This would be their blog post about this: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ https://code.facebook.com/posts/218678814984400/scaling-merc... and the code at http://selenic.com/repo/hg/shortlog/ http://selenic.com/repo/hg/shortlog/

ghuntley 12y ago

Does anyone have a .torrent of all the videos from F8? It's a shame that they are self-hosting otherwise it would be possible to youtube-dl it :-)

ghuntley 12y ago
Here's a bash script that will download all of the videos: $ cat download-facebook-f8-2015-videos.sh #!/usr/bin/env bash youtube-dl "https://www.facebook.com/video.php?v=10152795258318553" youtube-dl "https://www.facebook.com/video.php?v=10152795258318553" youtube-dl "https://www.facebook.com/video.php?v=10152795404193553" youtube-dl "https://www.facebook.com/video.php?v=10152795420103553" youtube-dl "https://www.facebook.com/video.php?v=10152795617423553" youtube-dl "https://www.facebook.com/video.php?v=10152795634488553" youtube-dl "https://www.facebook.com/video.php?v=10152795636318553" youtube-dl "https://www.facebook.com/video.php?v=10152795737378553" youtube-dl "https://www.facebook.com/video.php?v=10152795739268553" youtube-dl "https://www.facebook.com/video.php?v=10152795771278553" youtube-dl "https://www.facebook.com/video.php?v=10152795771278553" youtube-dl "https://www.facebook.com/video.php?v=10152795779043553" youtube-dl "https://www.facebook.com/video.php?v=10152795794003553" youtube-dl "https://www.facebook.com/video.php?v=10152797088108553" youtube-dl "https://www.facebook.com/video.php?v=10152797098293553" youtube-dl "https://www.facebook.com/video.php?v=10152797107658553" youtube-dl "https://www.facebook.com/video.php?v=10152797148263553" youtube-dl "https://www.facebook.com/video.php?v=10152797156538553" youtube-dl "https://www.facebook.com/video.php?v=10152797345373553" youtube-dl "https://www.facebook.com/video.php?v=10152797350653553" youtube-dl "https://www.facebook.com/video.php?v=10152797720553553" youtube-dl "https://www.facebook.com/video.php?v=10152797736763553" youtube-dl "https://www.facebook.com/video.php?v=10152800459113553" youtube-dl "https://www.facebook.com/video.php?v=10152800485138553" youtube-dl "https://www.facebook.com/video.php?v=10152800517193553" youtube-dl "https://www.facebook.com/video.php?v=10152800539128553" youtube-dl "https://www.facebook.com/video.php?v=10152800550043553" youtube-dl "https://www.facebook.com/video.php?v=10152800554083553" youtube-dl "https://www.facebook.com/video.php?v=10152800569888553" youtube-dl "https://www.facebook.com/video.php?v=10152800582978553" youtube-dl "https://www.facebook.com/video.php?v=10152800594638553" youtube-dl "https://www.facebook.com/video.php?v=10152800597428553" youtube-dl "https://www.facebook.com/video.php?v=10152800611948553" youtube-dl "https://www.facebook.com/video.php?v=10152800614133553" youtube-dl "https://www.facebook.com/video.php?v=10152800617653553" youtube-dl "https://www.facebook.com/video.php?v=10152800624928553" youtube-dl "https://www.facebook.com/video.php?v=10152800744793553" youtube-dl "https://www.facebook.com/video.php?v=10152800779103553" youtube-dl "https://www.facebook.com/video.php?v=10152800781003553" youtube-dl "https://www.facebook.com/video.php?v=10152800789943553" youtube-dl "https://www.facebook.com/video.php?v=10152800795178553" youtube-dl "https://www.facebook.com/video.php?v=10152800797663553"

freedrull 12y ago

"Push-pull-rebase bottleneck: if you rebase and push and someone beats you to it, you have to pull, rebase, and try again. This gets worse as commit rate increases and people do needless legwork. Facebook has moved to server-side rebasing on push to mostly eliminate this pain point. (This is part of a still-experimental feature in Mercurial, which should hopefully lose its experimental flag soon.)" What if there are merge conflicts? I don't know about Mercurial, but in Git there are tons of cases where rebasing cannot happen automatically.

evgen 12y ago
The process is optimistic. You submit your commit and it runs async in the background with success or failure sent to you via email and SMS. You still run into merge conflicts, especially if you are touching a frequently tweaked bit of core, but not having to babysit the process is almost always a win.

coldcode 12y ago

Although I love reading how big companies solve developmental problems I am also glad I don't work there. Programming at FB seems more like working in a salt mine.