9 ms·

A behind-the-scenes look at Facebook release engineering

196 points by 3lit3H4ck3r 15y ago

raphinou 15y ago
The article mentions that employees visiting Facebook from inside use an experimental build. Any idea how they manage this if the experimental build requires changes in the data structures used by the site?
- bonzoesc 15y ago
  I suspect that if new features require storage changes, the changes are strictly additive and won't affect old features, or old features are modified to use the new storage setup. With the amount of data Facebook has, they probably don't have the option of an "iterate over every row and change a thing" kind of migration.
  - flyt 15y ago
    No, but they do have an extremely fast and efficient tool for performing online schema changes to its huge MySQL deployment: https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932 https://www.facebook.com/notes/mysql-at-facebook/online-sche...
- HarrisonFisk 15y ago
  Quora question about that: http://www.quora.com/How-does-Facebook-implement-feature-switching-including-versioning/ http://www.quora.com/How-does-Facebook-implement-feature-swi...
- ahupp 15y ago
  Eventually you'll have users in the same state, either because the push is not atomic or because the change is rolled out slowly. This means you have to build the change with this in mind regardless of the employee testing.
benjaminwootton 15y ago
I'm surprised by the monolithic all-or-nothing deployments that they have off of the single binary. I prefer to componentise applications and allow those components to be deployed, released, and rolled back separately. I also don't agree with the 'rollback is for losers' message as hinted at in the article. A fast dependable rollback (measured in the seconds) is significantly preferable to getting a developer to implement a fix to some issue under pressure and push it out in a rush. Much better to roll back, take stock, then implement the right fix under the tested process.
- flyt 15y ago
  Facebook is updating tens of thousands of servers with every push. "Rolling back" a release could take as long as a regular push and contribute to problems as the version in use diverges. Instead, FB has an aggressive and flexible internal system for "ungating" features to groups based on different criteria. Usually a feature would be pushed out in a deactivated state, then a developer will slowly ramp up its exposure to actual traffic. This limits the ability for a push to insta-break the site and means they can come back around for the next day's push with tweaks, then increase the code's exposure.
  - benjaminwootton 15y ago
    Still though, the 1.5 GB binary takes 30 minutes to get out to the servers as per the article. In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30. I agree that you can and should have all of the gating and safety nets and checks in the world as you roll forward, but at the end of the day, you need the ultimate safety net of a very fast, very dependable rollback that you can run at will. IMO reverting isn't for losers. It should be the first thing you do in case of errors, if your architecture supports it.
    - rguldener 15y ago
      "In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30." According to Rossi the servers keep the old version around after rollout, so rollback wouldn't require them to push out the old binary again but just to restart the old process. This will probably only take a couple of seconds making rollback really fast if needed.
algolicious 15y ago
Facebook's testing practices and culture of developer accountability help to prevent serious bugs from being rolled out in production code. When a developer's code disrupts the website and necessitates a post-deployment fix, the incident is tracked and factored into Facebook's assessment of the developer's job performance. [...] Employees with low karma can regain their lost points over time by performing well—though some also try to help their odds by bringing Rossi goodies. Booze and cupcakes are Rossi's preferred currency of redemption; the release engineering team has an impressive supply of booze on hand, some of which was supplied by developers looking to restore their tarnished karma. This sounds like Facebook strongly rewards developers who work on trivial, low-risk features rather than larger, more important features. Also, it sounds like bribery factors into your overall job performance rating.
- binxbolling 15y ago
  I think you're reading too much into that part.
- nbm 15y ago
  Push karma primarily affects how likely the release engineering team will accept any breaking of the standard rules of getting your code into the push. It generally doesn't drop if you are responsive and responsible for any problems your change causes. The only way to restore points is to show respect and consideration for the hard work the release engineering team does. (I'm not 100% sure, but I think most of the booze and cupcakes come from people who were appreciative of the release engineering team for bringing potential issues to their attention or for being accommodating in terms of hours and in terms of delay to get things fixed.) Being irresponsible (not supporting your changes) will factor into your performance review, but working exclusively on low-risk features will most likely hurt it way more.
- Spellman 15y ago
  I was more annoyed that the karma system wasn't attributed to other sites who used it first. Which is to say not terribly.
brown9-2 15y ago
This tech talk video from Chuck Rossi on the same topic also has a lot of really interesting information https://www.facebook.com/video/video.php?v=10100259101684977 https://www.facebook.com/video/video.php?v=10100259101684977
zbuc 15y ago
How do you wind up with a 1.5gb binary? That's incredible -- especially considering all their static assets are on their CDN, so this is basically their code and all the libraries they're pulling in.
- jgrahamc 15y ago
  I expect you end up with that when you decide that you never, ever want to have to resolve a problem with an executable and a dynamic library having incompatible versions.
- steadicat 15y ago
  That binary includes the static assets. They're the origin for the CDN.
  - YuriNiyazov 15y ago
    The companies whose deployment processes I've seen did exactly that - after an update, a request to a CDN results in a fault which forces the CDN to query the original company server. In those cases, the static assets had to be distributed onto the servers. I would imagine Facebook does the same, but the article pretty explicitly says that they don't do that (which is why I imagine you got downvoted - for the record, I think that was inappropriate), which raises the question of what exactly do they do?
  - bonzoesc 15y ago
    > The binary executable is just one part of the Facebook application stack, of course. Many external resources are referenced from Facebook pages, including JavaScript, CSS, and graphical assets. Those files are hosted on geographically distributed content delivery networks (CDNs). It doesn't say whether or not these are part of the binary, but I'd suspect they aren't used as the CDN origin. You'd want these (randomly-but-uniquely named) resources to be pushed out to the CDN and warmed up before a deploy to avoid a thundering herd[1] problem from millions of users. 1: http://en.wikipedia.org/wiki/Thundering_herd_problem http://en.wikipedia.org/wiki/Thundering_herd_problem
cake 15y ago
The company has two separate sets of these tests; one does some conventional sanity checking on the code and the other simulates user interaction to make sure that the website's user interface behaves properly. Anyone know more about this ? How are the user interaction made ?
- rheide 15y ago
  I'm guessing they're talking about unit tests and integration tests, the integration tests probably simulating user input using a framework such as Selenium(RC).
  - alexgartrell 15y ago
    It's been a while since I've had to deal with it (bootcamp), but I'm pretty sure we still use Watir [1] [1] http://watir.com/ http://watir.com/
- catshirt 15y ago
  can't speak on facebook specifically, but one popular option is Selenium http://seleniumhq.org/ http://seleniumhq.org/
- thomaslangston 15y ago
  Others have already mentioned Selenium, but another integration test framework is CasperJS. http://casperjs.org/ http://casperjs.org/
jpeterson 15y ago
Why would they transfer the entire 1.5g binary each time, and not a delta? Surely it doesn't change that much between releases.
- monatron 15y ago
  further on in the article they explain that facebook is developing a hip hop virtual machine that allows them to run custom bytecode in their own runtime environment which would allow them to push deltas. im fairly certain that pushing deltas of compiled c++ code on this scale is impossible.
- shabble 15y ago
  I wonder whether they've considered (something similar to) Google Courgette[1], for distributing compiled binaries with many similarities, but translating into abstract basic blocks and rewriting pointers to match. Although, both the generation of the diff-set, and it's application on each individual server might end up eating more resources than just using torrents. [1] http://dev.chromium.org/developers/design-documents/software-updates-courgette http://dev.chromium.org/developers/design-documents/software...
- tedunangst 15y ago
  "bsdiff is quite memory-hungry. It requires max(17n,9n+m)+O(1) bytes of memory, where n is the size of the old file and m is the size of the new file." http://www.daemonology.net/bsdiff/ http://www.daemonology.net/bsdiff/
- jrockway 15y ago
  Because the continuous integration runs tests for all components at HEAD, not at every random possible combination that could end up on a machine. The key to releases is repeatability and consistency. Copying one big blob to every machine is repeatable and consistent. Installing a bunch of libraries and updating things piecemeal is much more difficult to do right. Internal bandwidth is cheap, so this is almost a no-brainer. Even without an internal bittorrent distribution mechanism, it's still easy. The reason why people tend to gravitate to incremental deployments for web applications is because the typical tools encourage it; modules get installed in separate directories, each part of the app is a separate file (back in the CGI days), etc. When you compile everything into one file, though, then you just copy that file around to deploy. It's easier. (Ask a PHP programmer how to change one file, and it will probably be "change that one file". Ask a Java programmer, and it will be "fix the file, build a WAR, and replace the WAR". Tools dictate process, and the "scripting language" default is to work on a file level instead of an application level.) I've always wanted one-file deployment for my personal applications, but I never saw anyone doing it so I assumed I was wrong. But nope, it turns out that everyone else was wrong :)
bonaldi 15y ago
I find it most interesting that they rely on irc internally. They work on one of the world's largest online communications platforms -- surely they could solve their problem in a way that gives it to their millions of users too?
- thomaslangston 15y ago
  Why? There is no need to re-invent IRC.
  - bonaldi 15y ago
    And yet a fair few people (including 37 Signals) have tried, because there's a recognition that irc is plain hostile to modern users. A real-time low-friction shared chat service would actually be a hugely compelling thing inside Facebook.
    - Florin_Andrei 15y ago
      > there's a recognition that irc is plain hostile to modern users How so? (just curious, I used it before in a corporate environment, it was fine for us)
      - bonaldi 15y ago
        In much the same way that if you live in a world of Word, vi is hostile. Don't get me wrong -- I use and love irc, for many of the reasons given above. I just think if I could have all or many of its benefits in a way that all my non-technical Facebook friends could use too, there'd be something exceptionally powerful and useful. It's almost like right now we're in the days of pine and mailman. What we could have is GMail.
        Florin_Andrei 15y ago
        I see what you mean. We were all Unix geeks, so perhaps that's why it worked well for us.
joshuahedlund 15y ago
So many gems in this article. To help spot problems, Facebook employees who access the social network from within the company's internal network will always see an experimental build of the site based on the very latest code, including proposed changes that haven't officially been accepted. Probably the only place where your excuse for checking Facebook at work can be "Looking for bugs!" The many data sources tracked by Facebook's internal monitoring tools even include tweets about Facebook. That information is displayed in a graph with separate trend lines to show the change in volume of positive and negative remarks Guess I need to tweet more about how slow their mobile app is getting... One of the major ongoing development efforts at Facebook is a project to replace the HipHop transpiler. Facebook's developers are creating their own bytecode format and custom runtime environment.... the company can push thin bytecode deltas representing just the parts that have changed. Facebook may even be able to splice the updated bytecode into the application while it's running, avoiding a process restart. Even though I'm doing nothing nearly this awesome, this article has done more to inspire and excite me about my own coding than anything I've read in a long time.
- mikeleeorg 15y ago
  I'm also really intrigued by the "karma" rating for all of their developers. Can anyone speak to how well that is working? Has it been effective? Any negative side-effects? Is it just for the release process, or for any development effort?
  - nbm 15y ago
    It is "Push karma", so generally only applies to the push process. It isn't visible to anyone but the pushers and yourself (at least, I can't see anyone else's push karma in the expected places). It isn't a complex rating system - there are probably 97% of people at the base karma level, maybe 0.1% at one rung higher, 2.7% at one rung lower, and 0.2% at lower than that. The "Push" tech talk at https://www.facebook.com/video/video.php?v=10100259101684977 https://www.facebook.com/video/video.php?v=10100259101684977 has more on it. Mostly it is a way of letting you know that you made people's lives difficult by holding up the push process by not being available to support your changes. You know that you won't get away with that, that you need to make up for it, and you also know when you've made up for it. (It also doesn't apply to a decent number of engineers, since they work on services and infrastructure that are not part of that process.)
stcredzero 15y ago
Facebook's developers are creating their own bytecode format and custom runtime environment.... the company can push thin bytecode deltas representing just the parts that have changed. Facebook may even be able to splice the updated bytecode into the application while it's running, avoiding a process restart. NB: Fast JIT byte code VMs running web app frameworks in high level languages that can do all of this have been around since the early 2000s. (Including the distribution of binary deltas that can be applied atomically to live servers.) Smalltalk web app servers had all of this tech, plus refactoring capabilities and distributed version control years ahead of the rest of the industry. It makes me wonder what else is out there beneath the radar today.

196 points by 3lit3H4ck3r 15y ago

The article mentions that employees visiting Facebook from inside use an experimental build. Any idea how they manage this if the experimental build requires changes in the data structures used by the site?

bonzoesc 15y ago
I suspect that if new features require storage changes, the changes are strictly additive and won't affect old features, or old features are modified to use the new storage setup. With the amount of data Facebook has, they probably don't have the option of an "iterate over every row and change a thing" kind of migration.
- flyt 15y ago
  No, but they do have an extremely fast and efficient tool for performing online schema changes to its huge MySQL deployment: https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932 https://www.facebook.com/notes/mysql-at-facebook/online-sche...
HarrisonFisk 15y ago
Quora question about that: http://www.quora.com/How-does-Facebook-implement-feature-switching-including-versioning/ http://www.quora.com/How-does-Facebook-implement-feature-swi...
ahupp 15y ago
Eventually you'll have users in the same state, either because the push is not atomic or because the change is rolled out slowly. This means you have to build the change with this in mind regardless of the employee testing.

benjaminwootton 15y ago

I'm surprised by the monolithic all-or-nothing deployments that they have off of the single binary. I prefer to componentise applications and allow those components to be deployed, released, and rolled back separately. I also don't agree with the 'rollback is for losers' message as hinted at in the article. A fast dependable rollback (measured in the seconds) is significantly preferable to getting a developer to implement a fix to some issue under pressure and push it out in a rush. Much better to roll back, take stock, then implement the right fix under the tested process.

flyt 15y ago
Facebook is updating tens of thousands of servers with every push. "Rolling back" a release could take as long as a regular push and contribute to problems as the version in use diverges. Instead, FB has an aggressive and flexible internal system for "ungating" features to groups based on different criteria. Usually a feature would be pushed out in a deactivated state, then a developer will slowly ramp up its exposure to actual traffic. This limits the ability for a push to insta-break the site and means they can come back around for the next day's push with tweaks, then increase the code's exposure.
- benjaminwootton 15y ago
  Still though, the 1.5 GB binary takes 30 minutes to get out to the servers as per the article. In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30. I agree that you can and should have all of the gating and safety nets and checks in the world as you roll forward, but at the end of the day, you need the ultimate safety net of a very fast, very dependable rollback that you can run at will. IMO reverting isn't for losers. It should be the first thing you do in case of errors, if your architecture supports it.
  - rguldener 15y ago
    "In my business, we would be crucified if we had to wait for 3 minutes to roll back a change let alone 30." According to Rossi the servers keep the old version around after rollout, so rollback wouldn't require them to push out the old binary again but just to restart the old process. This will probably only take a couple of seconds making rollback really fast if needed.

algolicious 15y ago

Facebook's testing practices and culture of developer accountability help to prevent serious bugs from being rolled out in production code. When a developer's code disrupts the website and necessitates a post-deployment fix, the incident is tracked and factored into Facebook's assessment of the developer's job performance. [...] Employees with low karma can regain their lost points over time by performing well—though some also try to help their odds by bringing Rossi goodies. Booze and cupcakes are Rossi's preferred currency of redemption; the release engineering team has an impressive supply of booze on hand, some of which was supplied by developers looking to restore their tarnished karma. This sounds like Facebook strongly rewards developers who work on trivial, low-risk features rather than larger, more important features. Also, it sounds like bribery factors into your overall job performance rating.

binxbolling 15y ago
I think you're reading too much into that part.
nbm 15y ago
Push karma primarily affects how likely the release engineering team will accept any breaking of the standard rules of getting your code into the push. It generally doesn't drop if you are responsive and responsible for any problems your change causes. The only way to restore points is to show respect and consideration for the hard work the release engineering team does. (I'm not 100% sure, but I think most of the booze and cupcakes come from people who were appreciative of the release engineering team for bringing potential issues to their attention or for being accommodating in terms of hours and in terms of delay to get things fixed.) Being irresponsible (not supporting your changes) will factor into your performance review, but working exclusively on low-risk features will most likely hurt it way more.
Spellman 15y ago
I was more annoyed that the karma system wasn't attributed to other sites who used it first. Which is to say not terribly.

brown9-2 15y ago

This tech talk video from Chuck Rossi on the same topic also has a lot of really interesting information https://www.facebook.com/video/video.php?v=10100259101684977 https://www.facebook.com/video/video.php?v=10100259101684977

zbuc 15y ago

How do you wind up with a 1.5gb binary? That's incredible -- especially considering all their static assets are on their CDN, so this is basically their code and all the libraries they're pulling in.

jgrahamc 15y ago
I expect you end up with that when you decide that you never, ever want to have to resolve a problem with an executable and a dynamic library having incompatible versions.
steadicat 15y ago
That binary includes the static assets. They're the origin for the CDN.
- YuriNiyazov 15y ago
  The companies whose deployment processes I've seen did exactly that - after an update, a request to a CDN results in a fault which forces the CDN to query the original company server. In those cases, the static assets had to be distributed onto the servers. I would imagine Facebook does the same, but the article pretty explicitly says that they don't do that (which is why I imagine you got downvoted - for the record, I think that was inappropriate), which raises the question of what exactly do they do?
- bonzoesc 15y ago
  > The binary executable is just one part of the Facebook application stack, of course. Many external resources are referenced from Facebook pages, including JavaScript, CSS, and graphical assets. Those files are hosted on geographically distributed content delivery networks (CDNs). It doesn't say whether or not these are part of the binary, but I'd suspect they aren't used as the CDN origin. You'd want these (randomly-but-uniquely named) resources to be pushed out to the CDN and warmed up before a deploy to avoid a thundering herd[1] problem from millions of users. 1: http://en.wikipedia.org/wiki/Thundering_herd_problem http://en.wikipedia.org/wiki/Thundering_herd_problem

cake 15y ago

The company has two separate sets of these tests; one does some conventional sanity checking on the code and the other simulates user interaction to make sure that the website's user interface behaves properly. Anyone know more about this ? How are the user interaction made ?

rheide 15y ago
I'm guessing they're talking about unit tests and integration tests, the integration tests probably simulating user input using a framework such as Selenium(RC).
- alexgartrell 15y ago
  It's been a while since I've had to deal with it (bootcamp), but I'm pretty sure we still use Watir [1] [1] http://watir.com/ http://watir.com/
catshirt 15y ago
can't speak on facebook specifically, but one popular option is Selenium http://seleniumhq.org/ http://seleniumhq.org/
thomaslangston 15y ago
Others have already mentioned Selenium, but another integration test framework is CasperJS. http://casperjs.org/ http://casperjs.org/

jpeterson 15y ago

Why would they transfer the entire 1.5g binary each time, and not a delta? Surely it doesn't change that much between releases.

monatron 15y ago
further on in the article they explain that facebook is developing a hip hop virtual machine that allows them to run custom bytecode in their own runtime environment which would allow them to push deltas. im fairly certain that pushing deltas of compiled c++ code on this scale is impossible.
shabble 15y ago
I wonder whether they've considered (something similar to) Google Courgette[1], for distributing compiled binaries with many similarities, but translating into abstract basic blocks and rewriting pointers to match. Although, both the generation of the diff-set, and it's application on each individual server might end up eating more resources than just using torrents. [1] http://dev.chromium.org/developers/design-documents/software-updates-courgette http://dev.chromium.org/developers/design-documents/software...
tedunangst 15y ago
"bsdiff is quite memory-hungry. It requires max(17n,9n+m)+O(1) bytes of memory, where n is the size of the old file and m is the size of the new file." http://www.daemonology.net/bsdiff/ http://www.daemonology.net/bsdiff/
jrockway 15y ago
Because the continuous integration runs tests for all components at HEAD, not at every random possible combination that could end up on a machine. The key to releases is repeatability and consistency. Copying one big blob to every machine is repeatable and consistent. Installing a bunch of libraries and updating things piecemeal is much more difficult to do right. Internal bandwidth is cheap, so this is almost a no-brainer. Even without an internal bittorrent distribution mechanism, it's still easy. The reason why people tend to gravitate to incremental deployments for web applications is because the typical tools encourage it; modules get installed in separate directories, each part of the app is a separate file (back in the CGI days), etc. When you compile everything into one file, though, then you just copy that file around to deploy. It's easier. (Ask a PHP programmer how to change one file, and it will probably be "change that one file". Ask a Java programmer, and it will be "fix the file, build a WAR, and replace the WAR". Tools dictate process, and the "scripting language" default is to work on a file level instead of an application level.) I've always wanted one-file deployment for my personal applications, but I never saw anyone doing it so I assumed I was wrong. But nope, it turns out that everyone else was wrong :)

bonaldi 15y ago

I find it most interesting that they rely on irc internally. They work on one of the world's largest online communications platforms -- surely they could solve their problem in a way that gives it to their millions of users too?

thomaslangston 15y ago
Why? There is no need to re-invent IRC.
- bonaldi 15y ago
  And yet a fair few people (including 37 Signals) have tried, because there's a recognition that irc is plain hostile to modern users. A real-time low-friction shared chat service would actually be a hugely compelling thing inside Facebook.
  - Florin_Andrei 15y ago
    > there's a recognition that irc is plain hostile to modern users How so? (just curious, I used it before in a corporate environment, it was fine for us)
    - bonaldi 15y ago
      In much the same way that if you live in a world of Word, vi is hostile. Don't get me wrong -- I use and love irc, for many of the reasons given above. I just think if I could have all or many of its benefits in a way that all my non-technical Facebook friends could use too, there'd be something exceptionally powerful and useful. It's almost like right now we're in the days of pine and mailman. What we could have is GMail.
      - Florin_Andrei 15y ago
        I see what you mean. We were all Unix geeks, so perhaps that's why it worked well for us.

joshuahedlund 15y ago

So many gems in this article. To help spot problems, Facebook employees who access the social network from within the company's internal network will always see an experimental build of the site based on the very latest code, including proposed changes that haven't officially been accepted. Probably the only place where your excuse for checking Facebook at work can be "Looking for bugs!" The many data sources tracked by Facebook's internal monitoring tools even include tweets about Facebook. That information is displayed in a graph with separate trend lines to show the change in volume of positive and negative remarks Guess I need to tweet more about how slow their mobile app is getting... One of the major ongoing development efforts at Facebook is a project to replace the HipHop transpiler. Facebook's developers are creating their own bytecode format and custom runtime environment.... the company can push thin bytecode deltas representing just the parts that have changed. Facebook may even be able to splice the updated bytecode into the application while it's running, avoiding a process restart. Even though I'm doing nothing nearly this awesome, this article has done more to inspire and excite me about my own coding than anything I've read in a long time.

mikeleeorg 15y ago
I'm also really intrigued by the "karma" rating for all of their developers. Can anyone speak to how well that is working? Has it been effective? Any negative side-effects? Is it just for the release process, or for any development effort?
- nbm 15y ago
  It is "Push karma", so generally only applies to the push process. It isn't visible to anyone but the pushers and yourself (at least, I can't see anyone else's push karma in the expected places). It isn't a complex rating system - there are probably 97% of people at the base karma level, maybe 0.1% at one rung higher, 2.7% at one rung lower, and 0.2% at lower than that. The "Push" tech talk at https://www.facebook.com/video/video.php?v=10100259101684977 https://www.facebook.com/video/video.php?v=10100259101684977 has more on it. Mostly it is a way of letting you know that you made people's lives difficult by holding up the push process by not being available to support your changes. You know that you won't get away with that, that you need to make up for it, and you also know when you've made up for it. (It also doesn't apply to a decent number of engineers, since they work on services and infrastructure that are not part of that process.)

stcredzero 15y ago

Facebook's developers are creating their own bytecode format and custom runtime environment.... the company can push thin bytecode deltas representing just the parts that have changed. Facebook may even be able to splice the updated bytecode into the application while it's running, avoiding a process restart. NB: Fast JIT byte code VMs running web app frameworks in high level languages that can do all of this have been around since the early 2000s. (Including the distribution of binary deltas that can be applied atomically to live servers.) Smalltalk web app servers had all of this tech, plus refactoring capabilities and distributed version control years ahead of the rest of the industry. It makes me wonder what else is out there beneath the radar today.