6 ms·

How do you wind up with a 1.5gb binary? That's incredible -- especially considering all their static assets are on their CDN, so this is basically their code an

by zbuc 15y ago

How do you wind up with a 1.5gb binary? That's incredible -- especially considering all their static assets are on their CDN, so this is basically their code and all the libraries they're pulling in.

jgrahamc 15y ago
I expect you end up with that when you decide that you never, ever want to have to resolve a problem with an executable and a dynamic library having incompatible versions.
steadicat 15y ago
That binary includes the static assets. They're the origin for the CDN.
- YuriNiyazov 15y ago
  The companies whose deployment processes I've seen did exactly that - after an update, a request to a CDN results in a fault which forces the CDN to query the original company server. In those cases, the static assets had to be distributed onto the servers. I would imagine Facebook does the same, but the article pretty explicitly says that they don't do that (which is why I imagine you got downvoted - for the record, I think that was inappropriate), which raises the question of what exactly do they do?
- bonzoesc 15y ago
  > The binary executable is just one part of the Facebook application stack, of course. Many external resources are referenced from Facebook pages, including JavaScript, CSS, and graphical assets. Those files are hosted on geographically distributed content delivery networks (CDNs). It doesn't say whether or not these are part of the binary, but I'd suspect they aren't used as the CDN origin. You'd want these (randomly-but-uniquely named) resources to be pushed out to the CDN and warmed up before a deploy to avoid a thundering herd[1] problem from millions of users. 1: http://en.wikipedia.org/wiki/Thundering_herd_problem http://en.wikipedia.org/wiki/Thundering_herd_problem
  - nbm 15y ago
    Ignoring technical ways to work around the thundering herd problem, the slow rollout (employees first for several days, then a subset of servers, then a larger subset) also mitigates the problem.
kmavm 15y ago
(I work on the HipHop compiler.) You start by compiling PHP source. Simple PHP statements take a lot more space in the binary than intuition suggests. E.g.: if ($a == $b) ... would seem like it should be cmp $rax, $rbx jz ... But! If type inference has failed, we don't know what types $a and $b are, so they might be strings or objects or something crazy. So we're going to have to indirectly dispatch to $a's '==' method. We also spend a ton of space on reference counting code; the semantics of the language basically force you to do naive reference counting, since refcounts can be witnessed in various ways, so every time we pass an argument, do an assignment, sometimes even evaluate expressions, we need to manipulate reference counts, and if they've gone to zero call a destructor. It ends up making the code really large, and one of the things that's unique about our efforts to run PHP fast relative to other dynamic language efforts is that sheer code bulk ends up being our largest enemy; if we're not careful, icache misses eat us alive. Finally, I'll note that it's not quite a 1.5GB binary. The actual ELF binary is something like 1.1GB, and the remainder of the package we bittorrent around production is stuff like static resources (javascript, css) and primed contents for the APC cache that we want prepopulated on boot.
- cookiecaper 15y ago
  Based on your work heretofore, do you think it's wise for Facebook to continue on the PHP path instead of working on a backend rewrite in C# or some other, saner language? I find it odd that Facebook is still using PHP and pouring lots of effort and cash into things like HipHop when they're obviously hiring people smart enough to use another language, and when they obviously have the runway to perform a dark horse rewrite into a much cleaner, saner backend.
  - kmavm 15y ago
    This is a long and deep subject. I wouldn't say that we've "continued on the PHP path." I'd say that we've refused to throw out the precious PHP parts of our application, while not being afraid to use more appropriate languages across Thrift boundaries when needed. Our search engine, newsfeed, and ad serving infrastructure, for instance, are in C++. A drop-everything-and-rewrite of the PHP code is entirely out of the question, for all the reasons covered in Spolsky's 12-year-old classic on the subject: http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html. Those of us working on making PHP perform better are a tiny fraction of Facebook engineering as a whole; this small overhead cost is nothing compared to the risks inherent in a ground-up rewrite. Most of PHP's language-level faults can be engineered around. For instance, we have a code-review-time script that parses (really parses) the code to warn engineers (and reviewers) about dangerous or deprecated idioms. PHP also has some affirmative virtues. The programming model is more productive than that of compiled languages, and even many interpreted languages; save/reload the web page is just a better, tighter loop to get work done in than save/compile/restart my server/reload the web page. I'm actually a fan of PHP's concurrency model, which naifs often mistake as "no concurrency allowed"; PHP's concurrency primitive is curl[1], and if you wrap a tiny bit of library around it, you can make it behave like actors. [1] Seriously. curl provides a shared-nothing way to asynchronously run code, and has the virtue of not caring what language the other side is written in to boot.

by zbuc 15y ago

How do you wind up with a 1.5gb binary? That's incredible -- especially considering all their static assets are on their CDN, so this is basically their code and all the libraries they're pulling in.

jgrahamc 15y ago

I expect you end up with that when you decide that you never, ever want to have to resolve a problem with an executable and a dynamic library having incompatible versions.

steadicat 15y ago

That binary includes the static assets. They're the origin for the CDN.

YuriNiyazov 15y ago
The companies whose deployment processes I've seen did exactly that - after an update, a request to a CDN results in a fault which forces the CDN to query the original company server. In those cases, the static assets had to be distributed onto the servers. I would imagine Facebook does the same, but the article pretty explicitly says that they don't do that (which is why I imagine you got downvoted - for the record, I think that was inappropriate), which raises the question of what exactly do they do?
bonzoesc 15y ago
> The binary executable is just one part of the Facebook application stack, of course. Many external resources are referenced from Facebook pages, including JavaScript, CSS, and graphical assets. Those files are hosted on geographically distributed content delivery networks (CDNs). It doesn't say whether or not these are part of the binary, but I'd suspect they aren't used as the CDN origin. You'd want these (randomly-but-uniquely named) resources to be pushed out to the CDN and warmed up before a deploy to avoid a thundering herd[1] problem from millions of users. 1: http://en.wikipedia.org/wiki/Thundering_herd_problem http://en.wikipedia.org/wiki/Thundering_herd_problem
- nbm 15y ago
  Ignoring technical ways to work around the thundering herd problem, the slow rollout (employees first for several days, then a subset of servers, then a larger subset) also mitigates the problem.

kmavm 15y ago

(I work on the HipHop compiler.) You start by compiling PHP source. Simple PHP statements take a lot more space in the binary than intuition suggests. E.g.: if ($a == $b) ... would seem like it should be cmp $rax, $rbx jz ... But! If type inference has failed, we don't know what types $a and $b are, so they might be strings or objects or something crazy. So we're going to have to indirectly dispatch to $a's '==' method. We also spend a ton of space on reference counting code; the semantics of the language basically force you to do naive reference counting, since refcounts can be witnessed in various ways, so every time we pass an argument, do an assignment, sometimes even evaluate expressions, we need to manipulate reference counts, and if they've gone to zero call a destructor. It ends up making the code really large, and one of the things that's unique about our efforts to run PHP fast relative to other dynamic language efforts is that sheer code bulk ends up being our largest enemy; if we're not careful, icache misses eat us alive. Finally, I'll note that it's not quite a 1.5GB binary. The actual ELF binary is something like 1.1GB, and the remainder of the package we bittorrent around production is stuff like static resources (javascript, css) and primed contents for the APC cache that we want prepopulated on boot.

cookiecaper 15y ago
Based on your work heretofore, do you think it's wise for Facebook to continue on the PHP path instead of working on a backend rewrite in C# or some other, saner language? I find it odd that Facebook is still using PHP and pouring lots of effort and cash into things like HipHop when they're obviously hiring people smart enough to use another language, and when they obviously have the runway to perform a dark horse rewrite into a much cleaner, saner backend.
- kmavm 15y ago
  This is a long and deep subject. I wouldn't say that we've "continued on the PHP path." I'd say that we've refused to throw out the precious PHP parts of our application, while not being afraid to use more appropriate languages across Thrift boundaries when needed. Our search engine, newsfeed, and ad serving infrastructure, for instance, are in C++. A drop-everything-and-rewrite of the PHP code is entirely out of the question, for all the reasons covered in Spolsky's 12-year-old classic on the subject: http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html. Those of us working on making PHP perform better are a tiny fraction of Facebook engineering as a whole; this small overhead cost is nothing compared to the risks inherent in a ground-up rewrite. Most of PHP's language-level faults can be engineered around. For instance, we have a code-review-time script that parses (really parses) the code to warn engineers (and reviewers) about dangerous or deprecated idioms. PHP also has some affirmative virtues. The programming model is more productive than that of compiled languages, and even many interpreted languages; save/reload the web page is just a better, tighter loop to get work done in than save/compile/restart my server/reload the web page. I'm actually a fan of PHP's concurrency model, which naifs often mistake as "no concurrency allowed"; PHP's concurrency primitive is curl[1], and if you wrap a tiny bit of library around it, you can make it behave like actors. [1] Seriously. curl provides a shared-nothing way to asynchronously run code, and has the virtue of not caring what language the other side is written in to boot.