9 ms·

Meta’s Hyperscale Infrastructure: Overview and Insights

302 points by sidcool 2y ago

yuliyp 2y ago
I find it interesting how they describe the PHP web front end as a "serverless" or "function as a service" architecture. I guess it's a matter of perspective. It's a service that has a monolithic codebase with lots of endpoints deployed to it. I guess from the perspective of the maintainer of one of those endpoints it's "serverless" but that abstraction (like all abstractions) has leaks: the teams responsible for the top endpoints and those working on shared libraries can't treat the infra as a given, but rather need to be acutely aware of its limitations and performance characteristics.
- whstl 2y ago
  “Serverless” is not synonymous with Lambda, it’s just a computing model. Lots of companies are hosting old monoliths on Amazon Fargate, for example.
  - falcor84 2y ago
    As I see it, it's just an abuse of terminology. By referring to Fargate as Serverless which previously referred to FaaS, they (apparently AWS's marketing team) have seemingly removed the distinction between FaaS and PaaS
- vineyardmike 2y ago
  I think the full description of “stateless, serverless functions” adds a bit of clarity. My read of this is that whatever code is running doesn’t maintain state between requests, and doesn’t touch the underlying operating system. Which seems pretty standard for highly managed environments anyways. It’s been years since I’ve written backend API code that touched the underlying system, or left objects in heap between requests. The flexibility of knowing that any machine can instantly run the code for your API gives a lot of flexibility to rapidly scale up an API. Nothing is “serverless” to everyone. Especially when you run the data center. But being “serverless” and even sitting above the “language runtime” gives API developers a lot of freedom to focus on business logic.
- ramblemonkey 2y ago
  There’s only one entry point for all requests. It’s php (well, Hack) all the way down from there. So all the routing is readable as php code, request handling etc.
- NBJack 2y ago
  It's serverless only in that there's a whole different organization at Meta handling the (monolith) hosting on actual hardware. By that definition, you could argue Amazon's detail page is "serverless".
- chubot 2y ago
  PHP is definitely the original serverless! You upload your code and someone else runs it! CGI is also serverless I wrote a couple articles with these comparisons, and related issues: Comments on Scripting, CGI, and FastCGI https://www.oilshell.org/blog/2024/06/cgi.html https://www.oilshell.org/blog/2024/06/cgi.html Comments on Shared Unix Hosting vs. the Cloud https://oils.pub/blog/2025/02/shared-hosting.html https://oils.pub/blog/2025/02/shared-hosting.html --- I'm not the only one who thinks this: https://www.devever.net/~hl/mildlydynamic https://www.devever.net/~hl/mildlydynamic AWS Lambda: CGI But It's Trendy. Recently we've seen the rise in popularity of AWS Lambda, a “functions as a service” provider. From my perspective this is literally a reinvention of CGI, except a) much more complicated for essentially the same functionality, b) with vendor lock-in, c) with a much more complex and bespoke deployment process which requires the use of special tools. People who don't think this is true probably never used PHP or CGI ... You don't manage the server; you just upload your code! Architecturally, it is the same. From the user experience, it is the same. PHP and CGI also scale infinitely, because they are stateless. Meta scales PHP to a user base of half the world's population. They could also scale CGI/FastCGI.
davedx 2y ago
Very interesting, in particular the explicit comparisons with hyperscalers. I almost wonder if this is preparation for them launching their own public cloud. Anyone from Meta care to comment?
- olivermuty 2y ago
  It would be pretty impressive to launch a public cloud I would trust even less than GCP :D
  - arjvik 2y ago
    GCP got this reputation because it’s a second class citizen within Google. Google’s own internal infra (Borg, Blaze) is top-notch. If Meta can pull off the public cloud correctly, I’d trust them greatly - they’ve shown significant engineering and product competence till now, even if they could use some more consistent and stable UI.
    - mathverse 2y ago
      Dont all new projects go to gcp within google?
      - Cthulhu_ 2y ago
        Yeah at this stage it doesn't make sense to have two tiers within Google.
      - kridsdale1 2y ago
        Depends on what you mean by new. None of the new features for the mature product I work on have touched it. To the side I have had GCP work but it’s been isolated and as if I were moonlighting.
  - nijave 2y ago
    A lot of Fortune 50 companies compete in some way with Microsoft, Amazon, and Google so are reluctant to use their services. I think Meta has the benefit of less corporate competition Anyway in that context, you don't need trust when you have a sufficiently large legal department
vasco 2y ago
> Moreover, once it [Threads] was developed, the infrastructure teams were given only two day’s notice to prepare for its production launch. Most large organizations would take longer than two days just to draft a project plan involving dozens of interdependent teams, let alone execute it. At Meta, however, we quickly established war rooms across distributed sites, bringing together both infrastructure and product teams to address issues in real-time. Despite the tight timeline, the app’s launch was highly successful, reaching 100 million users within just five days, making it the fastest-growing app in history. Kind of more impressive to have kept this ability to ship fast than anything else. A lot of work is needed to not let the bureaucracy increase and stop the lawyers or other functions from creating approval gates everywhere. Or at least to be able to have war rooms that can get it done so quickly.
- deleted 2y ago
  [deleted]
- silisili 2y ago
  I thought the same thing, but then realize that Threads was a huge flop. Is it really a skill to very quickly release a dud app? I don't know the answer to that. Bypassing bureaucracy seems like heaven, but it feels like it also bypassed the product folks entirely.
  - ianlevesque 2y ago
    What? It has more than 300M users.
    - silisili 2y ago
      Today, maybe? I haven't kept up. I'm talking specifically about its launch.
      - OtherShrezzing 2y ago
        They had 100m account creations within the first 5 days.
asdasd1234 2y ago
Is there any public information available for the deploy/observability tool?
- linkregister 2y ago
  Unfortunately I didn't see any code posted for Conveyor. The following USENIX paper is available: https://www.usenix.org/system/files/osdi23-grubic.pdf https://www.usenix.org/system/files/osdi23-grubic.pdf .
- _kb 2y ago
  I assume it's changes since then, but this is from 2021: https://atscaleconference.com/conveyor-continuous-deployment-at-facebook/ https://atscaleconference.com/conveyor-continuous-deployment.... There's also a good paper on the approaches to configuration management here: https://research.facebook.com/publications/holistic-configuration-management-at-facebook/ https://research.facebook.com/publications/holistic-configur....
kmdrpc 2y ago
Not a word about Thrift, perhaps it was too low level for an infra overview, but I would have expected it to make some technical impact from a global perspective.
- linkregister 2y ago
  I wouldn't be surprised if they have switched to GRPC for improved performance. They mentioned that RPC libraries are centrally maintained in their monorepo; a migration from Thrift to GRPC might have taken less than 6 months.
  - vitaut 2y ago
    gRPC and Thrift are comparable in performance and there is actually an opposite trend of switching from gRPC to Thrift in the few places where the former is still used.
    - linkregister 2y ago
      I participated in a painfully slow migration from Thrift to gRPC. I did not record the performance metrics, but it was internally advertised to be significantly more performant. There are still some Thrift services running at the organization, but most were migrated to gRPC and certainly not migrated back.
  - bagels 2y ago
    I haven't been there a while, but that is extremely unlikely.
linkregister 2y ago
> In a datacenter environment, we prefer centralized controllers over decentralized ones due to their simplicity and ability to make higher-quality decisions. In many cases, a hybrid approach—a centralized control plane combined with a decentralized data plane—provides the best of both worlds. This approach appears to be one of the most optimal designs for software networking (service mesh) and for storage (database operations) for organizations with large server counts. I was surprised to see their IP networking to follow the same model, rather than primarily relying on BGP. It was omitted in this paper, but I would expect for local caching to be used to reduce load on L7 routers and for improved latency for database queries. Clients can invalidate caches and perform another lookup to the service mesh after a reasonable timeout (100-500ms).
ribadeo 2y ago
At least half their gak is due to them NOT moving quickly and NOT wanting to break things. IIRC, graphql is a means of papering over a bunch of legacy APIs. They removed foreign keys from mysql using it as a column store db, a vestige of the original LAMP stack still on PHP. I don't think Meta infrastructural choices are applicable to most folk. What does serverless land your average dev? A high AWS bill. Elastic managed Kubernetes stack? A higher bill. Did you know that you can use YAML and provision actual cloud provider resources with boring tech? Welcome to Ansible. There is no need to recreate Linux network stack when you have the Linux network stack, and it actually works! Quite a lot of hacky gak is required when you run node.js as a production public facing web service. A statically compiled binary won't invent novel code execution paths 4 days into a memory leaking runtime bender. Boring tech is boring, I guess, even if it's new and shiny. Facebook creates tech to mitigate the pathologies their past continuously present.
- ForHackernews 2y ago
  > Facebook creates tech to mitigate the pathologies their past continuously present. Remember when they hacked a running Android Dalvik machine because their organizational constraints were such that they could never remove code or delete unused classes? https://archive.is/nIPlg https://archive.is/nIPlg https://engineering.fb.com/2013/03/04/android/under-the-hood-dalvik-patch-for-facebook-for-android/ https://engineering.fb.com/2013/03/04/android/under-the-hood... Facebook seems like a place where they do amazing engineering to temporarily stave off the disastrous consequences of their previous feat of amazing engineering.
  - kridsdale1 2y ago
    Basically yeah. You should see the FB iOS code base. Yikes. And I have seen Apple code bases that literally go back to the 80s.
- sofixa 2y ago
  > Did you know that you can use YAML and provision actual cloud provider resources with boring tech? Welcome to Ansible Anyone using Ansible for cloud infrastructure management is not to be taken seriously. It's among the worst tools for the job - not (always) idempotent, no state tracking, slow, very limited in the resources it can manage, very lacking templating, fun stuff like "state: absent", running, and then having to remove the corresponding lines to delete, etc etc. You're literally better of bash scripting the cloud provider's CLI than using Ansible. Terraform/OpenTofu, Pulumi/tfcdk if you hate your future self are just clearly so much better.
udev4096 2y ago
As much as I hate clownfare, it would be very interesting if they published something like that because of the sheer number of data centers they operate
- sofixa 2y ago
  They have published a lot of information on their blog[1]. It's piecemeal, and around other articles about optimisations or security fixes or failures/postmortems, but it's there. Stuff like: https://blog.cloudflare.com/how-we-use-hashicorp-nomad/ https://blog.cloudflare.com/how-we-use-hashicorp-nomad/ https://blog.cloudflare.com/cloudflare-deployment-in-guam/ https://blog.cloudflare.com/cloudflare-deployment-in-guam/ https://blog.cloudflare.com/behind-the-scenes-with-stream-live-cloudflares-live-streaming-service/ https://blog.cloudflare.com/behind-the-scenes-with-stream-li... 1 - https://blog.cloudflare.com/ https://blog.cloudflare.com/
lionkor 2y ago
> the image is not cached at CDN109 when the user requests it, CDN109 forwards the request to a nearby PoP. The PoP then forwards the request to the load balancer in a datacenter region, which retrieves the image from the storage system. Say I want a 1MB image, wouldn't it be faster to serve me the 1MB image over a slow connection with 100ms latency, than going through multiple hops of increasing latency, with multiple round trips? Say I request the image directly: me -- 100ms --> datacenter datacenter -- 100ms --> me Say I now go through Meta's system, assuming that goes to the same Datacenter in the end, and there's no FTL tech: me -- 10ms --> CDN CDN -- 10ms --> PoP PoP -- 90ms --> datacenter datacenter -- 90ms --> PoP PoP -- 10ms --> CDN CDN -- 10ms --> me
- karparov 2y ago
  [dead]
- deleted 2y ago
  [deleted]
- dijit 2y ago
  DC-to-DC networking is typically much higher throughput, and often lower latency (due to fewer hops). Consider also that the CDN can fetch the whole image async after the first request, but the nature of TCP (that you'll be using to fetch objects) is serial, meaning you can only fetch about 1k each round-trip. Now, in reality the situation is way more complex, because various hacks have been added over the years, for example your browser might grab the size header and just create a bunch of download threads for your image with various offsets, the TCP scaling window might come into play; but largely most people set 1500b for their MTU on consumer hardware, and there's some overhead from TCP/IP and so on.
- jusomg 2y ago
  Serving an image over HTTPS implies initiating the TCP connection (which requires minimum 3 packets) and the TLS connection (which requires many more, lets say 10). CDN <-> PoP <-> Datacenter communication doesn't require initiating connections. They reuse them because they centralize requests to serve different users (this is even explicitly called out in the article). Lets say your closest datacenter is 50ms away, and the closest PoP/CDN node is 10ms. Just initiating the image download through HTTPS all of sudden is 500ms vs 100ms. Sure, PoP/CDN might need to go to the datacenter to fetch the image (and only if the content is not cached there already) but that only happens once before it gets cached, and there's still a lot of ms to use on that to make the tradeoff worth it.
revskill 2y ago
Interesting that facebook built the whole thing but still useless to me.
karparov 2y ago
[dead]
nk8620 2y ago
> Moreover, for non-AI compute workloads, we offer only a single server type, equipped with one CPU and the same amount of DRAM (previously 64GB, now 256GB). I'm reading something like this for the first time. Is this common across industry or only typical to Meta? In contrast, we use multiple instances for sub-components of our ML training pipeline.
- HolyLampshade 2y ago
  At least in my experience in a somewhat bespoke industry on the non-cloud side, things have been moving in that direction due to most of the reasons they identify (it also ties nicely into the global "Datacenter as a Computer" concept they hit on). People in my industry have a tendency to over-optimize (technical folks love nothing more than to tinker), and in doing so create very unique deployments per group that require a significant amount of infra and operational support, and drastically slow down the rate of progress/change (not to mention the cost). When you peel back all the requirements it turns out we really only need three unique deployment options. Makes it all significantly less cumbersome.
- NBJack 2y ago
  Not to my knowledge, no. If it's a virtualized server, maybe this works, but to my knowledge most companies try to maximize their hardware usage through a variety of slices/sizes based on underlying resources. I am not a datacenter expert however.
- UltraSane 2y ago
  One reason for using a single CPU is that the number of cores per CPU has become very high and only using a single CPU avoids a lot of complexity around Non-Uniform Memory Architecture where the OS should allocate data to RAM that is "closest" to the CPU it is being used by for best performance.
- kridsdale1 2y ago
  This wasn’t true for the server side compiler jobs I was doing. Those were multithreaded af.
ThinkBeat 2y ago
Quickly developed serverless functions combined with continuous deployment, and anyone can make edits in the entire codebase sounds like a dystopian nightmare. The amount of logging that is required to debug and find bugs is extreme. Using Erlang to write serverless functions seems like avoiding all the huge benefits BEAM can offer. >Additionally, product engineers predominantly write code in stateless, serverless functions in PHP, Python, and Erlang for their benefits in simplicity, productivity, and iteration speed. >To boost developer productivity, Meta has adopted continuous deployment universally and enabled more developers to write serverless functions rather than traditional service code.
- HolyLampshade 2y ago
  > The amount of logging that is required to debug and find bugs is extreme. Ten or so years ago I remember going to a larger technical recruiting pitch at Facebook where they discussed their logging complex (I have only a vague recollection of the details). Honestly it was one of the most beefy implementations of system/application logging I remember having seen at the time.
  - kridsdale1 2y ago
    It powers the world’s best experimentation platform too. Which is their business superpower. Competitors can spend a year thinking about what to do. FB has already tried and measured all options and pivots to the optimal. I have been very disappointed since moving from FB to Google at how much worse the data and measurement platform is here.
- arbor_day 2y ago
  I was suspicious before I joined, but it works way better than you'd think!
- kridsdale1 2y ago
  Yes it’s dangerous. That’s why the hiring bar is very high, and everyone spends their first 8 weeks in training bootcamp before being let loose in their real project team.
factfindingisfn 2y ago
Sponsored by Brawndo - its what plants crave Seriously though, this is painful to watch. Reminds me of those articles/videos where people have to have their jobs "invest" or software/hardware partners in that website and all of a sudden this person is a "Person of Interest". Meh
- ImPostingOnHN 2y ago
  > articles/videos where people have to have their jobs "invest" > software/hardware partners in that website > all of a sudden this person is a "Person of Interest" Would you mind rephrasing these points? It's unclear what they mean or are referring to (e.g. what does it mean when a job "invests"?)
robblbobbl 2y ago
> PHP > Planetary > What's the math behind?
- arbor_day 2y ago
  facebook.com started out as a php app. It was easier to make php efficient (enough) than to rewrite the application.
philg_jr 2y ago
Amazing. All of this truly wild, impressive technology and some of the absolute best engineers in the world, just to shove more ads in people's eyeballs. Sigh.
- njovin 2y ago
  Hey now let's not downplay what a wonderful vehicle it is for state-sponsored political propaganda and misinformation.
- UltraSane 2y ago
  That was the old Facebook. The new Facebook is about manipulation of people's politics and opinions.
- umeshunni 2y ago
  The tragedy of our times is that some of our brightest minds are working tirelessly, not to cure diseases or to explore the stars, but to improve the probability that someone will click on an ad by 0.1%.
  - deleted 2y ago
    [deleted]
- kridsdale1 2y ago
  Take it up with capitalism and with DARPA. If we could earn $700k doing cool science shit in white shirts and black ties for Uncle Sam like our grandfathers maybe we would.
- nyclounge 2y ago
  >Amazing. All of this truly wild, impressive technology and some of the absolute best engineers in the world, just to shove more ads in people's eyeballs. Sigh. If only which that were true, it is more like building our own digital prison.
eclark 2y ago
While I was at FB (it wasn't Meta then), I saw what a superpower the infrastructure is there. Product engineers build things of a scale in days. While I was there, I got to be tech lead for several different teams (2x distributed dbs, 1x Dev Efficiency, 1x Ads), some of which are called out by name here. Shout out to the HBase and ZippyDB teams! This is the first public acknowledgment that ZippyDB was converged upon. It's also super cool to see the Developer Efficiency pushes called out. 10,000 Services pushed daily, or every commit is so impressive. When I left FB, I couldn't find anything close. So, I'm building the infra that I was missing as a startup. Batteries Included. https://www.batteriesincl.com/ https://www.batteriesincl.com/ https://github.com/batteries-included/batteries-included/ https://github.com/batteries-included/batteries-included/
- kridsdale1 2y ago
  Good luck to you. Maybe you’ll be the next StatSig.
  - eclark 2y ago
    Thanks! They have built an impressive business and tool.
ViktorRay 2y ago
It's unfortunate there is so much cynicism and negativity in these comments. I understand many people here strongly dislike Meta but the actual article is awe inspiring to me. I didn't know how extensive and complicated the infrastructure underpinning the modern digital world was. Reading this article and seeing the sense of scale is mindblowing to me. It feels almost like a miracle and like magic. A "wonder of the world" in terms of the amount of effort and complexity. Maybe the company in charge is bad in many ways but all of the things in the article are astounding to me. I am not an engineer like many of you so maybe the article is old news to you guys but I couldn't help but say "wow". I feel like if you took some old science fiction writers from back in the day and showed them this article, you would find sheer awe in their faces too.
- teachrdan 2y ago
  > I feel like if you took some old science fiction writers from back in the day and showed them this article, you would find sheer awe in their faces too. I can't tell if this is satire. The science fiction writers from back in the day wrote about humans exploring other planets, building sentient machines, encountering aliens or maybe developing psychic powers. (looking at you two, Philip K. Dick and Theodore Sturgeon) Building the world's biggest ad-serving infrastructure might impress them. It impresses me! But "sheer awe" seems a few orders of magnitude off the mark.
  - teachrdan 2y ago
    I'm getting downvoted here but can't for the life of me guess why. If you've actually read Isaac Asimov and his peers, as I have, they wrote about robots with complex inner lives and space ships that traveled to alien civilizations. There is no reason whatsoever to think they would experience any more "sheer awe" at Meta's infrastructure than any other random person from their time period. I honestly think they would be disappointed that it was all done to sell advertisements to people rather than to help humanity explore the universe -- or anything else of actual value.
    - jappgar 2y ago
aaroninsf 2y ago
So much money, so much effort, all in service, of so much banal evil. Get off Meta. Stay off Meta. Don't work for them. To integrate them. Don't share links to them.
robbale 2y ago
the level of scale with this infra is awesome. i liked how the offineline processing is decoupled from the frontend

302 points by sidcool 2y ago

I find it interesting how they describe the PHP web front end as a "serverless" or "function as a service" architecture. I guess it's a matter of perspective. It's a service that has a monolithic codebase with lots of endpoints deployed to it. I guess from the perspective of the maintainer of one of those endpoints it's "serverless" but that abstraction (like all abstractions) has leaks: the teams responsible for the top endpoints and those working on shared libraries can't treat the infra as a given, but rather need to be acutely aware of its limitations and performance characteristics.

whstl 2y ago
“Serverless” is not synonymous with Lambda, it’s just a computing model. Lots of companies are hosting old monoliths on Amazon Fargate, for example.
- falcor84 2y ago
  As I see it, it's just an abuse of terminology. By referring to Fargate as Serverless which previously referred to FaaS, they (apparently AWS's marketing team) have seemingly removed the distinction between FaaS and PaaS
vineyardmike 2y ago
I think the full description of “stateless, serverless functions” adds a bit of clarity. My read of this is that whatever code is running doesn’t maintain state between requests, and doesn’t touch the underlying operating system. Which seems pretty standard for highly managed environments anyways. It’s been years since I’ve written backend API code that touched the underlying system, or left objects in heap between requests. The flexibility of knowing that any machine can instantly run the code for your API gives a lot of flexibility to rapidly scale up an API. Nothing is “serverless” to everyone. Especially when you run the data center. But being “serverless” and even sitting above the “language runtime” gives API developers a lot of freedom to focus on business logic.
ramblemonkey 2y ago
There’s only one entry point for all requests. It’s php (well, Hack) all the way down from there. So all the routing is readable as php code, request handling etc.
NBJack 2y ago
It's serverless only in that there's a whole different organization at Meta handling the (monolith) hosting on actual hardware. By that definition, you could argue Amazon's detail page is "serverless".
chubot 2y ago
PHP is definitely the original serverless! You upload your code and someone else runs it! CGI is also serverless I wrote a couple articles with these comparisons, and related issues: Comments on Scripting, CGI, and FastCGI https://www.oilshell.org/blog/2024/06/cgi.html https://www.oilshell.org/blog/2024/06/cgi.html Comments on Shared Unix Hosting vs. the Cloud https://oils.pub/blog/2025/02/shared-hosting.html https://oils.pub/blog/2025/02/shared-hosting.html --- I'm not the only one who thinks this: https://www.devever.net/~hl/mildlydynamic https://www.devever.net/~hl/mildlydynamic AWS Lambda: CGI But It's Trendy. Recently we've seen the rise in popularity of AWS Lambda, a “functions as a service” provider. From my perspective this is literally a reinvention of CGI, except a) much more complicated for essentially the same functionality, b) with vendor lock-in, c) with a much more complex and bespoke deployment process which requires the use of special tools. People who don't think this is true probably never used PHP or CGI ... You don't manage the server; you just upload your code! Architecturally, it is the same. From the user experience, it is the same. PHP and CGI also scale infinitely, because they are stateless. Meta scales PHP to a user base of half the world's population. They could also scale CGI/FastCGI.

davedx 2y ago

Very interesting, in particular the explicit comparisons with hyperscalers. I almost wonder if this is preparation for them launching their own public cloud. Anyone from Meta care to comment?

olivermuty 2y ago
It would be pretty impressive to launch a public cloud I would trust even less than GCP :D
- arjvik 2y ago
  GCP got this reputation because it’s a second class citizen within Google. Google’s own internal infra (Borg, Blaze) is top-notch. If Meta can pull off the public cloud correctly, I’d trust them greatly - they’ve shown significant engineering and product competence till now, even if they could use some more consistent and stable UI.
  - mathverse 2y ago
    Dont all new projects go to gcp within google?
    - Cthulhu_ 2y ago
      Yeah at this stage it doesn't make sense to have two tiers within Google.
    - kridsdale1 2y ago
      Depends on what you mean by new. None of the new features for the mature product I work on have touched it. To the side I have had GCP work but it’s been isolated and as if I were moonlighting.
- nijave 2y ago
  A lot of Fortune 50 companies compete in some way with Microsoft, Amazon, and Google so are reluctant to use their services. I think Meta has the benefit of less corporate competition Anyway in that context, you don't need trust when you have a sufficiently large legal department

vasco 2y ago

> Moreover, once it [Threads] was developed, the infrastructure teams were given only two day’s notice to prepare for its production launch. Most large organizations would take longer than two days just to draft a project plan involving dozens of interdependent teams, let alone execute it. At Meta, however, we quickly established war rooms across distributed sites, bringing together both infrastructure and product teams to address issues in real-time. Despite the tight timeline, the app’s launch was highly successful, reaching 100 million users within just five days, making it the fastest-growing app in history. Kind of more impressive to have kept this ability to ship fast than anything else. A lot of work is needed to not let the bureaucracy increase and stop the lawyers or other functions from creating approval gates everywhere. Or at least to be able to have war rooms that can get it done so quickly.

deleted 2y ago
[deleted]
silisili 2y ago
I thought the same thing, but then realize that Threads was a huge flop. Is it really a skill to very quickly release a dud app? I don't know the answer to that. Bypassing bureaucracy seems like heaven, but it feels like it also bypassed the product folks entirely.
- ianlevesque 2y ago
  What? It has more than 300M users.
  - silisili 2y ago
    Today, maybe? I haven't kept up. I'm talking specifically about its launch.
    - OtherShrezzing 2y ago
      They had 100m account creations within the first 5 days.

asdasd1234 2y ago

Is there any public information available for the deploy/observability tool?

linkregister 2y ago
Unfortunately I didn't see any code posted for Conveyor. The following USENIX paper is available: https://www.usenix.org/system/files/osdi23-grubic.pdf https://www.usenix.org/system/files/osdi23-grubic.pdf .
_kb 2y ago
I assume it's changes since then, but this is from 2021: https://atscaleconference.com/conveyor-continuous-deployment-at-facebook/ https://atscaleconference.com/conveyor-continuous-deployment.... There's also a good paper on the approaches to configuration management here: https://research.facebook.com/publications/holistic-configuration-management-at-facebook/ https://research.facebook.com/publications/holistic-configur....

kmdrpc 2y ago

Not a word about Thrift, perhaps it was too low level for an infra overview, but I would have expected it to make some technical impact from a global perspective.

linkregister 2y ago
I wouldn't be surprised if they have switched to GRPC for improved performance. They mentioned that RPC libraries are centrally maintained in their monorepo; a migration from Thrift to GRPC might have taken less than 6 months.
- vitaut 2y ago
  gRPC and Thrift are comparable in performance and there is actually an opposite trend of switching from gRPC to Thrift in the few places where the former is still used.
  - linkregister 2y ago
    I participated in a painfully slow migration from Thrift to gRPC. I did not record the performance metrics, but it was internally advertised to be significantly more performant. There are still some Thrift services running at the organization, but most were migrated to gRPC and certainly not migrated back.
- bagels 2y ago
  I haven't been there a while, but that is extremely unlikely.

linkregister 2y ago

> In a datacenter environment, we prefer centralized controllers over decentralized ones due to their simplicity and ability to make higher-quality decisions. In many cases, a hybrid approach—a centralized control plane combined with a decentralized data plane—provides the best of both worlds. This approach appears to be one of the most optimal designs for software networking (service mesh) and for storage (database operations) for organizations with large server counts. I was surprised to see their IP networking to follow the same model, rather than primarily relying on BGP. It was omitted in this paper, but I would expect for local caching to be used to reduce load on L7 routers and for improved latency for database queries. Clients can invalidate caches and perform another lookup to the service mesh after a reasonable timeout (100-500ms).

ribadeo 2y ago

At least half their gak is due to them NOT moving quickly and NOT wanting to break things. IIRC, graphql is a means of papering over a bunch of legacy APIs. They removed foreign keys from mysql using it as a column store db, a vestige of the original LAMP stack still on PHP. I don't think Meta infrastructural choices are applicable to most folk. What does serverless land your average dev? A high AWS bill. Elastic managed Kubernetes stack? A higher bill. Did you know that you can use YAML and provision actual cloud provider resources with boring tech? Welcome to Ansible. There is no need to recreate Linux network stack when you have the Linux network stack, and it actually works! Quite a lot of hacky gak is required when you run node.js as a production public facing web service. A statically compiled binary won't invent novel code execution paths 4 days into a memory leaking runtime bender. Boring tech is boring, I guess, even if it's new and shiny. Facebook creates tech to mitigate the pathologies their past continuously present.

ForHackernews 2y ago
> Facebook creates tech to mitigate the pathologies their past continuously present. Remember when they hacked a running Android Dalvik machine because their organizational constraints were such that they could never remove code or delete unused classes? https://archive.is/nIPlg https://archive.is/nIPlg https://engineering.fb.com/2013/03/04/android/under-the-hood-dalvik-patch-for-facebook-for-android/ https://engineering.fb.com/2013/03/04/android/under-the-hood... Facebook seems like a place where they do amazing engineering to temporarily stave off the disastrous consequences of their previous feat of amazing engineering.
- kridsdale1 2y ago
  Basically yeah. You should see the FB iOS code base. Yikes. And I have seen Apple code bases that literally go back to the 80s.
sofixa 2y ago
> Did you know that you can use YAML and provision actual cloud provider resources with boring tech? Welcome to Ansible Anyone using Ansible for cloud infrastructure management is not to be taken seriously. It's among the worst tools for the job - not (always) idempotent, no state tracking, slow, very limited in the resources it can manage, very lacking templating, fun stuff like "state: absent", running, and then having to remove the corresponding lines to delete, etc etc. You're literally better of bash scripting the cloud provider's CLI than using Ansible. Terraform/OpenTofu, Pulumi/tfcdk if you hate your future self are just clearly so much better.

udev4096 2y ago

As much as I hate clownfare, it would be very interesting if they published something like that because of the sheer number of data centers they operate

sofixa 2y ago
They have published a lot of information on their blog[1]. It's piecemeal, and around other articles about optimisations or security fixes or failures/postmortems, but it's there. Stuff like: https://blog.cloudflare.com/how-we-use-hashicorp-nomad/ https://blog.cloudflare.com/how-we-use-hashicorp-nomad/ https://blog.cloudflare.com/cloudflare-deployment-in-guam/ https://blog.cloudflare.com/cloudflare-deployment-in-guam/ https://blog.cloudflare.com/behind-the-scenes-with-stream-live-cloudflares-live-streaming-service/ https://blog.cloudflare.com/behind-the-scenes-with-stream-li... 1 - https://blog.cloudflare.com/ https://blog.cloudflare.com/

lionkor 2y ago

> the image is not cached at CDN109 when the user requests it, CDN109 forwards the request to a nearby PoP. The PoP then forwards the request to the load balancer in a datacenter region, which retrieves the image from the storage system. Say I want a 1MB image, wouldn't it be faster to serve me the 1MB image over a slow connection with 100ms latency, than going through multiple hops of increasing latency, with multiple round trips? Say I request the image directly: me -- 100ms --> datacenter datacenter -- 100ms --> me Say I now go through Meta's system, assuming that goes to the same Datacenter in the end, and there's no FTL tech: me -- 10ms --> CDN CDN -- 10ms --> PoP PoP -- 90ms --> datacenter datacenter -- 90ms --> PoP PoP -- 10ms --> CDN CDN -- 10ms --> me

karparov 2y ago
[dead]
deleted 2y ago
[deleted]
dijit 2y ago
DC-to-DC networking is typically much higher throughput, and often lower latency (due to fewer hops). Consider also that the CDN can fetch the whole image async after the first request, but the nature of TCP (that you'll be using to fetch objects) is serial, meaning you can only fetch about 1k each round-trip. Now, in reality the situation is way more complex, because various hacks have been added over the years, for example your browser might grab the size header and just create a bunch of download threads for your image with various offsets, the TCP scaling window might come into play; but largely most people set 1500b for their MTU on consumer hardware, and there's some overhead from TCP/IP and so on.
jusomg 2y ago
Serving an image over HTTPS implies initiating the TCP connection (which requires minimum 3 packets) and the TLS connection (which requires many more, lets say 10). CDN <-> PoP <-> Datacenter communication doesn't require initiating connections. They reuse them because they centralize requests to serve different users (this is even explicitly called out in the article). Lets say your closest datacenter is 50ms away, and the closest PoP/CDN node is 10ms. Just initiating the image download through HTTPS all of sudden is 500ms vs 100ms. Sure, PoP/CDN might need to go to the datacenter to fetch the image (and only if the content is not cached there already) but that only happens once before it gets cached, and there's still a lot of ms to use on that to make the tradeoff worth it.

revskill 2y ago

Interesting that facebook built the whole thing but still useless to me.

karparov 2y ago

[dead]

nk8620 2y ago

> Moreover, for non-AI compute workloads, we offer only a single server type, equipped with one CPU and the same amount of DRAM (previously 64GB, now 256GB). I'm reading something like this for the first time. Is this common across industry or only typical to Meta? In contrast, we use multiple instances for sub-components of our ML training pipeline.

HolyLampshade 2y ago
At least in my experience in a somewhat bespoke industry on the non-cloud side, things have been moving in that direction due to most of the reasons they identify (it also ties nicely into the global "Datacenter as a Computer" concept they hit on). People in my industry have a tendency to over-optimize (technical folks love nothing more than to tinker), and in doing so create very unique deployments per group that require a significant amount of infra and operational support, and drastically slow down the rate of progress/change (not to mention the cost). When you peel back all the requirements it turns out we really only need three unique deployment options. Makes it all significantly less cumbersome.
NBJack 2y ago
Not to my knowledge, no. If it's a virtualized server, maybe this works, but to my knowledge most companies try to maximize their hardware usage through a variety of slices/sizes based on underlying resources. I am not a datacenter expert however.
UltraSane 2y ago
One reason for using a single CPU is that the number of cores per CPU has become very high and only using a single CPU avoids a lot of complexity around Non-Uniform Memory Architecture where the OS should allocate data to RAM that is "closest" to the CPU it is being used by for best performance.
kridsdale1 2y ago
This wasn’t true for the server side compiler jobs I was doing. Those were multithreaded af.

ThinkBeat 2y ago

Quickly developed serverless functions combined with continuous deployment, and anyone can make edits in the entire codebase sounds like a dystopian nightmare. The amount of logging that is required to debug and find bugs is extreme. Using Erlang to write serverless functions seems like avoiding all the huge benefits BEAM can offer. >Additionally, product engineers predominantly write code in stateless, serverless functions in PHP, Python, and Erlang for their benefits in simplicity, productivity, and iteration speed. >To boost developer productivity, Meta has adopted continuous deployment universally and enabled more developers to write serverless functions rather than traditional service code.

HolyLampshade 2y ago
> The amount of logging that is required to debug and find bugs is extreme. Ten or so years ago I remember going to a larger technical recruiting pitch at Facebook where they discussed their logging complex (I have only a vague recollection of the details). Honestly it was one of the most beefy implementations of system/application logging I remember having seen at the time.
- kridsdale1 2y ago
  It powers the world’s best experimentation platform too. Which is their business superpower. Competitors can spend a year thinking about what to do. FB has already tried and measured all options and pivots to the optimal. I have been very disappointed since moving from FB to Google at how much worse the data and measurement platform is here.
arbor_day 2y ago
I was suspicious before I joined, but it works way better than you'd think!
kridsdale1 2y ago
Yes it’s dangerous. That’s why the hiring bar is very high, and everyone spends their first 8 weeks in training bootcamp before being let loose in their real project team.

factfindingisfn 2y ago

Sponsored by Brawndo - its what plants crave Seriously though, this is painful to watch. Reminds me of those articles/videos where people have to have their jobs "invest" or software/hardware partners in that website and all of a sudden this person is a "Person of Interest". Meh

ImPostingOnHN 2y ago
> articles/videos where people have to have their jobs "invest" > software/hardware partners in that website > all of a sudden this person is a "Person of Interest" Would you mind rephrasing these points? It's unclear what they mean or are referring to (e.g. what does it mean when a job "invests"?)

robblbobbl 2y ago

> PHP > Planetary > What's the math behind?

arbor_day 2y ago
facebook.com started out as a php app. It was easier to make php efficient (enough) than to rewrite the application.

philg_jr 2y ago

Amazing. All of this truly wild, impressive technology and some of the absolute best engineers in the world, just to shove more ads in people's eyeballs. Sigh.

njovin 2y ago
Hey now let's not downplay what a wonderful vehicle it is for state-sponsored political propaganda and misinformation.
UltraSane 2y ago
That was the old Facebook. The new Facebook is about manipulation of people's politics and opinions.
umeshunni 2y ago
The tragedy of our times is that some of our brightest minds are working tirelessly, not to cure diseases or to explore the stars, but to improve the probability that someone will click on an ad by 0.1%.
- deleted 2y ago
  [deleted]
kridsdale1 2y ago
Take it up with capitalism and with DARPA. If we could earn $700k doing cool science shit in white shirts and black ties for Uncle Sam like our grandfathers maybe we would.
nyclounge 2y ago
>Amazing. All of this truly wild, impressive technology and some of the absolute best engineers in the world, just to shove more ads in people's eyeballs. Sigh. If only which that were true, it is more like building our own digital prison.

eclark 2y ago

While I was at FB (it wasn't Meta then), I saw what a superpower the infrastructure is there. Product engineers build things of a scale in days. While I was there, I got to be tech lead for several different teams (2x distributed dbs, 1x Dev Efficiency, 1x Ads), some of which are called out by name here. Shout out to the HBase and ZippyDB teams! This is the first public acknowledgment that ZippyDB was converged upon. It's also super cool to see the Developer Efficiency pushes called out. 10,000 Services pushed daily, or every commit is so impressive. When I left FB, I couldn't find anything close. So, I'm building the infra that I was missing as a startup. Batteries Included. https://www.batteriesincl.com/ https://www.batteriesincl.com/ https://github.com/batteries-included/batteries-included/ https://github.com/batteries-included/batteries-included/

kridsdale1 2y ago
Good luck to you. Maybe you’ll be the next StatSig.
- eclark 2y ago
  Thanks! They have built an impressive business and tool.

ViktorRay 2y ago

It's unfortunate there is so much cynicism and negativity in these comments. I understand many people here strongly dislike Meta but the actual article is awe inspiring to me. I didn't know how extensive and complicated the infrastructure underpinning the modern digital world was. Reading this article and seeing the sense of scale is mindblowing to me. It feels almost like a miracle and like magic. A "wonder of the world" in terms of the amount of effort and complexity. Maybe the company in charge is bad in many ways but all of the things in the article are astounding to me. I am not an engineer like many of you so maybe the article is old news to you guys but I couldn't help but say "wow". I feel like if you took some old science fiction writers from back in the day and showed them this article, you would find sheer awe in their faces too.

teachrdan 2y ago
> I feel like if you took some old science fiction writers from back in the day and showed them this article, you would find sheer awe in their faces too. I can't tell if this is satire. The science fiction writers from back in the day wrote about humans exploring other planets, building sentient machines, encountering aliens or maybe developing psychic powers. (looking at you two, Philip K. Dick and Theodore Sturgeon) Building the world's biggest ad-serving infrastructure might impress them. It impresses me! But "sheer awe" seems a few orders of magnitude off the mark.
- teachrdan 2y ago
  I'm getting downvoted here but can't for the life of me guess why. If you've actually read Isaac Asimov and his peers, as I have, they wrote about robots with complex inner lives and space ships that traveled to alien civilizations. There is no reason whatsoever to think they would experience any more "sheer awe" at Meta's infrastructure than any other random person from their time period. I honestly think they would be disappointed that it was all done to sell advertisements to people rather than to help humanity explore the universe -- or anything else of actual value.
  - jappgar 2y ago

aaroninsf 2y ago

So much money, so much effort, all in service, of so much banal evil. Get off Meta. Stay off Meta. Don't work for them. To integrate them. Don't share links to them.

robbale 2y ago

the level of scale with this infra is awesome. i liked how the offineline processing is decoupled from the frontend