5 ms·

Meta's serverless platform processing trillions of function calls a day (2023)

50 points by jbredeche 3y ago

shortrounddev2 3y ago
I never understood why these platforms, which clearly run on a server, were marketed as "serverless". They seem more akin to a distributed operating system, to me
- deleted 3y ago
  [deleted]
- duped 3y ago
  Firstly because you don't provision servers to run the code, the runtime will scale the infrastructure horizontally with response to traffic. Secondly because it's not a client/server architecture from the perspective of the code being run. Basically it's "serverless" because you the developer are not doing anything with servers. You're not writing one, maintaining one, or managing the infrastructure to run one. You drop off a blob of logic to the cloud provider and they handle everything else for you, and you pay on demand. Functions-as-a-Service (FaaS) is another name for the same idea but less catchy
- gary17the 3y ago
  I think the nomenclature comes from the fact that "traditionally", before you can upload a piece of code somewhere and have it executed, you have to first setup a server, as in: an instance of a particular operating system (a physical server or a VPS), hence the name "server-ful". In the "server-less" paradigm, you upload directly to an abstract cloud and have cloud infrastructure execute your code. Moreover, uploading to "a server" implies using a fixed, pre-allocated set of CPU/memory/disk resources that cannot grow dynamically, while "server-less" implies using dynamic infrastructure that can scale CPU/memory/disk resources up and down as needed and/or clone executed code across multiple internal compute instances controlled by a load balancer and so on - the notion of "a server" becomes no longer relevant.
- KaiserPro 3y ago
  because the only other viable analogies were: CGI-Bin Mainframes which were deeply out of fashion at the time.
- weberer 3y ago
  Well of course they have to run on something. But I guess since you don't have to configure the server yourself, the marketing guys figured "out of sight, out of mind".
- sneed_chucker 3y ago
  Ultimately it's a marketing term, but it's a good one. Of course ultimately the code runs on a server, but it's not a server that the customer has to buy, rent, maintain, or otherwise think about.
  - bob1029 3y ago
    > or otherwise think about. For us this is by far the biggest factor. Innovation tokens are the most expensive thing there is.
- derekdb 3y ago
  As a customer of serverless, you do not have to manage any servers, that is provided by the platform. Even with a platform like EKS, the customer is still responsible for selecting instance type, OS/AMI and any OS updates. With serverless, the customer doesn't worry about managing their servers, they just enqueue tasks (function invocations) and let the provider worry about right-sizing instances types, OS updates, etc.
frfl 3y ago
Sorry this is going to be slightly tangential to the primary topic of this post. But does anyone work on backends that heavily utilize a serverless platform? To be more specific, backend (micro)services that embrace serverless fully (for example AWS Lambda, Dynamodb, StepFunctions, etc) and minimal usage of any ec2/container based deployments in the architecture? And to be clear, I'm not talking about some small bot or single internal app running on a serverless platform, rather an entire complex core business product or service that embraces serverless architecture -- think Amazon Prime previously using AWS Lambda type stuff. I'd love to know people's (objective if possible) thoughts, experiences and opinions on such an approach.
- foofie 3y ago
  > But does anyone work on backends that heavily utilize a serverless platform? As far as I know, that scenario is not in the decision tree for when to go with function-as-a-service offerings. FaaS is for things like glue code for event handlers, low-traffic services like API Gateway/NLB calling Lambdas, and expensive tasks that can be offloaded from a worker thread to a Lambda. When traffic going into a FaaS-based service starts to go up, the cost of running Lambdas stops being competitive and you're better off just having a dedicated service handle the work. Nevertheless, the main selling point of FaaS is not performance, throughput, or even developer convenience. The main selling point is hardware utilization. If your dev teams deploy FaaS stuff instead of dedicated services, the company doesn't need to provision anything to get them to run. The functions are just deployed in the sea of FaaS things they have been running, and they can live with high hardware utilization just fine.
  - frfl 3y ago
    I ask because it's relevant to me, but not in my control. And still early stages and haven't hit that "When traffic going into a FaaS-basdd service starts to go up, the cost of running Lambdas stops being competitive" stage yet -- if what you're describing will be applicable and true in my situation. Sorry I can't go into specifics as it wouldn't be wise for me to discuss work stuff in more detail. So that's good to know. Do you have any references, in terms of further reading material, for the cross over point for when it's stops being competitive? I highly doubt $cloud_provider is going to give you such a analysis for obvious financial reasons.
throwaway3672 3y ago
What are functions in this context? How, for example, are they different from other kinds of computation, like database aggregation quieries?
- frfl 3y ago
  This maybe useful reading https://en.wikipedia.org/wiki/Function_as_a_service https://en.wikipedia.org/wiki/Function_as_a_service
- weberer 3y ago
  They are arbitrary Python functions (or whatever other language you want to use). Serverless platforms let developers deploy simple functions without having to provision a VPS and all the stuff that comes with it (operating system, networking, etc). You get charged by the millisecond as the function runs.
soamv 3y ago
Here's a link to the actual paper: https://tangchq74.github.io/XFaaS-SOSP23-Final.pdf https://tangchq74.github.io/XFaaS-SOSP23-Final.pdf
mbb70 3y ago
Seems like the biggest edge they have over aws/gcp/azure is the ability to delay function calls to smooth demand. Not much other providers can do to match that.
KaiserPro 3y ago
The big thing that makes it more efficient is that the only real guarantee is that your function will execute _sometime_ in the future. Now, those members of our fair technological coven, of a certain age (who are sadly dwindling in number) will certainly recognise sending off a program to be run by a big magic massive machine, and waiting for results later on. Thats batch computing. Now the key advantage of batch computing here, as also laid out by the paper is that you can group stuff together to cut out VM warmup time. BUT it also means that because you have the luxury of waiting, you can use spare CPU cycles when they are free. As with everything big sites have burst of traffic, so to handle that, you have lots of spare CPU cycles waiting around. if you clearly telegraph that your function is liable to be delayed, killed or re-run, then there are still lots of useful work that can be done without costing you more in capacity. Its a strongly under utilised pattern in modern computing.
- teacpde 3y ago
  Surely there is a lot of workload is delay-able, but there is equal, if not more, amount of workload that do not have the luxury of waiting. Is it fair to say the serverless platform has no advantage for the latter workload?
  - KaiserPro 3y ago
    Yeah, you can't do synchronous stuff on it, like you would say lambda, or what ever google calls theirs. I think calling it serverless is probably confusing to a lot of the outside world, as to them serverless is cgi-bin for the docker generation. this form of serverless is very much batch jobs with some modern bits sprinkled in. perhaps they should have called it serverless batch, or Batch-as-a-service, async-serverless or serverless-futures.
ChuckMcM 3y ago
Interesting article, I wondered about this: It fetches function calls from DurableQs, consolidates them into FuncBuffers (ordered by importance, then execution deadline), Since that is a classic source of priority inversions (which is that 'important' messages overwhelm the system resulting in less important messages missing their execution deadline) If there is someone reading this that knows this system can you tell me if they queue ranking is actually rank = f(priority, deadline) ? Where things close to their deadline can out rank high priority things? The other question that I had but didn't get from skimming was how they do data management. Do the functions carry around all their data? Is any of that data part of a state machine? (which is to say mutating that data by the function results in a new 'state' which would affect other functions using that data)
jauntywundrkind 3y ago
It might be cool to see a cloud platform that incentivized off-peak usage. There's presently no incentive structure for most folks to defer work to off peak on any major hosted platform I know. I'm so excited for the webassembly future, where a huge number of shared modules can be left in memory, sandboxed, and when my function calls such a module, it's already resident & loaded.
vkaku 3y ago
Some parts of the article didn't exactly talk about their setup, idle and teardown costs. Plus how exactly their I/O is optimized. I am skeptical until I see those numbers. Edit: Read the OG version here. https://soteris.github.io/publication/sahraei-2023-xfaas/sahraei-2023-xfaas.pdf https://soteris.github.io/publication/sahraei-2023-xfaas/sah...
brcmthrowaway 3y ago
Meta has definitely evolved from a LAMP stack!

50 points by jbredeche 3y ago

I never understood why these platforms, which clearly run on a server, were marketed as "serverless". They seem more akin to a distributed operating system, to me

deleted 3y ago
[deleted]
duped 3y ago
Firstly because you don't provision servers to run the code, the runtime will scale the infrastructure horizontally with response to traffic. Secondly because it's not a client/server architecture from the perspective of the code being run. Basically it's "serverless" because you the developer are not doing anything with servers. You're not writing one, maintaining one, or managing the infrastructure to run one. You drop off a blob of logic to the cloud provider and they handle everything else for you, and you pay on demand. Functions-as-a-Service (FaaS) is another name for the same idea but less catchy
gary17the 3y ago
I think the nomenclature comes from the fact that "traditionally", before you can upload a piece of code somewhere and have it executed, you have to first setup a server, as in: an instance of a particular operating system (a physical server or a VPS), hence the name "server-ful". In the "server-less" paradigm, you upload directly to an abstract cloud and have cloud infrastructure execute your code. Moreover, uploading to "a server" implies using a fixed, pre-allocated set of CPU/memory/disk resources that cannot grow dynamically, while "server-less" implies using dynamic infrastructure that can scale CPU/memory/disk resources up and down as needed and/or clone executed code across multiple internal compute instances controlled by a load balancer and so on - the notion of "a server" becomes no longer relevant.
KaiserPro 3y ago
because the only other viable analogies were: CGI-Bin Mainframes which were deeply out of fashion at the time.
weberer 3y ago
Well of course they have to run on something. But I guess since you don't have to configure the server yourself, the marketing guys figured "out of sight, out of mind".
sneed_chucker 3y ago
Ultimately it's a marketing term, but it's a good one. Of course ultimately the code runs on a server, but it's not a server that the customer has to buy, rent, maintain, or otherwise think about.
- bob1029 3y ago
  > or otherwise think about. For us this is by far the biggest factor. Innovation tokens are the most expensive thing there is.
derekdb 3y ago
As a customer of serverless, you do not have to manage any servers, that is provided by the platform. Even with a platform like EKS, the customer is still responsible for selecting instance type, OS/AMI and any OS updates. With serverless, the customer doesn't worry about managing their servers, they just enqueue tasks (function invocations) and let the provider worry about right-sizing instances types, OS updates, etc.

frfl 3y ago

Sorry this is going to be slightly tangential to the primary topic of this post. But does anyone work on backends that heavily utilize a serverless platform? To be more specific, backend (micro)services that embrace serverless fully (for example AWS Lambda, Dynamodb, StepFunctions, etc) and minimal usage of any ec2/container based deployments in the architecture? And to be clear, I'm not talking about some small bot or single internal app running on a serverless platform, rather an entire complex core business product or service that embraces serverless architecture -- think Amazon Prime previously using AWS Lambda type stuff. I'd love to know people's (objective if possible) thoughts, experiences and opinions on such an approach.

foofie 3y ago
> But does anyone work on backends that heavily utilize a serverless platform? As far as I know, that scenario is not in the decision tree for when to go with function-as-a-service offerings. FaaS is for things like glue code for event handlers, low-traffic services like API Gateway/NLB calling Lambdas, and expensive tasks that can be offloaded from a worker thread to a Lambda. When traffic going into a FaaS-based service starts to go up, the cost of running Lambdas stops being competitive and you're better off just having a dedicated service handle the work. Nevertheless, the main selling point of FaaS is not performance, throughput, or even developer convenience. The main selling point is hardware utilization. If your dev teams deploy FaaS stuff instead of dedicated services, the company doesn't need to provision anything to get them to run. The functions are just deployed in the sea of FaaS things they have been running, and they can live with high hardware utilization just fine.
- frfl 3y ago
  I ask because it's relevant to me, but not in my control. And still early stages and haven't hit that "When traffic going into a FaaS-basdd service starts to go up, the cost of running Lambdas stops being competitive" stage yet -- if what you're describing will be applicable and true in my situation. Sorry I can't go into specifics as it wouldn't be wise for me to discuss work stuff in more detail. So that's good to know. Do you have any references, in terms of further reading material, for the cross over point for when it's stops being competitive? I highly doubt $cloud_provider is going to give you such a analysis for obvious financial reasons.

throwaway3672 3y ago

What are functions in this context? How, for example, are they different from other kinds of computation, like database aggregation quieries?

frfl 3y ago
This maybe useful reading https://en.wikipedia.org/wiki/Function_as_a_service https://en.wikipedia.org/wiki/Function_as_a_service
weberer 3y ago
They are arbitrary Python functions (or whatever other language you want to use). Serverless platforms let developers deploy simple functions without having to provision a VPS and all the stuff that comes with it (operating system, networking, etc). You get charged by the millisecond as the function runs.

soamv 3y ago

Here's a link to the actual paper: https://tangchq74.github.io/XFaaS-SOSP23-Final.pdf https://tangchq74.github.io/XFaaS-SOSP23-Final.pdf

mbb70 3y ago

Seems like the biggest edge they have over aws/gcp/azure is the ability to delay function calls to smooth demand. Not much other providers can do to match that.

KaiserPro 3y ago

The big thing that makes it more efficient is that the only real guarantee is that your function will execute _sometime_ in the future. Now, those members of our fair technological coven, of a certain age (who are sadly dwindling in number) will certainly recognise sending off a program to be run by a big magic massive machine, and waiting for results later on. Thats batch computing. Now the key advantage of batch computing here, as also laid out by the paper is that you can group stuff together to cut out VM warmup time. BUT it also means that because you have the luxury of waiting, you can use spare CPU cycles when they are free. As with everything big sites have burst of traffic, so to handle that, you have lots of spare CPU cycles waiting around. if you clearly telegraph that your function is liable to be delayed, killed or re-run, then there are still lots of useful work that can be done without costing you more in capacity. Its a strongly under utilised pattern in modern computing.

teacpde 3y ago
Surely there is a lot of workload is delay-able, but there is equal, if not more, amount of workload that do not have the luxury of waiting. Is it fair to say the serverless platform has no advantage for the latter workload?
- KaiserPro 3y ago
  Yeah, you can't do synchronous stuff on it, like you would say lambda, or what ever google calls theirs. I think calling it serverless is probably confusing to a lot of the outside world, as to them serverless is cgi-bin for the docker generation. this form of serverless is very much batch jobs with some modern bits sprinkled in. perhaps they should have called it serverless batch, or Batch-as-a-service, async-serverless or serverless-futures.

ChuckMcM 3y ago

Interesting article, I wondered about this: It fetches function calls from DurableQs, consolidates them into FuncBuffers (ordered by importance, then execution deadline), Since that is a classic source of priority inversions (which is that 'important' messages overwhelm the system resulting in less important messages missing their execution deadline) If there is someone reading this that knows this system can you tell me if they queue ranking is actually rank = f(priority, deadline) ? Where things close to their deadline can out rank high priority things? The other question that I had but didn't get from skimming was how they do data management. Do the functions carry around all their data? Is any of that data part of a state machine? (which is to say mutating that data by the function results in a new 'state' which would affect other functions using that data)

jauntywundrkind 3y ago

It might be cool to see a cloud platform that incentivized off-peak usage. There's presently no incentive structure for most folks to defer work to off peak on any major hosted platform I know. I'm so excited for the webassembly future, where a huge number of shared modules can be left in memory, sandboxed, and when my function calls such a module, it's already resident & loaded.

vkaku 3y ago

Some parts of the article didn't exactly talk about their setup, idle and teardown costs. Plus how exactly their I/O is optimized. I am skeptical until I see those numbers. Edit: Read the OG version here. https://soteris.github.io/publication/sahraei-2023-xfaas/sahraei-2023-xfaas.pdf https://soteris.github.io/publication/sahraei-2023-xfaas/sah...

brcmthrowaway 3y ago

Meta has definitely evolved from a LAMP stack!