7 ms·

Dude where's my coldstart?

45 points by alexellisuk 5y ago

kayodelycaon 5y ago
Maybe I’m just turning into one of those old programmers who doesn’t keep with technology, but the incredible inefficiency described and the workarounds for it are just mind-boggling to me. Then again, maybe it is cost-effective to burn seconds of CPU time and tens to hundreds of megabytes per request handled. I’m sure some C programmer from the CGI days would be floored by how much resources multithreaded ruby and python app servers use.
- edgyquant 5y ago
  At this time it is more efficient to use the rich ecosystems provided by modern languages and just scale up than it is to microoptimize against any perceived inefficiencies of using containers and these ecosystems.
- GiorgioG 5y ago
  Agreed - the cloud providers are laughing all the way to the bank.
- erikerikson 5y ago
  On the other hand, I've seen organizations run an average of 4% utilization for their VMs.
- jeroenhd 5y ago
  Despite the absolutely terrible inefficiencies, the scaling benefit these """serverless""" technologies provide is hard to replicate with traditional software. Throwing a Javascript file into some *aaS platform and having that run on a datacenter near the user, automatically upload the output to storage near the user, and scale up dynamically if there's a sudden influx of thousands of users is very hard to accomplish manually without reserving tons of VMs or using some kind of data center API with auto provisioning. Most businesses have no need for such a system, of course, and the costs per function execution are astronomical in comparison to a native program, but for some use cases, the benefits outweigh the costs. As for Ruby and Python, the cost is measured in dev time rather than server costs. Servers are cheap compared to devs, and if a dev can pump out a feature three times as fast because they can make use of a simpler framework, you can afford the overhead of slower programming languages. For this reason I'm watching the Rust space with interest, because the speed is much closer to C++ than to the traditional Java/C# servers most big applications are written in, but the frameworks are slowly shaping up to be similarly complete. Of course, there's a huge difference between Rust and most other languages that cannot be overlooked, but services like Vaultwarden show an interesting opportunity in the server space. It'll still be a while before the tools and frameworks reach a similar level of usability, though.
  - reilly3000 5y ago
    > having that run on a datacenter near the user This is the biggest deal IMHO. FaaS in CDN edge DCs are remarkably fast and affordable. If you can respond to an API call faster than a request can get through to a load balancer, you're providing quite a UX improvement regardless of cost. For most people running stateful apps this requires some large tradeoffs in the form of the data layer, but they are getting more reasonable over time. In the case of CloudFlare which may have the strongest edge function hosting now, it used to only support its Worker KV data store (fast but KV only and globally eventually consistent ~10 secs). More recently they have added support for fast connections to their R2 (S3-like) service, and now they have a new service that can proxy to MySQL and Postgres databases. https://blog.cloudflare.com/relational-database-connectors/ https://blog.cloudflare.com/relational-database-connectors/ Its an exciting space, because its not just about CDNs: as 5G towers are rolling out, they are building in edge compute clusters along with them. So now you have a 5G iPhone with faster single-threaded performance than your desktop able to get 1Gbps down and faster response times because of its path to edge compute. ISPs are also getting in the game: https://www.coxedge.com/serverless https://www.coxedge.com/serverless https://www.limelight.com/resources/data-sheet/edge-compute/ https://www.limelight.com/resources/data-sheet/edge-compute/
    - invalidname 5y ago
      I think that's sort of against the pareto principle. Assuming you have the physical data near the user. Great. But most of these functions then go on to fetch stuff from elsewhere with its own delays and problems. Even if S3 or cloudflares equivalent is replicated everywhere there would still be a networking delay and context switch waiting for a response. If performance is THAT important you would be better served with a HUGE machine with a huge cache of the database. That's not something you can do with a FAAS. Other than saying "hello world" fast. I can't think of a proper use case where the performance of FAAS would be realistically better than a dumb monolith. The performance and complexity overheads of FAAS make the case even harder to make.
      - reilly3000 5y ago
- vasco 5y ago
  When you have a business where salaries are a much larger percentage than hosting on your overall costs, you're going to add to the second to squeeze more out of the first. When you have done the optimisation above, and your bosses keep asking for more features coming out per quarter, and you need to hire more, you're going to opt for the new things everyone is learning, so that you can hire enough without making your life all about hiring. These are trade-offs like any other, and you probably just haven't been exposed to similar constraints or all of this would make at least some sense. There may be better ways, of course.
fake-name 5y ago
In what potential world would it ever make sense to start an entire container for each request? The entire architecture that leads to such a concept is so dumb I have trouble believing anyone ever thought it could be a decent idea.
- tiffanyh 5y ago
  Thats how Erlang works essentially.
  - foepys 5y ago
    Yes but Erlang's BEAM as a purposely built architecture that makes this very efficient. Spawning OS processes and threads is comparatively expensive.
  - nesarkvechnep 5y ago
    Erlang doesn’t spawn containers for every request.
- lmeyerov 5y ago
  Extreme take: Numbers-wise, that should probably be the default ;-) In particular, the world runs on power laws, and I'm betting most server apps are boring occasional internal stuff at the not-even-1-qps level. Long tails add up to be the typical project. So yeah, if all the worlds internal django tools can do scale-to-zero with fast cold start on shared boxes with 100X more tenancy than today, that's saving a lot of electricity & hw. Sort of a virtualization equivalent of "640K ought to be enough for anybody".
- tyingq 5y ago
  >In what potential world would it ever make sense to start an entire container for each request? If you wait long enough between requests (~5 minutes), that is what AWS Lambda does, only with a VM instead of a container.
FounderBurr 5y ago
This is a joke right?
nithril 5y ago
The benefit from an app running in a container and already listening to an HTTP port with on top an horizontal autoscaler looks small. I might miss something in the approach but as watchdog requires an "HTTP-level multiplexing for requests" (so the app should listen for http), the app is already running in a container with watchdog: everything is in place to not need watchdog. From the FAQ: > Why do I need watchdog anyway? > Whilst we don’t recommend it, if you’re using a HTTP microservice that conforms to the Workloads spec, then technically you may be able to get away without using it.
option_greek 5y ago
So this serverless solution has a watchdog process always running and mapping requests to the correct process? It that more like a server than serverless? Will the next optimisation be launching one thread per request? In which case we are back to square one (:
- pdimitar 5y ago
  You're quite right but as a customer I frankly don't care about how do they it as long as management and deployment are easier from my side. At one point you do hit a limit on what you can do by cold-starting (almost) everything anyway, right? Let them optimize so they are competitive, and now they've given me a good reason to evaluate them.
- tyingq 5y ago
  >Will the next optimisation be launching one thread per request? That is what Cloudflare Workers do. One v8 isolate thread per request.
alexellisuk 5y ago
Disclosure: I'm the maintainer of the open source project - openfaas which this blog post talks about. It also covers other FaaS frameworks and why we see a "cold start" even when using Kubernetes as our platform.
- forty 5y ago
  Does't the proposed solution both solve the issue of FaaS and also remove the main benefit? (ie having stateless function with clean state when it starts running) Also it feels to me like FaaS is good when someone else is running the infrastructure (so you really don't have to know about the underlying infrastructure). Why are people self hosting this? Is it for local test platforms before deploying to a managed FaaS?
  - OJFord 5y ago
    One reason: you have an infrastructure team providing the FaaS solution, and multiple other teams that deploy to it; to whom it is 'a managed FaaS'.

45 points by alexellisuk 5y ago

Maybe I’m just turning into one of those old programmers who doesn’t keep with technology, but the incredible inefficiency described and the workarounds for it are just mind-boggling to me. Then again, maybe it is cost-effective to burn seconds of CPU time and tens to hundreds of megabytes per request handled. I’m sure some C programmer from the CGI days would be floored by how much resources multithreaded ruby and python app servers use.

edgyquant 5y ago
At this time it is more efficient to use the rich ecosystems provided by modern languages and just scale up than it is to microoptimize against any perceived inefficiencies of using containers and these ecosystems.
GiorgioG 5y ago
Agreed - the cloud providers are laughing all the way to the bank.
erikerikson 5y ago
On the other hand, I've seen organizations run an average of 4% utilization for their VMs.
jeroenhd 5y ago
Despite the absolutely terrible inefficiencies, the scaling benefit these """serverless""" technologies provide is hard to replicate with traditional software. Throwing a Javascript file into some *aaS platform and having that run on a datacenter near the user, automatically upload the output to storage near the user, and scale up dynamically if there's a sudden influx of thousands of users is very hard to accomplish manually without reserving tons of VMs or using some kind of data center API with auto provisioning. Most businesses have no need for such a system, of course, and the costs per function execution are astronomical in comparison to a native program, but for some use cases, the benefits outweigh the costs. As for Ruby and Python, the cost is measured in dev time rather than server costs. Servers are cheap compared to devs, and if a dev can pump out a feature three times as fast because they can make use of a simpler framework, you can afford the overhead of slower programming languages. For this reason I'm watching the Rust space with interest, because the speed is much closer to C++ than to the traditional Java/C# servers most big applications are written in, but the frameworks are slowly shaping up to be similarly complete. Of course, there's a huge difference between Rust and most other languages that cannot be overlooked, but services like Vaultwarden show an interesting opportunity in the server space. It'll still be a while before the tools and frameworks reach a similar level of usability, though.
- reilly3000 5y ago
  > having that run on a datacenter near the user This is the biggest deal IMHO. FaaS in CDN edge DCs are remarkably fast and affordable. If you can respond to an API call faster than a request can get through to a load balancer, you're providing quite a UX improvement regardless of cost. For most people running stateful apps this requires some large tradeoffs in the form of the data layer, but they are getting more reasonable over time. In the case of CloudFlare which may have the strongest edge function hosting now, it used to only support its Worker KV data store (fast but KV only and globally eventually consistent ~10 secs). More recently they have added support for fast connections to their R2 (S3-like) service, and now they have a new service that can proxy to MySQL and Postgres databases. https://blog.cloudflare.com/relational-database-connectors/ https://blog.cloudflare.com/relational-database-connectors/ Its an exciting space, because its not just about CDNs: as 5G towers are rolling out, they are building in edge compute clusters along with them. So now you have a 5G iPhone with faster single-threaded performance than your desktop able to get 1Gbps down and faster response times because of its path to edge compute. ISPs are also getting in the game: https://www.coxedge.com/serverless https://www.coxedge.com/serverless https://www.limelight.com/resources/data-sheet/edge-compute/ https://www.limelight.com/resources/data-sheet/edge-compute/
  - invalidname 5y ago
    I think that's sort of against the pareto principle. Assuming you have the physical data near the user. Great. But most of these functions then go on to fetch stuff from elsewhere with its own delays and problems. Even if S3 or cloudflares equivalent is replicated everywhere there would still be a networking delay and context switch waiting for a response. If performance is THAT important you would be better served with a HUGE machine with a huge cache of the database. That's not something you can do with a FAAS. Other than saying "hello world" fast. I can't think of a proper use case where the performance of FAAS would be realistically better than a dumb monolith. The performance and complexity overheads of FAAS make the case even harder to make.
    - reilly3000 5y ago
vasco 5y ago
When you have a business where salaries are a much larger percentage than hosting on your overall costs, you're going to add to the second to squeeze more out of the first. When you have done the optimisation above, and your bosses keep asking for more features coming out per quarter, and you need to hire more, you're going to opt for the new things everyone is learning, so that you can hire enough without making your life all about hiring. These are trade-offs like any other, and you probably just haven't been exposed to similar constraints or all of this would make at least some sense. There may be better ways, of course.

fake-name 5y ago

In what potential world would it ever make sense to start an entire container for each request? The entire architecture that leads to such a concept is so dumb I have trouble believing anyone ever thought it could be a decent idea.

tiffanyh 5y ago
Thats how Erlang works essentially.
- foepys 5y ago
  Yes but Erlang's BEAM as a purposely built architecture that makes this very efficient. Spawning OS processes and threads is comparatively expensive.
- nesarkvechnep 5y ago
  Erlang doesn’t spawn containers for every request.
lmeyerov 5y ago
Extreme take: Numbers-wise, that should probably be the default ;-) In particular, the world runs on power laws, and I'm betting most server apps are boring occasional internal stuff at the not-even-1-qps level. Long tails add up to be the typical project. So yeah, if all the worlds internal django tools can do scale-to-zero with fast cold start on shared boxes with 100X more tenancy than today, that's saving a lot of electricity & hw. Sort of a virtualization equivalent of "640K ought to be enough for anybody".
tyingq 5y ago
>In what potential world would it ever make sense to start an entire container for each request? If you wait long enough between requests (~5 minutes), that is what AWS Lambda does, only with a VM instead of a container.

FounderBurr 5y ago

This is a joke right?

nithril 5y ago

The benefit from an app running in a container and already listening to an HTTP port with on top an horizontal autoscaler looks small. I might miss something in the approach but as watchdog requires an "HTTP-level multiplexing for requests" (so the app should listen for http), the app is already running in a container with watchdog: everything is in place to not need watchdog. From the FAQ: > Why do I need watchdog anyway? > Whilst we don’t recommend it, if you’re using a HTTP microservice that conforms to the Workloads spec, then technically you may be able to get away without using it.

option_greek 5y ago

So this serverless solution has a watchdog process always running and mapping requests to the correct process? It that more like a server than serverless? Will the next optimisation be launching one thread per request? In which case we are back to square one (:

pdimitar 5y ago
You're quite right but as a customer I frankly don't care about how do they it as long as management and deployment are easier from my side. At one point you do hit a limit on what you can do by cold-starting (almost) everything anyway, right? Let them optimize so they are competitive, and now they've given me a good reason to evaluate them.
tyingq 5y ago
>Will the next optimisation be launching one thread per request? That is what Cloudflare Workers do. One v8 isolate thread per request.

alexellisuk 5y ago

Disclosure: I'm the maintainer of the open source project - openfaas which this blog post talks about. It also covers other FaaS frameworks and why we see a "cold start" even when using Kubernetes as our platform.

forty 5y ago
Does't the proposed solution both solve the issue of FaaS and also remove the main benefit? (ie having stateless function with clean state when it starts running) Also it feels to me like FaaS is good when someone else is running the infrastructure (so you really don't have to know about the underlying infrastructure). Why are people self hosting this? Is it for local test platforms before deploying to a managed FaaS?
- OJFord 5y ago
  One reason: you have an infrastructure team providing the FaaS solution, and multiple other teams that deploy to it; to whom it is 'a managed FaaS'.