The amount of waste we experience can be really high. At Remind we started tracking this using a metric we call OUCH (Overprivisioned Underutilized Cpu Hours).

The amount of waste we experience can be really high. At Remind we started tracking this using a metric we call OUCH (Overprivisioned Underutilized Cpu Hours). Consider a k8s/ECS installation that autoscales. Each application will have a CPU target. In order to prevent spiky traffic patterns from overwhelming the running containers, we target 70% CPU. As usage goes above 70% CPU, we will launch more containers. The k8s cluster will have a reservation target. In order to allow fast launching of new containers, we want the cluster to have only an 80% occupancy rate. If more than 80% of the cluster is reserved, we launch more instances and expand the cluster. So to run my autoscaling container-based applications in a way that will be reactive and respond to incoming load, I have to leave 44% (1 - .7*.8) of my hardware idle. If we also factor in that AWS itself doesn't target a 100% occupancy rate (because then nobody could launch new instances), each unit of CPU I actually use requires a significant amount of idle infrastructure. Easily double, possibly triple in the larger scheme of things. All of that is either directly paid for (k8s nodes) or indirectly paid for (ec2 pricing inevitably has idle capacity costs baked in). With serverless, we would eliminate many of those inefficiencies. With ultra fast launch times, we don't need to give containers headroom to handle spikes. By not running our compute cluster, we can instantly launch everything we need, eliminating k8s overprovisioning. We're left with AWS idle capacity as the only waste, and AWS mostly has solved that with the spot market. The math doesn't always come out ahead, but there's a lot of opportunity for serverless to be cheaper in many cases.