2 ms·
> Incident > Then comes Black Friday, and traffic goes up to over 300k per hour, and stays that way for 5 days. All alarms are going off, Pods becoming unhealt
by mrkeen 3y ago
> Incident
> Then comes Black Friday, and traffic goes up to over 300k per hour, and stays that way for 5 days. All alarms are going off, Pods becoming unhealthy and “dying”. APIs erroring out with 5xx all over the place. Containers coming up, and dying, and new containers coming up. The site experience is slowed to a crawl. All this while the devops team is frantically bringing up new pods.
You have load. It happens. There shouldn't be anything frantic about bringing up new pods. Just bump up your replica numbers in your config via git. Or, if you're doing it by hand for some reason:
kubectl scale deployment --replicas=5 my-app
> You're not Google. Stop making k8s happen.
This negates the incident story. You don't need k8s... unless you start losing services because of load and need to react quickly and/or automatically.