Prioritising observability and load stress mitigation mechanisms/playbooks is the right thing to do for modern cloud native application service clusters. In su

Prioritising observability and load stress mitigation mechanisms/playbooks is the right thing to do for modern cloud native application service clusters. In such an environment, you could fairly easily make it possible to subject your prod deployment to synthetically generated end-to-end scenario traffic in a safe manner (through request tainting) so that you can create controlled stress situations and see how your application behaves at breaking conditions. This is important for understanding if the observability and mitigation mechanisms/playbooks are in good shape or not. It is especially important to be able to do this periodically because the functionality is continuously evolving. Cloud can lull you into a false sense of safety – not everything in cloud scales or fails over the way you expect them to at all operating points. So it is important to continuously do controlled assessments at loads 2x higher than your current peak 6 months before that becomes your reality. Of course there is always live and learn by fire approach where you wait until the traffic growth (from that frenzy flash sale your business team sprung on you) topples your stack over.