I've always followed the "metrics driven development" school of thought. If you make capturing "application telemetry" part of the deployment and running of you

I've always followed the "metrics driven development" school of thought. If you make capturing "application telemetry" part of the deployment and running of your application, you make it super easy to monitor via a more or less full on integration test using production traffic. Api response time > 300ms for > 5 seconds? Pager duty!