Rather than load testing, do red line testing. That is – gradually increase load on your service with concurrent requests until it is saturated. Measure how

Rather than load testing, do red line testing. That is – gradually increase load on your service with concurrent requests until it is saturated. Measure how the latency and errors progress with more concurrent requests and understand at what point and how your service starts to breakdown under heavy load. Based on this you may have to do many things – 1. Can you optimize your service or your service's downstream dependencies or the application calling your service. 2. Can you build in graceful degradation into your service – functionality reduction to get more useful throughput out of your service – with same resources. 3. Build circuit breakers and throttlers before your downstream dependencies so that you don't overload them and cause them to fail or you don't fail totally when they do indeed fail. 4. If you do get overloaded for some reason (say your server pool suddenly became half its size), are you able to recover quickly. 5. Do you have monitoring and alerts for these load scenarios? If you do this for each of the services in the service call graph, then your distributed service application would be robust.