Another trick is failure injection, e.g. go for a 404/410 for a small % of all requests or clients to figure out potential impact and give an early warning, the

Another trick is failure injection, e.g. go for a 404/410 for a small % of all requests or clients to figure out potential impact and give an early warning, then slowly ramp up the %.