I saw something like that one time - the new version had one rarely-used broken API endpoint; clients who hit that would silently retry until eventually hitting

I saw something like that one time - the new version had one rarely-used broken API endpoint; clients who hit that would silently retry until eventually hitting an old instance which worked and then they’d be on their way. It was rarely used enough that the total number of retries didn’t trigger any alarms, and failed fast so that failing 19 times before working on the 20th attempt didn’t cause any timeouts. We now have monitoring of success-rate-per-endpoint, so even if 99.9% of requests are successful, if a single function crashes 100% of the time we should still notice :)