This sounds good, but I don’t understand how it meshes with building reliable systems on top of fallible components. I recently deployed a new service that has

This sounds good, but I don’t understand how it meshes with building reliable systems on top of fallible components. I recently deployed a new service that has three clusters inside it; customer queries hit a proxy layer that queries all three and returns the first non-error reply to the customer. In a typical day, I will have tens of thousands of errors but customers see none. My first deployed code logged the internal errors as errors, but then the SRE started in on a high number of failures, so I switched to counting the internal errors but not reporting as errors, saving that for customer errors.