3 ms·

Interestingly, RPCs failing is not really a problem at companies with heavy investment in microservice tooling. Everything is set up to retry transparently belo

Interestingly, RPCs failing is not really a problem at companies with heavy investment in microservice tooling. Everything is set up to retry transparently below the application layer. Obviously this costs in terms of infrastructure and latency but it's essentially transparent to the programmer.

mpweiher 8y ago
> RPCs failing is not really a problem. Hmm... 1. Software applications are written with little error-handling on networking errors. During a network outage, such applications may stall or infinitely wait for an answer packet, permanently consuming memory or other resources. When the failed network becomes available, those applications may also fail to retry any stalled operations or require a (manual) restart. https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... Infinite retries aren't "not really a problem". If the server isn't there or can't be reached, it isn't there or can't be reached. This is not a problem software/infrastructure can paper over transparently. You can try your best to mitigate, but that's hard, and often makes things worse. I remember a colleague trying out CouchDB. His takeaway was that all error-recovery attempts by distributed CouchDB servers made things worse. So you could use it as a semi-decent single-instance DB, but mercy on your soul should you be so brazen as to attempt to use the distribution features. > heavy investment in microservice tooling. And what percentage of microservices is that? > costs in terms of infrastructure and latency >> Or if they don't fail outright, have unpredictable latencies Hmm...
nostrademons 8y ago
It's still a problem, just not quite as big as one and often one that gets shunted off to a different team than the developers (eg. SRE or PMs). At Google one of the leading causes of cascading failures was RPC retries - service gets overloaded, RPCs start failing to return, client code automatically retries, service gets more overloaded, eventually everything grinds to a halt. Then you ameliorate that problem (eg. with timeouts or backpressure) and suddenly failing RPCs are visible to the application code again. You end up needing to make product decisions about what can fail, what can't, and what an acceptable level of latency is - for example, in search, if the Ad system doesn't return by the time the results do, the page just doesn't show ads. However, if a click on an Ad can't write to the logs, it blocks the redirect until it can because you can't just drop actions that you charged money for without ensuring the transaction goes through. If the RPC to either the cookie store or user database fails to go through, the page just won't show personalized results (and the rest of the search code needs to be prepared to fall back to not having that data as a result). If the search results themselves don't come back, the system retries until they do (and pages on-call SRE). This can also introduce big problems with 95th-percentile latency, because the retries aren't free, they trade failure for slowness. In user-facing services this often isn't a problem, because the user eats the slowness and grumbles. But in services way down in the stack (eg. BigTable), it can be a big problem. High 95th-percentile latency can make services that previously weren't on the critical path suddenly part of the critical path; it can trigger timeouts in higher-level code; it can trigger retries in higher-level code; and it can cause large increases in load (with the possibility of further cascading failures) if those additional retries put some other service over its capacity limits. TANSTAAFL. RPC failures still result in complexity, it's just that there are possible systems that can shunt that complexity into where it's less of a problem for the user. The only time you can outright ignore them is if you're a junior developer, though; anyone at senior/TL/architect level needs to think about them and be prepared.

mpweiher 8y ago

> RPCs failing is not really a problem. Hmm... 1. Software applications are written with little error-handling on networking errors. During a network outage, such applications may stall or infinitely wait for an answer packet, permanently consuming memory or other resources. When the failed network becomes available, those applications may also fail to retry any stalled operations or require a (manual) restart. https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... Infinite retries aren't "not really a problem". If the server isn't there or can't be reached, it isn't there or can't be reached. This is not a problem software/infrastructure can paper over transparently. You can try your best to mitigate, but that's hard, and often makes things worse. I remember a colleague trying out CouchDB. His takeaway was that all error-recovery attempts by distributed CouchDB servers made things worse. So you could use it as a semi-decent single-instance DB, but mercy on your soul should you be so brazen as to attempt to use the distribution features. > heavy investment in microservice tooling. And what percentage of microservices is that? > costs in terms of infrastructure and latency >> Or if they don't fail outright, have unpredictable latencies Hmm...

nostrademons 8y ago

It's still a problem, just not quite as big as one and often one that gets shunted off to a different team than the developers (eg. SRE or PMs). At Google one of the leading causes of cascading failures was RPC retries - service gets overloaded, RPCs start failing to return, client code automatically retries, service gets more overloaded, eventually everything grinds to a halt. Then you ameliorate that problem (eg. with timeouts or backpressure) and suddenly failing RPCs are visible to the application code again. You end up needing to make product decisions about what can fail, what can't, and what an acceptable level of latency is - for example, in search, if the Ad system doesn't return by the time the results do, the page just doesn't show ads. However, if a click on an Ad can't write to the logs, it blocks the redirect until it can because you can't just drop actions that you charged money for without ensuring the transaction goes through. If the RPC to either the cookie store or user database fails to go through, the page just won't show personalized results (and the rest of the search code needs to be prepared to fall back to not having that data as a result). If the search results themselves don't come back, the system retries until they do (and pages on-call SRE). This can also introduce big problems with 95th-percentile latency, because the retries aren't free, they trade failure for slowness. In user-facing services this often isn't a problem, because the user eats the slowness and grumbles. But in services way down in the stack (eg. BigTable), it can be a big problem. High 95th-percentile latency can make services that previously weren't on the critical path suddenly part of the critical path; it can trigger timeouts in higher-level code; it can trigger retries in higher-level code; and it can cause large increases in load (with the possibility of further cascading failures) if those additional retries put some other service over its capacity limits. TANSTAAFL. RPC failures still result in complexity, it's just that there are possible systems that can shunt that complexity into where it's less of a problem for the user. The only time you can outright ignore them is if you're a junior developer, though; anyone at senior/TL/architect level needs to think about them and be prepared.