4 ms·

You can, and you will. But the mental overhead in "rewrite this 500 line service in a way that doesn't suck" isn't that high. That's not true of a 700,000 line

You can, and you will. But the mental overhead in "rewrite this 500 line service in a way that doesn't suck" isn't that high. That's not true of a 700,000 line monolith.

TheCoelacanth 8y ago
Instead you have the overhead of "rewrite these 50 services to have an architecture that doesn't suck".
- mpweiher 8y ago
  And that architecture is written down nowhere, it is implicit in the different individual services. It's also an architecture that is now much more prone to failures, because while procedure calls essentially never fail, RPCs do. Or if they don't fail outright, have unpredictable latencies etc. (I turned a microservices/SOA code-base into monolith back in 2003/2004. It worked wonders.)
  - YawningAngel 8y ago
    Interestingly, RPCs failing is not really a problem at companies with heavy investment in microservice tooling. Everything is set up to retry transparently below the application layer. Obviously this costs in terms of infrastructure and latency but it's essentially transparent to the programmer.
    - mpweiher 8y ago
      > RPCs failing is not really a problem. Hmm... 1. Software applications are written with little error-handling on networking errors. During a network outage, such applications may stall or infinitely wait for an answer packet, permanently consuming memory or other resources. When the failed network becomes available, those applications may also fail to retry any stalled operations or require a (manual) restart. https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... Infinite retries aren't "not really a problem". If the server isn't there or can't be reached, it isn't there or can't be reached. This is not a problem software/infrastructure can paper over transparently. You can try your best to mitigate, but that's hard, and often makes things worse. I remember a colleague trying out CouchDB. His takeaway was that all error-recovery attempts by distributed CouchDB servers made things worse. So you could use it as a semi-decent single-instance DB, but mercy on your soul should you be so brazen as to attempt to use the distribution features. > heavy investment in microservice tooling. And what percentage of microservices is that? > costs in terms of infrastructure and latency >> Or if they don't fail outright, have unpredictable latencies Hmm...
    - nostrademons 8y ago
      It's still a problem, just not quite as big as one and often one that gets shunted off to a different team than the developers (eg. SRE or PMs). At Google one of the leading causes of cascading failures was RPC retries - service gets overloaded, RPCs start failing to return, client code automatically retries, service gets more overloaded, eventually everything grinds to a halt. Then you ameliorate that problem (eg. with timeouts or backpressure) and suddenly failing RPCs are visible to the application code again. You end up needing to make product decisions about what can fail, what can't, and what an acceptable level of latency is - for example, in search, if the Ad system doesn't return by the time the results do, the page just doesn't show ads. However, if a click on an Ad can't write to the logs, it blocks the redirect until it can because you can't just drop actions that you charged money for without ensuring the transaction goes through. If the RPC to either the cookie store or user database fails to go through, the page just won't show personalized results (and the rest of the search code needs to be prepared to fall back to not having that data as a result). If the search results themselves don't come back, the system retries until they do (and pages on-call SRE). This can also introduce big problems with 95th-percentile latency, because the retries aren't free, they trade failure for slowness. In user-facing services this often isn't a problem, because the user eats the slowness and grumbles. But in services way down in the stack (eg. BigTable), it can be a big problem. High 95th-percentile latency can make services that previously weren't on the critical path suddenly part of the critical path; it can trigger timeouts in higher-level code; it can trigger retries in higher-level code; and it can cause large increases in load (with the possibility of further cascading failures) if those additional retries put some other service over its capacity limits. TANSTAAFL. RPC failures still result in complexity, it's just that there are possible systems that can shunt that complexity into where it's less of a problem for the user. The only time you can outright ignore them is if you're a junior developer, though; anyone at senior/TL/architect level needs to think about them and be prepared.
- jhall1468 8y ago
  Yes, if we assume literally every microservice is just as bad as one monolith, that's true. That hasn't been my experience. Often time, the monolith encourages bad "new features", while the microservices seem to be more of a mixed bag. Given the choice, I'd rather re-write 50 microservices in a heartbeat.
  - mbrodersen 8y ago
    Well given the choice, and with 35+ years of software experience trying both, I'd rather re-write a monolith (by implementing 50 libraries) in a heartbeat.

You can, and you will. But the mental overhead in "rewrite this 500 line service in a way that doesn't suck" isn't that high. That's not true of a 700,000 line monolith.

TheCoelacanth 8y ago

Instead you have the overhead of "rewrite these 50 services to have an architecture that doesn't suck".

mpweiher 8y ago
And that architecture is written down nowhere, it is implicit in the different individual services. It's also an architecture that is now much more prone to failures, because while procedure calls essentially never fail, RPCs do. Or if they don't fail outright, have unpredictable latencies etc. (I turned a microservices/SOA code-base into monolith back in 2003/2004. It worked wonders.)
- YawningAngel 8y ago
  Interestingly, RPCs failing is not really a problem at companies with heavy investment in microservice tooling. Everything is set up to retry transparently below the application layer. Obviously this costs in terms of infrastructure and latency but it's essentially transparent to the programmer.
  - mpweiher 8y ago
    > RPCs failing is not really a problem. Hmm... 1. Software applications are written with little error-handling on networking errors. During a network outage, such applications may stall or infinitely wait for an answer packet, permanently consuming memory or other resources. When the failed network becomes available, those applications may also fail to retry any stalled operations or require a (manual) restart. https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... Infinite retries aren't "not really a problem". If the server isn't there or can't be reached, it isn't there or can't be reached. This is not a problem software/infrastructure can paper over transparently. You can try your best to mitigate, but that's hard, and often makes things worse. I remember a colleague trying out CouchDB. His takeaway was that all error-recovery attempts by distributed CouchDB servers made things worse. So you could use it as a semi-decent single-instance DB, but mercy on your soul should you be so brazen as to attempt to use the distribution features. > heavy investment in microservice tooling. And what percentage of microservices is that? > costs in terms of infrastructure and latency >> Or if they don't fail outright, have unpredictable latencies Hmm...
  - nostrademons 8y ago
    It's still a problem, just not quite as big as one and often one that gets shunted off to a different team than the developers (eg. SRE or PMs). At Google one of the leading causes of cascading failures was RPC retries - service gets overloaded, RPCs start failing to return, client code automatically retries, service gets more overloaded, eventually everything grinds to a halt. Then you ameliorate that problem (eg. with timeouts or backpressure) and suddenly failing RPCs are visible to the application code again. You end up needing to make product decisions about what can fail, what can't, and what an acceptable level of latency is - for example, in search, if the Ad system doesn't return by the time the results do, the page just doesn't show ads. However, if a click on an Ad can't write to the logs, it blocks the redirect until it can because you can't just drop actions that you charged money for without ensuring the transaction goes through. If the RPC to either the cookie store or user database fails to go through, the page just won't show personalized results (and the rest of the search code needs to be prepared to fall back to not having that data as a result). If the search results themselves don't come back, the system retries until they do (and pages on-call SRE). This can also introduce big problems with 95th-percentile latency, because the retries aren't free, they trade failure for slowness. In user-facing services this often isn't a problem, because the user eats the slowness and grumbles. But in services way down in the stack (eg. BigTable), it can be a big problem. High 95th-percentile latency can make services that previously weren't on the critical path suddenly part of the critical path; it can trigger timeouts in higher-level code; it can trigger retries in higher-level code; and it can cause large increases in load (with the possibility of further cascading failures) if those additional retries put some other service over its capacity limits. TANSTAAFL. RPC failures still result in complexity, it's just that there are possible systems that can shunt that complexity into where it's less of a problem for the user. The only time you can outright ignore them is if you're a junior developer, though; anyone at senior/TL/architect level needs to think about them and be prepared.
jhall1468 8y ago
Yes, if we assume literally every microservice is just as bad as one monolith, that's true. That hasn't been my experience. Often time, the monolith encourages bad "new features", while the microservices seem to be more of a mixed bag. Given the choice, I'd rather re-write 50 microservices in a heartbeat.
- mbrodersen 8y ago
  Well given the choice, and with 35+ years of software experience trying both, I'd rather re-write a monolith (by implementing 50 libraries) in a heartbeat.