3 ms·

> Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware. Isn't this

> Sure, in theory, the distributed system should win, but in practice, it fails more often (due to operator error or unforeseen bugs) than hardware. Isn't this because of the rampant introduction of accidental complexity whenever you attempt to make a system horizontally scalable - e.g. for whatever reason the developers or the people in charge suddenly try to cram in as many technological solutions as possible because apparently that's what the large companies are doing? There's no reason why you couldn't think about which data can or cannot be shared, and develop your system as one that's scalable, possibly modular, but with the codebase still being largely monolithic in nature. I'd argue that there's a large gray area between the the opposite ends of that spectrum - monoliths vs microservices. I actually wrote down some thoughts in a blog post of mine, called "Moduliths: because we need to scale, but we also cannot afford microservices": https://blog.kronis.dev/articles/modulith-because-we-need-to-scale-but-we-also-cannot-afford-micro-services https://blog.kronis.dev/articles/modulith-because-we-need-to...

Nextgrid 5y ago
I think another issue is that there are plenty of robust, battle-tested building blocks such as databases for monolithic, vertically-scalable applications, but much fewer for horizontally-scalable ones, meaning you'll need to roll your own (at the application layer) and most likely screw it up.
- KronisLV 5y ago
  I do agree with you in that regard, however, that's also a dangerous line of thinking. There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility https://docs.pingcap.com/tidb/stable/mysql-compatibility There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up. Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth. Alternatively, when you actually hit issues with the above approach (and only then), consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all with services that are separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms. If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best. To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible. Not all components can or should scale horizontally.
  - lgunsch 5y ago
    Have you ever seen CockroachDB, or Yugabyte? Both are not eventually consistent, but are horizontally scalable. It looks like to me it is a solved problem.
earleybird 5y ago
Not to say that some systems don't introduce accidental complexity, however, getting distributed systems right/correct is necessarily complicated. Design for failure and you will get a good picture of what tools you need to employ.
- chousuke 5y ago
  I think that you can keep systems simple and HA if you forget about multi-master components and design active/passive solutions instead, whose behaviour in failure scenarios is much easier to understand and much less likely to go wrong, especially if you also forgo attempting to autorecover failed components after a failover event. If no HA is fine 90% of the time, simple HA is fine 99% of the time.

Nextgrid 5y ago

I think another issue is that there are plenty of robust, battle-tested building blocks such as databases for monolithic, vertically-scalable applications, but much fewer for horizontally-scalable ones, meaning you'll need to roll your own (at the application layer) and most likely screw it up.

KronisLV 5y ago
I do agree with you in that regard, however, that's also a dangerous line of thinking. There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility https://docs.pingcap.com/tidb/stable/mysql-compatibility There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up. Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu... To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth. Alternatively, when you actually hit issues with the above approach (and only then), consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all with services that are separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms. If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best. To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible. Not all components can or should scale horizontally.
- lgunsch 5y ago
  Have you ever seen CockroachDB, or Yugabyte? Both are not eventually consistent, but are horizontally scalable. It looks like to me it is a solved problem.

earleybird 5y ago

Not to say that some systems don't introduce accidental complexity, however, getting distributed systems right/correct is necessarily complicated. Design for failure and you will get a good picture of what tools you need to employ.

chousuke 5y ago
I think that you can keep systems simple and HA if you forget about multi-master components and design active/passive solutions instead, whose behaviour in failure scenarios is much easier to understand and much less likely to go wrong, especially if you also forgo attempting to autorecover failed components after a failover event. If no HA is fine 90% of the time, simple HA is fine 99% of the time.