5 ms·

I think it's often a better strategy to do sharding or micro-services and if possible keep the service so small that it's state can fit in a single machine. If

by z3t4 10y ago

I think it's often a better strategy to do sharding or micro-services and if possible keep the service so small that it's state can fit in a single machine. If you can have data boundaries, like for example one customer do not need to access the data of another customer, then you can separate customers's data and place them in different databases.

elvinyung 10y ago
The state can never fit on just a single machine, unless you're perfectly confident that single machine will never fail :) Distributed consistency as discussed in this blog post is not really about scaling write throughput (which is the problem solved by horizontal/vertical partitioning), but rather about keeping replicas in sync with each other, seeing the same state. The usual suspects, MySQL and Postgres, don't really have any sort of replication that's totally synchronous (although they're getting there[1, 2]). As for scaling writes, I agree that chopping up your Postgres database is the safest choice for now, but it's generally true that this is hacky, and kind of a pain to architect your application around it. Solutions like Spanner and FaunaDB aim to free the application developer from this burden, without having to give up the large feature set of relational databases by using a NoSQL database. [1] https://dev.mysql.com/doc/refman/5.7/en/group-replication.html https://dev.mysql.com/doc/refman/5.7/en/group-replication.ht... [2] http://paquier.xyz/postgresql-2/postgres-10-quorum-sync/ http://paquier.xyz/postgresql-2/postgres-10-quorum-sync/
ngbronson 10y ago
Sharding your product so that no cross-shard links are possible is robust and scalable, but it requires you to understand your final product very well at design time. If you start down that path and then bolt on some cross-shard data, the end result is almost always worse then if you had planned for that from the start.

by z3t4 10y ago

elvinyung 10y ago

The state can never fit on just a single machine, unless you're perfectly confident that single machine will never fail :) Distributed consistency as discussed in this blog post is not really about scaling write throughput (which is the problem solved by horizontal/vertical partitioning), but rather about keeping replicas in sync with each other, seeing the same state. The usual suspects, MySQL and Postgres, don't really have any sort of replication that's totally synchronous (although they're getting there[1, 2]). As for scaling writes, I agree that chopping up your Postgres database is the safest choice for now, but it's generally true that this is hacky, and kind of a pain to architect your application around it. Solutions like Spanner and FaunaDB aim to free the application developer from this burden, without having to give up the large feature set of relational databases by using a NoSQL database. [1] https://dev.mysql.com/doc/refman/5.7/en/group-replication.html https://dev.mysql.com/doc/refman/5.7/en/group-replication.ht... [2] http://paquier.xyz/postgresql-2/postgres-10-quorum-sync/ http://paquier.xyz/postgresql-2/postgres-10-quorum-sync/

ngbronson 10y ago

Sharding your product so that no cross-shard links are possible is robust and scalable, but it requires you to understand your final product very well at design time. If you start down that path and then bolt on some cross-shard data, the end result is almost always worse then if you had planned for that from the start.