5 ms·

Avoiding sharding and complex replication is very smart to postpone as late as possible with any database (mysql, postgres, mongodb). It can be very fragile or

Avoiding sharding and complex replication is very smart to postpone as late as possible with any database (mysql, postgres, mongodb). It can be very fragile or fail in unexpected or unusual ways and most importantly it can take much longer to fix. E.g. 18 hours instead of 2 hours of downtime.

evanelias 6y ago
Once your data grows very large, a successfully-implemented sharding solution actually improves availability, rather than reducing it. With a huge monolithic database, a failure causes downtime for your entire product/company. Replica cloning and backups are slow. Major version upgrades are stressful because it's all-or-nothing. With a sharded environment, a single shard failure only impacts a portion of your userbase, and smaller databases are faster to perform operational actions on. There are definitely major downsides to sharding, but they tend to be more on the application side in my experience.
- codyb 6y ago
  Wouldn’t sharding generally result in overlap between the key ranges so that a database shard going down doesn’t have to result in any downtime? Then your issue is replication of writes I suppose. Probably depends on use case what configuration you choose.
  - Tostino 6y ago
    Totally depends on how you shard. In my case (b2b), i'd be sharding by tenant. Having a single tenant go down would not have the same impact as every single tenant going down.
  - evanelias 6y ago
    With a typical sharded relational database setup, each sharding key value maps to exactly one shard. There should be replicas of that shard, which can be promoted in case of a master failure. But in rare cases all replicas may also be degraded or inconsistent and therefore non-promotable. When this happens to a giant monolithic non-sharded database, the impact is far more catastrophic than when it happens to a single smaller shard. In any case, replication is a separate concern from sharding. Each sharded database replica set (traditionally 1 master + N replicas) has a data set that is unique / independent of other sharded database replica sets. That said, some of the "NewSQL" distributed databases may arrange their data in more complex ways. [Edit to clarify: I mean in respect to the combination of both sharding and replication. With a traditional sharded relational database, you have replica sets where each replica contains the exact same shard or shards; with a NewSQL distributed DB, data may be replicated in a more complex arrangement]
    - zinclozenge 6y ago
      Most, if not all, commercial newsql distributed databases do range based splitting of the data, with each range managed by a raft group. Raft groups get migrated between nodes to avoid hot spots, among other scheduling criteria. TiDB does this for sure, I'd be surprised if CockroachDB (and yugabyte and dgraph) doesn't do it.
  - wejick 6y ago
    Honestly I never found the case when this happens, data always falls into 1 shard according to the key. Then comes the concept of shard replica where the shard can live in several nodes and form a redundancy. However I'm noy sure how usually it's being setup on Postgres
- bmcahren 6y ago
  Shards increase the number of failure modes and increase the complexity of those failure modes. For most businesses, the recommendation holds true... keep it simple, don't shard until you need. I find it somewhat concerning that MongoDB has a better architecture for upgrades than Postgres. You add a replica to the cluster running the new major version and then switch that replica over as your primary once you've replaced enough instances in the cluster. Having worked in Oracle and MySQL for years then having switched to a company with a MongoDB framework I forgot how stressful upgrades would be with such archaic limitations.
  - evanelias 6y ago
    > Shards increase the number of failure modes and increase the complexity of those failure modes. I would only agree with this during the initial implementation of sharding. Once deployed and stable, I have not found this to be the case, at all. I say this as someone who has directly architected a sharded database layer that scaled to over a trillion rows, and later worked on core automation and operations for sharded databases that scaled to an incalculable number of rows (easily over 1 quadrillion). In both cases, each company's non-sharded databases were FAR more operationally problematic than the sharded ones. The sharded database tiers behave in common ways with relatively uniform workloads, and the non-sharded databases were each special snowflakes using different obscure features of the database. > keep it simple, don't shard until you need I would have wholeheartedly agreed with this until a few years ago. Cloud storage now permits many companies to run monster 10+ TB monolithic relational databases. Technically these companies no longer "need" to shard, possibly ever. But at these data sizes, many operations become extremely painful, problematic, and slow.
    - throwdbaaway 6y ago
      One of those companies is facebook right? I think this blog post provides a much more balanced view: http://yoshinorimatsunobu.blogspot.com/2017/11/towards-bigger-small-data.html http://yoshinorimatsunobu.blogspot.com/2017/11/towards-bigge...
      - 6y ago

evanelias 6y ago

Once your data grows very large, a successfully-implemented sharding solution actually improves availability, rather than reducing it. With a huge monolithic database, a failure causes downtime for your entire product/company. Replica cloning and backups are slow. Major version upgrades are stressful because it's all-or-nothing. With a sharded environment, a single shard failure only impacts a portion of your userbase, and smaller databases are faster to perform operational actions on. There are definitely major downsides to sharding, but they tend to be more on the application side in my experience.

codyb 6y ago
Wouldn’t sharding generally result in overlap between the key ranges so that a database shard going down doesn’t have to result in any downtime? Then your issue is replication of writes I suppose. Probably depends on use case what configuration you choose.
- Tostino 6y ago
  Totally depends on how you shard. In my case (b2b), i'd be sharding by tenant. Having a single tenant go down would not have the same impact as every single tenant going down.
- evanelias 6y ago
  With a typical sharded relational database setup, each sharding key value maps to exactly one shard. There should be replicas of that shard, which can be promoted in case of a master failure. But in rare cases all replicas may also be degraded or inconsistent and therefore non-promotable. When this happens to a giant monolithic non-sharded database, the impact is far more catastrophic than when it happens to a single smaller shard. In any case, replication is a separate concern from sharding. Each sharded database replica set (traditionally 1 master + N replicas) has a data set that is unique / independent of other sharded database replica sets. That said, some of the "NewSQL" distributed databases may arrange their data in more complex ways. [Edit to clarify: I mean in respect to the combination of both sharding and replication. With a traditional sharded relational database, you have replica sets where each replica contains the exact same shard or shards; with a NewSQL distributed DB, data may be replicated in a more complex arrangement]
  - zinclozenge 6y ago
    Most, if not all, commercial newsql distributed databases do range based splitting of the data, with each range managed by a raft group. Raft groups get migrated between nodes to avoid hot spots, among other scheduling criteria. TiDB does this for sure, I'd be surprised if CockroachDB (and yugabyte and dgraph) doesn't do it.
- wejick 6y ago
  Honestly I never found the case when this happens, data always falls into 1 shard according to the key. Then comes the concept of shard replica where the shard can live in several nodes and form a redundancy. However I'm noy sure how usually it's being setup on Postgres
bmcahren 6y ago
Shards increase the number of failure modes and increase the complexity of those failure modes. For most businesses, the recommendation holds true... keep it simple, don't shard until you need. I find it somewhat concerning that MongoDB has a better architecture for upgrades than Postgres. You add a replica to the cluster running the new major version and then switch that replica over as your primary once you've replaced enough instances in the cluster. Having worked in Oracle and MySQL for years then having switched to a company with a MongoDB framework I forgot how stressful upgrades would be with such archaic limitations.
- evanelias 6y ago
  > Shards increase the number of failure modes and increase the complexity of those failure modes. I would only agree with this during the initial implementation of sharding. Once deployed and stable, I have not found this to be the case, at all. I say this as someone who has directly architected a sharded database layer that scaled to over a trillion rows, and later worked on core automation and operations for sharded databases that scaled to an incalculable number of rows (easily over 1 quadrillion). In both cases, each company's non-sharded databases were FAR more operationally problematic than the sharded ones. The sharded database tiers behave in common ways with relatively uniform workloads, and the non-sharded databases were each special snowflakes using different obscure features of the database. > keep it simple, don't shard until you need I would have wholeheartedly agreed with this until a few years ago. Cloud storage now permits many companies to run monster 10+ TB monolithic relational databases. Technically these companies no longer "need" to shard, possibly ever. But at these data sizes, many operations become extremely painful, problematic, and slow.
  - throwdbaaway 6y ago
    One of those companies is facebook right? I think this blog post provides a much more balanced view: http://yoshinorimatsunobu.blogspot.com/2017/11/towards-bigger-small-data.html http://yoshinorimatsunobu.blogspot.com/2017/11/towards-bigge...
    - 6y ago