2 ms·

>For example, sharding is useful if writes heavily outnumber reads If you distribute your data carefully and don't need to do joins across shards, I fail to se

>For example, sharding is useful if writes heavily outnumber reads If you distribute your data carefully and don't need to do joins across shards, I fail to see how sharding doesn't help with reads.

YorickPeterse 3y ago
Every time we'd look into this, we'd look into the challenges we'd have to face and see if they changed since last time. One of the problems GitLab faces is that it JOINs all over the place. In addition, while most data can be shared on a group/namespace basis, GitLab also does a bunch of cross-group queries in frequently accessed pages. To put it differently, try to think of an application that does all the things you _don't_ want to be doing if you're going to (or want to) shard. Then imagine that that's basically GitLab, coupled with the load patterns simply not justifying it.
isbvhodnvemrwvn 3y ago
It adds a huge amount of complexity due to those limitations, and unlike writes, you can handle reads off of read replicas, as they did.
ahofmann 3y ago
Read replicas are better suited for this situation and way less complicated. And you said it yourself: you need to be careful and you loose the ability to do joins.
evanelias 3y ago
Totally agree. Overall the author's arguments against sharding seem short-sighted, at best. All large successful user-generated content applications eventually need to shard, and most of them are read-heavy just like gitlab.com. The two primary motivators for sharding in this case are database size and blast radius, rather than needing to scale writes. For database size, even if your DB still can fit on one beefy server, operational activities like backup/restore and new replica cloning take an increasingly long time. This in turn becomes a major risk to the business -- for example he even discusses how long the restore took when he had to restore the prod DB from the staging DB. For blast radius, it's the ability to have outages (e.g. hardware failure) only affect a small subset of users. Or for a more extreme situation, the massive difference between "I accidentally dropped the database" and "I accidentally dropped a database". Sharding does indeed add terrible complexity to the application as well as operations. But in any case, eventually the data size motivator becomes unavoidable for user generated content: you eventually must shard, and the longer you wait, the more difficult it will become.
- dalyons 3y ago
  Remember most of the business is on prem installs . Introducing the huge complexity of sharding to the codebase would have benefited these installs approximately zero.
  - evanelias 3y ago
    The existence and uptime of gitlab.com is essential to the company's customer acquisition. If gitlab.com goes offline for an extended period of time (due to database size issues or anything else), it's a huge hit to the reputation of their brand. This will directly affect their bottom line even if it isn't the main direct source of their revenue. And again, what's the alternative? If you offer a successful user-generated content platform, eventually you must shard. User-generated content only grows over time, it never shrinks, and at some point you exceed the maximum amount of storage that can be attached to a single server. And long before that point, you encounter horrendous operational headaches, such as full backups taking longer than 24 hours to make (let alone restore from). It's also conceivably possible that their largest on-prem customers would be interested in sharding their self-hosted installations, so theoretically this could be a valuable high-cost enterprise feature.
    - dalyons 3y ago
      I somewhat agree with what you're saying, but it still really depends. On growth rate in users and content, access patterns, UGC incremental size etc. Machines keep getting bigger, if your growth rate is low/moderate you might never reach storage limits. Or might theoretically reach them in 5+ years in which case many other things could change before then, in tech or the business. The alternative (to expensive front loaded investment in sharding) is just not doing it, until you have to, because its likely you might never actually have to. Of course if you're growing like a popular B2C ala early facebook or whatever, yes you have to, asap, otherwise you'll die. That probably isnt the case for companies like gitlab. Do your realistic growth projections, and your cost/benefit analysis, and pick the cheapest thing you can get away with.
      - 3y ago

>For example, sharding is useful if writes heavily outnumber reads If you distribute your data carefully and don't need to do joins across shards, I fail to see how sharding doesn't help with reads.

YorickPeterse 3y ago

Every time we'd look into this, we'd look into the challenges we'd have to face and see if they changed since last time. One of the problems GitLab faces is that it JOINs all over the place. In addition, while most data can be shared on a group/namespace basis, GitLab also does a bunch of cross-group queries in frequently accessed pages. To put it differently, try to think of an application that does all the things you _don't_ want to be doing if you're going to (or want to) shard. Then imagine that that's basically GitLab, coupled with the load patterns simply not justifying it.

isbvhodnvemrwvn 3y ago

It adds a huge amount of complexity due to those limitations, and unlike writes, you can handle reads off of read replicas, as they did.

ahofmann 3y ago

Read replicas are better suited for this situation and way less complicated. And you said it yourself: you need to be careful and you loose the ability to do joins.

evanelias 3y ago

Totally agree. Overall the author's arguments against sharding seem short-sighted, at best. All large successful user-generated content applications eventually need to shard, and most of them are read-heavy just like gitlab.com. The two primary motivators for sharding in this case are database size and blast radius, rather than needing to scale writes. For database size, even if your DB still can fit on one beefy server, operational activities like backup/restore and new replica cloning take an increasingly long time. This in turn becomes a major risk to the business -- for example he even discusses how long the restore took when he had to restore the prod DB from the staging DB. For blast radius, it's the ability to have outages (e.g. hardware failure) only affect a small subset of users. Or for a more extreme situation, the massive difference between "I accidentally dropped the database" and "I accidentally dropped a database". Sharding does indeed add terrible complexity to the application as well as operations. But in any case, eventually the data size motivator becomes unavoidable for user generated content: you eventually must shard, and the longer you wait, the more difficult it will become.

dalyons 3y ago
Remember most of the business is on prem installs . Introducing the huge complexity of sharding to the codebase would have benefited these installs approximately zero.
- evanelias 3y ago
  The existence and uptime of gitlab.com is essential to the company's customer acquisition. If gitlab.com goes offline for an extended period of time (due to database size issues or anything else), it's a huge hit to the reputation of their brand. This will directly affect their bottom line even if it isn't the main direct source of their revenue. And again, what's the alternative? If you offer a successful user-generated content platform, eventually you must shard. User-generated content only grows over time, it never shrinks, and at some point you exceed the maximum amount of storage that can be attached to a single server. And long before that point, you encounter horrendous operational headaches, such as full backups taking longer than 24 hours to make (let alone restore from). It's also conceivably possible that their largest on-prem customers would be interested in sharding their self-hosted installations, so theoretically this could be a valuable high-cost enterprise feature.
  - dalyons 3y ago
    I somewhat agree with what you're saying, but it still really depends. On growth rate in users and content, access patterns, UGC incremental size etc. Machines keep getting bigger, if your growth rate is low/moderate you might never reach storage limits. Or might theoretically reach them in 5+ years in which case many other things could change before then, in tech or the business. The alternative (to expensive front loaded investment in sharding) is just not doing it, until you have to, because its likely you might never actually have to. Of course if you're growing like a popular B2C ala early facebook or whatever, yes you have to, asap, otherwise you'll die. That probably isnt the case for companies like gitlab. Do your realistic growth projections, and your cost/benefit analysis, and pick the cheapest thing you can get away with.
    - 3y ago