4 ms·

Totally agree. Overall the author's arguments against sharding seem short-sighted, at best. All large successful user-generated content applications eventually

Totally agree. Overall the author's arguments against sharding seem short-sighted, at best. All large successful user-generated content applications eventually need to shard, and most of them are read-heavy just like gitlab.com. The two primary motivators for sharding in this case are database size and blast radius, rather than needing to scale writes. For database size, even if your DB still can fit on one beefy server, operational activities like backup/restore and new replica cloning take an increasingly long time. This in turn becomes a major risk to the business -- for example he even discusses how long the restore took when he had to restore the prod DB from the staging DB. For blast radius, it's the ability to have outages (e.g. hardware failure) only affect a small subset of users. Or for a more extreme situation, the massive difference between "I accidentally dropped the database" and "I accidentally dropped a database". Sharding does indeed add terrible complexity to the application as well as operations. But in any case, eventually the data size motivator becomes unavoidable for user generated content: you eventually must shard, and the longer you wait, the more difficult it will become.

dalyons 3y ago
Remember most of the business is on prem installs . Introducing the huge complexity of sharding to the codebase would have benefited these installs approximately zero.
- evanelias 3y ago
  The existence and uptime of gitlab.com is essential to the company's customer acquisition. If gitlab.com goes offline for an extended period of time (due to database size issues or anything else), it's a huge hit to the reputation of their brand. This will directly affect their bottom line even if it isn't the main direct source of their revenue. And again, what's the alternative? If you offer a successful user-generated content platform, eventually you must shard. User-generated content only grows over time, it never shrinks, and at some point you exceed the maximum amount of storage that can be attached to a single server. And long before that point, you encounter horrendous operational headaches, such as full backups taking longer than 24 hours to make (let alone restore from). It's also conceivably possible that their largest on-prem customers would be interested in sharding their self-hosted installations, so theoretically this could be a valuable high-cost enterprise feature.
  - dalyons 3y ago
    I somewhat agree with what you're saying, but it still really depends. On growth rate in users and content, access patterns, UGC incremental size etc. Machines keep getting bigger, if your growth rate is low/moderate you might never reach storage limits. Or might theoretically reach them in 5+ years in which case many other things could change before then, in tech or the business. The alternative (to expensive front loaded investment in sharding) is just not doing it, until you have to, because its likely you might never actually have to. Of course if you're growing like a popular B2C ala early facebook or whatever, yes you have to, asap, otherwise you'll die. That probably isnt the case for companies like gitlab. Do your realistic growth projections, and your cost/benefit analysis, and pick the cheapest thing you can get away with.
    - evanelias 3y ago
      Sure, I'm not advocating premature optimization here. Don't shard unless it's clear that you eventually need to. But we're talking about a specific company here, not sharding in general. I believe in GitLab's case that they absolutely will need to shard eventually. This is a decade-old company with a $12 billion market cap, not some early seed-stage venture with unproven success. And they're absolutely growing faster than hardware storage capacity limits are increasing. Even aside from data size motivations, they've clearly already had blast-radius problems from using a single monolithic production database. It's not like sharding is some major unsolvable problem in 2024. It was a lot harder 10-15+ years ago, when engineers with sharding experience were extremely difficult to find, frameworks didn't support it at all, premade solutions didn't exist, etc. But even then it was absolutely solvable, I can say from first-hand experience, even with much smaller teams than what GitLab can afford to throw at this.

dalyons 3y ago

Remember most of the business is on prem installs . Introducing the huge complexity of sharding to the codebase would have benefited these installs approximately zero.

evanelias 3y ago
The existence and uptime of gitlab.com is essential to the company's customer acquisition. If gitlab.com goes offline for an extended period of time (due to database size issues or anything else), it's a huge hit to the reputation of their brand. This will directly affect their bottom line even if it isn't the main direct source of their revenue. And again, what's the alternative? If you offer a successful user-generated content platform, eventually you must shard. User-generated content only grows over time, it never shrinks, and at some point you exceed the maximum amount of storage that can be attached to a single server. And long before that point, you encounter horrendous operational headaches, such as full backups taking longer than 24 hours to make (let alone restore from). It's also conceivably possible that their largest on-prem customers would be interested in sharding their self-hosted installations, so theoretically this could be a valuable high-cost enterprise feature.
- dalyons 3y ago
  I somewhat agree with what you're saying, but it still really depends. On growth rate in users and content, access patterns, UGC incremental size etc. Machines keep getting bigger, if your growth rate is low/moderate you might never reach storage limits. Or might theoretically reach them in 5+ years in which case many other things could change before then, in tech or the business. The alternative (to expensive front loaded investment in sharding) is just not doing it, until you have to, because its likely you might never actually have to. Of course if you're growing like a popular B2C ala early facebook or whatever, yes you have to, asap, otherwise you'll die. That probably isnt the case for companies like gitlab. Do your realistic growth projections, and your cost/benefit analysis, and pick the cheapest thing you can get away with.
  - evanelias 3y ago
    Sure, I'm not advocating premature optimization here. Don't shard unless it's clear that you eventually need to. But we're talking about a specific company here, not sharding in general. I believe in GitLab's case that they absolutely will need to shard eventually. This is a decade-old company with a $12 billion market cap, not some early seed-stage venture with unproven success. And they're absolutely growing faster than hardware storage capacity limits are increasing. Even aside from data size motivations, they've clearly already had blast-radius problems from using a single monolithic production database. It's not like sharding is some major unsolvable problem in 2024. It was a lot harder 10-15+ years ago, when engineers with sharding experience were extremely difficult to find, frameworks didn't support it at all, premade solutions didn't exist, etc. But even then it was absolutely solvable, I can say from first-hand experience, even with much smaller teams than what GitLab can afford to throw at this.