3 ms·

I somewhat agree with what you're saying, but it still really depends. On growth rate in users and content, access patterns, UGC incremental size etc. Machines

I somewhat agree with what you're saying, but it still really depends. On growth rate in users and content, access patterns, UGC incremental size etc. Machines keep getting bigger, if your growth rate is low/moderate you might never reach storage limits. Or might theoretically reach them in 5+ years in which case many other things could change before then, in tech or the business. The alternative (to expensive front loaded investment in sharding) is just not doing it, until you have to, because its likely you might never actually have to. Of course if you're growing like a popular B2C ala early facebook or whatever, yes you have to, asap, otherwise you'll die. That probably isnt the case for companies like gitlab. Do your realistic growth projections, and your cost/benefit analysis, and pick the cheapest thing you can get away with.

evanelias 3y ago
Sure, I'm not advocating premature optimization here. Don't shard unless it's clear that you eventually need to. But we're talking about a specific company here, not sharding in general. I believe in GitLab's case that they absolutely will need to shard eventually. This is a decade-old company with a $12 billion market cap, not some early seed-stage venture with unproven success. And they're absolutely growing faster than hardware storage capacity limits are increasing. Even aside from data size motivations, they've clearly already had blast-radius problems from using a single monolithic production database. It's not like sharding is some major unsolvable problem in 2024. It was a lot harder 10-15+ years ago, when engineers with sharding experience were extremely difficult to find, frameworks didn't support it at all, premade solutions didn't exist, etc. But even then it was absolutely solvable, I can say from first-hand experience, even with much smaller teams than what GitLab can afford to throw at this.

evanelias 3y ago

Sure, I'm not advocating premature optimization here. Don't shard unless it's clear that you eventually need to. But we're talking about a specific company here, not sharding in general. I believe in GitLab's case that they absolutely will need to shard eventually. This is a decade-old company with a $12 billion market cap, not some early seed-stage venture with unproven success. And they're absolutely growing faster than hardware storage capacity limits are increasing. Even aside from data size motivations, they've clearly already had blast-radius problems from using a single monolithic production database. It's not like sharding is some major unsolvable problem in 2024. It was a lot harder 10-15+ years ago, when engineers with sharding experience were extremely difficult to find, frameworks didn't support it at all, premade solutions didn't exist, etc. But even then it was absolutely solvable, I can say from first-hand experience, even with much smaller teams than what GitLab can afford to throw at this.