3 ms·

DISCLAIMER: I do some paid work with with Open Core Ventures[0] and I've been a GitLab fan for a while so I'm quite biased. Just tried to use some GitLab repos

DISCLAIMER: I do some paid work with with Open Core Ventures[0] and I've been a GitLab fan for a while so I'm quite biased. Just tried to use some GitLab repos I have and ran into a well specified 503 error page: > We are splitting our database into Main and CI! > > For more information about what we're up to, you can check out our blog article. For progress updates, please check our status page. https://about.gitlab.com/blog/2022/06/02/splitting-database-into-main-and-ci/ https://about.gitlab.com/blog/2022/06/02/splitting-database-... Don't know how I missed it, but it's pretty nice that the downtime is clearly specified, and the blog post (and related posts) were very informative. Some random thoughts below: > GitLab.com's database architecture uses a single PostgreSQL database cluster. This single cluster (let's call it main), consists of a single primary and multiple read-only replicas and stores the data generated by all GitLab features. Database reads can be scaled horizontally through read-only replicas, but writes cannot because PostgreSQL does not support active-active replication natively. All you need is Postgres. IIRC Reddit had a similar starting story. Excited to see how much faster gitlab.com will be and how much more reliable their CI gets (I don't have numbers on hand, but I think they've gone down less than GH Actions in recent memory at least -- maybe there's some recency bias there). In a world where Citus is now fully open source[1], I wonder if it needs to just get pulled into contrib. [0]: https://opencoreventures.com/ https://opencoreventures.com/ [1]: https://www.citusdata.com/blog/2022/06/17/citus-11-goes-fully-open-source/ https://www.citusdata.com/blog/2022/06/17/citus-11-goes-full...

tpetry 4y ago
Citus in PostgreSQL main would be great. Currently Citus is one of many solution but being in main would make it the solution. PostgreSQL would officially be a distributed database, not just by stacking some plugins on it.
- hardwaresofton 4y ago
  > Citus in PostgreSQL main would be great. Currently Citus is one of many solution but being in main would make it the solution. PostgreSQL would officially be a distributed database, not just by stacking some plugins on it. Scaling concurrent transactions (locally and horizontally across machines) has been one of the valid complaints about postgres for so long, it would be amazing to see it solved so well/reliably in-tree. This is one of the great things about pg to me -- the cottage industry of consultancies/companies that make truly awesome stuff and build viable businesses is seriously amazing. Most times they merge changes upstream (if only for the clout), and sometimes they just sprinkle gold dust on all of us. As a sidenote, I have my M$ misgivings like any other dev of a certain age, but it is amazing that they chose to do that. Not only did they provide an exit to one of the awesome companies in the space, they also released such a huge benefit to the community for free. They'll probably get their return back retaining the Citus people inside Microsoft, since obviously they're the ones making the magic.
chippiewill 4y ago
> All you need is Postgres. IIRC Gitlab uses Redis as well, it get used by CI for the streaming logs while a job is still running before it gets tossed up to object storage. Their statement that they only used a single cluster up to this point is a bit dated as well, I believe they've had a second cluster for nearly a year now for their new docker registry implementation. This new cluster for CI is more like a third.
- boleary-gl 4y ago
  > IIRC GitLab uses Redis as well GitLab employee here. You're correct. We also have Redis for a few things: Rails caching, Sidekiq jobs, session data, CI, and a few other things. You can learn more about GitLab.com's infrastructure architecture here [1] and more about what Redis is used for here [2]. [1]: https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/ https://about.gitlab.com/handbook/engineering/infrastructure... [2]: https://docs.gitlab.com/ee/development/redis.html https://docs.gitlab.com/ee/development/redis.html

tpetry 4y ago

Citus in PostgreSQL main would be great. Currently Citus is one of many solution but being in main would make it the solution. PostgreSQL would officially be a distributed database, not just by stacking some plugins on it.

hardwaresofton 4y ago
> Citus in PostgreSQL main would be great. Currently Citus is one of many solution but being in main would make it the solution. PostgreSQL would officially be a distributed database, not just by stacking some plugins on it. Scaling concurrent transactions (locally and horizontally across machines) has been one of the valid complaints about postgres for so long, it would be amazing to see it solved so well/reliably in-tree. This is one of the great things about pg to me -- the cottage industry of consultancies/companies that make truly awesome stuff and build viable businesses is seriously amazing. Most times they merge changes upstream (if only for the clout), and sometimes they just sprinkle gold dust on all of us. As a sidenote, I have my M$ misgivings like any other dev of a certain age, but it is amazing that they chose to do that. Not only did they provide an exit to one of the awesome companies in the space, they also released such a huge benefit to the community for free. They'll probably get their return back retaining the Citus people inside Microsoft, since obviously they're the ones making the magic.

chippiewill 4y ago

> All you need is Postgres. IIRC Gitlab uses Redis as well, it get used by CI for the streaming logs while a job is still running before it gets tossed up to object storage. Their statement that they only used a single cluster up to this point is a bit dated as well, I believe they've had a second cluster for nearly a year now for their new docker registry implementation. This new cluster for CI is more like a third.

boleary-gl 4y ago
> IIRC GitLab uses Redis as well GitLab employee here. You're correct. We also have Redis for a few things: Rails caching, Sidekiq jobs, session data, CI, and a few other things. You can learn more about GitLab.com's infrastructure architecture here [1] and more about what Redis is used for here [2]. [1]: https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/ https://about.gitlab.com/handbook/engineering/infrastructure... [2]: https://docs.gitlab.com/ee/development/redis.html https://docs.gitlab.com/ee/development/redis.html