7 ms·

We are splitting our database into Main and CI

153 points by hardwaresofton 4y ago

hardwaresofton 4y ago
DISCLAIMER: I do some paid work with with Open Core Ventures[0] and I've been a GitLab fan for a while so I'm quite biased. Just tried to use some GitLab repos I have and ran into a well specified 503 error page: > We are splitting our database into Main and CI! > > For more information about what we're up to, you can check out our blog article. For progress updates, please check our status page. https://about.gitlab.com/blog/2022/06/02/splitting-database-into-main-and-ci/ https://about.gitlab.com/blog/2022/06/02/splitting-database-... Don't know how I missed it, but it's pretty nice that the downtime is clearly specified, and the blog post (and related posts) were very informative. Some random thoughts below: > GitLab.com's database architecture uses a single PostgreSQL database cluster. This single cluster (let's call it main), consists of a single primary and multiple read-only replicas and stores the data generated by all GitLab features. Database reads can be scaled horizontally through read-only replicas, but writes cannot because PostgreSQL does not support active-active replication natively. All you need is Postgres. IIRC Reddit had a similar starting story. Excited to see how much faster gitlab.com will be and how much more reliable their CI gets (I don't have numbers on hand, but I think they've gone down less than GH Actions in recent memory at least -- maybe there's some recency bias there). In a world where Citus is now fully open source[1], I wonder if it needs to just get pulled into contrib. [0]: https://opencoreventures.com/ https://opencoreventures.com/ [1]: https://www.citusdata.com/blog/2022/06/17/citus-11-goes-fully-open-source/ https://www.citusdata.com/blog/2022/06/17/citus-11-goes-full...
- tpetry 4y ago
  Citus in PostgreSQL main would be great. Currently Citus is one of many solution but being in main would make it the solution. PostgreSQL would officially be a distributed database, not just by stacking some plugins on it.
  - hardwaresofton 4y ago
    > Citus in PostgreSQL main would be great. Currently Citus is one of many solution but being in main would make it the solution. PostgreSQL would officially be a distributed database, not just by stacking some plugins on it. Scaling concurrent transactions (locally and horizontally across machines) has been one of the valid complaints about postgres for so long, it would be amazing to see it solved so well/reliably in-tree. This is one of the great things about pg to me -- the cottage industry of consultancies/companies that make truly awesome stuff and build viable businesses is seriously amazing. Most times they merge changes upstream (if only for the clout), and sometimes they just sprinkle gold dust on all of us. As a sidenote, I have my M$ misgivings like any other dev of a certain age, but it is amazing that they chose to do that. Not only did they provide an exit to one of the awesome companies in the space, they also released such a huge benefit to the community for free. They'll probably get their return back retaining the Citus people inside Microsoft, since obviously they're the ones making the magic.
- chippiewill 4y ago
  > All you need is Postgres. IIRC Gitlab uses Redis as well, it get used by CI for the streaming logs while a job is still running before it gets tossed up to object storage. Their statement that they only used a single cluster up to this point is a bit dated as well, I believe they've had a second cluster for nearly a year now for their new docker registry implementation. This new cluster for CI is more like a third.
  - boleary-gl 4y ago
    > IIRC GitLab uses Redis as well GitLab employee here. You're correct. We also have Redis for a few things: Rails caching, Sidekiq jobs, session data, CI, and a few other things. You can learn more about GitLab.com's infrastructure architecture here [1] and more about what Redis is used for here [2]. [1]: https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/ https://about.gitlab.com/handbook/engineering/infrastructure... [2]: https://docs.gitlab.com/ee/development/redis.html https://docs.gitlab.com/ee/development/redis.html
hardwaresofton 4y ago
Sorry all -- accidentally left the URL in the input... Would love to delete this thread in favor of https://news.ycombinator.com/item?id=31956876 https://news.ycombinator.com/item?id=31956876 I guess deleting the other one in favor of this works as well, but feels like more discussion than a link per-say. [EDIT] paging @dang anyway though I assume everyone is enjoying their weekend
leipert 4y ago
The change is already completed: https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083... Looking forward to follow-up blog posts/data. (Disclaimer: working for GitLab, not in areas related to this)
- chippiewill 4y ago
  > Disclaimer: working for GitLab, not in areas related to this Just an FYI this is a disclosure rather than a disclaimer, you're disclosing you work for Gitlab rather than disclaiming legal responsibility for something (common mistake to mix these up).
top_sigrid 4y ago
Would be interesting to know if and to which extend the quite new Multiple Databases feature of ActiveRecord [1] (introduced with Rails 6) helped achieving this. - [1] https://guides.rubyonrails.org/active_record_multiple_databases.html https://guides.rubyonrails.org/active_record_multiple_databa...
- dnsmichi 4y ago
  GitLab team member here. Thanks for asking - I have forwarded your question; sharing a summary what I learned below. Note that I am not a Rails engineer; for deeper questions I'd suggest commenting into the linked issues and tag engineers directly :) Rails 6+ multiple database support is being used by GitLab. The major difference is that a hand-rolled DB load balancer is currently used. [0] tracks the effort to change the load balancer to the native Rails implementation for connection handling. This comment summarizes [1] how things worked, and how Rails' support of multiple databases can help. It also provides flow diagrams and code snippets for better understanding. Side note: The multiple database feature effort is also related to bringing Clickhouse as a datastore for Error Tracking [2] and more Observability data [3]. [0] https://gitlab.com/gitlab-org/gitlab/-/issues/296870 https://gitlab.com/gitlab-org/gitlab/-/issues/296870 [1] https://gitlab.com/gitlab-org/gitlab/-/issues/296870#note_889536832 https://gitlab.com/gitlab-org/gitlab/-/issues/296870#note_88... [2] https://gitlab.com/groups/gitlab-org/-/epics/7772 https://gitlab.com/groups/gitlab-org/-/epics/7772 [3] https://about.gitlab.com/direction/monitor/observability/#unified-data-store https://about.gitlab.com/direction/monitor/observability/#un...
  - top_sigrid 4y ago
    Wow, thanks a lot!
- semiquaver 4y ago
  Interestingly enough this functionality in rails was partly developed by GitHub employees while we went through a similar process a few years ago.
wdb 4y ago
Anyone else experience slow downs since Friday when running CI jobs?
- factorialboy 4y ago
  Based on my weekend hacking, CI experience has been the same for me. But MRs loaded up much faster. Disclosure: GitLab team member, but I was hacking on side projects last weekend.
  - wdb 4y ago
    Here it feels jobs got 1/4 slower than a few weeks ago. Also having disk space issues when saving caches when using the shared Gitlab.com runners
  - ctxc 4y ago
    Out of curiosity, what side projects? :)
KronisLV 4y ago
You know, this seems like a perfect example of what a "simple" approach to scaling is: realizing that CI is the most likely thing to cause issues in the future due to how popular it is and then splitting the database along that particular boundary, due to the actual need that has arisen. You don't always need to experiment with bleeding edge tech or very complicated multi-leader clusters (though those also have their own use cases at a certain scale), sometimes just splitting the whole thing, especially when using something as solid as PostgreSQL, is enough!
- kevincox 4y ago
  While I agree that this is a wonderfully simple case the main problem is that it isn't always that easy. As long as you get get away slicing tables into different "shards" and aren't to join-heavy it isn't too baf. However it still adds complexity to the application code that is a constant maintenance cost. And eventually if you keep scaling you will get to the point where you need to shard a single table, and that is very painful. Both in the risk and difficulty of the conversion and further ongoing development costs. I do dream of the day when distributed databases are the "default" for new projects. Likely running just a single instance at the beginning. But then you have a seamless path forward when (and if) you need it. Even at medium size three small nodes will be easy to manage and give easy HA and allow you to do version updates safely and with no downtime. I don't think we are there yet, but there are a few contenders in the running but Postgres is tough competition. Its years of stability and predictability give it huge points even if it has the downsides of a centralized system. But I think the turning point is steadily approaching.
  - herrkanin 4y ago
    Which databases are the main contenders, that you were referring to?
    - kevincox 4y ago
      Disclaimer: I now work at https://www.yugabyte.com/yugabytedb/ https://www.yugabyte.com/yugabytedb/, I joined because I liked their approach and think it has a lot of merit. But there are also https://www.cockroachlabs.com/product/ https://www.cockroachlabs.com/product/, https://github.com/pingcap/tidb https://github.com/pingcap/tidb and https://ydb.tech/ https://ydb.tech/ which each has promise. There is definitely lots of progress to make in this space. From what I have seen they are all significantly slower on a single node and are not as battle-hardened. But I think with time we will have a few really nice options to pick from.
bram2w 4y ago
Is anyone also experiencing problems with purchasing additional CI minutes? I bought extra minutes yesterday and this morning, but they've not been added to my account. Before this worked instantly. It's blocking some of the development processes for us because the pipelines don't run right now, we've ran out of minutes. I'm curious whether it's related to splitting the database. If anyone from GitLab is reading this, it's for the public repository of Baserow https://gitlab.com/bramw/baserow https://gitlab.com/bramw/baserow. Help would be much appreciated.
- dnsmichi 4y ago
  GitLab team member here. Sorry for the troubles, and thanks for sharing here. I have forwarded your comment to engineering teams to investigate if related to the migration. To avoid blockers, I'd also suggest opening a support ticket [0] to let support and billing teams know and escalate. Include this HN comment and my name if you like. [0] https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293 https://support.gitlab.com/hc/en-us/requests/new?ticket_form...
  - bram2w 4y ago
    Thanks for forwarding my comment, it's much appreciated! I opened a ticket a while ago ago, I'll add this comment to the conversation. We're working on setting up our runners in the meantime.
    - dnsmichi 4y ago
      Thanks for opening the ticket already :-) I have talked with the fulfillment team at GitLab [0], and there was a problem identified with workers not syncing the purchased CI minutes after the DB migration. Corrections have been made, and potentially you should see the synced CI minutes soon. Support teams have access and can check with you, I do not have access. Setting up your own runners is a sensible approach, and allows you to scale them for your own needs too. Maybe this workshop about pipeline efficiency can help with more ideas and insights [1]. [0] https://about.gitlab.com/handbook/engineering/development/fulfillment/ https://about.gitlab.com/handbook/engineering/development/fu... [1] https://about.gitlab.com/handbook/marketing/community-relations/developer-evangelism/projects/#efficient-devsecops-pipelines-in-a-cloud-native-world https://about.gitlab.com/handbook/marketing/community-relati...
ismayilzadan 4y ago
Why something like this requires a downtime? Could not both databases be utilised at the same time while backfilling older entries in parallel and once all data migrated flip a feature switch to stop writing to old db? Reades can be switched with a flip too, or some smart logic to check both databases if data is not found. This approach is hard to implement and overall migration would take longer, but considering many companies depending on Gitlab SC, this should be the preferred approach I think.
- dnsmichi 4y ago
  GitLab team member here. The comment [0] provides more insights into the planning and downtime requirements. The epic itself may be helpful too, it is linked from the blog post. [0] https://gitlab.com/groups/gitlab-org/-/epics/7791#note_941021426 https://gitlab.com/groups/gitlab-org/-/epics/7791#note_94102...
- terom 4y ago
  Swallowing an hour or two of pre-scheduled downtime can be worth it if you can significantly reduce the complexity and risks associated with a migration, and get it over and done with sooner. Particularly if you're already hurting from whatever it is that you're fixing, and it's about to cause you unscheduled downtime.
- colechristensen 4y ago
  It is always possible but it can be very much more complex and if a mistake is made you could end up with much more downtime or data loss. I’m fine with planned downtime and usually not willing to pay for service providers to do everything for absolute minimum downtime.

DISCLAIMER: I do some paid work with with Open Core Ventures[0] and I've been a GitLab fan for a while so I'm quite biased. Just tried to use some GitLab repos I have and ran into a well specified 503 error page: > We are splitting our database into Main and CI! > > For more information about what we're up to, you can check out our blog article. For progress updates, please check our status page. https://about.gitlab.com/blog/2022/06/02/splitting-database-into-main-and-ci/ https://about.gitlab.com/blog/2022/06/02/splitting-database-... Don't know how I missed it, but it's pretty nice that the downtime is clearly specified, and the blog post (and related posts) were very informative. Some random thoughts below: > GitLab.com's database architecture uses a single PostgreSQL database cluster. This single cluster (let's call it main), consists of a single primary and multiple read-only replicas and stores the data generated by all GitLab features. Database reads can be scaled horizontally through read-only replicas, but writes cannot because PostgreSQL does not support active-active replication natively. All you need is Postgres. IIRC Reddit had a similar starting story. Excited to see how much faster gitlab.com will be and how much more reliable their CI gets (I don't have numbers on hand, but I think they've gone down less than GH Actions in recent memory at least -- maybe there's some recency bias there). In a world where Citus is now fully open source[1], I wonder if it needs to just get pulled into contrib. [0]: https://opencoreventures.com/ https://opencoreventures.com/ [1]: https://www.citusdata.com/blog/2022/06/17/citus-11-goes-fully-open-source/ https://www.citusdata.com/blog/2022/06/17/citus-11-goes-full...

tpetry 4y ago
Citus in PostgreSQL main would be great. Currently Citus is one of many solution but being in main would make it the solution. PostgreSQL would officially be a distributed database, not just by stacking some plugins on it.
- hardwaresofton 4y ago
  > Citus in PostgreSQL main would be great. Currently Citus is one of many solution but being in main would make it the solution. PostgreSQL would officially be a distributed database, not just by stacking some plugins on it. Scaling concurrent transactions (locally and horizontally across machines) has been one of the valid complaints about postgres for so long, it would be amazing to see it solved so well/reliably in-tree. This is one of the great things about pg to me -- the cottage industry of consultancies/companies that make truly awesome stuff and build viable businesses is seriously amazing. Most times they merge changes upstream (if only for the clout), and sometimes they just sprinkle gold dust on all of us. As a sidenote, I have my M$ misgivings like any other dev of a certain age, but it is amazing that they chose to do that. Not only did they provide an exit to one of the awesome companies in the space, they also released such a huge benefit to the community for free. They'll probably get their return back retaining the Citus people inside Microsoft, since obviously they're the ones making the magic.
chippiewill 4y ago
> All you need is Postgres. IIRC Gitlab uses Redis as well, it get used by CI for the streaming logs while a job is still running before it gets tossed up to object storage. Their statement that they only used a single cluster up to this point is a bit dated as well, I believe they've had a second cluster for nearly a year now for their new docker registry implementation. This new cluster for CI is more like a third.
- boleary-gl 4y ago
  > IIRC GitLab uses Redis as well GitLab employee here. You're correct. We also have Redis for a few things: Rails caching, Sidekiq jobs, session data, CI, and a few other things. You can learn more about GitLab.com's infrastructure architecture here [1] and more about what Redis is used for here [2]. [1]: https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/ https://about.gitlab.com/handbook/engineering/infrastructure... [2]: https://docs.gitlab.com/ee/development/redis.html https://docs.gitlab.com/ee/development/redis.html

hardwaresofton 4y ago

Sorry all -- accidentally left the URL in the input... Would love to delete this thread in favor of https://news.ycombinator.com/item?id=31956876 https://news.ycombinator.com/item?id=31956876 I guess deleting the other one in favor of this works as well, but feels like more discussion than a link per-say. [EDIT] paging @dang anyway though I assume everyone is enjoying their weekend

leipert 4y ago

The change is already completed: https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083... Looking forward to follow-up blog posts/data. (Disclaimer: working for GitLab, not in areas related to this)

chippiewill 4y ago
> Disclaimer: working for GitLab, not in areas related to this Just an FYI this is a disclosure rather than a disclaimer, you're disclosing you work for Gitlab rather than disclaiming legal responsibility for something (common mistake to mix these up).

top_sigrid 4y ago

Would be interesting to know if and to which extend the quite new Multiple Databases feature of ActiveRecord [1] (introduced with Rails 6) helped achieving this. - [1] https://guides.rubyonrails.org/active_record_multiple_databases.html https://guides.rubyonrails.org/active_record_multiple_databa...

dnsmichi 4y ago
GitLab team member here. Thanks for asking - I have forwarded your question; sharing a summary what I learned below. Note that I am not a Rails engineer; for deeper questions I'd suggest commenting into the linked issues and tag engineers directly :) Rails 6+ multiple database support is being used by GitLab. The major difference is that a hand-rolled DB load balancer is currently used. [0] tracks the effort to change the load balancer to the native Rails implementation for connection handling. This comment summarizes [1] how things worked, and how Rails' support of multiple databases can help. It also provides flow diagrams and code snippets for better understanding. Side note: The multiple database feature effort is also related to bringing Clickhouse as a datastore for Error Tracking [2] and more Observability data [3]. [0] https://gitlab.com/gitlab-org/gitlab/-/issues/296870 https://gitlab.com/gitlab-org/gitlab/-/issues/296870 [1] https://gitlab.com/gitlab-org/gitlab/-/issues/296870#note_889536832 https://gitlab.com/gitlab-org/gitlab/-/issues/296870#note_88... [2] https://gitlab.com/groups/gitlab-org/-/epics/7772 https://gitlab.com/groups/gitlab-org/-/epics/7772 [3] https://about.gitlab.com/direction/monitor/observability/#unified-data-store https://about.gitlab.com/direction/monitor/observability/#un...
- top_sigrid 4y ago
  Wow, thanks a lot!
semiquaver 4y ago
Interestingly enough this functionality in rails was partly developed by GitHub employees while we went through a similar process a few years ago.

wdb 4y ago

Anyone else experience slow downs since Friday when running CI jobs?

factorialboy 4y ago
Based on my weekend hacking, CI experience has been the same for me. But MRs loaded up much faster. Disclosure: GitLab team member, but I was hacking on side projects last weekend.
- wdb 4y ago
  Here it feels jobs got 1/4 slower than a few weeks ago. Also having disk space issues when saving caches when using the shared Gitlab.com runners
- ctxc 4y ago
  Out of curiosity, what side projects? :)

KronisLV 4y ago

You know, this seems like a perfect example of what a "simple" approach to scaling is: realizing that CI is the most likely thing to cause issues in the future due to how popular it is and then splitting the database along that particular boundary, due to the actual need that has arisen. You don't always need to experiment with bleeding edge tech or very complicated multi-leader clusters (though those also have their own use cases at a certain scale), sometimes just splitting the whole thing, especially when using something as solid as PostgreSQL, is enough!

kevincox 4y ago
While I agree that this is a wonderfully simple case the main problem is that it isn't always that easy. As long as you get get away slicing tables into different "shards" and aren't to join-heavy it isn't too baf. However it still adds complexity to the application code that is a constant maintenance cost. And eventually if you keep scaling you will get to the point where you need to shard a single table, and that is very painful. Both in the risk and difficulty of the conversion and further ongoing development costs. I do dream of the day when distributed databases are the "default" for new projects. Likely running just a single instance at the beginning. But then you have a seamless path forward when (and if) you need it. Even at medium size three small nodes will be easy to manage and give easy HA and allow you to do version updates safely and with no downtime. I don't think we are there yet, but there are a few contenders in the running but Postgres is tough competition. Its years of stability and predictability give it huge points even if it has the downsides of a centralized system. But I think the turning point is steadily approaching.
- herrkanin 4y ago
  Which databases are the main contenders, that you were referring to?
  - kevincox 4y ago
    Disclaimer: I now work at https://www.yugabyte.com/yugabytedb/ https://www.yugabyte.com/yugabytedb/, I joined because I liked their approach and think it has a lot of merit. But there are also https://www.cockroachlabs.com/product/ https://www.cockroachlabs.com/product/, https://github.com/pingcap/tidb https://github.com/pingcap/tidb and https://ydb.tech/ https://ydb.tech/ which each has promise. There is definitely lots of progress to make in this space. From what I have seen they are all significantly slower on a single node and are not as battle-hardened. But I think with time we will have a few really nice options to pick from.

bram2w 4y ago

Is anyone also experiencing problems with purchasing additional CI minutes? I bought extra minutes yesterday and this morning, but they've not been added to my account. Before this worked instantly. It's blocking some of the development processes for us because the pipelines don't run right now, we've ran out of minutes. I'm curious whether it's related to splitting the database. If anyone from GitLab is reading this, it's for the public repository of Baserow https://gitlab.com/bramw/baserow https://gitlab.com/bramw/baserow. Help would be much appreciated.

dnsmichi 4y ago
GitLab team member here. Sorry for the troubles, and thanks for sharing here. I have forwarded your comment to engineering teams to investigate if related to the migration. To avoid blockers, I'd also suggest opening a support ticket [0] to let support and billing teams know and escalate. Include this HN comment and my name if you like. [0] https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293 https://support.gitlab.com/hc/en-us/requests/new?ticket_form...
- bram2w 4y ago
  Thanks for forwarding my comment, it's much appreciated! I opened a ticket a while ago ago, I'll add this comment to the conversation. We're working on setting up our runners in the meantime.
  - dnsmichi 4y ago
    Thanks for opening the ticket already :-) I have talked with the fulfillment team at GitLab [0], and there was a problem identified with workers not syncing the purchased CI minutes after the DB migration. Corrections have been made, and potentially you should see the synced CI minutes soon. Support teams have access and can check with you, I do not have access. Setting up your own runners is a sensible approach, and allows you to scale them for your own needs too. Maybe this workshop about pipeline efficiency can help with more ideas and insights [1]. [0] https://about.gitlab.com/handbook/engineering/development/fulfillment/ https://about.gitlab.com/handbook/engineering/development/fu... [1] https://about.gitlab.com/handbook/marketing/community-relations/developer-evangelism/projects/#efficient-devsecops-pipelines-in-a-cloud-native-world https://about.gitlab.com/handbook/marketing/community-relati...

ismayilzadan 4y ago

Why something like this requires a downtime? Could not both databases be utilised at the same time while backfilling older entries in parallel and once all data migrated flip a feature switch to stop writing to old db? Reades can be switched with a flip too, or some smart logic to check both databases if data is not found. This approach is hard to implement and overall migration would take longer, but considering many companies depending on Gitlab SC, this should be the preferred approach I think.

dnsmichi 4y ago
GitLab team member here. The comment [0] provides more insights into the planning and downtime requirements. The epic itself may be helpful too, it is linked from the blog post. [0] https://gitlab.com/groups/gitlab-org/-/epics/7791#note_941021426 https://gitlab.com/groups/gitlab-org/-/epics/7791#note_94102...
terom 4y ago
Swallowing an hour or two of pre-scheduled downtime can be worth it if you can significantly reduce the complexity and risks associated with a migration, and get it over and done with sooner. Particularly if you're already hurting from whatever it is that you're fixing, and it's about to cause you unscheduled downtime.
colechristensen 4y ago
It is always possible but it can be very much more complex and if a mistake is made you could end up with much more downtime or data loss. I’m fine with planned downtime and usually not willing to pay for service providers to do everything for absolute minimum downtime.