3 ms·
Ask HN: Best practices for multi-datacenter infrastructure?
I'm deploying a set of applications on top of Kubernetes like GitLab and Argo suite (Workflows, CD, Rollouts), Harbor, Prometheus, and more and I need to have an active-active solution distributed across two data centers for all (if possible) of these tools.
One approach I'm investigating having Kafka queueing the commits (and other requests) and one consumer on each data center writing the commit into GitLab to execute CI/CD pipelines and deploy applications based on that commit. (CI/CD will only execute once)
This way I don't have to worry about replicating each application data from one data center to another and worry about where data is being written and so on.
I want to ask HN, are there best practices or documentation on how to do this kind of architecture that you can point me at?
Or is there a better way to work on this solution?
- wmf 5y agoModifying each application to add active-active replication sounds like a nightmare. I would seriously question where these requirements are coming from.
- __warlord__ 5y agoIs not modifying for active-active __per se__ but we want to have as much uptime as possible. Async replication is another option but that needs to be enabled per application and on each database cluster we use. That's why I'm thinking to have active-active using a queue (to simplify replication and being able to have a "journal" that we can do replays from.
- NicoJuicy 5y agoI did a quick check, but I'm not an expert. Make one the main and the backup in readonly. https://docs.gitlab.com/ee/administration/read_only_gitlab.html https://docs.gitlab.com/ee/administration/read_only_gitlab.h... You can stop the readonly from doing CI. Do a healthcheck on the main repo and let a gateway in front it. If the main is down, undo the readonly access. I'm not 100% sure what to do when the main goes backup though. That aside, I'm not sure if you want this. What duration/year was a downtime on the datacenter blocking deployments/coding?
- __warlord__ 5y ago> I'm not 100% sure what to do when the main goes backup though. I'm wondering this as well, it becomes very cumbersome to bring both instances at the same data consistency without a third party (like a queue)
- NicoJuicy 5y agoI don't think a queue is possible concerning commits. This could be part of the solution though. But automating everything seems not worth the effort. https://docs.gitlab.com/ee/administration/geo/disaster_recovery/ https://docs.gitlab.com/ee/administration/geo/disaster_recov...