6 ms·

This was a fascinating read but I feel like I’m left with some more questions about what’s going on under the hood in gitlab cloud: 1. I thought gitlab was hos

This was a fascinating read but I feel like I’m left with some more questions about what’s going on under the hood in gitlab cloud: 1. I thought gitlab was hosted in google cloud, but there’s a lot of references to e.g., a hand rolled consensus system and self-managed database clusters. I’m wondering if this event changes the math on build vs. buy at all for gitlab, sounds like a lot of money has gone into this solution. How did that solution come up, is it about some specific Postgres features that aren’t available in googles hosted dbs or pricing? 2. Again on the google cloud angle, why are servers being hand-managed and rebooted? Elasticity in the cloud would make me think that the safest option would be to stand up parallel infrastructure (like in a DR plan) and migrate traffic. Was this just about speed of solution rollout? Does gitlab have plans to harden DR plans so that you can execute in cases like this? Whenever someone says they’re “in the cloud” and yet unable to treat servers like cattle, I get a bit worried.

scarface74 7y ago
I led my first from the ground up greenfield infrastructure + development project on Consul+Nomad+Fabio and I became a big Hashicorp fanboy then. But, if I were in a cloud environment, I definitely wouldn’t use any of those products. As far as cloud, I only have experience with AWS, but there is nothing Consul+Nomad+Fabio would give me that I couldn’t do better with managed services. I’m sure GCP abd Azure are the same.
- vidarh 7y ago
  Then comes the day when you need a hybrid system that spans clouds, and your managed services have suddenly become a hindrance. That is where these services are valuable. I've run systems with Consul + a VPN that we did zero downtime migration of first from AWS to GCP, and then to managed servers, and once you do that you start appreciating platforms that you can replicate. This was prior to Nomad. Note that this of course means you can use managed services if you're careful and they're set up in ways that lets you do seamless migration - most importantly avoiding any proprietary features. [the way we did the migrations were nothing fancy: establish a VPN between the two sites; use load balancers updating from the health checks for all services; add the new servers into the cluster; drain the old ones once the new ones were up; point DNS at the new external load balancers; but being able to easily have Consul span different cloud environments made it a lot easier]
  - scarface74 7y ago
    So you’re avoiding “proprietary features” of the cloud. Congratulations! Now you have the worse of all worlds. You’re spending more on resources than you would at a colo, spending just as much on babysitting infrastructure and you’re not saving any money or time by outsourcing “the undifferentiated heavy lifting”. You can’t imagine how many times I have seen “software architects” use the “repository pattern”, interfaces, and be sure not to use any “proprietary features” just in case one day their CTO decides to move away from their six figure a year Oracle installation to Mysql. If being cross platform or cloud is a requirement in the beginning, you have to constantly test on all of your supported platforms. If not, it’s a waste of time to design for some unknown eventuality most of the time. In reality, at a certain size, you’re always tied into your infrastructure and few companies move from one cloud provider to another. The expense, the chance of regressions are too high and the rewards are too low. But to the Consul+Fabio use explicitly. I’ve duplicated the functionality: The KV Store: A DynamoDB table behind an API. The API can be used from anywhere. Health Checks/restarts/service discovery: Fargate (Serverless docker) with a load balancer/target groups that do health checks and automatically launch new Docker instances or just regular EC2 instances behind an autoscaling group. Nomad: we used Nomad because I didn’t want to introduce Docker and it can orchestrate anything. These days I would just use Docker or of it was simple enough lambda triggered by either scheduled events or other AWS events like queues or SNS messages.
    - vidarh 7y ago
      I tend to avoid cloud deployments in the first place, exactly because it is usually far more expensive than even managed hosting, even if you don't go all the way to a colo. And I've yet to do a migration off cloud services that didn't result in substantial cost reductions (including when factoring in devops costs). But sometimes it's impossible to convince people of this, even with detailed cost breakdowns, and/or people have other reasons for using cloud providers, some more rational than others. My current system is hosted on AWS because it's convenient and it won't grow large enough (in terms of resource use) anytime soon to be a cost issue relative to the engineering resources put into developing it. But what you can do in those instances is to plan out an architecture that ensures you maintain flexibility and minimizes the impact of changes while taking advantage of what you can. That does not imply avoiding all proprietary solutions is always the right choice. It means avoiding solutions that are particularly onerous to transition off, and to limit different services exposure to the proprietary aspects. How far you take this certainly greatly depends on how likely you are to migrate. My experience is different than yours - a very substantial proportion of the systems I've been contracted to work on have involved migrations between providers sooner or later. (But part of that may well have been architectures that made migrations simple and cheap. In one case the company in question was explicitly constantly chasing the cheapest deal, and so were on AWS and then GCP thanks to huge amounts of free credits, and moved to Hetzner the moment they'd used them up; the credits added up to at least two magnitudes more than the cost of the migrations) E.g. for anything deployed in AWS I'll happily use RDS because it's no harder to migrate off an RDS Postgres instance than migrating any other postgres instance, and the servers talking to it does not need to know whether they're talking to an RDS based instance or something that has been moved to another cloud provider. And I'll happily use many services that are fully AWS proprietary too like SNS ans SQS because their interfaces are narrow enough that migration tends to be relatively easy, and that the actual interaction with AWS can be narrowed down to a limited set of services. The point is not to be able to move on a seconds notice, but to avoid pointless coupling. Being cross platform/cloud is rarely a requirement; not making decisions that make it harder to migrate in the future when faced with choices where picking the cloud agnostic solution is about the same cost/complexity as the cloud specific one, often is. Using things like Fargate can fit perfectly fine into that kind of architecture.
- brodock 7y ago
  We just run the same thing our customers run on-premise. Some companies that allow you to run one instance of whatever they make, would usually tie that to a single cloud provider. That simplifies their operation but also reduces the potential customers. Not everyone may be using the same cloud provider, not everyone may be able to run on the cloud or even may have OS/distro restrictions. As always it's a matter of tradeoffs.
thaniri 7y ago
1. The consensus system you are talking about is Raft (https://raft.github.io/ https://raft.github.io/) and it's baked into Consul. 2. There are two ways to interpret parallel infrastructure. I will post my thoughts on both. 2a. Standing up a parallel Consul cluster. This is problematic because typically what people do with Consul is to put a Consul agent on every server (or pod) which registers its services for discovery to the Consul servers. When you make a parallel Consul cluster you need to also restart the Consul agents on every other service. They only mention Postgres in this blog post but there can potentially be a LOT of other servers registered to the Consul cluster. 2b. Standing up everything it takes to run Gitlab in parallel and then diverting traffic. Honestly sounds great. The reason a team wouldn't do this is due to either a) not having infrastructure code which allows for one-click deployment of whatever it takes to have Gitlab running. or b) it's actually pretty expensive to do if you're not Google or Amazon. The blog post mentions 255 clients (and 5 Consul servers). That's a lot of servers to rebuild! Now, I would love to hear from anyone else who uses Consul, because I have my own thoughts on how they decided to handle the issue. I will focus my attention entirely on the Consul and not the Postgres portion. The blog post mentions two limitations: 1) Reloading the configuration of the running service, which worked fine and did not drop connections, but the certificate settings are not included in the reloadable settings for our version of Consul. 2) Simultaneous restarts of various services, which worked, but our tools wouldn't allow us to do that with ALL of the nodes at once. We don't need to reload. We can run a rolling systemctl restart whicn ansible is perfect for. The nice thing about this is that their stop-gap solution is to disable TLS verification. This means that servers with TLS verification ON in the meantime should be able to continue validating certs while other servers can have a rolling restart that disables TLS verification one server at a time. If we want to minimize downtime we would do every non-leader server in the cluster, then finally the leader, then every client in a serial manner. With 260 servers to deal with it would be slow but it shouldn't break Raft at any point. There is no reason for quorum to be broken. The gossip will still be communicated over TLS, just that some of the servers/clients wouldn't be validating the certs. Then, we would follow exactly the same process for rolling out valid certificates with TLS validation turned back on. One non-leader server at a time, then the leader, then every client. I could be missing some critical piece here, and it looks like the Gitlab team did run a lab test before making their change in prod. It's easy to miss a possibility when under pressure, and also easy for an online commentator like myself to think they are so much smarter. They still managed to get out of the crisis with no downtime and congrats to the operators who pulled it off!
lima 7y ago
Keep in mind that Gitlab can also be deployed on-premise, using (presumably) the same HA mechanism: https://docs.gitlab.com/ee/administration/high_availability/README.html https://docs.gitlab.com/ee/administration/high_availability/... Locking a product and company like Gitlab into a single cloud vendor's proprietary orchestrator would be a massive risk. Sure, if you're an early stage SaaS startup whose only metric is growth, use whatever gets your infrastructure up and running with as little downtime as possible. In any other situation, locking yourself into a single cloud provider is a risk that needs to be carefully evaluated.

scarface74 7y ago

I led my first from the ground up greenfield infrastructure + development project on Consul+Nomad+Fabio and I became a big Hashicorp fanboy then. But, if I were in a cloud environment, I definitely wouldn’t use any of those products. As far as cloud, I only have experience with AWS, but there is nothing Consul+Nomad+Fabio would give me that I couldn’t do better with managed services. I’m sure GCP abd Azure are the same.

vidarh 7y ago
Then comes the day when you need a hybrid system that spans clouds, and your managed services have suddenly become a hindrance. That is where these services are valuable. I've run systems with Consul + a VPN that we did zero downtime migration of first from AWS to GCP, and then to managed servers, and once you do that you start appreciating platforms that you can replicate. This was prior to Nomad. Note that this of course means you can use managed services if you're careful and they're set up in ways that lets you do seamless migration - most importantly avoiding any proprietary features. [the way we did the migrations were nothing fancy: establish a VPN between the two sites; use load balancers updating from the health checks for all services; add the new servers into the cluster; drain the old ones once the new ones were up; point DNS at the new external load balancers; but being able to easily have Consul span different cloud environments made it a lot easier]
- scarface74 7y ago
  So you’re avoiding “proprietary features” of the cloud. Congratulations! Now you have the worse of all worlds. You’re spending more on resources than you would at a colo, spending just as much on babysitting infrastructure and you’re not saving any money or time by outsourcing “the undifferentiated heavy lifting”. You can’t imagine how many times I have seen “software architects” use the “repository pattern”, interfaces, and be sure not to use any “proprietary features” just in case one day their CTO decides to move away from their six figure a year Oracle installation to Mysql. If being cross platform or cloud is a requirement in the beginning, you have to constantly test on all of your supported platforms. If not, it’s a waste of time to design for some unknown eventuality most of the time. In reality, at a certain size, you’re always tied into your infrastructure and few companies move from one cloud provider to another. The expense, the chance of regressions are too high and the rewards are too low. But to the Consul+Fabio use explicitly. I’ve duplicated the functionality: The KV Store: A DynamoDB table behind an API. The API can be used from anywhere. Health Checks/restarts/service discovery: Fargate (Serverless docker) with a load balancer/target groups that do health checks and automatically launch new Docker instances or just regular EC2 instances behind an autoscaling group. Nomad: we used Nomad because I didn’t want to introduce Docker and it can orchestrate anything. These days I would just use Docker or of it was simple enough lambda triggered by either scheduled events or other AWS events like queues or SNS messages.
  - vidarh 7y ago
    I tend to avoid cloud deployments in the first place, exactly because it is usually far more expensive than even managed hosting, even if you don't go all the way to a colo. And I've yet to do a migration off cloud services that didn't result in substantial cost reductions (including when factoring in devops costs). But sometimes it's impossible to convince people of this, even with detailed cost breakdowns, and/or people have other reasons for using cloud providers, some more rational than others. My current system is hosted on AWS because it's convenient and it won't grow large enough (in terms of resource use) anytime soon to be a cost issue relative to the engineering resources put into developing it. But what you can do in those instances is to plan out an architecture that ensures you maintain flexibility and minimizes the impact of changes while taking advantage of what you can. That does not imply avoiding all proprietary solutions is always the right choice. It means avoiding solutions that are particularly onerous to transition off, and to limit different services exposure to the proprietary aspects. How far you take this certainly greatly depends on how likely you are to migrate. My experience is different than yours - a very substantial proportion of the systems I've been contracted to work on have involved migrations between providers sooner or later. (But part of that may well have been architectures that made migrations simple and cheap. In one case the company in question was explicitly constantly chasing the cheapest deal, and so were on AWS and then GCP thanks to huge amounts of free credits, and moved to Hetzner the moment they'd used them up; the credits added up to at least two magnitudes more than the cost of the migrations) E.g. for anything deployed in AWS I'll happily use RDS because it's no harder to migrate off an RDS Postgres instance than migrating any other postgres instance, and the servers talking to it does not need to know whether they're talking to an RDS based instance or something that has been moved to another cloud provider. And I'll happily use many services that are fully AWS proprietary too like SNS ans SQS because their interfaces are narrow enough that migration tends to be relatively easy, and that the actual interaction with AWS can be narrowed down to a limited set of services. The point is not to be able to move on a seconds notice, but to avoid pointless coupling. Being cross platform/cloud is rarely a requirement; not making decisions that make it harder to migrate in the future when faced with choices where picking the cloud agnostic solution is about the same cost/complexity as the cloud specific one, often is. Using things like Fargate can fit perfectly fine into that kind of architecture.
brodock 7y ago
We just run the same thing our customers run on-premise. Some companies that allow you to run one instance of whatever they make, would usually tie that to a single cloud provider. That simplifies their operation but also reduces the potential customers. Not everyone may be using the same cloud provider, not everyone may be able to run on the cloud or even may have OS/distro restrictions. As always it's a matter of tradeoffs.

thaniri 7y ago

1. The consensus system you are talking about is Raft (https://raft.github.io/ https://raft.github.io/) and it's baked into Consul. 2. There are two ways to interpret parallel infrastructure. I will post my thoughts on both. 2a. Standing up a parallel Consul cluster. This is problematic because typically what people do with Consul is to put a Consul agent on every server (or pod) which registers its services for discovery to the Consul servers. When you make a parallel Consul cluster you need to also restart the Consul agents on every other service. They only mention Postgres in this blog post but there can potentially be a LOT of other servers registered to the Consul cluster. 2b. Standing up everything it takes to run Gitlab in parallel and then diverting traffic. Honestly sounds great. The reason a team wouldn't do this is due to either a) not having infrastructure code which allows for one-click deployment of whatever it takes to have Gitlab running. or b) it's actually pretty expensive to do if you're not Google or Amazon. The blog post mentions 255 clients (and 5 Consul servers). That's a lot of servers to rebuild! Now, I would love to hear from anyone else who uses Consul, because I have my own thoughts on how they decided to handle the issue. I will focus my attention entirely on the Consul and not the Postgres portion. The blog post mentions two limitations: 1) Reloading the configuration of the running service, which worked fine and did not drop connections, but the certificate settings are not included in the reloadable settings for our version of Consul. 2) Simultaneous restarts of various services, which worked, but our tools wouldn't allow us to do that with ALL of the nodes at once. We don't need to reload. We can run a rolling systemctl restart whicn ansible is perfect for. The nice thing about this is that their stop-gap solution is to disable TLS verification. This means that servers with TLS verification ON in the meantime should be able to continue validating certs while other servers can have a rolling restart that disables TLS verification one server at a time. If we want to minimize downtime we would do every non-leader server in the cluster, then finally the leader, then every client in a serial manner. With 260 servers to deal with it would be slow but it shouldn't break Raft at any point. There is no reason for quorum to be broken. The gossip will still be communicated over TLS, just that some of the servers/clients wouldn't be validating the certs. Then, we would follow exactly the same process for rolling out valid certificates with TLS validation turned back on. One non-leader server at a time, then the leader, then every client. I could be missing some critical piece here, and it looks like the Gitlab team did run a lab test before making their change in prod. It's easy to miss a possibility when under pressure, and also easy for an online commentator like myself to think they are so much smarter. They still managed to get out of the crisis with no downtime and congrats to the operators who pulled it off!

lima 7y ago

Keep in mind that Gitlab can also be deployed on-premise, using (presumably) the same HA mechanism: https://docs.gitlab.com/ee/administration/high_availability/README.html https://docs.gitlab.com/ee/administration/high_availability/... Locking a product and company like Gitlab into a single cloud vendor's proprietary orchestrator would be a massive risk. Sure, if you're an early stage SaaS startup whose only metric is growth, use whatever gets your infrastructure up and running with as little downtime as possible. In any other situation, locking yourself into a single cloud provider is a risk that needs to be carefully evaluated.