5 ms·

At GitLab we're making the application ready for Kubernetes, we have not switched yet. It required us to untangle many parts. For example we used to store uploa

by sytse 9y ago

At GitLab we're making the application ready for Kubernetes, we have not switched yet. It required us to untangle many parts. For example we used to store uploads on disk before moving them to object storage, now it goes directly to object storage. There where many interactions between our applications https://docs.gitlab.com/ee/development/architecture.html#components https://docs.gitlab.com/ee/development/architecture.html#com... over disk or sockets that we need to clean up. When its done we expect to be able to scale the different parts of our application independently. This also makes it easier to detect problems (why did Gitaly suddenly autoscale to twice the number of containers).

hatred 9y ago
Curious, did you evaluate any other similar technologies like Swarm/(Mesos+Marathon) etc.? What made you eventually go ahead with K8s other than the main obvious reason (the community) itself?
- twk3 9y ago
  At GitLab, we did evaluate Mesosphere DC/OS. What turned our focus to Kubernetes as our primary cluster install target was the speed at which it was developing, and after watching the space and talking to partners/customers we formed the opinion that Kubernetes was going to lead the pack. We've been looking at these technologies for two years now, with our focus being Kubernetes for the last one.
- joshlambert 9y ago
  GitLab has been investing significantly in Kubernetes, both because we believe in the platform but also because we see significant demand from our customers as well. It's ability to run on-premise as well as availability in a wide variety of managed cloud flavors is a huge benefit, and likely a driver of the demand. We also try to use the same deployment tools for GitLab.com that we provide to customers, and this lets us offer a scalable production-grade deployment method that can run nearly anywhere.
- WarheadsSE 9y ago
  As twk3 has said, we have examined multiple other options. One big reason we've chosen to implement with Kubernetes is community adoption. You can now run Kubernetes on AWS, GCP, Azure, IBM cloud, as well as a slew of other platforms both on-premise and as a PaaS. With the expanding ecosystem, our decision has only been further confirmed, as the list of available solutions and providers continues to grow rapidly (https://kubernetes.io/docs/setup/pick-right-solution/ https://kubernetes.io/docs/setup/pick-right-solution/) Another big reason is far simpler: ease of use and configuration. A permanent fixture in our work for Cloud Native GitLab has been that this method must be as easy, if not easier to provide a scalable, resilient, highly-available deployment of GitLab at scale. We can’t say, “This is our new suggested method. By the way, it’s harder.” What we have found is that many other solutions require a much larger initial investment of time to understand and configure GitLab as a whole solution, as compared to the combination of Kubernetes with Helm. Helm provides us templating, default value handling, component enable/disable logic, and many other extremely useful features. These allow us to provide our users with a practical, streamlined method of installation and configuration, without the need to spend countless hours reding documentation, deciding on the architecture, and making edits to YAML.
twk3 9y ago
At GitLab, our installation method for the product was also tightly based around chef-solo and chef omnibus. At first we were trying to continue using these technologies inside our containers, but this required our containers to run as root. So a lot of our effort in moving to kubernetes has required us to try and duplicate all of our previous installation work in ways that don't require root access while the containers are running. To help, we've chosen to use Helm to manage versioning/upgrading our application in the cluster: https://helm.sh/ https://helm.sh/
leetbulb 9y ago
Running GitLab CI specifically is absolutely amazing on K8S... Thank you so much for your efforts.
- joshlambert 9y ago
  Thanks, glad you are finding it helpful! We have a lot of improvements coming as well: https://about.gitlab.com/direction/#ci--cd https://about.gitlab.com/direction/#ci--cd
  - ossm1db 9y ago
    If Gitlab could deploy to a play-with-k8s temporary cluster with four hours to test and evaluate, that would be pretty cool.
    - joshlambert 9y ago
      I agree, that would be awesome! While it's not a single button, GitLab is getting closer to that goal with the addition of our cluster integration feature: https://docs.gitlab.com/ee/user/project/clusters/#adding-and-creating-a-new-gke-cluster-via-gitlab https://docs.gitlab.com/ee/user/project/clusters/#adding-and... If you create a trial account on GCP, you can then use GitLab to make a new GKE cluster and it will be automatically linked. You can then click a single button to deploy the Helm Tiller, a GitLab Runner, and then run your CI/CD jobs on your brand new cluster! The GCP trial is pretty nice, you get $300 in credits and you won't be automatically billed at the end: https://cloud.google.com/free/ https://cloud.google.com/free/
WarheadsSE 9y ago
I recently present at Cloud Expo Europe, describing the how and why GitLab has been working to produce a cloud native distribution. I've outlined some of the blockers we've faced, and what we've been working on to resolve these going forward. Those changes benefit both our traditional deployment methodologies, as well as our efforts for deployment on Kubernetes. Journey to Cloud Native: Breaking the monolith and scaling towards tomorrow - https://docs.google.com/presentation/d/1fsgvSuGpn-MnMqKaTOoiRvUvdJFHd3VpoAAfBoMtFYg https://docs.google.com/presentation/d/1fsgvSuGpn-MnMqKaTOoi... Our biggest blockers have been how to separate file system dependency for our various parts. For Git content, we've implemented Gitaly (https://gitlab.com/gitlab-org/gitaly/ https://gitlab.com/gitlab-org/gitaly/). For various other parts, we've been implementing support for object storage across the board. Doing this while keeping up with the rapid growth of Kubernetes and Helm has been a challenge, but entirely worth the effort. We are also bringing all of these changes to GitLab CE, as a part of our Stewardship directive (https://about.gitlab.com/stewardship/ https://about.gitlab.com/stewardship/, https://about.gitlab.com/2016/01/11/being-a-good-open-source-steward/ https://about.gitlab.com/2016/01/11/being-a-good-open-source...). We don't feel that our efforts to allow for greater scalability, resiliency, and platform agnosticism belong only in the Enterprise offering, so we're actively ensuring that everyone can benefit, just as they can contribute.

by sytse 9y ago

hatred 9y ago

Curious, did you evaluate any other similar technologies like Swarm/(Mesos+Marathon) etc.? What made you eventually go ahead with K8s other than the main obvious reason (the community) itself?

twk3 9y ago
At GitLab, we did evaluate Mesosphere DC/OS. What turned our focus to Kubernetes as our primary cluster install target was the speed at which it was developing, and after watching the space and talking to partners/customers we formed the opinion that Kubernetes was going to lead the pack. We've been looking at these technologies for two years now, with our focus being Kubernetes for the last one.
joshlambert 9y ago
GitLab has been investing significantly in Kubernetes, both because we believe in the platform but also because we see significant demand from our customers as well. It's ability to run on-premise as well as availability in a wide variety of managed cloud flavors is a huge benefit, and likely a driver of the demand. We also try to use the same deployment tools for GitLab.com that we provide to customers, and this lets us offer a scalable production-grade deployment method that can run nearly anywhere.
WarheadsSE 9y ago
As twk3 has said, we have examined multiple other options. One big reason we've chosen to implement with Kubernetes is community adoption. You can now run Kubernetes on AWS, GCP, Azure, IBM cloud, as well as a slew of other platforms both on-premise and as a PaaS. With the expanding ecosystem, our decision has only been further confirmed, as the list of available solutions and providers continues to grow rapidly (https://kubernetes.io/docs/setup/pick-right-solution/ https://kubernetes.io/docs/setup/pick-right-solution/) Another big reason is far simpler: ease of use and configuration. A permanent fixture in our work for Cloud Native GitLab has been that this method must be as easy, if not easier to provide a scalable, resilient, highly-available deployment of GitLab at scale. We can’t say, “This is our new suggested method. By the way, it’s harder.” What we have found is that many other solutions require a much larger initial investment of time to understand and configure GitLab as a whole solution, as compared to the combination of Kubernetes with Helm. Helm provides us templating, default value handling, component enable/disable logic, and many other extremely useful features. These allow us to provide our users with a practical, streamlined method of installation and configuration, without the need to spend countless hours reding documentation, deciding on the architecture, and making edits to YAML.

twk3 9y ago

At GitLab, our installation method for the product was also tightly based around chef-solo and chef omnibus. At first we were trying to continue using these technologies inside our containers, but this required our containers to run as root. So a lot of our effort in moving to kubernetes has required us to try and duplicate all of our previous installation work in ways that don't require root access while the containers are running. To help, we've chosen to use Helm to manage versioning/upgrading our application in the cluster: https://helm.sh/ https://helm.sh/

leetbulb 9y ago

Running GitLab CI specifically is absolutely amazing on K8S... Thank you so much for your efforts.

joshlambert 9y ago
Thanks, glad you are finding it helpful! We have a lot of improvements coming as well: https://about.gitlab.com/direction/#ci--cd https://about.gitlab.com/direction/#ci--cd
- ossm1db 9y ago
  If Gitlab could deploy to a play-with-k8s temporary cluster with four hours to test and evaluate, that would be pretty cool.
  - joshlambert 9y ago
    I agree, that would be awesome! While it's not a single button, GitLab is getting closer to that goal with the addition of our cluster integration feature: https://docs.gitlab.com/ee/user/project/clusters/#adding-and-creating-a-new-gke-cluster-via-gitlab https://docs.gitlab.com/ee/user/project/clusters/#adding-and... If you create a trial account on GCP, you can then use GitLab to make a new GKE cluster and it will be automatically linked. You can then click a single button to deploy the Helm Tiller, a GitLab Runner, and then run your CI/CD jobs on your brand new cluster! The GCP trial is pretty nice, you get $300 in credits and you won't be automatically billed at the end: https://cloud.google.com/free/ https://cloud.google.com/free/

WarheadsSE 9y ago

I recently present at Cloud Expo Europe, describing the how and why GitLab has been working to produce a cloud native distribution. I've outlined some of the blockers we've faced, and what we've been working on to resolve these going forward. Those changes benefit both our traditional deployment methodologies, as well as our efforts for deployment on Kubernetes. Journey to Cloud Native: Breaking the monolith and scaling towards tomorrow - https://docs.google.com/presentation/d/1fsgvSuGpn-MnMqKaTOoiRvUvdJFHd3VpoAAfBoMtFYg https://docs.google.com/presentation/d/1fsgvSuGpn-MnMqKaTOoi... Our biggest blockers have been how to separate file system dependency for our various parts. For Git content, we've implemented Gitaly (https://gitlab.com/gitlab-org/gitaly/ https://gitlab.com/gitlab-org/gitaly/). For various other parts, we've been implementing support for object storage across the board. Doing this while keeping up with the rapid growth of Kubernetes and Helm has been a challenge, but entirely worth the effort. We are also bringing all of these changes to GitLab CE, as a part of our Stewardship directive (https://about.gitlab.com/stewardship/ https://about.gitlab.com/stewardship/, https://about.gitlab.com/2016/01/11/being-a-good-open-source-steward/ https://about.gitlab.com/2016/01/11/being-a-good-open-source...). We don't feel that our efforts to allow for greater scalability, resiliency, and platform agnosticism belong only in the Enterprise offering, so we're actively ensuring that everyone can benefit, just as they can contribute.