3 ms·

Kubernetes had won the container scheduler wars. At GitLab we're all in on making a PaaS based on k8s and our CI/CD and the container registry that is part of G

by sytse 9y ago

Kubernetes had won the container scheduler wars. At GitLab we're all in on making a PaaS based on k8s and our CI/CD and the container registry that is part of GitLab.

candiodari 9y ago
It feels awfully 19th century though that despite k8s having "won", by far the biggest container schedulers by containers scheduled are, no doubt: (I think this is the correct order, not 100% sure of course) 1) google borg (maybe omega) [1] 2) amazon ec2 3) whatever microsoft is using (large gap) 4) all the rest of the world combined, a small portion of which is k8s [1] https://www.quora.com/Does-Google-use-the-Open-Source-Kubernetes-or-a-version-of-Borg-for-their-own-container-management https://www.quora.com/Does-Google-use-the-Open-Source-Kubern... (one might even say [1] seems to imply it'll never happen, or at least take a very long time. Also if you read the papers it becomes very clear that "Google Borg" includes a lot of things these days at many levels, from custom ASICs, device firmware (as in standard device, google borg firmware), BIOS firmware, entirely custom sub-kernel code, custom kernels, custom userspace (ie. Google-specific libc that's not optional), ... all of these will turn out to have dependencies on eachother that have to be redone for k8s, could take a while to migrate over) (although I have not read any papers on it (I'd love some though), I'd bet amazon is in a similar boat, and of course Microsoft is Microsoft)
- gaius 9y ago
  EC2 is not a container scheduler - it's an IaaS for VMs. The Amazon container PaaS (ECS/EKS) is a layer on top of EC2. And that is being superseded by Fargate which will make the underlying EC2 invisible. If you need a Fargate-like capability now, Azure AKS does it. See https://azure.microsoft.com/en-us/services/container-service/ https://azure.microsoft.com/en-us/services/container-service... and https://aws.amazon.com/fargate/ https://aws.amazon.com/fargate/
  - candiodari 9y ago
    So what is the EC2 container scheduler before Fargate called ? Any papers on it ?
    - gaius 9y ago
      ECS and EKS.
  - coredog64 9y ago
    Fargate is expensive as hell for long running services. You should only be using it for something that creates value 100% of the time that it is running.
sandGorgon 9y ago
> At GitLab we're all in on making a PaaS based on k8s This is very interesting. Could you talk more on this ? There is definitely space for an "opinionated k8s distro with batteries included". I have wished for Swarm to become this....
- sytse 9y ago
  It is not a Kubernetes distribution. You can use any distribution or CaaS you want. The beginning of it is in GitLab Auto DevOps https://about.gitlab.com/2017/10/04/devops-strategy/ https://about.gitlab.com/2017/10/04/devops-strategy/
komuW 9y ago
Interesting. Is there a blog post where i can read more about this?
- sytse 9y ago
  https://docs.gitlab.com/ee/topics/autodevops/ https://docs.gitlab.com/ee/topics/autodevops/ and https://about.gitlab.com/2017/10/04/devops-strategy/ https://about.gitlab.com/2017/10/04/devops-strategy/

by sytse 9y ago

Kubernetes had won the container scheduler wars. At GitLab we're all in on making a PaaS based on k8s and our CI/CD and the container registry that is part of GitLab.

candiodari 9y ago

It feels awfully 19th century though that despite k8s having "won", by far the biggest container schedulers by containers scheduled are, no doubt: (I think this is the correct order, not 100% sure of course) 1) google borg (maybe omega) [1] 2) amazon ec2 3) whatever microsoft is using (large gap) 4) all the rest of the world combined, a small portion of which is k8s [1] https://www.quora.com/Does-Google-use-the-Open-Source-Kubernetes-or-a-version-of-Borg-for-their-own-container-management https://www.quora.com/Does-Google-use-the-Open-Source-Kubern... (one might even say [1] seems to imply it'll never happen, or at least take a very long time. Also if you read the papers it becomes very clear that "Google Borg" includes a lot of things these days at many levels, from custom ASICs, device firmware (as in standard device, google borg firmware), BIOS firmware, entirely custom sub-kernel code, custom kernels, custom userspace (ie. Google-specific libc that's not optional), ... all of these will turn out to have dependencies on eachother that have to be redone for k8s, could take a while to migrate over) (although I have not read any papers on it (I'd love some though), I'd bet amazon is in a similar boat, and of course Microsoft is Microsoft)

gaius 9y ago
EC2 is not a container scheduler - it's an IaaS for VMs. The Amazon container PaaS (ECS/EKS) is a layer on top of EC2. And that is being superseded by Fargate which will make the underlying EC2 invisible. If you need a Fargate-like capability now, Azure AKS does it. See https://azure.microsoft.com/en-us/services/container-service/ https://azure.microsoft.com/en-us/services/container-service... and https://aws.amazon.com/fargate/ https://aws.amazon.com/fargate/
- candiodari 9y ago
  So what is the EC2 container scheduler before Fargate called ? Any papers on it ?
  - gaius 9y ago
    ECS and EKS.
- coredog64 9y ago
  Fargate is expensive as hell for long running services. You should only be using it for something that creates value 100% of the time that it is running.

sandGorgon 9y ago

> At GitLab we're all in on making a PaaS based on k8s This is very interesting. Could you talk more on this ? There is definitely space for an "opinionated k8s distro with batteries included". I have wished for Swarm to become this....

sytse 9y ago
It is not a Kubernetes distribution. You can use any distribution or CaaS you want. The beginning of it is in GitLab Auto DevOps https://about.gitlab.com/2017/10/04/devops-strategy/ https://about.gitlab.com/2017/10/04/devops-strategy/

komuW 9y ago

Interesting. Is there a blog post where i can read more about this?

sytse 9y ago
https://docs.gitlab.com/ee/topics/autodevops/ https://docs.gitlab.com/ee/topics/autodevops/ and https://about.gitlab.com/2017/10/04/devops-strategy/ https://about.gitlab.com/2017/10/04/devops-strategy/