6 ms·

> k8s, for example, incorporates service discovery. As far as I can tell, swarmkit does not. k8s incorporates networking, containerd does not. Similar for thing

by shykes 10y ago

> k8s, for example, incorporates service discovery. As far as I can tell, swarmkit does not. k8s incorporates networking, containerd does not. Similar for things like ingress load balancing. Swarmkit does in fact implement service discovery, networking, and ingress load-balancing. It also implements out-of-the-box node security and mutual TLS, secure secrets management, a built-in raft store, infrastructure-agnostic overlay networking, and various goodies which we needed to make Docker work great out of the box. containerd is a different type of component entirely - in fact it is very complementary to kubernetes. > There are certainly potential customers debating, directly, k8s vs your full Docker platform They typically debate Docker vs kubernetes-based platforms (among other possible alternatives). If they're a Red Hat shop, they typically evaluate Openshift. Sometimes there's a team building an in-house platform. Nobody ever deploys kub alone in production. There is always some form of platform on top.

tyingq 10y ago
>containerd is a different type of component entirely - in fact it is very complementary to kubernetes You made the initial comparison. > Nobody ever deploys kub alone in production. There is always some form of platform on top. Which is another way of saying "even if there are some gaps they have to fill with other software". Sure, some customers pick a platform where the gaps are prefilled. Not terribly different from some of your customers that pair docker pieces with pieces from other vendors.
- shykes 10y ago
  >> containerd is a different type of component entirely - in fact it is very complementary to kubernetes > You made the initial comparison. Yes, sorry I wasn't clear. I meant that they were both components, as opposed to complete products. On the other hand, SwarmKit and Kubernetes are comparable in functionality. > Which is another way of saying "even if there are some gaps they have to fill with other software". I agree. > Sure, some customers pick a platform where the gaps are prefilled. Not terribly different from some of your customers that pair docker pieces with pieces from other vendors. I also agree. It seems that there is nothing left to disagree on :)
  - tyingq 10y ago
    Sure. Said more concisely, my beef was that you compared k8s to things narrower in purpose, then compared Docker to things broader in purpose. It made the gap between the two seem larger than what it is. I agree there's a gap. Edit: Re: "Try developing on Docker for Mac/Windows and deploying to production with Docker for AWS/Azure" Developing on a platform that isn't the same as your production platform is an avoidable situation. I could develop on .Net and deploy on mono as well, but...
    - shykes 10y ago
      The gap is in fact huge. Try developing on Docker for Mac/Windows and deploying to production with Docker for AWS/Azure. Then try to build the same level of functionality yourself from naked kubernetes and only open-source tools. Let me know when you're done :)
      - btmiller 10y ago
        The difference is that I can put together components of Kubernetes that actually work. Sorry to be inflammatory, but I've been working on the bleeding edge of Swarm and it simply doesn't deliver what's promised (ex: flaky management of iptables, missing idioms/abstractions, etc.). Single-host Docker in production, yeah that's not terribly complex, but the new price of admission for production is a complete orchestration layer, and there is 0% chance I consider taking a Swarm cluster to production any time soon. And that's to say nothing of the DDC disaster...
      - thockingoog 10y ago
        MiniKube as a local env is getting great feedback filling this niche.
        tmzt 10y ago
        It's absolutely awesome in this role. I don't know if a few hundred lines of shell scripting and Helm charts are considered building around it though, but that's enough for a Git hosting service (Gogs) and a CI (Drone 0.5) with persistent storage with vboxsf.
thockingoog 10y ago
> Nobody ever deploys kub alone in production That doesn't gel with any reality I know of, sorry. Lots of people use higher level platforms, or develop bespoke tooling, to be sure, but that's going to be true forever, for every platform. (disclosure for others: I'm a Kubernetes founder)
- thockingoog 10y ago
  Replying to myself: The fact that someone builds tooling on top of your offering is not an indictment of your product. It simply means people are using it. Be worried when people STOP trying to wrap your stuff with their own opinions.
AndyNemmity 10y ago
> Nobody ever deploys kub alone in production. Uhh, we do. And we're not alone as we work with several other companies that do. That's quite a big nobody tent.
- shykes 10y ago
  What do you use for provisioning and upgrading Kub? For monitoring it? For CICD? For access control? Image registry? Image build? The collection of these things, and everything else required for deploying and managing containers in production, is what I call a container platform. Is kub core not working on spinning out built-in volume plugins out of the core to keep it smaller? Are the maintainers not recommending using third-party tools + 3PR for all major new features going forward? It seems like my comment was misinterpreted as a criticism of kub, or a sign of ignorance. It was neither. I stand by my comment that nobody runs naked kub in production. That doesn't make it any less good, stable or useful. It just means it's not meant to be a complete product.
bketelsen 10y ago
I beg to differ, having done Kubernetes training for nearly a year now. None of the shops I've trained at had any platforms on "top" of Kubernetes. They all ran Kubernetes directly from the source. And we're talking about some big enterprises here, not mom and pop shops.
- thickice 10y ago
  When an enterprise runs Kubernetes directly from the source how do they deal with bugs and support ? Doesn't that need some vendor backing with tight SLAs ?
hjacobs 10y ago
> Nobody ever deploys kub alone in production. We do: https://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html https://kubernetes-on-aws.readthedocs.io/en/latest/admin-gui... There are only a few AWS integration components we added, e.g. to use the "new" AWS ALB (see the linked text for details).
majewsky 10y ago
> Nobody ever deploys kub alone in production. We do. I work at SAP, and we run our company-wide OpenStack on pure Kubernetes on CoreOS's Container Linux. [1] We do use Docker (the container runtime) because it comes with CoreOS, but no other Docker product as far as I'm aware. I've been working with Kubernetes for quite some time now, and honestly don't know what else you would need on top of it (except for some Continuous Integration tool, of course, but that's already a staple of any well-organized agile team, no matter the platform). [1] I mean the API and orchestrator parts, not the customer VMs themselves. These sit on traditional hypervisors.
- jacques_chester 10y ago
  Of interest, SAP is a Platinum member of the Cloud Foundry Foundation and has a certified CF distribution, SAP Cloud Platform. Disclosure: I work for Pivotal, another Foundation member, which also sells a commercial CF distribution.
  - majewsky 10y ago
    Ah, yeah, the CF guys are from another team. They deploy their stuff on (among other things) the VMs created by the OpenStacks running within our Kuberneteses, though. It's turtles all the way down. :)
    - jacques_chester 10y ago
      That's more turtles than I've typically seen. How're you deploying k8s?
      - majewsky 10y ago
        Bare-metal with homebrew automation. We install CoreOS via PXE boot, and during the installation it also sets up a Kubelet as an rkt container. The Kubelet then spins up the other k8s components via manifests. The pod and service networks are routed via BGP using our own https://github.com/sapcc/parrot https://github.com/sapcc/parrot Later this year, we will go back and evaluate the maturing k8s administration landscape. Our current approach has a few drawbacks, e.g. it requires a CoreOS reinstall to upgrade k8s cleanly (since all the magic happens in cloud-init and Ignition).

by shykes 10y ago

tyingq 10y ago

>containerd is a different type of component entirely - in fact it is very complementary to kubernetes You made the initial comparison. > Nobody ever deploys kub alone in production. There is always some form of platform on top. Which is another way of saying "even if there are some gaps they have to fill with other software". Sure, some customers pick a platform where the gaps are prefilled. Not terribly different from some of your customers that pair docker pieces with pieces from other vendors.

shykes 10y ago
>> containerd is a different type of component entirely - in fact it is very complementary to kubernetes > You made the initial comparison. Yes, sorry I wasn't clear. I meant that they were both components, as opposed to complete products. On the other hand, SwarmKit and Kubernetes are comparable in functionality. > Which is another way of saying "even if there are some gaps they have to fill with other software". I agree. > Sure, some customers pick a platform where the gaps are prefilled. Not terribly different from some of your customers that pair docker pieces with pieces from other vendors. I also agree. It seems that there is nothing left to disagree on :)
- tyingq 10y ago
  Sure. Said more concisely, my beef was that you compared k8s to things narrower in purpose, then compared Docker to things broader in purpose. It made the gap between the two seem larger than what it is. I agree there's a gap. Edit: Re: "Try developing on Docker for Mac/Windows and deploying to production with Docker for AWS/Azure" Developing on a platform that isn't the same as your production platform is an avoidable situation. I could develop on .Net and deploy on mono as well, but...
  - shykes 10y ago
    The gap is in fact huge. Try developing on Docker for Mac/Windows and deploying to production with Docker for AWS/Azure. Then try to build the same level of functionality yourself from naked kubernetes and only open-source tools. Let me know when you're done :)
    - btmiller 10y ago
      The difference is that I can put together components of Kubernetes that actually work. Sorry to be inflammatory, but I've been working on the bleeding edge of Swarm and it simply doesn't deliver what's promised (ex: flaky management of iptables, missing idioms/abstractions, etc.). Single-host Docker in production, yeah that's not terribly complex, but the new price of admission for production is a complete orchestration layer, and there is 0% chance I consider taking a Swarm cluster to production any time soon. And that's to say nothing of the DDC disaster...
    - thockingoog 10y ago
      MiniKube as a local env is getting great feedback filling this niche.
      - tmzt 10y ago
        It's absolutely awesome in this role. I don't know if a few hundred lines of shell scripting and Helm charts are considered building around it though, but that's enough for a Git hosting service (Gogs) and a CI (Drone 0.5) with persistent storage with vboxsf.

thockingoog 10y ago

> Nobody ever deploys kub alone in production That doesn't gel with any reality I know of, sorry. Lots of people use higher level platforms, or develop bespoke tooling, to be sure, but that's going to be true forever, for every platform. (disclosure for others: I'm a Kubernetes founder)

thockingoog 10y ago
Replying to myself: The fact that someone builds tooling on top of your offering is not an indictment of your product. It simply means people are using it. Be worried when people STOP trying to wrap your stuff with their own opinions.

AndyNemmity 10y ago

> Nobody ever deploys kub alone in production. Uhh, we do. And we're not alone as we work with several other companies that do. That's quite a big nobody tent.

shykes 10y ago
What do you use for provisioning and upgrading Kub? For monitoring it? For CICD? For access control? Image registry? Image build? The collection of these things, and everything else required for deploying and managing containers in production, is what I call a container platform. Is kub core not working on spinning out built-in volume plugins out of the core to keep it smaller? Are the maintainers not recommending using third-party tools + 3PR for all major new features going forward? It seems like my comment was misinterpreted as a criticism of kub, or a sign of ignorance. It was neither. I stand by my comment that nobody runs naked kub in production. That doesn't make it any less good, stable or useful. It just means it's not meant to be a complete product.

bketelsen 10y ago

I beg to differ, having done Kubernetes training for nearly a year now. None of the shops I've trained at had any platforms on "top" of Kubernetes. They all ran Kubernetes directly from the source. And we're talking about some big enterprises here, not mom and pop shops.

thickice 10y ago
When an enterprise runs Kubernetes directly from the source how do they deal with bugs and support ? Doesn't that need some vendor backing with tight SLAs ?

hjacobs 10y ago

> Nobody ever deploys kub alone in production. We do: https://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html https://kubernetes-on-aws.readthedocs.io/en/latest/admin-gui... There are only a few AWS integration components we added, e.g. to use the "new" AWS ALB (see the linked text for details).

majewsky 10y ago

> Nobody ever deploys kub alone in production. We do. I work at SAP, and we run our company-wide OpenStack on pure Kubernetes on CoreOS's Container Linux. [1] We do use Docker (the container runtime) because it comes with CoreOS, but no other Docker product as far as I'm aware. I've been working with Kubernetes for quite some time now, and honestly don't know what else you would need on top of it (except for some Continuous Integration tool, of course, but that's already a staple of any well-organized agile team, no matter the platform). [1] I mean the API and orchestrator parts, not the customer VMs themselves. These sit on traditional hypervisors.

jacques_chester 10y ago
Of interest, SAP is a Platinum member of the Cloud Foundry Foundation and has a certified CF distribution, SAP Cloud Platform. Disclosure: I work for Pivotal, another Foundation member, which also sells a commercial CF distribution.
- majewsky 10y ago
  Ah, yeah, the CF guys are from another team. They deploy their stuff on (among other things) the VMs created by the OpenStacks running within our Kuberneteses, though. It's turtles all the way down. :)
  - jacques_chester 10y ago
    That's more turtles than I've typically seen. How're you deploying k8s?
    - majewsky 10y ago
      Bare-metal with homebrew automation. We install CoreOS via PXE boot, and during the installation it also sets up a Kubelet as an rkt container. The Kubelet then spins up the other k8s components via manifests. The pod and service networks are routed via BGP using our own https://github.com/sapcc/parrot https://github.com/sapcc/parrot Later this year, we will go back and evaluate the maturing k8s administration landscape. Our current approach has a few drawbacks, e.g. it requires a CoreOS reinstall to upgrade k8s cleanly (since all the magic happens in cloud-init and Ignition).