6 ms·

I'm not surprised the Kubernetes primitives got copied into this. I had stated before that Kubernetes was what Docker wants to be when it grows up -- and maybe

by hosh 10y ago

I'm not surprised the Kubernetes primitives got copied into this. I had stated before that Kubernetes was what Docker wants to be when it grows up -- and maybe that time is here. Having used Kubernetes in production, I don't know robust the Docker orchestration primitives are in comparison. But I'll probably find out soon. The big advantage having a built-in Docker orchestration is that Kubernetes is painful to install from scratch. (Yes, there are scripts to help mitigate this; yes, GKE is effectively hosted Kubernetes). I'm involved in another Dockerization projection, but we don't know if we want to invest the time into setting up Kubernetes (though GKE is an option). This would be a good time to check things out. Just to show how quickly things change. In 2015, I tried the docker compose, AWS ECS, and Kubernetes for production deployment, and of all of them, Kubernetes addressed the pain points in a practical way. The Kubernetes project ended in late 2015, and now six months later, things changed again ...

jacques_chester 10y ago
I'll seed what will quickly be an enthusiastic crowd of spruikers for various platforms: have you looked at Cloud Foundry? It's mature, has been scaled to 10,000 real app containers in a single install with a 250,000-container milestone coming soon, belongs to an independent foundation instead of a single company, runs on AWS, Azure, GCP, OpenStack, vSphere with more coming. It handles routing, service injection, logging, container orchestration and a whole bunch of other bits and bobs. You can get started by just pushing your sourcecode, or you can provide container images, whichever works better for you. We even made a version you can test drive on your laptop[0] and we've been working on a "bootloader" for spinning up a full AWS version quickly[1]. Disclaimer: by "we" I mean Pivotal, my employer, which donates the majority of engineering effort to Cloud Foundry. [0] http://pivotal.io/pcf-dev http://pivotal.io/pcf-dev [1] https://github.com/pivotal-cf-experimental/bosh-bootloader https://github.com/pivotal-cf-experimental/bosh-bootloader
- Veratyr 10y ago
  Back when I looked at CloudFoundry (years ago now), it didn't work out because there was no middle ground deployment between a single PC and a full blown production deployment. You either ran it on one machine or you spun up what would have costed ~$1200 in servers at the time. It doesn't appear this has changed: http://docs.pivotal.io/pivotalcf/1-7/customizing/openstack.html http://docs.pivotal.io/pivotalcf/1-7/customizing/openstack.h... Even at a cheap provider like DigitalOcean, this is 16x10 + 6x160 = $1120/month in __overhead__, without even running any useful applications.
  - jacques_chester 10y ago
    I can see your point, insofar at CF is biased to provide HA for all its foundational components. It's actually possible to consolidate it down to fewer machines if you want. But it requires BOSH knowledge and the willingness to surrender the HA guarantees. If you only need to deploy a handful of apps, it'd be easier to start with a hosted service and go from there. We have one in Pivotal Web Services[0], pricing is based on RAM usage. IBM have BlueMix, there are also service installations managed by CenturyLink, AnyNines and Swisscom. [0] https://run.pivotal.io/ https://run.pivotal.io/
    - Veratyr 10y ago
      BOSH knowledge isn't easy to come by unfortunately and I don't think the problem is HA for the foundational components so much as there being so many foundational components. Look at a HA Kubernetes cluster. You've got kubelets and 2-3 masters and you're done. Look at CF and you need 22 servers total just to get started. It may be easier/cheaper to use a hosted service compared to setting up your own CloudFoundry cluster but it's still extremely expensive and doesn't compete with alternatives like hosted Kubernetes. Say I want 32GB total of apps with 8 CPUs for a few development environments. For CF I pay ~$1600, for Pivotal I pay ~$700, for self hosted Kubernetes I pay ~$400 (4x8GB DigitalOcean instances + 3x2GB for masters) and for managed Kubernetes I pay ~$205 (4x n1-standard-2 Google Compute Engine instances + free cluster management since under 6 nodes).
      - jacques_chester 10y ago
        I'm not sure it's apples to oranges, though. Out of the box Kubernetes provides some, but not all, of what Cloud Foundry provides. When you say "so many foundational components", you're right: CF is featuresome and opinionated. That's the whole idea. With Kubernetes you can absolutely assemble all the other parts yourself and wire them together to taste, but that will chew up some of your 32Gb. And our calculations should expand to include engineering time, which is, last time I checked, quite expensive. I'll definitely give Google due credit for competitive pricing though. For that price, and given their expertise, it'd be silly to self-host Kubernetes if you could use GCP.
- hosh 10y ago
  One of the things I liked about CF was how it encoded application dependencies into the app package. Chef has just recently announced Habitat, which apparently brings that kind of an idea into play. The thing is though, Kubernetes is a set of lego blocks from which you can compose into CF, but you couldn't create Kubernetes from CF. That makes Kubernetes more interesting to me.
  - jacques_chester 10y ago
    I think that's a fair call. CF is an already assembled kit and we're constantly looking for ways to have better and better fit-and-finish. The whole point is to allow operators and developers to have smooth "Day 0", "Day 1" and "Day 2" experiences. For those who really want a prebuilt Kubernete PaaS, Red Hat built OpenShift 3. That said, Diego (our scheduler tech) was deliberately architected to be ignorant of Cloud Foundry. Right now it's pretty much serving the historical bias of Cloud Foundry's initial architecture -- applications -- but as the v3 API arrives we're breaking it down into finer grained concepts, some familiar to Kubernetes folk. At one point, Diego was extracted out into a Kubernetes-like package with the Lattice project, which Pivotal supported for a while. But once we went and talked to developers, they said "well I don't really trust it because it's not a real Cloud Foundry". Hence we pivoted to PCFDev. If Docker and Kubernetes or Mesos had emerged earlier, it's likely we'd have used them as our building blocks (as we've adopted other components). But when Cloud Foundry was born, none of these components really existed outside of private systems owned by Google or Facebook, or in some respects at Heroku. Disclaimer: I was not responsible for any of these decisions, or even near to them, so take it all of the above with a grain of salt.
    - hosh 10y ago
      Yep, I remember when CF first came out. It was really interesting tech.
      - jacques_chester 10y ago
        It still is! :)
jpgvm 10y ago
If you want a simpler alternative to Kubernetes have a look at Flynn (https://flynn.io https://flynn.io).
- hosh 10y ago
  I have, wasn't interested. I'm more interested in Dies since it was rewritten against Kubernetes, and the same with Helm. Thanks for the suggestion though, I had not thought of Flynn in a while.
  - jpgvm 10y ago
    Fair enough. :) Disclaimer regarding the above, I work on the Flynn project.
    - hosh 10y ago
      Cool! Looking forward to seeing what you guys got. We seem to have a Cambrian explosion of tools and approaches. It'll also be interesting to see who survives after the die-off.
  - bogomipz 10y ago
    I think you mean Deis :)
- sciurus 10y ago
  Kubernetes really stole Flynn's thunder. I wonder, if the Flynn team could travel back in time to 2013, what they would do differently knowing how things turned out.
  - danielsiders 10y ago
    Flynn co-founder here. Thanks for the question. We view the work of the Kubernetes (and Mesos, Docker Swarm, and other scheduler projects) team as incredibly valuable to Flynn. We try to solve problems and create value at a the platform layer. Schedulers are one technology we use, but aren't at the core of the value that we (or most other platforms) deliver. We want to make development and management of production apps simple and invisible. For most users that should mean never needing to know about or interact with a scheduler directly. We closely monitor progress on the major scheduler projects and look forward to a day when we can utilize one of them in Flynn instead of maintaining our own. As these technologies mature we'll be able to deliver even more value on top of them, while saving users from interacting with lower level components.
lobster_johnson 10y ago
Having set up Kubernetes to test, I'd say it's a bit less complicated than many people claim, though still obviously complex. I had a two-node cluster up and running in an hour using Flannel and Kismatic's prepackaged binaries for Debian/Ubuntu [1]. The biggest hurdle, ironically, is the official documentation, which seems to have been written by someone with no understanding of explaining things from first principles, or any talent for organizing ifnroamtion. You'd think the "setup from scratch" guide [2] would be the most detailed and useful for someone who wants to see how each piece fits into the next one, but it's actually terrible. It skips a lot of implicit information and makes some odd assumptions/suggestions. Almost every part of Kubernetes is multiple choice, which affects later steps. For example, if you use Flannel and you're not on GCloud, much of what Google's docs say about networking is moot. Everything is also made harder by the fact that Kubernetes is changing awfully fast in non-backwards-compatible ways, and chances are that if you google some config problem, the answer is long out of date — the necessary kubectl option no longer exists or whatever. A lot of config recipes are useless now. [1] https://github.com/kismatic/repo_server/find/master https://github.com/kismatic/repo_server/find/master [2] http://kubernetes.io/docs/getting-started-guides/scratch/ http://kubernetes.io/docs/getting-started-guides/scratch/
- hosh 10y ago
  I set up Kubernetes on AWS, a staging and a production cluster. Staging had 1 master, 3 worker nodes, Production has 3 masters, 5 worker nodes, with two compute classes (fixed and burst, so pods could be scheduled to what was needed). All of this was on CoreOS, and included independent etcd clusters. So all total, we're talking 18 AWS EC2 instances. I put live users on it, with real traffic and tinkered with it as I learned things from a production workflow. This was back in mid-2015, which meant figuring out things like how to get overlay networking with AWS's routing, fiddling with AWS's IAM settings, VPC settings, figuring out how to split the nodes across availability zones. This included integrating with AWS AutoScalingGroups to bring up new CoreOS/K8S nodes whenever something goes down. I never got EBS working as a Kubernetes volume type (at the time, you have to add hackish tools into CoreOS to detect unformatted EBS blocks and format them when they get mounted; forget it), but fortunately, used AWS RDS for persistent stores. Persistent storage is something I look forward to trying on GKE. I could have tried creating CloudFormation scripts but I was making my way through documentation, blog posts, and Github issues to pull it all together. It was worth it when I was done. I didn't say it was complicated. I said it was a pain in the ass, and it's something that Kubernetes developers have acknowledged and are working on it.
wstrange 10y ago
> Kubernetes is painful to install from scratch Fair comment, and AFAIK, this is recognised and being addressed. For example, minikube [https://github.com/kubernetes/minikube https://github.com/kubernetes/minikube] to easily run Kubernetes on the desktop. There are efforts underway to simplify cluster bring up, but this is not an easy problem because Kubernetes has to abstract away the underlying differences in O/S and cloud providers (example: load balancers on AWS vs. GCE). I think they are getting there, but if you want dead simple "it just works", GKE is a great option.
- hosh 10y ago
  I would love to see equivalent for Docker Toolkit, Docker for Mac, and Docker for Windows. It would make it easier to convince my team. I'll check out Minikube. But yes, I know it is not an easy problem. I had looked at the automation involved as well as deployed Kubernetes by hand to AWS, having to translate what works on AWS. And as I have mentioned in my comment, GKE is an option. Having said that, on the desktop, I think all I really care about would be the equivalent of a docker-machine setup for Mac and Windows. Something that brings up a single-node CoreOS running stable, with networking working, and kubelet running via Docker. (Back in 2015, I had to rely on kubelet being baked into the CoreOS release). Probably not a big deal on Mac if I put a vagrant script, or use the existing ones that CoreOS has. But my teammates that are using Windows will want something that works well. I'm curious -- what's the underlying distro being used for minikube?
  - rodgerd 10y ago
    > I would love to see equivalent for Docker Toolkit, Docker for Mac, and Docker for Windows Take a gander at https://www.openshift.org/vm/ https://www.openshift.org/vm/ ? Kubernetes based.
  - ssijak 10y ago
    https://github.com/TheNewNormal/kube-solo-osx https://github.com/TheNewNormal/kube-solo-osx and https://github.com/TheNewNormal/kube-cluster-osx https://github.com/TheNewNormal/kube-cluster-osx
illumin8 10y ago
You should check out Rancher - they fully automate/orchestrate the installation of Kubernetes (along with Mesos, Swarm, or Cattle) and support some advanced features like blue/green deployments by integrating with the Kubernetes ingress feature: http://rancher.com/rancher-controller-for-the-kubernetes-ingress-feature/ http://rancher.com/rancher-controller-for-the-kubernetes-ing... Rancher Labs definitely seems to be further along the path of a production-ready container hosting platform than any other one I've tried so far.
- ecliptik 10y ago
  I'm surprised I don't read more about Rancher qurg everyone focusing on k8s. Not only can Rancher use k8s for scheduling, but mixing and matching environments with different scheduling technologies is extremely powerful instead of pigeon holing yourself on a one-size-fits all approach for you applications.

by hosh 10y ago

jacques_chester 10y ago

I'll seed what will quickly be an enthusiastic crowd of spruikers for various platforms: have you looked at Cloud Foundry? It's mature, has been scaled to 10,000 real app containers in a single install with a 250,000-container milestone coming soon, belongs to an independent foundation instead of a single company, runs on AWS, Azure, GCP, OpenStack, vSphere with more coming. It handles routing, service injection, logging, container orchestration and a whole bunch of other bits and bobs. You can get started by just pushing your sourcecode, or you can provide container images, whichever works better for you. We even made a version you can test drive on your laptop[0] and we've been working on a "bootloader" for spinning up a full AWS version quickly[1]. Disclaimer: by "we" I mean Pivotal, my employer, which donates the majority of engineering effort to Cloud Foundry. [0] http://pivotal.io/pcf-dev http://pivotal.io/pcf-dev [1] https://github.com/pivotal-cf-experimental/bosh-bootloader https://github.com/pivotal-cf-experimental/bosh-bootloader

Veratyr 10y ago
Back when I looked at CloudFoundry (years ago now), it didn't work out because there was no middle ground deployment between a single PC and a full blown production deployment. You either ran it on one machine or you spun up what would have costed ~$1200 in servers at the time. It doesn't appear this has changed: http://docs.pivotal.io/pivotalcf/1-7/customizing/openstack.html http://docs.pivotal.io/pivotalcf/1-7/customizing/openstack.h... Even at a cheap provider like DigitalOcean, this is 16x10 + 6x160 = $1120/month in __overhead__, without even running any useful applications.
- jacques_chester 10y ago
  I can see your point, insofar at CF is biased to provide HA for all its foundational components. It's actually possible to consolidate it down to fewer machines if you want. But it requires BOSH knowledge and the willingness to surrender the HA guarantees. If you only need to deploy a handful of apps, it'd be easier to start with a hosted service and go from there. We have one in Pivotal Web Services[0], pricing is based on RAM usage. IBM have BlueMix, there are also service installations managed by CenturyLink, AnyNines and Swisscom. [0] https://run.pivotal.io/ https://run.pivotal.io/
  - Veratyr 10y ago
    BOSH knowledge isn't easy to come by unfortunately and I don't think the problem is HA for the foundational components so much as there being so many foundational components. Look at a HA Kubernetes cluster. You've got kubelets and 2-3 masters and you're done. Look at CF and you need 22 servers total just to get started. It may be easier/cheaper to use a hosted service compared to setting up your own CloudFoundry cluster but it's still extremely expensive and doesn't compete with alternatives like hosted Kubernetes. Say I want 32GB total of apps with 8 CPUs for a few development environments. For CF I pay ~$1600, for Pivotal I pay ~$700, for self hosted Kubernetes I pay ~$400 (4x8GB DigitalOcean instances + 3x2GB for masters) and for managed Kubernetes I pay ~$205 (4x n1-standard-2 Google Compute Engine instances + free cluster management since under 6 nodes).
    - jacques_chester 10y ago
      I'm not sure it's apples to oranges, though. Out of the box Kubernetes provides some, but not all, of what Cloud Foundry provides. When you say "so many foundational components", you're right: CF is featuresome and opinionated. That's the whole idea. With Kubernetes you can absolutely assemble all the other parts yourself and wire them together to taste, but that will chew up some of your 32Gb. And our calculations should expand to include engineering time, which is, last time I checked, quite expensive. I'll definitely give Google due credit for competitive pricing though. For that price, and given their expertise, it'd be silly to self-host Kubernetes if you could use GCP.
hosh 10y ago
One of the things I liked about CF was how it encoded application dependencies into the app package. Chef has just recently announced Habitat, which apparently brings that kind of an idea into play. The thing is though, Kubernetes is a set of lego blocks from which you can compose into CF, but you couldn't create Kubernetes from CF. That makes Kubernetes more interesting to me.
- jacques_chester 10y ago
  I think that's a fair call. CF is an already assembled kit and we're constantly looking for ways to have better and better fit-and-finish. The whole point is to allow operators and developers to have smooth "Day 0", "Day 1" and "Day 2" experiences. For those who really want a prebuilt Kubernete PaaS, Red Hat built OpenShift 3. That said, Diego (our scheduler tech) was deliberately architected to be ignorant of Cloud Foundry. Right now it's pretty much serving the historical bias of Cloud Foundry's initial architecture -- applications -- but as the v3 API arrives we're breaking it down into finer grained concepts, some familiar to Kubernetes folk. At one point, Diego was extracted out into a Kubernetes-like package with the Lattice project, which Pivotal supported for a while. But once we went and talked to developers, they said "well I don't really trust it because it's not a real Cloud Foundry". Hence we pivoted to PCFDev. If Docker and Kubernetes or Mesos had emerged earlier, it's likely we'd have used them as our building blocks (as we've adopted other components). But when Cloud Foundry was born, none of these components really existed outside of private systems owned by Google or Facebook, or in some respects at Heroku. Disclaimer: I was not responsible for any of these decisions, or even near to them, so take it all of the above with a grain of salt.
  - hosh 10y ago
    Yep, I remember when CF first came out. It was really interesting tech.
    - jacques_chester 10y ago
      It still is! :)

jpgvm 10y ago

If you want a simpler alternative to Kubernetes have a look at Flynn (https://flynn.io https://flynn.io).

hosh 10y ago
I have, wasn't interested. I'm more interested in Dies since it was rewritten against Kubernetes, and the same with Helm. Thanks for the suggestion though, I had not thought of Flynn in a while.
- jpgvm 10y ago
  Fair enough. :) Disclaimer regarding the above, I work on the Flynn project.
  - hosh 10y ago
    Cool! Looking forward to seeing what you guys got. We seem to have a Cambrian explosion of tools and approaches. It'll also be interesting to see who survives after the die-off.
- bogomipz 10y ago
  I think you mean Deis :)
sciurus 10y ago
Kubernetes really stole Flynn's thunder. I wonder, if the Flynn team could travel back in time to 2013, what they would do differently knowing how things turned out.
- danielsiders 10y ago
  Flynn co-founder here. Thanks for the question. We view the work of the Kubernetes (and Mesos, Docker Swarm, and other scheduler projects) team as incredibly valuable to Flynn. We try to solve problems and create value at a the platform layer. Schedulers are one technology we use, but aren't at the core of the value that we (or most other platforms) deliver. We want to make development and management of production apps simple and invisible. For most users that should mean never needing to know about or interact with a scheduler directly. We closely monitor progress on the major scheduler projects and look forward to a day when we can utilize one of them in Flynn instead of maintaining our own. As these technologies mature we'll be able to deliver even more value on top of them, while saving users from interacting with lower level components.

lobster_johnson 10y ago

Having set up Kubernetes to test, I'd say it's a bit less complicated than many people claim, though still obviously complex. I had a two-node cluster up and running in an hour using Flannel and Kismatic's prepackaged binaries for Debian/Ubuntu [1]. The biggest hurdle, ironically, is the official documentation, which seems to have been written by someone with no understanding of explaining things from first principles, or any talent for organizing ifnroamtion. You'd think the "setup from scratch" guide [2] would be the most detailed and useful for someone who wants to see how each piece fits into the next one, but it's actually terrible. It skips a lot of implicit information and makes some odd assumptions/suggestions. Almost every part of Kubernetes is multiple choice, which affects later steps. For example, if you use Flannel and you're not on GCloud, much of what Google's docs say about networking is moot. Everything is also made harder by the fact that Kubernetes is changing awfully fast in non-backwards-compatible ways, and chances are that if you google some config problem, the answer is long out of date — the necessary kubectl option no longer exists or whatever. A lot of config recipes are useless now. [1] https://github.com/kismatic/repo_server/find/master https://github.com/kismatic/repo_server/find/master [2] http://kubernetes.io/docs/getting-started-guides/scratch/ http://kubernetes.io/docs/getting-started-guides/scratch/

hosh 10y ago
I set up Kubernetes on AWS, a staging and a production cluster. Staging had 1 master, 3 worker nodes, Production has 3 masters, 5 worker nodes, with two compute classes (fixed and burst, so pods could be scheduled to what was needed). All of this was on CoreOS, and included independent etcd clusters. So all total, we're talking 18 AWS EC2 instances. I put live users on it, with real traffic and tinkered with it as I learned things from a production workflow. This was back in mid-2015, which meant figuring out things like how to get overlay networking with AWS's routing, fiddling with AWS's IAM settings, VPC settings, figuring out how to split the nodes across availability zones. This included integrating with AWS AutoScalingGroups to bring up new CoreOS/K8S nodes whenever something goes down. I never got EBS working as a Kubernetes volume type (at the time, you have to add hackish tools into CoreOS to detect unformatted EBS blocks and format them when they get mounted; forget it), but fortunately, used AWS RDS for persistent stores. Persistent storage is something I look forward to trying on GKE. I could have tried creating CloudFormation scripts but I was making my way through documentation, blog posts, and Github issues to pull it all together. It was worth it when I was done. I didn't say it was complicated. I said it was a pain in the ass, and it's something that Kubernetes developers have acknowledged and are working on it.

wstrange 10y ago

> Kubernetes is painful to install from scratch Fair comment, and AFAIK, this is recognised and being addressed. For example, minikube [https://github.com/kubernetes/minikube https://github.com/kubernetes/minikube] to easily run Kubernetes on the desktop. There are efforts underway to simplify cluster bring up, but this is not an easy problem because Kubernetes has to abstract away the underlying differences in O/S and cloud providers (example: load balancers on AWS vs. GCE). I think they are getting there, but if you want dead simple "it just works", GKE is a great option.

hosh 10y ago
I would love to see equivalent for Docker Toolkit, Docker for Mac, and Docker for Windows. It would make it easier to convince my team. I'll check out Minikube. But yes, I know it is not an easy problem. I had looked at the automation involved as well as deployed Kubernetes by hand to AWS, having to translate what works on AWS. And as I have mentioned in my comment, GKE is an option. Having said that, on the desktop, I think all I really care about would be the equivalent of a docker-machine setup for Mac and Windows. Something that brings up a single-node CoreOS running stable, with networking working, and kubelet running via Docker. (Back in 2015, I had to rely on kubelet being baked into the CoreOS release). Probably not a big deal on Mac if I put a vagrant script, or use the existing ones that CoreOS has. But my teammates that are using Windows will want something that works well. I'm curious -- what's the underlying distro being used for minikube?
- rodgerd 10y ago
  > I would love to see equivalent for Docker Toolkit, Docker for Mac, and Docker for Windows Take a gander at https://www.openshift.org/vm/ https://www.openshift.org/vm/ ? Kubernetes based.
- ssijak 10y ago
  https://github.com/TheNewNormal/kube-solo-osx https://github.com/TheNewNormal/kube-solo-osx and https://github.com/TheNewNormal/kube-cluster-osx https://github.com/TheNewNormal/kube-cluster-osx

illumin8 10y ago

You should check out Rancher - they fully automate/orchestrate the installation of Kubernetes (along with Mesos, Swarm, or Cattle) and support some advanced features like blue/green deployments by integrating with the Kubernetes ingress feature: http://rancher.com/rancher-controller-for-the-kubernetes-ingress-feature/ http://rancher.com/rancher-controller-for-the-kubernetes-ing... Rancher Labs definitely seems to be further along the path of a production-ready container hosting platform than any other one I've tried so far.

ecliptik 10y ago
I'm surprised I don't read more about Rancher qurg everyone focusing on k8s. Not only can Rancher use k8s for scheduling, but mixing and matching environments with different scheduling technologies is extremely powerful instead of pigeon holing yourself on a one-size-fits all approach for you applications.