8 ms·

I've since moved on to Kubernetes and SaltStack, but I got into production Docker containers with Deis[1] a year ago. With full Heroku buildpack support, Kubern

I've since moved on to Kubernetes and SaltStack, but I got into production Docker containers with Deis[1] a year ago. With full Heroku buildpack support, Kubernetes backed, and great container tools. I feel like it's sad that this project doesn't get more love. It's a 20 minute deploy to GKE or AWS (I recommend GKE). If you run OSX, TheNewNormal's solo and cluster Kube setup is a breeze and has the Deis Workflow as a one command install, then you have full local parity with deployment. When I started with straight Docker figuring out the transition to the real world was be really daunting. Getting into a real production environment that I could compost quickly as I learned learn the paradigms was invaluable. Articles like this seem to buzzword their way into making the path muddier than it actually is. Another big help for me was any time I needed to set something up, first thing, I'd look at how the bleeding edge was being done with Alpine. Practically you might not always run that light, but that community has all the answers. [1]https://deis.com/ https://deis.com/ [2]https://github.com/TheNewNormal/kube-solo-osx https://github.com/TheNewNormal/kube-solo-osx [3]https://github.com/TheNewNormal/kube-cluster-osx https://github.com/TheNewNormal/kube-cluster-osx

tedmiston 10y ago
When I first heard about Deis it was heavily compared to Flynn [1]. It's a little hard to tell from the outside if the two are still competitors or if Deis is focusing on a lower level of abstraction. https://flynn.io/ https://flynn.io/
- hackerboos 10y ago
  They both support Heroku-like buildpacks. Major difference IMO is that Flynn supports databases and other stateful services. I evaluated Flynn, Deis and Dokku and wasn't entirely satisfied with any of them opting for Heroku instead. I've promised people a blogpost on this and I will do it once I get time to evaluate RancherOS too.
  - baronseng 10y ago
    Can you evaluate Kontena as well while you are at it?
- tuananh 10y ago
  Flynn was much less polished than Deis. At least, that was when i tried it last year. I also started with Deis and move on to k8s later on.
bogomipz 10y ago
>"I've since moved on to Kubernetes and SaltStack" I curious how those relate to each other. Are you using SaltStack to bootstrap Kubernetes?
- lobster_johnson 10y ago
  Not the OP, but we use Google Container Engine (hosted Kubernetes), with Salt for the non-GKE VMs. This is needed because K8s is not mature enough to host all the things. In particular, stateful sets are still in beta. Not sure if I would trust K8s to run our production databases even with stateful sets. We've had K8s kill pods for unknown reasons, for example, and the volume handling has also been historically a bit flaky. Fine for completely redundant, stateless containers, less fine for stateless ones.
  - bogomipz 10y ago
    Sure that makes sense. Statefuf especially databases would make me nerovous as well. am curious, what are the advantages of GKE over deploying K8 on AWS? Or were you already GCE?
    - lobster_johnson 10y ago
      Our production setup is actually still on Digital Ocean; we're currently testing GCP (GKE + Salt-managed VMs) with a staging cluster to see how it works in practice. Before GCP, we set up a staging cluster on AWS. It was okay. The biggest pain point is that AWS's VPC does not match Kubernetes' requirement for per-pod IPs, so anyone who installs on AWS ends up setting up their own overlay network (such as Flannel or Calico). That's a big downside, because VPC isn't fun to deal with. You don't really notice how antiquated AWS is until you move to GCP. Everything — networking, UI, CLI tools, etc. — feels more modern, sleeker and less creaky. One area where GCP is particularly superior is networking. Google gives you a Layer 3 SDN (virtual network) that's more flexible than the rigid subnet-carving you need to do with AWS VPC. The tag-based firewall rules are also a breath of fresh air after AWS's weird, aging "security group" model. It's not all innovation, of course. Some services are carbon-copy clones of AWS counterparts: Pub/Sub is essentially SQS, Cloud Storage is S3, and so on, with only minor improvements along the way. Cloud Storage, for example, doesn't fix S3's lack of queryability. I've also been distinctly unimpressed with the "StackDriver"-branded suite of services, which do things like logging and metrics. I don't know why anyone in this day and age would bother making something that doesn't compare favourably to Prometheus. I should add that the security situation on GKE could be better: * GKE's Docker containers run in privileged mode. * There's still no role-based authentication. * Containers end up getting access to a lot of privileged APIs because they inherit the same IAM role as the machine. * You can't disable the automatic K8s service account mount [1]. Another scary thing, unrelated to GKE, is that the VMs run a daemon that automatically creates users with SSH keys. As a team member, you can SSH into any box with any user name. Not sure if real security weakness, but I don't like it. I love the CLI tools ("gcloud" and so on), much nicer than awscli etc, and hopefully much friendlier to junior devs. [1] You can disable the mount by mounting an "emptyDir" on top, but that makes K8s think the pod has local data, which causes the autoscaler to refuse to tear down a node. Fortunately there's an option coming to disable the service account.

tedmiston 10y ago

When I first heard about Deis it was heavily compared to Flynn [1]. It's a little hard to tell from the outside if the two are still competitors or if Deis is focusing on a lower level of abstraction. https://flynn.io/ https://flynn.io/

hackerboos 10y ago
They both support Heroku-like buildpacks. Major difference IMO is that Flynn supports databases and other stateful services. I evaluated Flynn, Deis and Dokku and wasn't entirely satisfied with any of them opting for Heroku instead. I've promised people a blogpost on this and I will do it once I get time to evaluate RancherOS too.
- baronseng 10y ago
  Can you evaluate Kontena as well while you are at it?
tuananh 10y ago
Flynn was much less polished than Deis. At least, that was when i tried it last year. I also started with Deis and move on to k8s later on.

bogomipz 10y ago

>"I've since moved on to Kubernetes and SaltStack" I curious how those relate to each other. Are you using SaltStack to bootstrap Kubernetes?

lobster_johnson 10y ago
Not the OP, but we use Google Container Engine (hosted Kubernetes), with Salt for the non-GKE VMs. This is needed because K8s is not mature enough to host all the things. In particular, stateful sets are still in beta. Not sure if I would trust K8s to run our production databases even with stateful sets. We've had K8s kill pods for unknown reasons, for example, and the volume handling has also been historically a bit flaky. Fine for completely redundant, stateless containers, less fine for stateless ones.
- bogomipz 10y ago
  Sure that makes sense. Statefuf especially databases would make me nerovous as well. am curious, what are the advantages of GKE over deploying K8 on AWS? Or were you already GCE?
  - lobster_johnson 10y ago
    Our production setup is actually still on Digital Ocean; we're currently testing GCP (GKE + Salt-managed VMs) with a staging cluster to see how it works in practice. Before GCP, we set up a staging cluster on AWS. It was okay. The biggest pain point is that AWS's VPC does not match Kubernetes' requirement for per-pod IPs, so anyone who installs on AWS ends up setting up their own overlay network (such as Flannel or Calico). That's a big downside, because VPC isn't fun to deal with. You don't really notice how antiquated AWS is until you move to GCP. Everything — networking, UI, CLI tools, etc. — feels more modern, sleeker and less creaky. One area where GCP is particularly superior is networking. Google gives you a Layer 3 SDN (virtual network) that's more flexible than the rigid subnet-carving you need to do with AWS VPC. The tag-based firewall rules are also a breath of fresh air after AWS's weird, aging "security group" model. It's not all innovation, of course. Some services are carbon-copy clones of AWS counterparts: Pub/Sub is essentially SQS, Cloud Storage is S3, and so on, with only minor improvements along the way. Cloud Storage, for example, doesn't fix S3's lack of queryability. I've also been distinctly unimpressed with the "StackDriver"-branded suite of services, which do things like logging and metrics. I don't know why anyone in this day and age would bother making something that doesn't compare favourably to Prometheus. I should add that the security situation on GKE could be better: * GKE's Docker containers run in privileged mode. * There's still no role-based authentication. * Containers end up getting access to a lot of privileged APIs because they inherit the same IAM role as the machine. * You can't disable the automatic K8s service account mount [1]. Another scary thing, unrelated to GKE, is that the VMs run a daemon that automatically creates users with SSH keys. As a team member, you can SSH into any box with any user name. Not sure if real security weakness, but I don't like it. I love the CLI tools ("gcloud" and so on), much nicer than awscli etc, and hopefully much friendlier to junior devs. [1] You can disable the mount by mounting an "emptyDir" on top, but that makes K8s think the pod has local data, which causes the autoscaler to refuse to tear down a node. Fortunately there's an option coming to disable the service account.