6 ms·

This is the problem I have with almost all container orchestration systems (DC/OS, K8s, etc.) None of them can go from 1 node running all your apps to 100, in a

This is the problem I have with almost all container orchestration systems (DC/OS, K8s, etc.) None of them can go from 1 node running all your apps to 100, in an easy straight forward manner. I wrote about this not too long ago: https://penguindreams.org/blog/my-love-hate-relationship-with-docker-and-container-orchestration-systems/#zero-to-a-hundred https://penguindreams.org/blog/my-love-hate-relationship-wit... I've also worked at both Kubernetes shops and DCOS/marathon shops and I honestly don't understand why people are choosing K8s. I'm sure a lot of it is the marketing and the power of Google behind it, but our DC/OS cluster had hundreds of nodes and could scale applications really well. Plus I found the marathon json files way less confusing. (Plus if you really want, you can use k8s to schedule containers on DC/OS, alongside marathon). That being said, that DCOS platform team was 11 people and they had tons of custom scripts, wiki pages, labels and specific networking ingress/egress points that all worked together. (You could even use a label to specify if you wanted to run containers on the local VMWare cluster or in AWS; with nodes that auto-scaled out into AWS in high load). I've met people at smaller startups who've gone the Nomad route instead, which seems a lot more sane in many ways, but still requires pretty careful planning and setup for large deploys.

asdkhadsj 8y ago
So I'm heading a shop with some serious legacy deployment woes. I've got plans for everything but scaling. Now, we're likely to hire out Ops eventually, but for now I'd like to learn devops (backend eng currently) and cut my teeth by getting the ball rolling on a slightly more robust setup. With that said, what do you recommend? We're a small shop, and our auto scaling needs a very very minimal. We're non-public facing, so our user base is pretty predictable. Yet, I'd like to migrate us to a more modern and mature system. Naturally I was looking towards K8s because of the overwhelming support I see for K8s on HN/etc, but your comment makes me question that. Honestly, it's the vast options that really screw with my head. There are just so many tools. And where they lack is not immediately evident from the outside. It's difficult to pick a direction. At the moment I'm reviewing Nomad, as it sounds interesting to work with our existing by-hand infrastructure.
- politelemon 8y ago
  I'd suggest going in baby steps. Start with the absolute basics - standalone docker containers + docker-compose. Work in those containers, deploy those same containers. Build your deployment pipeline. That'll get you familiar with the workings of containers in the docker context, and you'll be forced to iron out issues you may find with this way of working. Then slowly start to spread further - you could start with docker swarm, which is a smaller next-step for devs. Again it's to help you figure out what kind of deployment pipelines you would create and what constraints you need to introduce. The reason I call it a smaller next-step is because the docker compose file used for development is the same one you can use for deployment to standalone or swarm. By the time you start hitting the limitations of swarm, you should be ready to move to k8s and look at the various options available to you.
  - djsumdog 8y ago
    I like this. This is good advice. Containers are still fairly portable and I've worked at a shop where we moved from CoreOS+Consul to Marathon/DCOS and there were almost no app changes. That is one of the big benefits of Docker. We had some stuff just running on nodes with Docker/docker-compose and even that stuff slowly got migrated over to the big platform.
    - asdkhadsj 8y ago
      Agreed, this method sounds nice. The only thing I'd like to find on top of it is some software to manage apps across container resources and physical resources. Ie, I've got some non-cloud old windows boxes that I want to deploy to, so ideally I need a mechanism further abstracts that. I'm currently looking at Nomad, though i'm very early in research.
  - JonasJSchreiber 8y ago
    Currently on this path almost exactly. Problem description and steps all the way up to limitations of swarm. Our swarm currently handles 1M/requests per day we haven't really hit any limitations yet. Good advice though, really
fizx 8y ago
GitLab's "Auto DevOps" with GCP Kubernetes Engine is pretty good at the 1 to 100 story. I clicked several buttons and got a google-managed k8s cluster that will pretend to be Heroku for me. I've since outgrown the basic setup, but their docs showed me how to copy their "magic" configuration into my repo, and it was easy enough to edit to get what I wanted. I kinda figured I'd be bypassing Kubernetes for this job, because I didn't want to spend a week setting it up, but they made it into an afternoon.
- dsumenkovic 8y ago
  Thanks for the feedback, it's great to hear that our product is that easy to use.
specialp 8y ago
I have used Mesos/Marathon from nearly the beginning and K8s. We currently use K8s. In your third paragraph about the 11 person team with custom scripts is why K8s has won out. It wasn't really the marketing by Google, it was that K8s provided by default what pretty much everyone needs (Discovery and storage management). Mesos took the approach that big shops like where everything can be extended and call APIs but if you are not very huge scale you really do not want to be writing scripts or doing any of that. You want an opinionated full stack. Scaling from one node to 100s is not trivial in almost every stack. I just think that the requirement that something like that is trivial does not fit the use cases of almost everyone. There's always the "Can this be web scale?" question. We are able to add nodes on the fly but we would certainly face other non-k8s scaling issues with databases that are more complex. And as for moving k8s to 100 nodes with the touch of a button pretty easily that is coming/is here. That is the power of a full stack that many people are on with. It will only get easier. Writing ad-hoc scripts and combining solutions will not.
humbleMouse 8y ago
> None of them can go from 1 node running all your apps to 100, in an easy straight forward manner. Redhat's Openshift platform can do that with one click.
- Aeolun 8y ago
  Once you get it to install at least. Seriously. Installing open shift has been the most frustrating exercise in my life. I ended up finding rancher, and even though their 2.0 version is pretty much beta at the moment the installation is so much more painless than open shift.
scarface74 8y ago
I've met people at smaller startups who've gone the Nomad route instead, which seems a lot more sane in many ways, but still requires pretty careful planning and setup for large deploys. As an admitted Hashicorp fanboy who has used Consul, Nomad, and Vault why do you say that? The beauty of Nomad for me is that it can orchestrate anything - shell scripts, executables, Docker containers, etc. Once you have a Consul cluster set up as a server, it’s dead simple. Both the Consul apps and the Nomad apps are small self contained executables that can run in server mode or client mode. You set your Consul server cluster up first (crazy simple) and then your Nomad cluster automatically configures and registers itself using Consul. Adding more Nomad nodes is a simple matter of copying the Consul app and the Nomad app and running them both as clients. These days we do everything on AWS. It’s just as easy for us to spin up a bunch of small EC2 instances using autoscaling and we haven’t needed to use Docker for anything except for custom build environments for CodeBuild so I don’t have anything to compare it to.
gnur 8y ago
Unless you are using DC/OS enterprise, DC/OS is incredibly insecure by default. Any container running in the cluster has access to the full marathon API (which has NO auth). So any compromised container will be able to: stop and delete everything, see all environment variables set for other containers (often contain secrets for databases or other external apis) and even start arbitrary new containers on your cluster. So think about it, any compromise in anything you run on the cluster has the potential to access all your data. This alone should be reason enough to never use the free DC/OS, and the enterprise offering has no public pricing information.

asdkhadsj 8y ago

So I'm heading a shop with some serious legacy deployment woes. I've got plans for everything but scaling. Now, we're likely to hire out Ops eventually, but for now I'd like to learn devops (backend eng currently) and cut my teeth by getting the ball rolling on a slightly more robust setup. With that said, what do you recommend? We're a small shop, and our auto scaling needs a very very minimal. We're non-public facing, so our user base is pretty predictable. Yet, I'd like to migrate us to a more modern and mature system. Naturally I was looking towards K8s because of the overwhelming support I see for K8s on HN/etc, but your comment makes me question that. Honestly, it's the vast options that really screw with my head. There are just so many tools. And where they lack is not immediately evident from the outside. It's difficult to pick a direction. At the moment I'm reviewing Nomad, as it sounds interesting to work with our existing by-hand infrastructure.

politelemon 8y ago
I'd suggest going in baby steps. Start with the absolute basics - standalone docker containers + docker-compose. Work in those containers, deploy those same containers. Build your deployment pipeline. That'll get you familiar with the workings of containers in the docker context, and you'll be forced to iron out issues you may find with this way of working. Then slowly start to spread further - you could start with docker swarm, which is a smaller next-step for devs. Again it's to help you figure out what kind of deployment pipelines you would create and what constraints you need to introduce. The reason I call it a smaller next-step is because the docker compose file used for development is the same one you can use for deployment to standalone or swarm. By the time you start hitting the limitations of swarm, you should be ready to move to k8s and look at the various options available to you.
- djsumdog 8y ago
  I like this. This is good advice. Containers are still fairly portable and I've worked at a shop where we moved from CoreOS+Consul to Marathon/DCOS and there were almost no app changes. That is one of the big benefits of Docker. We had some stuff just running on nodes with Docker/docker-compose and even that stuff slowly got migrated over to the big platform.
  - asdkhadsj 8y ago
    Agreed, this method sounds nice. The only thing I'd like to find on top of it is some software to manage apps across container resources and physical resources. Ie, I've got some non-cloud old windows boxes that I want to deploy to, so ideally I need a mechanism further abstracts that. I'm currently looking at Nomad, though i'm very early in research.
- JonasJSchreiber 8y ago
  Currently on this path almost exactly. Problem description and steps all the way up to limitations of swarm. Our swarm currently handles 1M/requests per day we haven't really hit any limitations yet. Good advice though, really

fizx 8y ago

GitLab's "Auto DevOps" with GCP Kubernetes Engine is pretty good at the 1 to 100 story. I clicked several buttons and got a google-managed k8s cluster that will pretend to be Heroku for me. I've since outgrown the basic setup, but their docs showed me how to copy their "magic" configuration into my repo, and it was easy enough to edit to get what I wanted. I kinda figured I'd be bypassing Kubernetes for this job, because I didn't want to spend a week setting it up, but they made it into an afternoon.

dsumenkovic 8y ago
Thanks for the feedback, it's great to hear that our product is that easy to use.

specialp 8y ago

I have used Mesos/Marathon from nearly the beginning and K8s. We currently use K8s. In your third paragraph about the 11 person team with custom scripts is why K8s has won out. It wasn't really the marketing by Google, it was that K8s provided by default what pretty much everyone needs (Discovery and storage management). Mesos took the approach that big shops like where everything can be extended and call APIs but if you are not very huge scale you really do not want to be writing scripts or doing any of that. You want an opinionated full stack. Scaling from one node to 100s is not trivial in almost every stack. I just think that the requirement that something like that is trivial does not fit the use cases of almost everyone. There's always the "Can this be web scale?" question. We are able to add nodes on the fly but we would certainly face other non-k8s scaling issues with databases that are more complex. And as for moving k8s to 100 nodes with the touch of a button pretty easily that is coming/is here. That is the power of a full stack that many people are on with. It will only get easier. Writing ad-hoc scripts and combining solutions will not.

humbleMouse 8y ago

> None of them can go from 1 node running all your apps to 100, in an easy straight forward manner. Redhat's Openshift platform can do that with one click.

Aeolun 8y ago
Once you get it to install at least. Seriously. Installing open shift has been the most frustrating exercise in my life. I ended up finding rancher, and even though their 2.0 version is pretty much beta at the moment the installation is so much more painless than open shift.

scarface74 8y ago

I've met people at smaller startups who've gone the Nomad route instead, which seems a lot more sane in many ways, but still requires pretty careful planning and setup for large deploys. As an admitted Hashicorp fanboy who has used Consul, Nomad, and Vault why do you say that? The beauty of Nomad for me is that it can orchestrate anything - shell scripts, executables, Docker containers, etc. Once you have a Consul cluster set up as a server, it’s dead simple. Both the Consul apps and the Nomad apps are small self contained executables that can run in server mode or client mode. You set your Consul server cluster up first (crazy simple) and then your Nomad cluster automatically configures and registers itself using Consul. Adding more Nomad nodes is a simple matter of copying the Consul app and the Nomad app and running them both as clients. These days we do everything on AWS. It’s just as easy for us to spin up a bunch of small EC2 instances using autoscaling and we haven’t needed to use Docker for anything except for custom build environments for CodeBuild so I don’t have anything to compare it to.

gnur 8y ago

Unless you are using DC/OS enterprise, DC/OS is incredibly insecure by default. Any container running in the cluster has access to the full marathon API (which has NO auth). So any compromised container will be able to: stop and delete everything, see all environment variables set for other containers (often contain secrets for databases or other external apis) and even start arbitrary new containers on your cluster. So think about it, any compromise in anything you run on the cluster has the potential to access all your data. This alone should be reason enough to never use the free DC/OS, and the enterprise offering has no public pricing information.