4 ms·

My goal for 2017 is to fully figure out Docker. The majority of information out there, including tutorials and blog articles about others' successful deploymen

My goal for 2017 is to fully figure out Docker. The majority of information out there, including tutorials and blog articles about others' successful deployments, comes in the form of very high-level overviews. Everything I've found is an introduction to getting a basic docker instance running. There is very little useful information out there as to how to run a proper multi-host cluster. There is core Docker. Tack on docker-machine, docker-compose, Swarm, and the dozens of 3rd-party cluster management abstractions such as Rancher - and the intensity of the headache never stops growing. It sounds wonderful, but there is so much to learn to be able to tackle a full production stack. It's one thing to successfully launch a working cluster after hours of manual tinkering. It's a separate beast altogether to fully automate setting up a new cluster by issuing a single command, taking into account consistent configuration of: secure networking, persistent volumes with backups, deployment of container configuration and VCS codebases (ex: nginx vhosts and your code itself), etc. My goal is to set up an entire project in such a way that there is a single suite of automation that can deploy all environments: development VM, staging, and production.

cutety 10y ago
This is also one of my goals for 2017. I've been using Docker on my project at work for about 8 months, but only in development and CI. I really want to start a push to begin testing usage in production, but have no idea where to begin. It's hard to find information that strings everything together. I feel comfortable with using it in development, but production seems to be an entire different beast.
JBerlinsky 10y ago
Are you looking to automate infrastructure-as-code in there, as well? I've lost many a night to trying to get Rancher to play nice with Terraform, especially in high-availability mode. Luckily, it appears that the process has been vastly simplified in recent rancher-manager releases.
- developer2 10y ago
  Oh great, you've just added Terraform[1] to my list of possible tie-in softwares. Thanks for that, you monster. /s Yes, at least for development, setting up the entire environment must be a one-command execution. Every new developer to a team will obviously need to progressively learn the entire stack, but they should be up and running after a single VCS checkout and installation command. I expect staging and production to be a perfect replica of the same environment developers use. As to how realistic it is to "launch the entire production cluster with a single command", with remote server provisioning, IP allocations, multiple hosts for various load-balanced pools, etc... I'll have to see when that time comes. [1] https://www.terraform.io/ https://www.terraform.io/
  - JBerlinsky 10y ago
    You're welcome ;) Terraform is designed for something close to one-command execution. You're going to have to swap in variables (e.g. AWS access keys for a particular account, domain names, IP addresses, etc.), but Terraform is designed for that. I would advise looking into Terraform Modules[0], which encapsulate this kind of work nicely. [0]: https://www.terraform.io/docs/modules/create.html https://www.terraform.io/docs/modules/create.html
  - golovast 10y ago
    You might take a look at troposphere[1] too. I've used both extensively and each definitely has it's strengths. In some cases troposphere is a better fit. [1] https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere
- ellius 10y ago
  Are there good ways to use Terraform internally / without online providers like AWS? I've just started using it with DigitalOcean, but I'd like to use it at home to do IAC with my whole setup. As far as I can tell the only good route for this is to throw OpenStack on bare metal, but I'm a newbie so I wonder if there's a better way.
  - JBerlinsky 10y ago
    You can write your own providers for Terraform[0] and bake whatever you want into its HCL syntax. In your case, this might not have significant returns over bootstrapping with Ansible/Chef/Puppet -- it all depends on what you want to do. [0]: https://www.hashicorp.com/blog/terraform-custom-providers.html https://www.hashicorp.com/blog/terraform-custom-providers.ht...
ellius 10y ago
As a self-taught programmer without much in the way of "official" pro experience (I have another job and program just to increase capabilities in that role), Docker was a revelation to me. I've programmed lots of tools at home, but getting anywhere in production is a nightmare of trying to wrangle managers and sysadmins into helping me translate my vision into a full-blown, deployed app within the business's stack. This year was the first time I was able to throw something together, Dockerize it, and have it deployed by a sysadmin with one or two minor code changes to work through our proxies. There are a lot of things that make me nervous about containers (persistence for example), but that was a game-changing experience for me and opened a whole new window of possibility.
overint 10y ago
I'm going to learn it as well, a fair few jobs seems to note Docker as a requirement, or suggestion. I've never used containers before, so it should be fun.
andrewstuart2 10y ago
My advice for container orchestration: pick Kubernetes and don't look back (or at least give it a solid try). It falls at the right layer of abstraction and gets so much right. You get automatic container scaling (by CPU usage for now), container composability, service discovery, configuration and (kind of) secret management, portability, it's open source, and really a whole lot more. It's spearheaded by Google, RedHat, CoreOS, and other organizations, so it's fairly safe to say that it won't completely disappear a la Google reader if it's abandoned. I recently migrated a whole microservice stack of a half dozen services to OpenID connect and Kubernetes in two weeks. This is with about a year of casual familiarity and playing with Kubernetes, and the same migration to OpenID connect would have easily taken me 5 or 6 weeks to do in Amazon ECS, which is what we currently use in production. Not to mention I can run a cluster on my three computers at home at no extra charge beyond electricity and play around for free. (See, dear, I'm not a hoarder!) Setting up a cluster is even simpler now with tools like kops and kubeadm. Or just get one provisioned for you by Google or Red Hat with GKE or OpenShift. I would highly recommend at the very least making it one of the solutions you try.
shrikrishna 10y ago
Start slow, and work your way up. I started with docker early, so I can see how its ecosystem can seem intimidating now, with all the different "tools" and "workflows" to know about. So my recommendation would be to go bottom up. Start with just plain old docker - learn what containers are, what makes them work, then see how to manage containers in a single host with compose. Then move on to clustering with swarm, and then move on to other cluster-mgmnt projects like k8s. Try to make your containers more space-efficient by basing your images on Alpine. Make your host more robust by using CoreOS/Rancher, and so on. You won't need all the steps (or even in the order) I've listed here. But once you've started with a base, you'll have some idea as to where you want to proceed next. If you need any help, you are free to ping me with any queries at the email in my profile. My authority on the subject: having written a book [1] and a Udemy course [2] [1]: https://www.packtpub.com/virtualization-and-cloud/orchestrating-docker https://www.packtpub.com/virtualization-and-cloud/orchestrat... [2]: https://www.udemy.com/mastering-docker/ https://www.udemy.com/mastering-docker/
xMudrii 10y ago
Absolutely same, I really want to get into it. I already tried to do so, but for some reason I was out of luck finding good resources for learning. Can someone recommend some great resources for beginners? TBH, would like a free resource. :D Beside that I want to get around HashiCorp tools, especially Vault for storing and Consul for service discovery.
echlebek 10y ago
Given your goals, you might be better off learning kubernetes instead; just treat docker as an implementation detail of that.
laithshadeed 10y ago
I found this presentation helpful: 'A workshop on Linux containers: Rebuild Docker from Scratch' https://github.com/Fewbytes/rubber-docker https://github.com/Fewbytes/rubber-docker

cutety 10y ago

This is also one of my goals for 2017. I've been using Docker on my project at work for about 8 months, but only in development and CI. I really want to start a push to begin testing usage in production, but have no idea where to begin. It's hard to find information that strings everything together. I feel comfortable with using it in development, but production seems to be an entire different beast.

JBerlinsky 10y ago

Are you looking to automate infrastructure-as-code in there, as well? I've lost many a night to trying to get Rancher to play nice with Terraform, especially in high-availability mode. Luckily, it appears that the process has been vastly simplified in recent rancher-manager releases.

developer2 10y ago
Oh great, you've just added Terraform[1] to my list of possible tie-in softwares. Thanks for that, you monster. /s Yes, at least for development, setting up the entire environment must be a one-command execution. Every new developer to a team will obviously need to progressively learn the entire stack, but they should be up and running after a single VCS checkout and installation command. I expect staging and production to be a perfect replica of the same environment developers use. As to how realistic it is to "launch the entire production cluster with a single command", with remote server provisioning, IP allocations, multiple hosts for various load-balanced pools, etc... I'll have to see when that time comes. [1] https://www.terraform.io/ https://www.terraform.io/
- JBerlinsky 10y ago
  You're welcome ;) Terraform is designed for something close to one-command execution. You're going to have to swap in variables (e.g. AWS access keys for a particular account, domain names, IP addresses, etc.), but Terraform is designed for that. I would advise looking into Terraform Modules[0], which encapsulate this kind of work nicely. [0]: https://www.terraform.io/docs/modules/create.html https://www.terraform.io/docs/modules/create.html
- golovast 10y ago
  You might take a look at troposphere[1] too. I've used both extensively and each definitely has it's strengths. In some cases troposphere is a better fit. [1] https://github.com/cloudtools/troposphere https://github.com/cloudtools/troposphere
ellius 10y ago
Are there good ways to use Terraform internally / without online providers like AWS? I've just started using it with DigitalOcean, but I'd like to use it at home to do IAC with my whole setup. As far as I can tell the only good route for this is to throw OpenStack on bare metal, but I'm a newbie so I wonder if there's a better way.
- JBerlinsky 10y ago
  You can write your own providers for Terraform[0] and bake whatever you want into its HCL syntax. In your case, this might not have significant returns over bootstrapping with Ansible/Chef/Puppet -- it all depends on what you want to do. [0]: https://www.hashicorp.com/blog/terraform-custom-providers.html https://www.hashicorp.com/blog/terraform-custom-providers.ht...

ellius 10y ago

As a self-taught programmer without much in the way of "official" pro experience (I have another job and program just to increase capabilities in that role), Docker was a revelation to me. I've programmed lots of tools at home, but getting anywhere in production is a nightmare of trying to wrangle managers and sysadmins into helping me translate my vision into a full-blown, deployed app within the business's stack. This year was the first time I was able to throw something together, Dockerize it, and have it deployed by a sysadmin with one or two minor code changes to work through our proxies. There are a lot of things that make me nervous about containers (persistence for example), but that was a game-changing experience for me and opened a whole new window of possibility.

overint 10y ago

I'm going to learn it as well, a fair few jobs seems to note Docker as a requirement, or suggestion. I've never used containers before, so it should be fun.

andrewstuart2 10y ago

My advice for container orchestration: pick Kubernetes and don't look back (or at least give it a solid try). It falls at the right layer of abstraction and gets so much right. You get automatic container scaling (by CPU usage for now), container composability, service discovery, configuration and (kind of) secret management, portability, it's open source, and really a whole lot more. It's spearheaded by Google, RedHat, CoreOS, and other organizations, so it's fairly safe to say that it won't completely disappear a la Google reader if it's abandoned. I recently migrated a whole microservice stack of a half dozen services to OpenID connect and Kubernetes in two weeks. This is with about a year of casual familiarity and playing with Kubernetes, and the same migration to OpenID connect would have easily taken me 5 or 6 weeks to do in Amazon ECS, which is what we currently use in production. Not to mention I can run a cluster on my three computers at home at no extra charge beyond electricity and play around for free. (See, dear, I'm not a hoarder!) Setting up a cluster is even simpler now with tools like kops and kubeadm. Or just get one provisioned for you by Google or Red Hat with GKE or OpenShift. I would highly recommend at the very least making it one of the solutions you try.

shrikrishna 10y ago

Start slow, and work your way up. I started with docker early, so I can see how its ecosystem can seem intimidating now, with all the different "tools" and "workflows" to know about. So my recommendation would be to go bottom up. Start with just plain old docker - learn what containers are, what makes them work, then see how to manage containers in a single host with compose. Then move on to clustering with swarm, and then move on to other cluster-mgmnt projects like k8s. Try to make your containers more space-efficient by basing your images on Alpine. Make your host more robust by using CoreOS/Rancher, and so on. You won't need all the steps (or even in the order) I've listed here. But once you've started with a base, you'll have some idea as to where you want to proceed next. If you need any help, you are free to ping me with any queries at the email in my profile. My authority on the subject: having written a book [1] and a Udemy course [2] [1]: https://www.packtpub.com/virtualization-and-cloud/orchestrating-docker https://www.packtpub.com/virtualization-and-cloud/orchestrat... [2]: https://www.udemy.com/mastering-docker/ https://www.udemy.com/mastering-docker/

xMudrii 10y ago

Absolutely same, I really want to get into it. I already tried to do so, but for some reason I was out of luck finding good resources for learning. Can someone recommend some great resources for beginners? TBH, would like a free resource. :D Beside that I want to get around HashiCorp tools, especially Vault for storing and Consul for service discovery.

echlebek 10y ago

Given your goals, you might be better off learning kubernetes instead; just treat docker as an implementation detail of that.

laithshadeed 10y ago

I found this presentation helpful: 'A workshop on Linux containers: Rebuild Docker from Scratch' https://github.com/Fewbytes/rubber-docker https://github.com/Fewbytes/rubber-docker