7 ms·

As someone who stepped out of the devops world for a minute and is now trying to convert my companies infrastructure to use these tools, this is very useful and

As someone who stepped out of the devops world for a minute and is now trying to convert my companies infrastructure to use these tools, this is very useful and I'm reading through the whole thing. However, I'm still confused by how the tools in the ecosystem interact with the capabilities of various cloud providers. That is, we're using DigitalOcean and Docker and I want to get our infra to a point where I can easily spin up a brand new staging environment (staging-2 say) with an isolated postgres node using an attached volume which also say runs redis, a proxy node, a couple of nodes for application servers, and a couple of nodes for background jobs, all w/private networking, secure, w/non-root access, and run a quick task to seed dbs. I just can't seem to find guides which put the whole thing together, just pieces and I'm lost in researching an overwhelming number of tools from Ansible to Terraform to Kubernetes to Helm, etc, etc, etc.

jalfresi 9y ago
Agreed! The documentation is sparse, mostly blog posts and frequently out of date. I am finding some success with terraform style kubernetes cluster setups - there was a good one I came across for DigitalOcean that helped me trying to provision to Scaleway
rocgf 9y ago
I'm not a very versed DevOps guy, but I have used Ansible, Terraform and Kubernetes. It should go a bit like this: - Use Terraform to provision VMs, networking resources and storage from DigitalOcean. Basically, write the scripts that make your whole infrastructure available with a single command. - Then use Ansible for anything you might want installed on those machines - Kubernetes, security packages, SSH keys for your team. - Use Kubernetes to then deploy your application on top of your secure and replicable infrastructure. Each of the steps above should be roughly one shell command. If you are disciplined enough to always provision machines, install packages and deploy your app via config files, this should be very much achievable.
- jdc 9y ago
  Could you point to any articles that go into a bit more depth on this provisioning-configuration-deployment flow?
- ransom1538 9y ago
  Hey just curious: With a dockerfile is there any point to using Ansible/Puppet/Chef any more? I have used those tools to keep 'how machines are installed' in code in the past, but with dockerfiles these days I don't get the point anymore. Seems like the future will be having a few dockerfiles (staging, prod, dev, db, etc) then just finding a place to run them (digital ocean, docker swarm, aws container service - or your macbook).
  - dullgiulio 9y ago
    I don't think it's a good idea to have different dockerfiles for your staging, dev and prod environments. The point of using Docker containers is that you get to deploy to production your dev environment as is.
  - bdcravens 9y ago
    You still have to bootstrap the instances/droplets/etc with Docker, Kubernetes, etc before Dockerfiles can be used. Unless you're starting with images where those are already baked in, Ansible/Chef/Puppet are good ways to accomplish this on a clean OS.
    - zimbatm 9y ago
      It's debatable. Ansible/Chef/Puppet themselves need to be bootstrapped as well. Bash is perfectly fine if the machine can be installed with less than 100 lines of code and immutable infrastructure is being used (so no need for idempotent operations). Just make sure to add ShellCheck to the CI pipeline.
      - icebraining 9y ago
        Ansible/Chef/Puppet themselves need to be bootstrapped as well. Nope, Ansible just needs SSH access.
        Whitestrake 9y ago
        Remote hosts need Python, but you can use the raw command functionality to figure out what the remote package manager is and install Python as required.
- dominotw 9y ago
  >- Use Terraform to provision VMs, networking resources and storage from DigitalOcean. Basically, write the scripts that make your whole infrastructure available with a single command. How does autoscaling work if terraform/ansible are involved before kubernetes can take over?
  - deleted 9y ago
    [deleted]
  - joseph 9y ago
    IMO it's better for autoscaling to bake VM images and use cloud-init or similar to add per-instance config tweaks if necessary. Using Ansible to do runtime configuration is slow and doesn't play well with autoscaling. I do like Ansible, however, and use it for building EC2 AMIs. I even wrote a tool for this purpose at https://github.com/cloudboss/bossimage https://github.com/cloudboss/bossimage.
  - dalailambda 9y ago
    There are generally two levels of autoscaling involved with kubernetes. Firstly, kubernetes is able to create multiple instances of your app as load increases and scale them out over all the nodes. Secondly, the nodes your kubernetes cluster is running on can also autoscale. With terraform you can set up an AWS autoscaling group for example to automatically increase the size of your cluster as load increases.
- smw 9y ago
  You can use Ansible for step one, too. And I think I'd recommend it highly over Terraform for production provisioning.
  - purvis 9y ago
    > I think I'd recommend it highly over Terraform for production provisioning. I'm curious why you recommend this? I'd argue the opposite. With the declarative nature of Terraform, you can know exactly which part of your infrastructure is going to change before you actually apply it. With Ansible, this information is a bit more opaque. You basically just run the playbook and pray it goes as planned and things haven't drifted too much.
    - icebraining 9y ago
      You can run Ansible in check mode and see what steps will make changes.
honi 9y ago
Do you need to run staging-2 in a complete new cluster? If not, maybe namespaces can be what you need.
- andrewstuart2 9y ago
  I would very strongly recommend this approach. It's worked quite well for me. There are so many ways to split out application environments in k8s (namespaces being one, generated prefixes in a helm template being another) that you should really try to use your cluster as a single compute resource. Maybe segregate for prod if you're not confident in your ability to lock it down, but everything else should just be creating/scaling pods, then adding nodes to the cluster when it runs out of compute.
old-gregg 9y ago
If you (or someone on your team) is truly enjoying assembling DIY solutions using the best of what OSS has to offer, the possibilities of how to accomplish this are endless. But for a single-command fire-and-forget commercial solution, take a look at http://gravitational.com/telekube http://gravitational.com/telekube It allows you to, basically, define your own "Kubernetes flavor" with pre-installed components like CI/CD pipeline, databases, redis, etc, call it "developer cluster foo" and have X replicas of it running on any infrastructure. Telekube runs a single process per box and manages dozens of parts (kube services, docker, systemd, terraform) automatically for you, does atomic in-place version upgrades of the entire stack, etc. It is basically a "kubernetes appliance" and that's how many of our customers use it: ship it in a box into on-premise environments. Yes, yes. It was a shameless plug since I work at Gravitational. We believe that DevOps folks have better things to do that spend any time provisioning Kubernetes and keeping it (and a dozen of components it now consists of) alive.
- raesene9 9y ago
  Interesting! Do you document what security options you use with Telekube anywhere? One of the things I've noticed with the wide variety of k8s distros and installers that there are is that it's not always easy to establish what they've chosen to do in that regard, and as some of the options like RBAC are newer and have some operational overhead, not every distro uses them.
  - old-gregg 9y ago
    Yes, the security portion of our clusters is handled by Teleport, which is OSS and its architecture is well-documented here: http://gravitational.com/teleport/docs/2.0/architecture/ http://gravitational.com/teleport/docs/2.0/architecture/ Teleport maps your corporate groups/roles to both SSH and Kubernetes.
    - raesene6 9y ago
      thanks for that. I was perhaps more thinking of areas around how you configure kubernetes. for example, do you use RBAC, do you configure kubelet authentication/authorization, what certificates are used by the API server for etcd authentication? that kind of thing :) If there's manifests/pod speccs for the components that'd probably have the answers...
SEJeff 9y ago
Amusingly, much of the backend of digitalocean runs via kubernetes. They way they do k8s secrets in hashicorp vault is pretty nice actually: https://blog.digitalocean.com/vault-and-kubernetes/ https://blog.digitalocean.com/vault-and-kubernetes/
pstadler 9y ago
You might enjoy this guide[1] to get started. It discusses how to run a cluster on virtually any cloud provider and offers fully automated provisioning using Terraform[2]. [1] https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide [2] https://github.com/hobby-kube/provisioning https://github.com/hobby-kube/provisioning

jalfresi 9y ago

Agreed! The documentation is sparse, mostly blog posts and frequently out of date. I am finding some success with terraform style kubernetes cluster setups - there was a good one I came across for DigitalOcean that helped me trying to provision to Scaleway

rocgf 9y ago

I'm not a very versed DevOps guy, but I have used Ansible, Terraform and Kubernetes. It should go a bit like this: - Use Terraform to provision VMs, networking resources and storage from DigitalOcean. Basically, write the scripts that make your whole infrastructure available with a single command. - Then use Ansible for anything you might want installed on those machines - Kubernetes, security packages, SSH keys for your team. - Use Kubernetes to then deploy your application on top of your secure and replicable infrastructure. Each of the steps above should be roughly one shell command. If you are disciplined enough to always provision machines, install packages and deploy your app via config files, this should be very much achievable.

jdc 9y ago
Could you point to any articles that go into a bit more depth on this provisioning-configuration-deployment flow?
ransom1538 9y ago
Hey just curious: With a dockerfile is there any point to using Ansible/Puppet/Chef any more? I have used those tools to keep 'how machines are installed' in code in the past, but with dockerfiles these days I don't get the point anymore. Seems like the future will be having a few dockerfiles (staging, prod, dev, db, etc) then just finding a place to run them (digital ocean, docker swarm, aws container service - or your macbook).
- dullgiulio 9y ago
  I don't think it's a good idea to have different dockerfiles for your staging, dev and prod environments. The point of using Docker containers is that you get to deploy to production your dev environment as is.
- bdcravens 9y ago
  You still have to bootstrap the instances/droplets/etc with Docker, Kubernetes, etc before Dockerfiles can be used. Unless you're starting with images where those are already baked in, Ansible/Chef/Puppet are good ways to accomplish this on a clean OS.
  - zimbatm 9y ago
    It's debatable. Ansible/Chef/Puppet themselves need to be bootstrapped as well. Bash is perfectly fine if the machine can be installed with less than 100 lines of code and immutable infrastructure is being used (so no need for idempotent operations). Just make sure to add ShellCheck to the CI pipeline.
    - icebraining 9y ago
      Ansible/Chef/Puppet themselves need to be bootstrapped as well. Nope, Ansible just needs SSH access.
      - Whitestrake 9y ago
        Remote hosts need Python, but you can use the raw command functionality to figure out what the remote package manager is and install Python as required.
dominotw 9y ago
>- Use Terraform to provision VMs, networking resources and storage from DigitalOcean. Basically, write the scripts that make your whole infrastructure available with a single command. How does autoscaling work if terraform/ansible are involved before kubernetes can take over?
- deleted 9y ago
  [deleted]
- joseph 9y ago
  IMO it's better for autoscaling to bake VM images and use cloud-init or similar to add per-instance config tweaks if necessary. Using Ansible to do runtime configuration is slow and doesn't play well with autoscaling. I do like Ansible, however, and use it for building EC2 AMIs. I even wrote a tool for this purpose at https://github.com/cloudboss/bossimage https://github.com/cloudboss/bossimage.
- dalailambda 9y ago
  There are generally two levels of autoscaling involved with kubernetes. Firstly, kubernetes is able to create multiple instances of your app as load increases and scale them out over all the nodes. Secondly, the nodes your kubernetes cluster is running on can also autoscale. With terraform you can set up an AWS autoscaling group for example to automatically increase the size of your cluster as load increases.
smw 9y ago
You can use Ansible for step one, too. And I think I'd recommend it highly over Terraform for production provisioning.
- purvis 9y ago
  > I think I'd recommend it highly over Terraform for production provisioning. I'm curious why you recommend this? I'd argue the opposite. With the declarative nature of Terraform, you can know exactly which part of your infrastructure is going to change before you actually apply it. With Ansible, this information is a bit more opaque. You basically just run the playbook and pray it goes as planned and things haven't drifted too much.
  - icebraining 9y ago
    You can run Ansible in check mode and see what steps will make changes.

honi 9y ago

Do you need to run staging-2 in a complete new cluster? If not, maybe namespaces can be what you need.

andrewstuart2 9y ago
I would very strongly recommend this approach. It's worked quite well for me. There are so many ways to split out application environments in k8s (namespaces being one, generated prefixes in a helm template being another) that you should really try to use your cluster as a single compute resource. Maybe segregate for prod if you're not confident in your ability to lock it down, but everything else should just be creating/scaling pods, then adding nodes to the cluster when it runs out of compute.

old-gregg 9y ago

If you (or someone on your team) is truly enjoying assembling DIY solutions using the best of what OSS has to offer, the possibilities of how to accomplish this are endless. But for a single-command fire-and-forget commercial solution, take a look at http://gravitational.com/telekube http://gravitational.com/telekube It allows you to, basically, define your own "Kubernetes flavor" with pre-installed components like CI/CD pipeline, databases, redis, etc, call it "developer cluster foo" and have X replicas of it running on any infrastructure. Telekube runs a single process per box and manages dozens of parts (kube services, docker, systemd, terraform) automatically for you, does atomic in-place version upgrades of the entire stack, etc. It is basically a "kubernetes appliance" and that's how many of our customers use it: ship it in a box into on-premise environments. Yes, yes. It was a shameless plug since I work at Gravitational. We believe that DevOps folks have better things to do that spend any time provisioning Kubernetes and keeping it (and a dozen of components it now consists of) alive.

raesene9 9y ago
Interesting! Do you document what security options you use with Telekube anywhere? One of the things I've noticed with the wide variety of k8s distros and installers that there are is that it's not always easy to establish what they've chosen to do in that regard, and as some of the options like RBAC are newer and have some operational overhead, not every distro uses them.
- old-gregg 9y ago
  Yes, the security portion of our clusters is handled by Teleport, which is OSS and its architecture is well-documented here: http://gravitational.com/teleport/docs/2.0/architecture/ http://gravitational.com/teleport/docs/2.0/architecture/ Teleport maps your corporate groups/roles to both SSH and Kubernetes.
  - raesene6 9y ago
    thanks for that. I was perhaps more thinking of areas around how you configure kubernetes. for example, do you use RBAC, do you configure kubelet authentication/authorization, what certificates are used by the API server for etcd authentication? that kind of thing :) If there's manifests/pod speccs for the components that'd probably have the answers...

SEJeff 9y ago

Amusingly, much of the backend of digitalocean runs via kubernetes. They way they do k8s secrets in hashicorp vault is pretty nice actually: https://blog.digitalocean.com/vault-and-kubernetes/ https://blog.digitalocean.com/vault-and-kubernetes/

pstadler 9y ago

You might enjoy this guide[1] to get started. It discusses how to run a cluster on virtually any cloud provider and offers fully automated provisioning using Terraform[2]. [1] https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide [2] https://github.com/hobby-kube/provisioning https://github.com/hobby-kube/provisioning