20 ms·

$15 Production Kubernetes Cluster on DigitalOcean

219 points by discordianfish 10y ago

vittore 10y ago
So he basically followed it's the future story? (https://circleci.com/blog/its-the-future/ https://circleci.com/blog/its-the-future/)
- gkop 10y ago
  For what it's worth, CircleCI is going all-in on containers: https://circleci.com/blog/say-hello-to-circleci-2-0/ https://circleci.com/blog/say-hello-to-circleci-2-0/
- devoply 10y ago
  Now he can finally deploy WordPress.
  - vittore 10y ago
    You did not read it lol, he deployed Ghost!
    - devoply 10y ago
      You caught me :( But close enough. Point being overkill.
- tracker1 10y ago
  While funny, there are uses for the types of orchestration that containers can provide... for me, it's kind of awesome that finding a script for an automated deployment of any subject is now as simple as searching: <subject> dockerfile And voila, I have the steps to install <subject> ... ;-) I'm being a little snarky, but I do think that the dockerfile is pretty cool by itself, and wouldn't even mind seeing other automation frameworks use the dockerfile and the swarm configs for other purposes.
  - toomuchtodo 10y ago
    For sure! But please, for the love of Vint Cerf, don't go into production with boilerplate containerization configs.
    - tracker1 10y ago
      I usually just start with the published dockerfiles and modify as needed... Really depends on my own usage.
  - vittore 10y ago
    Well, it is funny, because after all that he just installed Ghost on it. We are using containers for pretty much same purpose - try it out different things and just throw it onto one of our dokku boxes - fast and easy way to play with something without investing heavily.
- joesmo 10y ago
  Lol. I was thinking similarly. If this is what you need to do to get Kubernets up in production, Kubernetes is far from production ready. Sadly, this type of UX is the norm in the ops world and everyone is expected to build out a custom in-house solution on top even though everyone is doing the exact same thing. AWS was built on this insane model, promising companies that they don't need operations when the need was shifted from hardware onto software, not eliminated. The devops movement has decided it is also fine with this as long as they can automate the dozens of moving parts using dozens of other moving parts. Surely nothing will fail then. Even products that manage a really great UX like Docker itself fail when it comes to providing a proper UX for higher level tools like docker swarm. I'd like to think that the industry is moving forward towards better, higher-level abstractions (like Kubernets or Heroku) but it seems to me it's only moving sideways, just like it did with the adoption of VMs.
- discordianfish 10y ago
  Pretty much! :) No, for real: It looks like I failed to communicate the intent very well. I do NOT recommend setting up such infrastructure for running Ghost. I do recommend such setup (not necessarily on DO though), if you have a sufficiently complex stack and you want to be able to deploy to it quickly. This is often the case if you're have multiple teams who want to deploy independently and you can't wait for building and spinning up VM images. Or if you're running on bare metal and can't get away with something like Docker + a few shell scripts. In my case I just run this way-to-complex-stack because I've worked in the space (at Docker and Mesosphere to be specific) and want to stay in touch with the space.
ohstopitu 10y ago
I've had a few questions about Kubernetes and production grade environments and container orchestration. 1. Why Kubernetes? (when mesosphere/dcos is available?) 2. Any available best practices when setting the entire environment from scratch? (right from server level security, to doing data backups for example) --- As a critic, I thought that if you were running it in production, you'd need 3 masters? (to ensure that if the master goes down, another takes over?) So with 3 masters, and 2 slaves...the production cost go up to about $25/m. I feel that's too expensive for what you do get (price vs performance) compared to others (I personally feel a production environment running 5 servers from packet.net @ ~180$/m is the way to go [0]) [0] - https://www.packet.net/bare-metal/servers/type-0/ https://www.packet.net/bare-metal/servers/type-0/
- vittore 10y ago
  Why DCOS if swarm is available? Not trolling, just want to put my head around it too.
  - ohstopitu 10y ago
    From what I understand, Swarm is not yet in it's prime state and seems to have a lot of issues that are not around. DCOS is based of mesos which is supposedly battle tested (I know it's the same for Kubernetes) Also, at least in mesos, you can set up time scheduled tasks which I don't think is possible with Swarm.
- avitzurel 10y ago
  I commented about DCOS vs Kubernetes many times before here on HN. I had a production cluster running on mesos/marathon (mesosphere). Also have an open source project heavily invested in it [1]. Recently, I refactored the entire environment to work on Kubernets and the open source project is following it as well (It is basically a miror of my production ENV). After this long story I will say this. They are both hard to set up correctly, especially if you follow best-practices of closed networks and publicly accessed resources (VPN, Internet Gateway etc...). Kubernetes has a lot of "force" around it right now with the community booming. Posts like this (with it's many problems), is somewhat of a proof becuase it shows a vibrant community around a project. Kube is more stable from my experience. Like you mentioned, I would not run Kube production on anything less than 200-250$ which means multiple masters and multiple slaves for the pod runners. It's nice to run something cheap to check the capabilities but not more than that. Regarding DO, I'm a big fan and run a lot of projects (personal) with them, for production grade stacks I would use AWS/Google/Azure, those give you better capabilities around hardening a cluster (security, networking) for production workloads. [1] http://docs.the-startup-stack.com http://docs.the-startup-stack.com
avitzurel 10y ago
Continuing some of the feelings here in the comments. If you want to run Ghost/Wordpress or any other container for that matter, there are much simpler orchestration tools than kubernetes. All of my blogs are hosted on a DO machine, all of them are dockerized and I "orchestrate" them with simple chef scripts. For this simple workload, I would go with swarm or something as simple as bash with upstart.
- vittore 10y ago
  That is pretty much what dokku is, with some additional bells and whistles.
  - avitzurel 10y ago
    Yeah. Point is. You don't have to go all Googly on your infrastructure, start simple. Deploying wordpress/ghost to DO takes 10 minutes.
    - josegonzalez 10y ago
      Actually, probably a bit less time if you use the project I recently wrote: https://github.com/josegonzalez/dokku-wordpress https://github.com/josegonzalez/dokku-wordpress Note: I am one of the Dokku maintainers.
- discordianfish 10y ago
  Author here: Yes, this is not a setup for running Ghost/Wordpress. It's for getting started with a production(ish) kubernetes environment which would be a complete overkill for a simple stack like the one I deployed on.
pst 10y ago
What people call production nowadays...
willcodeforfoo 10y ago
This is a great guide! Very thorough. However there are so many moving pieces it makes me think I should just use a hosted Kubernetes solution like GKE instead of trying to roll my own...
- discordianfish 10y ago
  Shameless plug: I also blogged about what you should roll on your own and what not: https://5pi.de/2015/04/22/scope-and-ownership-in-tech-companies/ https://5pi.de/2015/04/22/scope-and-ownership-in-tech-compan...
johnwheeler 10y ago
How many people are actually orchestrating a fleet of containers vs. pushing simple apps to a single box or two with Capistrano, Maven, or Fabric? I don't understand the popularity of Kubernetes and Docker relative to the aforementioned simpler technologies. Certainly some organizations have use for container orchestration, but are startups beginning with that vs. just simple deployment scripts in hopes of scaling up? What's going on? Why are these in the news so much?
- nine_k 10y ago
  Containers are also about development and testing, afaict.
- zwischenzug 10y ago
  I work for a 100K+ company and we are doing that, though far from google scale. I talk about this here: https://www.youtube.com/watch?v=Yj5I-yOaDao https://www.youtube.com/watch?v=Yj5I-yOaDao and blog on related things here: https://medium.com/@zwischenzugs https://medium.com/@zwischenzugs but I do understand your skepticism - the hype is far from reality.
- discordianfish 10y ago
  Something really simple like capistrano works usually only for a small subset of your services. With Kubernetes, you can run everything in there. And you can automate it to a very high degree: Imagine pushing a branch not only deploys it to some test system, but it actually bring up depending databases, a queue and a load balancing layer which then gets load tested. This is partially possible in the cloud as well, yet it requires building machine images which is slow. And you end up with a provider specific solution especially if you try to integrate it very well. But in short: It makes sense very soon if you run on bare metal (and arguably makes it more feasible option to run on bare metal) while it makes only sense much later if you're running in the cloud. I'd say you should have at least more than one team deploying more than every few days.
  - johnwheeler 10y ago
    I'm not super familiar with Capistrano, but you can provision systems and databases with Fabric. I'm still very doubtful of the utility of these more heavyweight technologies relative to the simpler ones for all but a very small subset of organizations. To me, it reeks of people trying to make simple things much more complicated. I haven't used Kubernetes. I tried docker and never understood why people would use it over an ordinary VM and some reusable deployment scripts.
zokier 10y ago
I notice that credential/secret management is kinda lackluster in this otherwise pretty sweet looking setup. It would be great to have Vault thrown in too for good measure. Maybe a followup article?
- discordianfish 10y ago
  Yes definitely an interesting topic. Next follow up article is probably around testing though.
nikon 10y ago
Why does Rancher[0] never get mentioned on here? I've been using it the last few months and it's been great. It handles everything for you without the huge learning curve. I'm using Gitlab.com pipelines also and it means my CI/CD pipeline is now free, and once by build/tests are run it pings the Rancher API to upgrade the service. [0] http://rancher.com/rancher/ http://rancher.com/rancher/
- chadscira 10y ago
  We have been using Rancher as well... It allowed us to move away from DO and AWS. Now most of our infra is from OVH :). It's been smooth sailing. Because of massive costs savings we were able to just reinvest it in our own redundancy. Also 12-factor apps are pretty damn resilient.
  - nikon 10y ago
    That's cool. I've actually just moved from ESXI hosts to OVH too. How are you handling persistent storage? I have opted to not "dockerize" Postgres for example after reading horror stories.
    - chadscira 10y ago
      Because theres no perfect solution at the moment we have opted for a 4 disk setup, and mirror them all using ZFS. We also do 10 minute snapshots, and incremental snapshot backups (to S3). Feels safe enough for our needs. Before this we didn't have 10 minute snapshots of all our data so this has actually allowed us to recover from dev errors as well (if they were to happen). We do have our DB running in a container + mounted volumes that use the above configuration.
    - esseti 10y ago
      Which kind of setup do you have i OVH? do you rent servers or VMs? Do you deploy docker? via kubernets or what? quite interested in this, if you have material/links on OVH and docker/kube I would not mind if you can share.
cgag 10y ago
I've been working on setting up something similar up, but it's a pain because I'm trying out a fork of kubernetes with support for digitalocean volumes. The author has a PR upstream and it looks like it'll probably get merged. I've played with kubernetes on DO in the past and had a lot of trouble with the controller node using more than 512mb of ram. Maybe I was running into a bug, but I'd be more comfortable with a 20 dollar cluster running a $10 controller node. I really look forward to more provider volume support landing upstream. If we end up with support for digitalocean, vultr, and packet volumes, we're going to be in a great place. You can run a whole lot on a 200 dollar packet cluster. Maybe most people use more of AWS than I do, but blockstorage is really the final thing I need kubernetes to abstract over for the provider to not matter to me.
- discordianfish 10y ago
  I've wrote a simple flex volume driver in bash which is included: https://github.com/5pi/infra/blob/master/packer/files/usr/libexec/kubernetes/kubelet-plugins/volume/exec/5pi.de~do-volume/do-volume https://github.com/5pi/infra/blob/master/packer/files/usr/li... but will probably rewrite it in golang once the new flex volume spec is final.
  - cgag 10y ago
    I actually hadn't heard of flex volumes until now, interesting. If you're interested here's the digitalocean pr upstream: https://github.com/kubernetes/kubernetes/pull/36894 https://github.com/kubernetes/kubernetes/pull/36894
  - cgag 10y ago
    I actually hadn't heard of flex volumes until now, interesting. If you're interested here's the digitalocean pr upstream: https://github.com/kubernetes/kubernetes/pull/36894 https://github.com/kubernetes/kubernetes/pull/36894
Axsuul 10y ago
You can also use kubeadm to bring up a cluster really quickly. We use this in production.
hackerboos 10y ago
Flynn offers a UI installer that works with Digital Ocean. 1. Login to DO and get your API keys 2. Select what images you want and how many instances 3. Click install and Flynn will setup your load-balanced environment. 4. Push apps like Heroku I've been evaluating dokku, Flynn and Deis. Just rancherOS left to do.
- jbhatab 10y ago
  I would love to hear you overall review of everything. I'm really into all the options. Dokku seems too simple for the stuff I want to do. Flynn is pretty awesome, but I've been loving the steady progress Deis makes. Rancher is also interesting. I'd love to know what was the smoothest process for something like a phoenix/node/rails app out of these. Feel free to email me at jbhatab@gmail.com if you feel like breaking it down over chat too :D
  - josegonzalez 10y ago
    Just curious, outside of multi-host support - I'm not in the business of writing a scheduler, so we'll most certainly use an existing one if we ever get to this feature - what is too simple about Dokku?
  - hackerboos 10y ago
    I'm going to write a three-part blog posts that discusses the strengths and weaknesses of each system. I'll email you when I'm done. We use Cloud Foundry in production and we've used Dokku in development. CF is similar to Dokku and Flynn as it can provide services (databases etc) and wire them to your container. Everyone running hobby projects should be using Dokku. It's very easy to get started and can replace Heroku in many cases. It is missing HA but you can always scale horizontally or load balance multiple dokku nodes (behind say nginx/haproxy) against an external db/redis although you are essentially running commands for each node. Dokku should look into Docker Swarm to do orchestration but that is a mammoth task for a pure open source project (no support business to fund it) and as josegonzalez mentions he needs the cash to be motivated to do it. Dokku also lacks a stable way to ship logs to external services. dokku-logspout is available as a plugin but it frequently loses it's connection and needs to restart. But then excels past any of the other PaaS solutions when it comes to SSL integration via Let's Encrypt. It's a truly remarkable project. Flynn provides services and sets up Postgres to balance between nodes so it too is HA. Flynn works with Google, Amazon and Digital Ocean via it's web installer (I had trouble getting it to run on a third-party provider, SSH was broken). Flynn provides Postgres, redis, MySQL and MongoDB out of the box and HA but with the caveat that you can't really customize these services. I also killed one of the three nodes via the DO console and it still carried on with no issues. Deis was the hardest of the bunch. It essential is a manual process to get Kubernetes up and running unless you use the cozy Google Cloud Container which I did but isn't cheap ($100/month before bandwidth for 2 containers). Once up and running it is as easy as the other two. They don't have any services though - people suggested running either running Postgres in a container. I think a PaaS should include the DB but that's debatable. Things I haven't tried: - adding a node to Flynn or Deis once up and running - killing a node on Deis and checking if it still runs (I assume Kubernetes takes care of this) - incorporating these systems with Gitlab to provide a full continuous delivery system
cweagans 10y ago
Okay, so there's 1.5GB of RAM total in the cluster. How much of that is taken up by the OS and how much is taken up by the Kubernetes components? Can I reasonably expect to run more than just one application on this setup?
- discordianfish 10y ago
  Not much but this is what I'm running: https://github.com/5pi/services https://github.com/5pi/services Basically postgres + two ghost instances. I'm actually surprised it works that well given the constraints. With Cloudflare in front and agressive caching it even survived this HN posting without issues (at least none I'm aware of). But sure, it's overkill if you're not mainly interested in Kubernetes itself.
sandGorgon 10y ago
I think k8s is "cheating" the private cloud story quite a bit. There are very few on-metal production deployments using k8s. This is because some of the harder problems of setting up load balancers, multiple masters, etc has not been really solved. in fact, just yesterday, there was a new post on the group to create a SIG-Metal ( https://docs.google.com/document/d/1oYtW7fgSJsQDl-ln6ETvAQrNdne4w_0am_qHTxtd3Yw https://docs.google.com/document/d/1oYtW7fgSJsQDl-ln6ETvAQrN...) because there's so less information. load balancer integration is written for all cloud providers... but not for metal (https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider https://github.com/kubernetes/kubernetes/tree/master/pkg/clo...). there's experimental integration with haproxy/traefik in contrib... but not in production. the reason is evident - a lot of the difficult parts of k8s is being "outsourced" to the underlying cloud platform. k8s needs to fix this fast... because the real disruption is being able to eliminate a cloud platform (or build your own).
- hubert123 10y ago
  that sounds bad to say the least. Isnt that the entire selling point, rolling your own cloud... I always wondered why Azure and AWS were seemingly more of a priority for a software solution that supposedly is made to replace the cloud or compete with exacxtly those services.
m3adow 10y ago
Very interesting. I'm currently working on a nearly identical project, except I started what you're only considering: Using CoreOS and cloud-config (plus some small extras). It's not done yet, but if you're interested, have a look: https://github.com/m3adow/coreos-kubernetes-tinc-cluster https://github.com/m3adow/coreos-kubernetes-tinc-cluster
arturoochoa 10y ago
I guess the definition of "Production" is not the same for everyone, I think you provided a good design reference on how to bootstrap an initial state of a kubernetes cluster. For new people, I think it still fuzzy to see the benefits of kubernetes/messos/dcos at long term view. Some of the comments here put emphasis on the deployment of Ghost/Wordpress/Custom Apps, but I think the main benefit comes on the life-cycle of those apps. What would you do if you needed to provide a auto-scaling mechanism to your apps? I think this is what the real question is... In your example you talked about a Ghost app, but on the comments you agreed this is not a good case scenario, why? how would you implement Ghost/Wordpress at large scale in top of kubernetes?

219 points by discordianfish 10y ago

So he basically followed it's the future story? (https://circleci.com/blog/its-the-future/ https://circleci.com/blog/its-the-future/)

gkop 10y ago
For what it's worth, CircleCI is going all-in on containers: https://circleci.com/blog/say-hello-to-circleci-2-0/ https://circleci.com/blog/say-hello-to-circleci-2-0/
devoply 10y ago
Now he can finally deploy WordPress.
- vittore 10y ago
  You did not read it lol, he deployed Ghost!
  - devoply 10y ago
    You caught me :( But close enough. Point being overkill.
tracker1 10y ago
While funny, there are uses for the types of orchestration that containers can provide... for me, it's kind of awesome that finding a script for an automated deployment of any subject is now as simple as searching: <subject> dockerfile And voila, I have the steps to install <subject> ... ;-) I'm being a little snarky, but I do think that the dockerfile is pretty cool by itself, and wouldn't even mind seeing other automation frameworks use the dockerfile and the swarm configs for other purposes.
- toomuchtodo 10y ago
  For sure! But please, for the love of Vint Cerf, don't go into production with boilerplate containerization configs.
  - tracker1 10y ago
    I usually just start with the published dockerfiles and modify as needed... Really depends on my own usage.
- vittore 10y ago
  Well, it is funny, because after all that he just installed Ghost on it. We are using containers for pretty much same purpose - try it out different things and just throw it onto one of our dokku boxes - fast and easy way to play with something without investing heavily.
joesmo 10y ago
Lol. I was thinking similarly. If this is what you need to do to get Kubernets up in production, Kubernetes is far from production ready. Sadly, this type of UX is the norm in the ops world and everyone is expected to build out a custom in-house solution on top even though everyone is doing the exact same thing. AWS was built on this insane model, promising companies that they don't need operations when the need was shifted from hardware onto software, not eliminated. The devops movement has decided it is also fine with this as long as they can automate the dozens of moving parts using dozens of other moving parts. Surely nothing will fail then. Even products that manage a really great UX like Docker itself fail when it comes to providing a proper UX for higher level tools like docker swarm. I'd like to think that the industry is moving forward towards better, higher-level abstractions (like Kubernets or Heroku) but it seems to me it's only moving sideways, just like it did with the adoption of VMs.
discordianfish 10y ago
Pretty much! :) No, for real: It looks like I failed to communicate the intent very well. I do NOT recommend setting up such infrastructure for running Ghost. I do recommend such setup (not necessarily on DO though), if you have a sufficiently complex stack and you want to be able to deploy to it quickly. This is often the case if you're have multiple teams who want to deploy independently and you can't wait for building and spinning up VM images. Or if you're running on bare metal and can't get away with something like Docker + a few shell scripts. In my case I just run this way-to-complex-stack because I've worked in the space (at Docker and Mesosphere to be specific) and want to stay in touch with the space.

ohstopitu 10y ago

I've had a few questions about Kubernetes and production grade environments and container orchestration. 1. Why Kubernetes? (when mesosphere/dcos is available?) 2. Any available best practices when setting the entire environment from scratch? (right from server level security, to doing data backups for example) --- As a critic, I thought that if you were running it in production, you'd need 3 masters? (to ensure that if the master goes down, another takes over?) So with 3 masters, and 2 slaves...the production cost go up to about $25/m. I feel that's too expensive for what you do get (price vs performance) compared to others (I personally feel a production environment running 5 servers from packet.net @ ~180$/m is the way to go [0]) [0] - https://www.packet.net/bare-metal/servers/type-0/ https://www.packet.net/bare-metal/servers/type-0/

vittore 10y ago
Why DCOS if swarm is available? Not trolling, just want to put my head around it too.
- ohstopitu 10y ago
  From what I understand, Swarm is not yet in it's prime state and seems to have a lot of issues that are not around. DCOS is based of mesos which is supposedly battle tested (I know it's the same for Kubernetes) Also, at least in mesos, you can set up time scheduled tasks which I don't think is possible with Swarm.
avitzurel 10y ago
I commented about DCOS vs Kubernetes many times before here on HN. I had a production cluster running on mesos/marathon (mesosphere). Also have an open source project heavily invested in it [1]. Recently, I refactored the entire environment to work on Kubernets and the open source project is following it as well (It is basically a miror of my production ENV). After this long story I will say this. They are both hard to set up correctly, especially if you follow best-practices of closed networks and publicly accessed resources (VPN, Internet Gateway etc...). Kubernetes has a lot of "force" around it right now with the community booming. Posts like this (with it's many problems), is somewhat of a proof becuase it shows a vibrant community around a project. Kube is more stable from my experience. Like you mentioned, I would not run Kube production on anything less than 200-250$ which means multiple masters and multiple slaves for the pod runners. It's nice to run something cheap to check the capabilities but not more than that. Regarding DO, I'm a big fan and run a lot of projects (personal) with them, for production grade stacks I would use AWS/Google/Azure, those give you better capabilities around hardening a cluster (security, networking) for production workloads. [1] http://docs.the-startup-stack.com http://docs.the-startup-stack.com

avitzurel 10y ago

Continuing some of the feelings here in the comments. If you want to run Ghost/Wordpress or any other container for that matter, there are much simpler orchestration tools than kubernetes. All of my blogs are hosted on a DO machine, all of them are dockerized and I "orchestrate" them with simple chef scripts. For this simple workload, I would go with swarm or something as simple as bash with upstart.

vittore 10y ago
That is pretty much what dokku is, with some additional bells and whistles.
- avitzurel 10y ago
  Yeah. Point is. You don't have to go all Googly on your infrastructure, start simple. Deploying wordpress/ghost to DO takes 10 minutes.
  - josegonzalez 10y ago
    Actually, probably a bit less time if you use the project I recently wrote: https://github.com/josegonzalez/dokku-wordpress https://github.com/josegonzalez/dokku-wordpress Note: I am one of the Dokku maintainers.
discordianfish 10y ago
Author here: Yes, this is not a setup for running Ghost/Wordpress. It's for getting started with a production(ish) kubernetes environment which would be a complete overkill for a simple stack like the one I deployed on.

pst 10y ago

What people call production nowadays...

willcodeforfoo 10y ago

This is a great guide! Very thorough. However there are so many moving pieces it makes me think I should just use a hosted Kubernetes solution like GKE instead of trying to roll my own...

discordianfish 10y ago
Shameless plug: I also blogged about what you should roll on your own and what not: https://5pi.de/2015/04/22/scope-and-ownership-in-tech-companies/ https://5pi.de/2015/04/22/scope-and-ownership-in-tech-compan...

johnwheeler 10y ago

How many people are actually orchestrating a fleet of containers vs. pushing simple apps to a single box or two with Capistrano, Maven, or Fabric? I don't understand the popularity of Kubernetes and Docker relative to the aforementioned simpler technologies. Certainly some organizations have use for container orchestration, but are startups beginning with that vs. just simple deployment scripts in hopes of scaling up? What's going on? Why are these in the news so much?

nine_k 10y ago
Containers are also about development and testing, afaict.
zwischenzug 10y ago
I work for a 100K+ company and we are doing that, though far from google scale. I talk about this here: https://www.youtube.com/watch?v=Yj5I-yOaDao https://www.youtube.com/watch?v=Yj5I-yOaDao and blog on related things here: https://medium.com/@zwischenzugs https://medium.com/@zwischenzugs but I do understand your skepticism - the hype is far from reality.
discordianfish 10y ago
Something really simple like capistrano works usually only for a small subset of your services. With Kubernetes, you can run everything in there. And you can automate it to a very high degree: Imagine pushing a branch not only deploys it to some test system, but it actually bring up depending databases, a queue and a load balancing layer which then gets load tested. This is partially possible in the cloud as well, yet it requires building machine images which is slow. And you end up with a provider specific solution especially if you try to integrate it very well. But in short: It makes sense very soon if you run on bare metal (and arguably makes it more feasible option to run on bare metal) while it makes only sense much later if you're running in the cloud. I'd say you should have at least more than one team deploying more than every few days.
- johnwheeler 10y ago
  I'm not super familiar with Capistrano, but you can provision systems and databases with Fabric. I'm still very doubtful of the utility of these more heavyweight technologies relative to the simpler ones for all but a very small subset of organizations. To me, it reeks of people trying to make simple things much more complicated. I haven't used Kubernetes. I tried docker and never understood why people would use it over an ordinary VM and some reusable deployment scripts.

zokier 10y ago

I notice that credential/secret management is kinda lackluster in this otherwise pretty sweet looking setup. It would be great to have Vault thrown in too for good measure. Maybe a followup article?

discordianfish 10y ago
Yes definitely an interesting topic. Next follow up article is probably around testing though.

nikon 10y ago

Why does Rancher[0] never get mentioned on here? I've been using it the last few months and it's been great. It handles everything for you without the huge learning curve. I'm using Gitlab.com pipelines also and it means my CI/CD pipeline is now free, and once by build/tests are run it pings the Rancher API to upgrade the service. [0] http://rancher.com/rancher/ http://rancher.com/rancher/

chadscira 10y ago
We have been using Rancher as well... It allowed us to move away from DO and AWS. Now most of our infra is from OVH :). It's been smooth sailing. Because of massive costs savings we were able to just reinvest it in our own redundancy. Also 12-factor apps are pretty damn resilient.
- nikon 10y ago
  That's cool. I've actually just moved from ESXI hosts to OVH too. How are you handling persistent storage? I have opted to not "dockerize" Postgres for example after reading horror stories.
  - chadscira 10y ago
    Because theres no perfect solution at the moment we have opted for a 4 disk setup, and mirror them all using ZFS. We also do 10 minute snapshots, and incremental snapshot backups (to S3). Feels safe enough for our needs. Before this we didn't have 10 minute snapshots of all our data so this has actually allowed us to recover from dev errors as well (if they were to happen). We do have our DB running in a container + mounted volumes that use the above configuration.
  - esseti 10y ago
    Which kind of setup do you have i OVH? do you rent servers or VMs? Do you deploy docker? via kubernets or what? quite interested in this, if you have material/links on OVH and docker/kube I would not mind if you can share.

cgag 10y ago

I've been working on setting up something similar up, but it's a pain because I'm trying out a fork of kubernetes with support for digitalocean volumes. The author has a PR upstream and it looks like it'll probably get merged. I've played with kubernetes on DO in the past and had a lot of trouble with the controller node using more than 512mb of ram. Maybe I was running into a bug, but I'd be more comfortable with a 20 dollar cluster running a $10 controller node. I really look forward to more provider volume support landing upstream. If we end up with support for digitalocean, vultr, and packet volumes, we're going to be in a great place. You can run a whole lot on a 200 dollar packet cluster. Maybe most people use more of AWS than I do, but blockstorage is really the final thing I need kubernetes to abstract over for the provider to not matter to me.

discordianfish 10y ago
I've wrote a simple flex volume driver in bash which is included: https://github.com/5pi/infra/blob/master/packer/files/usr/libexec/kubernetes/kubelet-plugins/volume/exec/5pi.de~do-volume/do-volume https://github.com/5pi/infra/blob/master/packer/files/usr/li... but will probably rewrite it in golang once the new flex volume spec is final.
- cgag 10y ago
  I actually hadn't heard of flex volumes until now, interesting. If you're interested here's the digitalocean pr upstream: https://github.com/kubernetes/kubernetes/pull/36894 https://github.com/kubernetes/kubernetes/pull/36894
- cgag 10y ago
  I actually hadn't heard of flex volumes until now, interesting. If you're interested here's the digitalocean pr upstream: https://github.com/kubernetes/kubernetes/pull/36894 https://github.com/kubernetes/kubernetes/pull/36894

Axsuul 10y ago

You can also use kubeadm to bring up a cluster really quickly. We use this in production.

hackerboos 10y ago

Flynn offers a UI installer that works with Digital Ocean. 1. Login to DO and get your API keys 2. Select what images you want and how many instances 3. Click install and Flynn will setup your load-balanced environment. 4. Push apps like Heroku I've been evaluating dokku, Flynn and Deis. Just rancherOS left to do.

jbhatab 10y ago
I would love to hear you overall review of everything. I'm really into all the options. Dokku seems too simple for the stuff I want to do. Flynn is pretty awesome, but I've been loving the steady progress Deis makes. Rancher is also interesting. I'd love to know what was the smoothest process for something like a phoenix/node/rails app out of these. Feel free to email me at jbhatab@gmail.com if you feel like breaking it down over chat too :D
- josegonzalez 10y ago
  Just curious, outside of multi-host support - I'm not in the business of writing a scheduler, so we'll most certainly use an existing one if we ever get to this feature - what is too simple about Dokku?
- hackerboos 10y ago
  I'm going to write a three-part blog posts that discusses the strengths and weaknesses of each system. I'll email you when I'm done. We use Cloud Foundry in production and we've used Dokku in development. CF is similar to Dokku and Flynn as it can provide services (databases etc) and wire them to your container. Everyone running hobby projects should be using Dokku. It's very easy to get started and can replace Heroku in many cases. It is missing HA but you can always scale horizontally or load balance multiple dokku nodes (behind say nginx/haproxy) against an external db/redis although you are essentially running commands for each node. Dokku should look into Docker Swarm to do orchestration but that is a mammoth task for a pure open source project (no support business to fund it) and as josegonzalez mentions he needs the cash to be motivated to do it. Dokku also lacks a stable way to ship logs to external services. dokku-logspout is available as a plugin but it frequently loses it's connection and needs to restart. But then excels past any of the other PaaS solutions when it comes to SSL integration via Let's Encrypt. It's a truly remarkable project. Flynn provides services and sets up Postgres to balance between nodes so it too is HA. Flynn works with Google, Amazon and Digital Ocean via it's web installer (I had trouble getting it to run on a third-party provider, SSH was broken). Flynn provides Postgres, redis, MySQL and MongoDB out of the box and HA but with the caveat that you can't really customize these services. I also killed one of the three nodes via the DO console and it still carried on with no issues. Deis was the hardest of the bunch. It essential is a manual process to get Kubernetes up and running unless you use the cozy Google Cloud Container which I did but isn't cheap ($100/month before bandwidth for 2 containers). Once up and running it is as easy as the other two. They don't have any services though - people suggested running either running Postgres in a container. I think a PaaS should include the DB but that's debatable. Things I haven't tried: - adding a node to Flynn or Deis once up and running - killing a node on Deis and checking if it still runs (I assume Kubernetes takes care of this) - incorporating these systems with Gitlab to provide a full continuous delivery system

cweagans 10y ago

Okay, so there's 1.5GB of RAM total in the cluster. How much of that is taken up by the OS and how much is taken up by the Kubernetes components? Can I reasonably expect to run more than just one application on this setup?

discordianfish 10y ago
Not much but this is what I'm running: https://github.com/5pi/services https://github.com/5pi/services Basically postgres + two ghost instances. I'm actually surprised it works that well given the constraints. With Cloudflare in front and agressive caching it even survived this HN posting without issues (at least none I'm aware of). But sure, it's overkill if you're not mainly interested in Kubernetes itself.

sandGorgon 10y ago

I think k8s is "cheating" the private cloud story quite a bit. There are very few on-metal production deployments using k8s. This is because some of the harder problems of setting up load balancers, multiple masters, etc has not been really solved. in fact, just yesterday, there was a new post on the group to create a SIG-Metal ( https://docs.google.com/document/d/1oYtW7fgSJsQDl-ln6ETvAQrNdne4w_0am_qHTxtd3Yw https://docs.google.com/document/d/1oYtW7fgSJsQDl-ln6ETvAQrN...) because there's so less information. load balancer integration is written for all cloud providers... but not for metal (https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider https://github.com/kubernetes/kubernetes/tree/master/pkg/clo...). there's experimental integration with haproxy/traefik in contrib... but not in production. the reason is evident - a lot of the difficult parts of k8s is being "outsourced" to the underlying cloud platform. k8s needs to fix this fast... because the real disruption is being able to eliminate a cloud platform (or build your own).

hubert123 10y ago
that sounds bad to say the least. Isnt that the entire selling point, rolling your own cloud... I always wondered why Azure and AWS were seemingly more of a priority for a software solution that supposedly is made to replace the cloud or compete with exacxtly those services.

m3adow 10y ago

Very interesting. I'm currently working on a nearly identical project, except I started what you're only considering: Using CoreOS and cloud-config (plus some small extras). It's not done yet, but if you're interested, have a look: https://github.com/m3adow/coreos-kubernetes-tinc-cluster https://github.com/m3adow/coreos-kubernetes-tinc-cluster

arturoochoa 10y ago

I guess the definition of "Production" is not the same for everyone, I think you provided a good design reference on how to bootstrap an initial state of a kubernetes cluster. For new people, I think it still fuzzy to see the benefits of kubernetes/messos/dcos at long term view. Some of the comments here put emphasis on the deployment of Ghost/Wordpress/Custom Apps, but I think the main benefit comes on the life-cycle of those apps. What would you do if you needed to provide a auto-scaling mechanism to your apps? I think this is what the real question is... In your example you talked about a Ghost app, but on the comments you agreed this is not a good case scenario, why? how would you implement Ghost/Wordpress at large scale in top of kubernetes?