25 ms·

Kubernetes Is a Surprisingly Affordable Platform for Personal Projects

652 points by cdoxsey 8y ago

ngrilly 8y ago
GKE documentation says 25 % of the first 4 GB are reserved for GKE: https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu https://cloud.google.com/kubernetes-engine/docs/concepts/clu... Is it what you observe?
sheepz 8y ago
Looking to try out Kubernetes, will definitely give this a try. Although the article starts with kind of a strawman (Kubernetes is robust section): you could do most of those with Docker (and compose) or with a non-containerized setup altogether?
topher200 8y ago
This seems awesome. I love the automated way to update Cloudflare DNS addresses via a trigger from the Kubernetes informer API.
- donmcronald 8y ago
  Yeah, that's pretty cool. I don't know much about K8S, but it looks like that might be a good strategy to build your own API gateway if `crystal-www-example.default.svc.cluster.local:8080` is a portable URI. I've never used them, but always thought the API gateway products seem like a way you could get locked into a specific platform.
07d046 8y ago
It looks like it isn't surprisingly affordable on Amazon. If I'm reading this right, EKS is 20c an hour per cluster, or about $150/month, and that's before EC2 costs for the actual machines in the cluster. https://aws.amazon.com/eks/pricing/ https://aws.amazon.com/eks/pricing/
- joseph 8y ago
  EKS is indeed expensive. I started creating https://github.com/cloudboss/keights https://github.com/cloudboss/keights for automating Kubernetes stacks before EKS was a thing, but I still prefer it over EKS. You can use it to set up a cluster using plain CloudFormation, or with an Ansible role which wraps the CloudFormation stack.
- chrissnell 8y ago
  You don't need EKS. Just use kops to build your cluster on plain EC2 instances.
- throwaway2016a 8y ago
  You can run Kubernetes on AWS without EKS and if you use t2.micro instances it would be around $28 per month if you use the load balancing technique here (where it is DNS based on an actual LB). $21 a month if you account for the free tier (and this article uses Google's free tier). If you further use spot instances (the equivalent here to "preemptible") you can save a ton more. The real savings (or $7 per month) is that on GCE you won't need to run your control server. GCE does that for you for free. Edit: now another issue here is (and I have no idea if this is true for GCE like it is AWS). t2 instances will throttle you if you use too much CPU which can be an issue when running a lot of containers.
  - tannhaeuser 8y ago
    Sure but the context is personal projects or portfolio/small business sites. A VPS on a discount provider such as Hetzner or OVH costs around 3-4 USD/EUR per month. A redundant VPS (two VPS and Ceph) costs 10 EUR on OVH. Not to speak about the complexity of setting up and running k8s. You'll be needing a second k8s development/testing environment for staging as well. So, no, I don't think k8s makes sense economically.
deckarep 8y ago
When picking preemptive machines for this demo project is the intention here to simply save money for a pet project? I guess my main question is would anybody actually use preemptive for k8s knowing that if a worker node gets reaped k8s will magically route only to healthy nodes while the preempted instance replacement is standing up?
- tmaier 8y ago
  Sure! First of all, GK3 would create new nodes when one of them gets killed. Second, builidng “cloud-native” applications means to build stateless applications, e.g. using 12-factor. So it does not really matter if your application gets killed, restarted or horizontally scaled. For databases, i always recommend to use a PaaS offering to have an easy and worry-free life
  - donmcronald 8y ago
    What if all 3 nodes get killed at the same time? Can you use the always free instance + 2 preemptible instances?
    - DuskStar 8y ago
      You can set GKE up to use multiple groups of nodes, yes. In this case you'd want to configure one group with a single non-preemptible micro node, and a second with two preemptible micro nodes. It's not something I'd really worry about for a hobby project though.
- halbritt 8y ago
  I use preemptible nodes extensively in my production environment.
- maktouch 8y ago
  We pretty much run 98% of our stuff in preemptibles. It's a free chaos monkey that makes your stuff cheaper too! Just make sure your app has proper probes, and make sure the probes returns 500 when the process receives a sigterm, and it should be invisible to you.
quaunaut 8y ago
I'm beginning to think that what Kubernetes needs by far, is a better development environment story. 1. Something that makes it easy to run the application(s) you're working on locally... 2. ...but have it interact with services in a local kubernetes cluster easily. 3. Something that makes your entire development environment effectively immutable, read-from-files the same way as the immutable infrastructure we're talking so much about. 4. Something that gives you a good start to running in production, so the first day of bringing your service up isn't a mess of trying to find/remember what environment variables/config options to set. 5. Something that's one tool, instead of a mixture of `minikube`, `kubectl`, `docker`, `draft`, and `telepresence`. 6. Something that's opinionated, and follows "convention over configuration", with conventions that make for the best future designs. Basically, we need something with a Rails-like impact but in the Kubernetes ecosystem.
- rememberlenny 8y ago
  Check out BOSH from pivotal labs.
- thebluehawk 8y ago
  Docker (at least on my mac) now comes with kubernetes built in, so you can just click a button and instantly have a kubernetes cluster running. As for an app deployment tool I've had good success with Helm. It allows you to define all the services, app, variables, etc. that your app needs. Though the templates do seem to be a bit convoluted and hard to read at times if you aren't careful.
  - GauntletWizard 8y ago
    +1 for docker for Mac, -1 for Helm. Helms templating is just too simple for the actual complexity of k8s, but it can make a decent starter too.
    - DuskStar 8y ago
      Helm + Kustomize [0] to add in whatever wasn't in the helm template works pretty well IMO. Kustomize is pretty new and not the most user friendly though. 0: https://github.com/kubernetes-sigs/kustomize https://github.com/kubernetes-sigs/kustomize
lostmsu 8y ago
Why do you even need Kubernetes when you can get free hosting from Azure (AppServices?) or Google (Cloud Engine?)?
- thrower123 8y ago
  Azure AppServices are hard to beat. One-click deployments from Visual Studio, or it takes about five minutes to setup continuous deployment from git if you use a free private VSTS repo. If you want to use the tier that allows a real domain instead of the azurewebsites domain, it costs about $10/month. Kubernetes seems like a lot of unnecessary complexity for small workloads. Kind of plinking tin cans with a nuclear bazooka. What is the break-even point where all of this stuff starts to really be worth bothering with?
  - sp527 8y ago
    > What is the break-even point where all of this stuff starts to really be worth bothering with? Series B
- mohamedmansour 8y ago
  You don’t, behind the scenes they use orchestrations like kubernetes or service fabric. So much easier deploying a docker container as a Azure Web App, so easy to auto scale based on thresholds of certain resources, so easy to debug what’s wrong since they give you all the tools. I use that in my current projects today. But Azure Kubernetes in my last project was easy too, it comes with a sophisticated monitoring solution, it integrates nicely with cheap Azure Load Balancer, but still too much overhead than Azure Web App.
tln 8y ago
The k8s system containers are quite resource hungry. The author is clearly very good at k8s (I enjoyed the article), cos running it cheap ain't easy
pstadler 8y ago
Great article! I‘m running a full-featured three node cluster for less than $10/month on Hetzner Cloud. This kind of setup requires a little more effort, but it can be fully automated and you learn a couple of things about Kubernetes on the way. I published and maintain this guide[1], including provisioning using Terraform and a bunch of example manifests to get you started. [1] https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide
- rtp 8y ago
  This guide is great. Thank you!
- raesene9 8y ago
  One thing you might want to look at based on a quick scan over your guide is that, at the moment ,it looks like you're running your etcd cluster with no authentication. Any attacker who was able to get on to the pod network (e.g. if there's a vulnerability in one of the apps running on the cluster) could hit the etcd endpoint and dump the contents of the database, which generally includes sensitive information like k8s secrets.
  - pstadler 8y ago
    Indeed, see https://github.com/hobby-kube/guide/issues/6 https://github.com/hobby-kube/guide/issues/6 Edit: I think this was you.
    - raesene9 8y ago
      ah yeah sorry I'd forgotten I mentioned it before.
- merb 8y ago
  nice guide, the only problem i see with hetzner cloud is that they do not allow to have a operating system like coreos or even the coreos (Container Linux) fork flatcar or even RancherOS.
- mahesh_rm 8y ago
erikb 8y ago
Sure if you put enough effort in you can make everything usable and cheap. But why put in the effort into that instead of investing it into building a system that naturally fufills such requirements? Kubernetes was never intended as a system for hobbyists, but for bringing more users onto google cloud. At that it is incredibly successful. But in exchange a lot of other stuff it does really badly. It is much better to start with a single-node system in mind that expects getting more nodes at some point and develop from there. This will bring in a lot of hobbyists and really carves out how to solve the developer story. Then this will grow automatically with the size of popular projects building on it. That said, of course it's nice if people invest energy into finding other usecases for k8s as well. But it shouldn't be the community's main effort.
superkuh 8y ago
This article, and a lot of others, seem to be using an overloaded meaning of "personal". All the arguments stem from the assumption that personal projects have the same uptime, scaling, and deployment requirements as some large commercial site. Some personal projects might but I'd argue most of those are really commercial in the sense that you're developing them for a portfolio or the like. They're not personal. It makes sense to use business tools like kubernetes when you're making a demo business site. It doesn't make it a good platform for personal projects. The best platform for those is hosting from home on a single computer, for free, with total control. ie, personal.
- daurnimator 8y ago
  I disagree. Even for personal projects I want to be able to access them from my phone while out and about. Something running on my home computer (which is turned off when I'm not there!) doesn't help. Nevermind that my home internet isn't fast or reliable. If you include running my own blog/mail/XMPP/matrix servers in "personal projects", then I would love to have perfect uptime. Additionally, being able to reuse high quality helm charts here just makes things simple instead of having one set of instructions for "personal" use and one set of instructions for everyone else.
  - wafflesindeed 8y ago
    Why would you want to be able to access your personal projects from your phone? What context requires this over waiting and solving the problem at home? What about work life balance? (yes, it's a personal project, but still)
    - rickycook 8y ago
      i have a paperless filing system that i frequently access from the office for things like scans of bills that aren’t yet electronic, etc. it’s fantastic to be able to access it all (via VPN; never direct) wherever i like
    - jamessb 8y ago
stephanlindauer 8y ago
we, at rebuy, built this little thing to run kubernetes on just one node: https://github.com/rebuy-de/terraform-aws-coreos-k8s-single-node https://github.com/rebuy-de/terraform-aws-coreos-k8s-single-... my colleague and i mainly use it to run our private hobby-projects.
- StavrosK 8y ago
  I freaking love Dokku for this setup. All the ease of use with none of the headache.
sklivvz1971 8y ago
The list of things mentioned in the article to do and learn for a simple, personal project with k8s is absolutely staggering, in my opinion. Having used it, there's a sizeable amount of further work needed which the article doesn't mention (e.g. learning how to use the pretty confusing google interface, finding the right logs and using their tools). So the overhead is really huge. Furthermore, the whole system is slow. Want to run a SQL query against your postgres? You need to use a google cloud command that changes the firewall and ssh's you in on the machine... and this takes a couple of minutes, just enough to make me desist unless I _really_ need to run that query. Abysmal. Finally, and this is a pet peeve against many advocacy blog posts, they just show you the happy path! Sure, _in the best of cases_ you just edit a file. In a more realistic case, you'll be stuck with a remote management system which is incredibly rich but also a really steep learning curve. Your setup is not performant? Good luck. Need to tweak or fine tune? Again, best of luck. We've tried to adopt k8s 3-4 times at work and every single time productivity dropped significantly without having significant benefits over normal provisioning of machines. {Edit: this does not mean k8s is bad, but rather that we are probably not the right use case for it!} ...which in turn is usually significantly slower than building your own home server (but that's another story!)
- arountheworld 8y ago
  If you design your app properly you won't ever need to run SQL queries by hand. This is a bad practice and it is good that k8s discourages such behaviour. I agree with your other points though.
  - barrkel 8y ago
    You have no idea what you're talking about to make a blanket statement like that.
  - reacharavindh 8y ago
    SQL is too low level?! I don't think you understand what is going on with the tools you use.
    - 8y ago
snaky 8y ago
> Kubernetes, right now, supports some workloads really really well, like stateless workloads that are CPU intensive that can be easily scaled horizontally, with easy to understand patterns that we currently put behind load balancers. > Other workloads, Kubernetes just doesn’t support well and likely will not support well in the near future because it’s really hard to do. It’s not magic, as some of the hype surrounding it would have you believe. https://gravitational.com/blog/running-postgresql-on-kubernetes/ https://gravitational.com/blog/running-postgresql-on-kuberne...
deleted 8y ago
[deleted]
ratiolat 8y ago
I'm trying to set up a deployment pipeline, quite from scratch. That also includes picking source repository software. Self-hosted is a must for every component. So I chose gitlab and after that I dived into it. Oh boy was I in for a surprise. For CI/CD one needs docker, kubernetes and whatnot: https://docs.gitlab.com/ee/topics/autodevops/ https://docs.gitlab.com/ee/topics/autodevops/ But the alternatives don't look better either. So, fellow hn'ians. My requirements ATM are quite simple: Submit code via git, run automated test, if all good, put the files into the live system. Pull-requests are must and everything must be self-hosted because cloud is just another persons computer. Code is written in python. What would you recommend? Edit: grammar.
- bthornbury 8y ago
  Sounds like you can get away with a bash script that calls rsync, and restarts the code on the server with ssh. (I deploy one of my websites this way, simple & reliable) If you're looking for something more legit: Ansible
- te_chris 8y ago
  Heroku. It is everything you want.
- jl-gitlab 8y ago
  Hey there, I'm the product manager for CI/CD at GitLab. AutoDevOps is one of our more advanced features to get up and running in a k8s environment quickly, but is not required if you just want basic CI. Our documentation on getting started with CI/CD is here: https://docs.gitlab.com/ee/ci/ https://docs.gitlab.com/ee/ci/
  - ratiolat 8y ago
    Thank You! I will look into that. OTOH this is how I came to conclusion that I must go the AutoDevOps way: This is my first installation of gitlab ever. So naturally, after somehow successfully installing it on the 2nd try, everything seems to work. So - lets explore configuration options. For example we should disable registration since, well, we don't want to be a public gitlab host. After a while, after the Pages section, we come to a section named "Continuous Integration and Deployment". And there's a checkbox named "Default to Auto DevOps pipeline for all projects". And there does not seem to be any other kind of "Continuous Integration and Deployment" available so one naturally comes to a conclusion that the only way seems to be Auto DevOps. My bad.
deleted 8y ago
[deleted]
TekMol 8y ago
What does it give me over a bare VM?
- movedx 8y ago
  Are you serious right now?
  - TekMol 8y ago
    Yes
    - movedx 8y ago
      OK, let me reverse the question and put it to you: what does a bare VM get you over a VM with Minikube on it?
    - movedx 8y ago
      (can't edit my original comment at this point) Didn't think so.
- axaxs 8y ago
  for the vast majority of folks, resume fodder.
heipei 8y ago
I'm surprised in all of these posts about k8s on HN the Nomad project (by Hashicorp) rarely comes up. I've found it an absolute joy (yeah, I said it) to use and a breeze to setup. Unless you're setting up a container orchestration system where you want to customise every single aspect while you're setting it up, my guess is you're better off with Nomad at first. At the very least you'll learn what you need from a container scheduler and can make a more informed decision if you re-evaluate your options down the line. Nomad also integrates perfectly with Consul and Vault, so you're never left wondering "is this is how I'm supposed to do it?" which is something that happens if you're just starting out. URL: https://www.nomadproject.io/ https://www.nomadproject.io/
- shaklee3 8y ago
  Check out that hashicorp blog over the last three weeks. Consul integrates tightly to kubernetes now. Vault already did.
  - josegonzalez 8y ago
    Consul only integrated "tightly" with kubernetes in the past few weeks. The vault integration is pretty barebones, and anything done there has been spearheaded by the Kubernetes community.
aksels 8y ago
From my experience it's quite expensive, at least in GKE. As Kubernetes takes a lot of cpu & memory by itself. I run a pool of one node n1-highcpu-2 (2 vCPUs, 1.8 GB memory) and it costs me about 72 euros per month. kube-system is taking about 730 mCPU of the node's 2000 mCPU. The issue is that the kube-system is deployed on each node of the pool so (tell me if I misunderstood/misconfigured something). If you have a pool of 3 nodes which have 1 vCPU each, with kube-system taking approximately 700 mCPU on each you have only 300 * 3 = 900 mCPU allocatable. If you have any tips on how I could reduce the costs of my personal projects I'm listening !
- ngrilly 8y ago
  When creating to disable the cluster, have you tried to disable Kubernetes dashboard and StackDriver monitoring and logging services? For logging, you could use FluentBit which is more lightweight: https://docs.fluentbit.io/manual/output/stackdriver https://docs.fluentbit.io/manual/output/stackdriver
- molf 8y ago
  As a data point, on a busy GKE production cluster with 12 highcpu-16 nodes we have kube-system pods taking a total of ~700mCPU on all servers combined. I'm not sure how the CPU usage scales, but probably not linearly. Most of the CPU usage is kube-dns, which I think requires much less CPU if you make few internal network connections.
- puzzle 8y ago
  I just saw a change the other day which makes system pods fit better on nodes with one vCPU. It reduced their numbers by the same factor (i.e. keeping the same proportions). It was meant for GCE and I am not sure if/when it will make it to GKE, but it looks like Google is looking at the tiny node scenario. Also, Tim Hockin led a discussion about something similar at last year's Kubecon.
  - puzzle 8y ago
    Here it is. It's in 1.12, actually, not on HEAD as I recalled: https://github.com/kubernetes/kubernetes/pull/67504 https://github.com/kubernetes/kubernetes/pull/67504 Still not sure if these are the configuration files that GKE uses behind the scenes, but the author is a Google employee.
leandot 8y ago
For personal projects I am using multi-node Hetzner instances managed with a Terraform script heavily inspired from https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide + RKE for K8S setup and it works really well.
- hobofan 8y ago
  For personal projects, I am using a single dedicated server at Hetzner with Docker & HAProxy. It's beefy enough to be able to handle as many personal projects as I have time to dedicate to. I've tried to replace my Docker setup with Kubernetes multiple, but it was always too much of a hassle. Things that were easy with a "bare" Docker setup became hard, and single-node clusters always felt like a second-class citizen.
  - ngrilly 8y ago
    I almost do the same (Docker and nginx as a reverse proxy on Google Compute Engine or Digital Ocean virtual machines). But how do you manage deployments of new versions without downtime? I use Docker Compose, but I had to complete it with some scripts to manage the deployment (start the new containers, wait for new containers to be ready, switch traffic to new containers, shutdown old containers), and it's a bit messy and non-standard. I'm thinking of using using Docker in swarm mode which offers the concept of services with automated deployments.
    - leandot 8y ago
      I would not recommend Docker Swarm, it's way too simplistic and my feeling is that it lags so much behind k8s that it might get discontinued. I've done Ansible, Docker Swarm, custom scripts and nothing is as clean as using standard K8S components + it gives you a good skill on your resume.
      - ngrilly 8y ago
        I'm afraid too of a discontinuation of Docker swarm mode... And agreed about having Kubernetes on your resume being a good thing.
nicolaslem 8y ago
Having worked with Mesos, Nomad and an in-house container scheduler, I would not pick any of them for personal projects. I've not tried Kubernetes yet but I don't think it would make me change my mind. I recently went through the infrastructure that runs my personal projects (Django, PostgreSQL, Redis, a few static sites...). I decided to go with plain Docker + Ansible. These tools give me enough automation to easily scale up or change hosting provider if I need, without introducing too much magic and complexity.
- djsumdog 8y ago
  I'm in the exact same boat with you. I was at a shop that used DC/OS at work, and to be fair, it's probably the easiest to get initially setup. But even a basic setup is going to be really expensive, as I discovered: https://penguindreams.org/blog/installing-mesosphere-dcos-on-small-digital-ocean-droplets/ https://penguindreams.org/blog/installing-mesosphere-dcos-on... I ended up doing what you did, having Ruby provision my machines, configure them with Ansible and deploy my docker containers (calling the Docker API from Ruby). The whole thing can be found here: https://github.com/sumdog/bee2 https://github.com/sumdog/bee2 It was a lot of work setting all that up. I should have probably just used Terraform + Ansible + docker-compose. Still, I'm glad I wrote it. I learned a lot about the complexities of running a docker infrastructure.
- ngrilly 8y ago
  I do the same (Docker + Ansible). I also use Docker Compose. How do you deploy new versions of your apps without downtime (start the new containers, wait for them to be ready, switch traffic to new containers, shutdown old containers)? I'm using custom Ansible scripts coupled to docker-compose but it's a bit messy and non-standard.
  - nicolaslem 8y ago
    To update an app container to a new version I just run Ansible again: - docker_container: image: foo/bar:latest state: started restart_policy: always pull: yes This pulls the new version and restarts the container. The downtime is probably a second maximum. It's small enough that it is not worth the trouble of a rolling release or whatnot. It's a personal project after all, no one will complain for getting a 502 once a month :) I do not use docker-compose as I never felt the need to and I read on multiple occasions that it's meant for development only.
Can_K 8y ago
I have not yet hosted anything in a container/cloud. Therefore maybe a stupid question, how do you keep your system up to date? What am I required to do in order to receive new operating system and software versions?
- minimaxir 8y ago
  A hosted provider like GKE will update the OS of the master/nodes for you. (although the OS itself doesn't do much; Google uses a container-optimized OS that only handles containers and has a much smaller footprint)
bwm 8y ago
Why not just outsource hosting to Heroku or something similar? Unless you're building something with extremely exotic requirements, it seems like you're just wasting time trying to build out your own infra.
paulcarroty 8y ago
Well, let's talk about bad side: 1) Google Cloud - hello privacy and support issues. 2) Traditional enterprise-style overcomplexed configs and architecture - the shitty trick "we want your losted time as justify to stay here" 3) Forget about quick deployment and scaling.
geku 8y ago
At https://www.kubebox.com https://www.kubebox.com we are working on affordable Kubernetes clusters for developers and personal usage. It's not as cheap as the mentioned solution but also comes with more memory and CPU power. Around $36 for 8GB and maybe less for just a 4GB node, not yet decided. Master is fully managed and hosted by us. What do you think?
gizzlon 8y ago
Or you can push your webapp to Appengine standard and it's 0$ per month =) Appengine requires you to learn some new stuff, but so does k8. Not the right choice for all apps, obviously, but something to compare this setup to.
pawurb 8y ago
Dokku is a viable infrastructure solution for side projects, even if you don't have much dev ops exp https://pawelurbanek.com/rails-heroku-dokku-migration https://pawelurbanek.com/rails-heroku-dokku-migration
parsadotsh 8y ago
While I don't have experience with large production systems, for hobby projects I found Nomad to be much easier to start with. Even the K8s distros that were supposedly "minimal" took a huge amount of time to setup in a reproducible manner. Plus Nomad's fantastic integration with consul makes it more than just a scheduler and a more complete alternative to Kubernetes.
danieldk 8y ago
I use NixOS, which allows you to define machines declaratively. To the points that he raises: How do you deploy your application? Just rsync it to the server? Define an overlay for your application. For a typical application, this is just a few lines of code with stdenv. What about dependencies? If you use python or ruby you're going to have to install them on the server. Do you intend to just run the commands manually? If you have an overlay for your applications, you can specify dependencies in the derivation of your application. Even though these are not web apps (I primarily work on local applications), I have derivations for some of my applications, so that I can install them on every machine with little effort. Two examples: https://github.com/danieldk/nix-home/blob/master/overlays/30-alpinocorpus.nix https://github.com/danieldk/nix-home/blob/master/overlays/30... https://github.com/danieldk/nix-home/blob/master/overlays/20-finalfrontier.nix https://github.com/danieldk/nix-home/blob/master/overlays/20... If I want to develop one of the applications, say alpinocorpus, I can just drop in a shell with all the dependencies (cmake, boost, Berkeley DB XML, etc.) available: nix-shell '<nixpkgs>' -A alpinocorpus When you exit the shell, you are again in your usual environment. Another option is to create a default.nix file in a project directory and use direnv to automatically load the environment when you switch to a directory. How are you going to run the application? Will you simply start the binary in the background and nohup it? You can define (systemd) services in a NixOS configuration. How will you handle running multiple applications with different domain names or http paths? (you'll probably need to setup haproxy or nginx) NixOS has support for defining nginx virtual hosts. You can also enable Let's Encrypt and Nix will set up systemd timers to automatically renew the certificates. E.g. here is a configuration of one of my machines with a bunch of virtual hosts with Let's Encrypt: https://github.com/danieldk/nixos-config/blob/master/machines/castle.nix https://github.com/danieldk/nixos-config/blob/master/machine... Suppose you update your application. How do you rollout the change? Stop the service, deploy the code, restart the service? How do you avoid downtime? nixos-rebuild switch --upgrade Or with nixops you can deploy and update machines remotely. What if you screw up the deployment? Any way to rollback? (Symlink a folder...? this simple script isn't sounding so simple anymore) Yes, every change creates a new generation. You can always switch into or boot into an older generation as long as it isn't garbage collected. And this works for the whole system, including kernel updates, etc. Does your application use other services like redis? How do you configure those services? Declaratively ;). Adding services.redis.enable = true; to your NixOS configuration does the trick. As a bonus, with nixops you can deploy a machine configurations on real-hardware, VPS, Virtual Box, KVM, etc.
nathan_f77 8y ago
No-one has mentioned Convox yet: https://convox.com https://convox.com Kubernetes is great, but Convox is way easier to set up. It's much closer to having Heroku in your own AWS account. It also leverages a lot of AWS services (ECS, RDS, etc.) instead of re-inventing the wheel. The only downside is that it only supports AWS. I'd recommend Convox for personal projects, but it can be a bit expensive because you have to run a minimum of 3 EC2 instances.
- bdcravens 8y ago
  In the past I've had similar success with Cloud 66.
nickjj 8y ago
I don't think you're being fair with your comparisons to DO when it comes to price. The f1 micro $5/month Google servers are SO much worse. Those "always free" f1 micros have 600mb of RAM, do not have SSDs and according to some tests[0] you only get 20% of 1 vCPU. There's also some pretty severe write limitations. That's a really big difference vs DO's 1gb of RAM, SSD and 100% of 1 vCPU. The performance difference alone with SSD vs no-SSD is massive, especially on a database and while most web apps aren't CPU bound, 20% of 1 vCPU is bound to run into problems beyond a simple hello world that you set up in your example. Also what happens if your app doesn't even fit in that 600mb? On DO you could get a 600mb app + 100mb OS + 150mb DB + whatever else running on that $5/month server since it has 1gb of ram. In a real world application I wouldn't be surprised if that cluster performs way worse than a single DO server in the $5/month range. In some cases your app wouldn't even be able to run. [0]: https://www.opsdash.com/blog/google-cloud-f1-micro.html https://www.opsdash.com/blog/google-cloud-f1-micro.html
- minimaxir 8y ago
  This is part of the reason why I use a similar setup as OP, but with a single n1-standard-1 instance (1 vCPU/3.75 GB RAM) instead ($7.30/mo w/ preemptible). The 3-node requirement of GKE is not a hard requirement apparently. Of course, that's mostly for running Jobs; a single node + preemptible with a production app is not a great idea.
ksajadi 8y ago
In my experience, having a reliable, configurable and flexible enough pipeline to generate the needed configuration files for Kubernetes makes a lot of difference in the adoption. Tools like Helm or Cloud 66 Skycap are some examples but I'm sure there are more.
bcoughlan 8y ago
I know a mid-sized company who have a team of 3 people dedicated to managing their Kubernetes clusters. For a small company that can't afford the overhead of that, it's an extra layer of risk and complexity. Amazon EKS launched last month in the EU west region, which I was excited about. But each cluster is $150 per month (20c per hour) before you even add any servers. To have a dev, qa and production cluster we're looking at $450 per month before the cost of EC2 instances to actually run the application. Our EC2 costs for backend services are about $800 per month, so it instantly negates the selling point that we can use reduce costs by using resources more efficiently. Our current deployment setup is groups of Docker services deployed to a set of EC2 instances via Docker Compose with a load balancer in front of each server group. Having isolated server groups means that outages are isolated to particular areas. I don't know how long we'd be down if there was an issue with the Kubernetes cluster. If we introduced Kubernetes it would be left in the hands of a couple of developers to manage, which is the opposite of the DevOps culture we strive for. We stay cloud-agnostic so it's easy for a developer to spin up a stack of dependent services with Docker Compose (Nginx, Redis, RabbitMQ etc.) when working on changes to a service. For a developer to run services locally with Kubernetes, they need to set up MiniKube. Many corporate environments have tight integration with Windows and our developers run Linux in a VM. I experimented with getting MiniKube to run in a VM but found it hard to get going with the vm-driver=none option, and it's something I definitely didn't want to add to our onboarding process. Kubernetes has a serious hype train at the moment reminiscent of the MongoDB days. My inner conspiracy theorist says that this is because Google has a vested interest in preventing vendor lock-in on cloud providers. It solves a Google-sized problem and introduces some complexity with the promise of removing more complexity than it adds. For small to mid sized companies, however, it adds far more complexity than it removes.
- StavrosK 8y ago
  To be fair, you don't need three clusters, you only need one and you can namespace things.
  - bcoughlan 8y ago
    Maybe I'm oldschool but I'd be quite wary about mixing anything in production, qa and dev environments.
maxhallinan 8y ago
I tried out Kubernetes (managed with kops) for a personal project. I had a single websocket endpoint. I wanted to terminate SSL at the load balancer. Found this to be extremely complicated and the troubleshooting resulted in a ~$100 AWS bill.
djhworld 8y ago
I looked at Kubernetes for personal stuff but the overhead was too much for something I'd rather just set and forget. My current solution is to just run docker on some raspberry pi's and have scripts to replace the container when I do an update, with volumes regularly backed up to S3. Works fine, although you don't get the clustering capability (I might try docker swarm one day but for now it's fine)
IshKebab 8y ago
This is so over-complicated for a personal project.
nova22033 8y ago
How are there only 2-3 references to minikube in the comments?
chuckdries 8y ago
I prefer simple tools. - Heroku is cheap for personal projects. - A single Digital Ocean droplet where you manage everything yourself, is cheap for personal projects. - If you're working in node, Glitch.com is literally free node hosting
JAdamMoore 8y ago
Stop trying to make Kubernetes cool. Guess what? People don't need to scale everything, or even better, most people don't need to learn or use Kubernetes, like, ever. Get over it. You made a tool you thought would scale, it didn't, because not everything needs to scale.
xaduha 8y ago
What kind of personal projects need a cluster? I'm pretty sure docker-compose + swarm can satisfy 99% percent of your needs.
rbanffy 8y ago
It really depends on your personal project. Kubernetes is a complicated, complex beast. Even on managed K8s on a cloud provider, you still need to learn a lot of stuff that has absolutely nothing to do with what you intend to accomplish (unless your personal project is learning to use K8s. Most of my personal projects get deployed on Google's App Engine. It's easy, simple and unbelievably cheap. There was a fantastic presentation at the last IO about riding as much as you can on free tiers: https://www.youtube.com/watch?v=N2OG1w6bGFo https://www.youtube.com/watch?v=N2OG1w6bGFo
betoh5 8y ago
For those who wanted to try this on GKE don't forget to open port 80 on your node firewall. https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/ https://kubernetes.io/docs/tasks/access-application-cluster/...
GordonS 8y ago
I need to host a web API and Postgres database in a high availability configuration. I only need a single instance each of the web API and database, but uptime is crucial. I'd like to use containers, partially because it gives an easily reproducible environment for running locally. For such a small setup, what's the recommended way of handling this, such that I get automatic failover and can easily do zero-downtime updates? I did look a bit into k8s, but quickly came to the conclusion that it's far too complex for this. Oh, I'm also constrained to using the Azure cloud, if it matters.
edpichler 8y ago
My personal conclusion using Kubernetes for personal projects: I think it's very good when you have lots of deployments, iterate fastand also a CDI environment. For this, K8s is excellent.
dekhn 8y ago
I really wanted k8s to be a nice replacement for everything I could enjoy about borg. After using it for a while, I found it far more cumbersome to work with, and the architecture to be confusing. I'm quite comfortable with complicated environments, typically all I really want is a small cluster running my long-running infrastructure, and expansion to spot instance worker nodes to meet burst capability. Just setting that up in k8s, keeping it up to date, and monitoring the various components that fail because you configured just-not-enough-RAM for them became tiresome.
jhabdas 8y ago
That guy's contention is written directly above a Website with a copyright dated 2017 and they personally admitted they couldn't remember how to deploy their own site. Was it possible they couldn't do it because it was too complex? My contention is the answer to that question is yes.
- cdoxsey 8y ago
  If you're curious, I used google app engine and google recently changed the build tools for go on app engine. I have a lot of go source code examples on my blog which aren't intended to be compiled by app engine, but with the recent changes I couldn't get it to deploy. After a couple of hours I decided I was fed up and tried k8s. Its a bit silly, but sadly that's often how software goes, at least in my experience. (also strictly speaking I couldn't do a static site cause there are some other things I run on here that are more than just html) I fixed the copyright date :)
vlucas 8y ago
The issue is not cost. The issue is conceptual overhead and sheer complexity.
mschaef 8y ago
> You don't have to learn systemd; you don't have to know what runlevels are or whether it was groupadd or addgroup; you don't have to format a disk, or learn how to use ps, or, God help you, vim. All this stuff is important and useful and none of it is going away. I have a great deal of respect for sysadmins who can code-golf their way around a unix environment. But wouldn't it be great if developers could productively provision infrastructure without having to know all of this? No. At least for small applications, Unix already has functional, established, and documented mechanisms for the vast majority of how they need to be managed. (Windows does too.) At the end of the day, I think people would be far better off learning and using those, until there is a specific documented need for some feature unique to some more powerful platform. Otherwise, you're continually paying a complexity cost for features you might never use. Going back to the author's original list of 7 questions, I run a few personal apps in AWS, and here's how I answered: > How do you deploy your application? Just rsync it to the server? I scp a JAR file to a staging directory and run an install script. The install script is pretty much standard across the handful of apps I run. > What about dependencies? If you use python or ruby you're going to have to install them on the server. Do you intend to just run the commands manually? The JAR file is an uberjar, so the dependencies are included. > How are you going to run the application? Will you simply start the binary in the background and nohup it? That's probably not great, so if you go the service route, do you need to learn systemd? I learned systemd. It gives me easy service management and a standardized way to ensure the process comes up at server startup. The service script is pretty much standard across the handful of apps I run. > How will you handle running multiple applications with different domain names or http paths? (you'll probably need to setup haproxy or nginx) I installed nginx. Took about 30 minutes. > Suppose you update your application. How do you rollout the change? Stop the service, deploy the code, restart the service? Pretty much. > How do you avoid downtime? I don't. Many small apps can tolerate small downtime windows, including the ones I run, so I don't need to worry about continual operation. If I did have to worry about that, I'd have additional things to worry about in my application design. > What if you screw up the deployment? Any way to rollback? (Symlink a folder...? this simple script isn't sounding so simple anymore) Select the previous jar file, which I will have retained. Schema migrations complicate rollback, so I keep them to a minimum, test a bunch locally, and make sure I have backups of the database taken immediately before a deployment. (So that I can roll back if absolutely necessary.) > Does your application use other services like redis? How do you configure those services? Persistence handled through an embedded SQL database, and a library I've written to managed schema migrations. For nginx and git, I have source code snapshots of the configuration, but that's only needed if I rebuild the server. (Once in over five years.) Generally, for small apps, I try to keep the dependency footprint as small as possible. (Which is pretty much the entire gist of this comment, now that I think about it.)
humbleMouse 8y ago
I'm going to say it again, just like in every kubernetes thread. People should check out openshift. It's built by redhat and is an abstraction layer above k8's. I love it.
nqzero 8y ago
great post - thank you for convincing me not to spend another moment considering using k8s ! /s
rb808 8y ago
This is great. I always thought k8s was perfect for personal projects but too expensive, I'm going to have to double down now.
cdelsolar 8y ago
I set up a single-node cluster for a personal project on DO and it took me some time, but it's up and running and has been fine for over a year. I used a decently beefy instance ($20/monthly on DO) because the Kubernetes processes are actually too resource intensive for the $5 instance. It runs several processes (the main Python app and a helper Go-based API), automatic SSL renewal, a daily cleanup Cronjob, etc. I really like being able to push a completely new container whenever I push to master - this is done with CircleCI, which builds the image, pushes to Docker Hub, and just does the kubectl apply. I couldn't think of any other way to do it that wouldn't result in some downtime if I had to change or upgrade packages. I wrote about it here a bit: https://hackernoon.com/lessons-learned-from-moving-my-side-project-to-kubernetes-c28161a16c69 https://hackernoon.com/lessons-learned-from-moving-my-side-p...
- Rotareti 8y ago
  > I couldn't think of any other way to do it that wouldn't result in some downtime if I had to change or upgrade packages. https://github.com/helm/helm https://github.com/helm/helm ? You push the new version of your app as a helm chart to your helm repo and then run: helm upgrade my-app
  - cdelsolar 8y ago
    no - I meant, outside of Kubernetes, in the past (sorry, my wording in my post was a bit confusing). My app used to just be something I would ssh into and then do a git pull, but whenever I had to upgrade packages it would be tricky. Then I tried Docker with docker-compose and I had similar issues with environment variable changes. Kubernetes is great; all I do is kubectl apply -f new_config.yml and everything just works without downtime. I only very recently learned about Helm when doing automatic SSL upgrading, but it looks cool too. In general I really don't think k8s is the monster people make it out to be. I did all this on the side over the course of a few weeks, a few hours here and there. I know I'm only scratching the surface of what is possible, but everything just works fine, and my .yaml files are pretty self-explanatory.
Tehchops 8y ago
I feel like Kubernetes solves a lot of problems. I don't feel like any of them have to do with personal hosting/blogs. If you really want to go simple/cheap for that, why not eschewing compute entirely and go serverless/static?
- nkassis 8y ago
  I tend to agree with this. Keeping thing as simple as possible to start is ideal. Also out of the box docker offers a lot of functionality for single machine and development environments that might be good enough. I've seen kubernetes make more sense when you need to grow to multiple machines and are trying to optimize resource utilization.
alexnewman 8y ago
I've never met someone who didn't love k8s unless they already knew how to run machines
sngz 8y ago
"Out of college I spent 5 years working in the Windows ecosystem. I can tell you my first job at a startup using linux was not an easy transition." more interested in how he manage to do that transition. been struggling with that part myself when competing with other candidates who have years of linux experience.
Draiken 8y ago
People keep saying k8s is so complex and the learning curve is steep but I honestly struggle to see it. The only hard part about k8s is setting up the cluster itself. Using it is so easy it boggles my mind how can someone say it's hard. It's a simple API with an amazing CLI that immediately solves dozens of issues you would have sooner or later. Take for example a very simple rails application. I'll keep it simple with 3 routes: manual, automated and with k8s (assuming you're not building the cluster itself). Manually setting up would require you to: - Create a VPS - SSH into the VPS - Install at least Ruby, NGINX (or some webserver), the DB and all Rails dependencies - If you want easy deployments you'd probably use something like Capistrano and have to set it up - If you want the system to be resilient you will have to add some sort of process monitoring and automatic startup This is very easy to get wrong. One missing step and you're having to manually SSH into the machine and trying to figure out what went wrong. An automated setup would essentially do everything you did on the manual setup, but you'd have to write that into something reproducible like Ansible, Chef, Puppet or similar. Every single one of these tools has a gotcha and a learning curve and that's on top of all the base knowledge of setting it all up by hand. Finally with k8s (assuming we're using Google's GKE offering): - Dockerize the app (which is extremely simple with tons of tutorials) - Download the gcloud CLI (alternatively it can all be done through the website if that's your thing) - Create the cluster - Either create some manual yaml files or just use kubectl to create a deployment for: rails, nginx and the database (again, there are millions of examples of this even on the official docs). - Create an ingress and point my DNS to the ingress' IP All of these methods have learning curves and are in many ways complex. However I don't think Kubernetes is a lot different than the others. It's just a different way of doing the same. * Instead of manually finding out how to setup load balancing and reverse proxying on NGINX, you learn how to use the k8s services. * Instead of setting up Capistrano (or any other deployment tool) you learn how a k8s Deployment and rollouts work * Instead of figuring out how to setup L7 load balancing you learn to use k8s Ingress On top of all that with GKE (or any other managed k8s solution, I presume) you'd automatically have: logging, metrics, scaling, rolling updates and much, much more. People are resistant to change and it's definitely not an "one size fits all" situation, but as soon as you need anything beyond an amateur website, k8s provides the best ROI. I've had to do this on all of these scenarios and I can confidently say Kubernetes changed the game. I got a 10 year old Rails application with tons of dependencies from an old dedicated server setup to Kubernetes on a few days (on Kubernetes 1.1 at the time, which was a lot harder) and I'm not some kind of genius. If I can do it, any developer can.
module0000 8y ago
>> You don't have to learn systemd; you don't have to know what runlevels are or whether it was groupadd or addgroup; you don't have to format a disk, or learn how to use ps, or, God help you, vim That's a quote from the article. The thought that someone with that mindset is responsible for anything more than a network-connected waffle iron is terrifying. This article advocates for willful ignorance of Unix/Linux, because you don't need any of those things if you know k8s. That said, k8s is nice for some things. I admin a large deployment at work, and it's relatively painless for developers to be productive with. One of the reasons for that stability isn't k8s itself though - it's the legion of seasoned Unix/Linux pros working alongside me. The reason "it just works" is because under the covers, there are a whole lot of us doing all the boring stuff to hundreds of physical hosts.
- Spivak 8y ago
  I'm sympathetic to the author's argument, UNIX administration is a full time job on top of whatever system you're using to do deployments. You leverage your ops team at work; is ops-as-a-service really all that different? If one of my devs wants to deploy a rails app I don't expect them to suddenly be a sysadmin or network engineer or infosec expert. I expect their application to be well written but the rest of the stack is up to ops.
- blablabla123 8y ago
  True, deploying stuff in an actual Unix environment while ignoring need of these tools is the perfect recipe for unstable setups. Although I guess for personal projects it could actually work as long as the load is low enough and when there is no data involved that needs to be protected. ;)
erulabs 8y ago
I had a project about this size on GKE, in an attempt to get a feel for Google hosted Kubernetes product. I went on vacation for two weeks while a container was printing an error (unbeknownst to me). Google wanted to charge me ~700 dollars for StackDriver logging volume (a feature I was not aware was connected to my Kube pod stdout). Google Cloud refused to refund or provide any guide as to how to entirely disable / purge StackDriver, so I'm back on AWS and won't be recommending anyone move to GKE any time soon (it's extremely extremely easy to suddenly more than 100x your bill...) With my hand-built Kube on AWS I have 15gb assigned for my ELK logging cluster, and it costs me nothing because it falls under the free tier. Looking forward to trying out Digital Ocean's product next!
- trumpeta 8y ago
  to be fair, if you have CloudWatch and send logs there it can get pretty expensive too
fridgamarator 8y ago
+1 for using Crystal
rconti 8y ago
"Kubernetes is free" FTFY

652 points by cdoxsey 8y ago

GKE documentation says 25 % of the first 4 GB are reserved for GKE: https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu https://cloud.google.com/kubernetes-engine/docs/concepts/clu... Is it what you observe?

sheepz 8y ago

Looking to try out Kubernetes, will definitely give this a try. Although the article starts with kind of a strawman (Kubernetes is robust section): you could do most of those with Docker (and compose) or with a non-containerized setup altogether?

topher200 8y ago

This seems awesome. I love the automated way to update Cloudflare DNS addresses via a trigger from the Kubernetes informer API.

donmcronald 8y ago
Yeah, that's pretty cool. I don't know much about K8S, but it looks like that might be a good strategy to build your own API gateway if `crystal-www-example.default.svc.cluster.local:8080` is a portable URI. I've never used them, but always thought the API gateway products seem like a way you could get locked into a specific platform.

07d046 8y ago

It looks like it isn't surprisingly affordable on Amazon. If I'm reading this right, EKS is 20c an hour per cluster, or about $150/month, and that's before EC2 costs for the actual machines in the cluster. https://aws.amazon.com/eks/pricing/ https://aws.amazon.com/eks/pricing/

joseph 8y ago
EKS is indeed expensive. I started creating https://github.com/cloudboss/keights https://github.com/cloudboss/keights for automating Kubernetes stacks before EKS was a thing, but I still prefer it over EKS. You can use it to set up a cluster using plain CloudFormation, or with an Ansible role which wraps the CloudFormation stack.
chrissnell 8y ago
You don't need EKS. Just use kops to build your cluster on plain EC2 instances.
throwaway2016a 8y ago
You can run Kubernetes on AWS without EKS and if you use t2.micro instances it would be around $28 per month if you use the load balancing technique here (where it is DNS based on an actual LB). $21 a month if you account for the free tier (and this article uses Google's free tier). If you further use spot instances (the equivalent here to "preemptible") you can save a ton more. The real savings (or $7 per month) is that on GCE you won't need to run your control server. GCE does that for you for free. Edit: now another issue here is (and I have no idea if this is true for GCE like it is AWS). t2 instances will throttle you if you use too much CPU which can be an issue when running a lot of containers.
- tannhaeuser 8y ago
  Sure but the context is personal projects or portfolio/small business sites. A VPS on a discount provider such as Hetzner or OVH costs around 3-4 USD/EUR per month. A redundant VPS (two VPS and Ceph) costs 10 EUR on OVH. Not to speak about the complexity of setting up and running k8s. You'll be needing a second k8s development/testing environment for staging as well. So, no, I don't think k8s makes sense economically.

deckarep 8y ago

When picking preemptive machines for this demo project is the intention here to simply save money for a pet project? I guess my main question is would anybody actually use preemptive for k8s knowing that if a worker node gets reaped k8s will magically route only to healthy nodes while the preempted instance replacement is standing up?

tmaier 8y ago
Sure! First of all, GK3 would create new nodes when one of them gets killed. Second, builidng “cloud-native” applications means to build stateless applications, e.g. using 12-factor. So it does not really matter if your application gets killed, restarted or horizontally scaled. For databases, i always recommend to use a PaaS offering to have an easy and worry-free life
- donmcronald 8y ago
  What if all 3 nodes get killed at the same time? Can you use the always free instance + 2 preemptible instances?
  - DuskStar 8y ago
    You can set GKE up to use multiple groups of nodes, yes. In this case you'd want to configure one group with a single non-preemptible micro node, and a second with two preemptible micro nodes. It's not something I'd really worry about for a hobby project though.
halbritt 8y ago
I use preemptible nodes extensively in my production environment.
maktouch 8y ago
We pretty much run 98% of our stuff in preemptibles. It's a free chaos monkey that makes your stuff cheaper too! Just make sure your app has proper probes, and make sure the probes returns 500 when the process receives a sigterm, and it should be invisible to you.

quaunaut 8y ago

I'm beginning to think that what Kubernetes needs by far, is a better development environment story. 1. Something that makes it easy to run the application(s) you're working on locally... 2. ...but have it interact with services in a local kubernetes cluster easily. 3. Something that makes your entire development environment effectively immutable, read-from-files the same way as the immutable infrastructure we're talking so much about. 4. Something that gives you a good start to running in production, so the first day of bringing your service up isn't a mess of trying to find/remember what environment variables/config options to set. 5. Something that's one tool, instead of a mixture of `minikube`, `kubectl`, `docker`, `draft`, and `telepresence`. 6. Something that's opinionated, and follows "convention over configuration", with conventions that make for the best future designs. Basically, we need something with a Rails-like impact but in the Kubernetes ecosystem.

rememberlenny 8y ago
Check out BOSH from pivotal labs.
thebluehawk 8y ago
Docker (at least on my mac) now comes with kubernetes built in, so you can just click a button and instantly have a kubernetes cluster running. As for an app deployment tool I've had good success with Helm. It allows you to define all the services, app, variables, etc. that your app needs. Though the templates do seem to be a bit convoluted and hard to read at times if you aren't careful.
- GauntletWizard 8y ago
  +1 for docker for Mac, -1 for Helm. Helms templating is just too simple for the actual complexity of k8s, but it can make a decent starter too.
  - DuskStar 8y ago
    Helm + Kustomize [0] to add in whatever wasn't in the helm template works pretty well IMO. Kustomize is pretty new and not the most user friendly though. 0: https://github.com/kubernetes-sigs/kustomize https://github.com/kubernetes-sigs/kustomize

lostmsu 8y ago

Why do you even need Kubernetes when you can get free hosting from Azure (AppServices?) or Google (Cloud Engine?)?

thrower123 8y ago
Azure AppServices are hard to beat. One-click deployments from Visual Studio, or it takes about five minutes to setup continuous deployment from git if you use a free private VSTS repo. If you want to use the tier that allows a real domain instead of the azurewebsites domain, it costs about $10/month. Kubernetes seems like a lot of unnecessary complexity for small workloads. Kind of plinking tin cans with a nuclear bazooka. What is the break-even point where all of this stuff starts to really be worth bothering with?
- sp527 8y ago
  > What is the break-even point where all of this stuff starts to really be worth bothering with? Series B
mohamedmansour 8y ago
You don’t, behind the scenes they use orchestrations like kubernetes or service fabric. So much easier deploying a docker container as a Azure Web App, so easy to auto scale based on thresholds of certain resources, so easy to debug what’s wrong since they give you all the tools. I use that in my current projects today. But Azure Kubernetes in my last project was easy too, it comes with a sophisticated monitoring solution, it integrates nicely with cheap Azure Load Balancer, but still too much overhead than Azure Web App.

tln 8y ago

The k8s system containers are quite resource hungry. The author is clearly very good at k8s (I enjoyed the article), cos running it cheap ain't easy

pstadler 8y ago

Great article! I‘m running a full-featured three node cluster for less than $10/month on Hetzner Cloud. This kind of setup requires a little more effort, but it can be fully automated and you learn a couple of things about Kubernetes on the way. I published and maintain this guide[1], including provisioning using Terraform and a bunch of example manifests to get you started. [1] https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide

rtp 8y ago
This guide is great. Thank you!
raesene9 8y ago
One thing you might want to look at based on a quick scan over your guide is that, at the moment ,it looks like you're running your etcd cluster with no authentication. Any attacker who was able to get on to the pod network (e.g. if there's a vulnerability in one of the apps running on the cluster) could hit the etcd endpoint and dump the contents of the database, which generally includes sensitive information like k8s secrets.
- pstadler 8y ago
  Indeed, see https://github.com/hobby-kube/guide/issues/6 https://github.com/hobby-kube/guide/issues/6 Edit: I think this was you.
  - raesene9 8y ago
    ah yeah sorry I'd forgotten I mentioned it before.
merb 8y ago
nice guide, the only problem i see with hetzner cloud is that they do not allow to have a operating system like coreos or even the coreos (Container Linux) fork flatcar or even RancherOS.
mahesh_rm 8y ago

erikb 8y ago

Sure if you put enough effort in you can make everything usable and cheap. But why put in the effort into that instead of investing it into building a system that naturally fufills such requirements? Kubernetes was never intended as a system for hobbyists, but for bringing more users onto google cloud. At that it is incredibly successful. But in exchange a lot of other stuff it does really badly. It is much better to start with a single-node system in mind that expects getting more nodes at some point and develop from there. This will bring in a lot of hobbyists and really carves out how to solve the developer story. Then this will grow automatically with the size of popular projects building on it. That said, of course it's nice if people invest energy into finding other usecases for k8s as well. But it shouldn't be the community's main effort.

superkuh 8y ago

This article, and a lot of others, seem to be using an overloaded meaning of "personal". All the arguments stem from the assumption that personal projects have the same uptime, scaling, and deployment requirements as some large commercial site. Some personal projects might but I'd argue most of those are really commercial in the sense that you're developing them for a portfolio or the like. They're not personal. It makes sense to use business tools like kubernetes when you're making a demo business site. It doesn't make it a good platform for personal projects. The best platform for those is hosting from home on a single computer, for free, with total control. ie, personal.

daurnimator 8y ago
I disagree. Even for personal projects I want to be able to access them from my phone while out and about. Something running on my home computer (which is turned off when I'm not there!) doesn't help. Nevermind that my home internet isn't fast or reliable. If you include running my own blog/mail/XMPP/matrix servers in "personal projects", then I would love to have perfect uptime. Additionally, being able to reuse high quality helm charts here just makes things simple instead of having one set of instructions for "personal" use and one set of instructions for everyone else.
- wafflesindeed 8y ago
  Why would you want to be able to access your personal projects from your phone? What context requires this over waiting and solving the problem at home? What about work life balance? (yes, it's a personal project, but still)
  - rickycook 8y ago
    i have a paperless filing system that i frequently access from the office for things like scans of bills that aren’t yet electronic, etc. it’s fantastic to be able to access it all (via VPN; never direct) wherever i like
  - jamessb 8y ago

stephanlindauer 8y ago

we, at rebuy, built this little thing to run kubernetes on just one node: https://github.com/rebuy-de/terraform-aws-coreos-k8s-single-node https://github.com/rebuy-de/terraform-aws-coreos-k8s-single-... my colleague and i mainly use it to run our private hobby-projects.

StavrosK 8y ago
I freaking love Dokku for this setup. All the ease of use with none of the headache.

sklivvz1971 8y ago

The list of things mentioned in the article to do and learn for a simple, personal project with k8s is absolutely staggering, in my opinion. Having used it, there's a sizeable amount of further work needed which the article doesn't mention (e.g. learning how to use the pretty confusing google interface, finding the right logs and using their tools). So the overhead is really huge. Furthermore, the whole system is slow. Want to run a SQL query against your postgres? You need to use a google cloud command that changes the firewall and ssh's you in on the machine... and this takes a couple of minutes, just enough to make me desist unless I _really_ need to run that query. Abysmal. Finally, and this is a pet peeve against many advocacy blog posts, they just show you the happy path! Sure, _in the best of cases_ you just edit a file. In a more realistic case, you'll be stuck with a remote management system which is incredibly rich but also a really steep learning curve. Your setup is not performant? Good luck. Need to tweak or fine tune? Again, best of luck. We've tried to adopt k8s 3-4 times at work and every single time productivity dropped significantly without having significant benefits over normal provisioning of machines. {Edit: this does not mean k8s is bad, but rather that we are probably not the right use case for it!} ...which in turn is usually significantly slower than building your own home server (but that's another story!)

arountheworld 8y ago
If you design your app properly you won't ever need to run SQL queries by hand. This is a bad practice and it is good that k8s discourages such behaviour. I agree with your other points though.
- barrkel 8y ago
  You have no idea what you're talking about to make a blanket statement like that.
- reacharavindh 8y ago
  SQL is too low level?! I don't think you understand what is going on with the tools you use.
  - 8y ago

snaky 8y ago

> Kubernetes, right now, supports some workloads really really well, like stateless workloads that are CPU intensive that can be easily scaled horizontally, with easy to understand patterns that we currently put behind load balancers. > Other workloads, Kubernetes just doesn’t support well and likely will not support well in the near future because it’s really hard to do. It’s not magic, as some of the hype surrounding it would have you believe. https://gravitational.com/blog/running-postgresql-on-kubernetes/ https://gravitational.com/blog/running-postgresql-on-kuberne...

deleted 8y ago

[deleted]

ratiolat 8y ago

I'm trying to set up a deployment pipeline, quite from scratch. That also includes picking source repository software. Self-hosted is a must for every component. So I chose gitlab and after that I dived into it. Oh boy was I in for a surprise. For CI/CD one needs docker, kubernetes and whatnot: https://docs.gitlab.com/ee/topics/autodevops/ https://docs.gitlab.com/ee/topics/autodevops/ But the alternatives don't look better either. So, fellow hn'ians. My requirements ATM are quite simple: Submit code via git, run automated test, if all good, put the files into the live system. Pull-requests are must and everything must be self-hosted because cloud is just another persons computer. Code is written in python. What would you recommend? Edit: grammar.

bthornbury 8y ago
Sounds like you can get away with a bash script that calls rsync, and restarts the code on the server with ssh. (I deploy one of my websites this way, simple & reliable) If you're looking for something more legit: Ansible
te_chris 8y ago
Heroku. It is everything you want.
jl-gitlab 8y ago
Hey there, I'm the product manager for CI/CD at GitLab. AutoDevOps is one of our more advanced features to get up and running in a k8s environment quickly, but is not required if you just want basic CI. Our documentation on getting started with CI/CD is here: https://docs.gitlab.com/ee/ci/ https://docs.gitlab.com/ee/ci/
- ratiolat 8y ago
  Thank You! I will look into that. OTOH this is how I came to conclusion that I must go the AutoDevOps way: This is my first installation of gitlab ever. So naturally, after somehow successfully installing it on the 2nd try, everything seems to work. So - lets explore configuration options. For example we should disable registration since, well, we don't want to be a public gitlab host. After a while, after the Pages section, we come to a section named "Continuous Integration and Deployment". And there's a checkbox named "Default to Auto DevOps pipeline for all projects". And there does not seem to be any other kind of "Continuous Integration and Deployment" available so one naturally comes to a conclusion that the only way seems to be Auto DevOps. My bad.

deleted 8y ago

[deleted]

TekMol 8y ago

What does it give me over a bare VM?

movedx 8y ago
Are you serious right now?
- TekMol 8y ago
  Yes
  - movedx 8y ago
    OK, let me reverse the question and put it to you: what does a bare VM get you over a VM with Minikube on it?
  - movedx 8y ago
    (can't edit my original comment at this point) Didn't think so.
axaxs 8y ago
for the vast majority of folks, resume fodder.

heipei 8y ago

I'm surprised in all of these posts about k8s on HN the Nomad project (by Hashicorp) rarely comes up. I've found it an absolute joy (yeah, I said it) to use and a breeze to setup. Unless you're setting up a container orchestration system where you want to customise every single aspect while you're setting it up, my guess is you're better off with Nomad at first. At the very least you'll learn what you need from a container scheduler and can make a more informed decision if you re-evaluate your options down the line. Nomad also integrates perfectly with Consul and Vault, so you're never left wondering "is this is how I'm supposed to do it?" which is something that happens if you're just starting out. URL: https://www.nomadproject.io/ https://www.nomadproject.io/

shaklee3 8y ago
Check out that hashicorp blog over the last three weeks. Consul integrates tightly to kubernetes now. Vault already did.
- josegonzalez 8y ago
  Consul only integrated "tightly" with kubernetes in the past few weeks. The vault integration is pretty barebones, and anything done there has been spearheaded by the Kubernetes community.

aksels 8y ago

From my experience it's quite expensive, at least in GKE. As Kubernetes takes a lot of cpu & memory by itself. I run a pool of one node n1-highcpu-2 (2 vCPUs, 1.8 GB memory) and it costs me about 72 euros per month. kube-system is taking about 730 mCPU of the node's 2000 mCPU. The issue is that the kube-system is deployed on each node of the pool so (tell me if I misunderstood/misconfigured something). If you have a pool of 3 nodes which have 1 vCPU each, with kube-system taking approximately 700 mCPU on each you have only 300 * 3 = 900 mCPU allocatable. If you have any tips on how I could reduce the costs of my personal projects I'm listening !

ngrilly 8y ago
When creating to disable the cluster, have you tried to disable Kubernetes dashboard and StackDriver monitoring and logging services? For logging, you could use FluentBit which is more lightweight: https://docs.fluentbit.io/manual/output/stackdriver https://docs.fluentbit.io/manual/output/stackdriver
molf 8y ago
As a data point, on a busy GKE production cluster with 12 highcpu-16 nodes we have kube-system pods taking a total of ~700mCPU on all servers combined. I'm not sure how the CPU usage scales, but probably not linearly. Most of the CPU usage is kube-dns, which I think requires much less CPU if you make few internal network connections.
puzzle 8y ago
I just saw a change the other day which makes system pods fit better on nodes with one vCPU. It reduced their numbers by the same factor (i.e. keeping the same proportions). It was meant for GCE and I am not sure if/when it will make it to GKE, but it looks like Google is looking at the tiny node scenario. Also, Tim Hockin led a discussion about something similar at last year's Kubecon.
- puzzle 8y ago
  Here it is. It's in 1.12, actually, not on HEAD as I recalled: https://github.com/kubernetes/kubernetes/pull/67504 https://github.com/kubernetes/kubernetes/pull/67504 Still not sure if these are the configuration files that GKE uses behind the scenes, but the author is a Google employee.

leandot 8y ago

For personal projects I am using multi-node Hetzner instances managed with a Terraform script heavily inspired from https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide + RKE for K8S setup and it works really well.

hobofan 8y ago
For personal projects, I am using a single dedicated server at Hetzner with Docker & HAProxy. It's beefy enough to be able to handle as many personal projects as I have time to dedicate to. I've tried to replace my Docker setup with Kubernetes multiple, but it was always too much of a hassle. Things that were easy with a "bare" Docker setup became hard, and single-node clusters always felt like a second-class citizen.
- ngrilly 8y ago
  I almost do the same (Docker and nginx as a reverse proxy on Google Compute Engine or Digital Ocean virtual machines). But how do you manage deployments of new versions without downtime? I use Docker Compose, but I had to complete it with some scripts to manage the deployment (start the new containers, wait for new containers to be ready, switch traffic to new containers, shutdown old containers), and it's a bit messy and non-standard. I'm thinking of using using Docker in swarm mode which offers the concept of services with automated deployments.
  - leandot 8y ago
    I would not recommend Docker Swarm, it's way too simplistic and my feeling is that it lags so much behind k8s that it might get discontinued. I've done Ansible, Docker Swarm, custom scripts and nothing is as clean as using standard K8S components + it gives you a good skill on your resume.
    - ngrilly 8y ago
      I'm afraid too of a discontinuation of Docker swarm mode... And agreed about having Kubernetes on your resume being a good thing.

nicolaslem 8y ago

Having worked with Mesos, Nomad and an in-house container scheduler, I would not pick any of them for personal projects. I've not tried Kubernetes yet but I don't think it would make me change my mind. I recently went through the infrastructure that runs my personal projects (Django, PostgreSQL, Redis, a few static sites...). I decided to go with plain Docker + Ansible. These tools give me enough automation to easily scale up or change hosting provider if I need, without introducing too much magic and complexity.

djsumdog 8y ago
I'm in the exact same boat with you. I was at a shop that used DC/OS at work, and to be fair, it's probably the easiest to get initially setup. But even a basic setup is going to be really expensive, as I discovered: https://penguindreams.org/blog/installing-mesosphere-dcos-on-small-digital-ocean-droplets/ https://penguindreams.org/blog/installing-mesosphere-dcos-on... I ended up doing what you did, having Ruby provision my machines, configure them with Ansible and deploy my docker containers (calling the Docker API from Ruby). The whole thing can be found here: https://github.com/sumdog/bee2 https://github.com/sumdog/bee2 It was a lot of work setting all that up. I should have probably just used Terraform + Ansible + docker-compose. Still, I'm glad I wrote it. I learned a lot about the complexities of running a docker infrastructure.
ngrilly 8y ago
I do the same (Docker + Ansible). I also use Docker Compose. How do you deploy new versions of your apps without downtime (start the new containers, wait for them to be ready, switch traffic to new containers, shutdown old containers)? I'm using custom Ansible scripts coupled to docker-compose but it's a bit messy and non-standard.
- nicolaslem 8y ago
  To update an app container to a new version I just run Ansible again: - docker_container: image: foo/bar:latest state: started restart_policy: always pull: yes This pulls the new version and restarts the container. The downtime is probably a second maximum. It's small enough that it is not worth the trouble of a rolling release or whatnot. It's a personal project after all, no one will complain for getting a 502 once a month :) I do not use docker-compose as I never felt the need to and I read on multiple occasions that it's meant for development only.

Can_K 8y ago

I have not yet hosted anything in a container/cloud. Therefore maybe a stupid question, how do you keep your system up to date? What am I required to do in order to receive new operating system and software versions?

minimaxir 8y ago
A hosted provider like GKE will update the OS of the master/nodes for you. (although the OS itself doesn't do much; Google uses a container-optimized OS that only handles containers and has a much smaller footprint)

bwm 8y ago

Why not just outsource hosting to Heroku or something similar? Unless you're building something with extremely exotic requirements, it seems like you're just wasting time trying to build out your own infra.

paulcarroty 8y ago

Well, let's talk about bad side: 1) Google Cloud - hello privacy and support issues. 2) Traditional enterprise-style overcomplexed configs and architecture - the shitty trick "we want your losted time as justify to stay here" 3) Forget about quick deployment and scaling.

geku 8y ago

At https://www.kubebox.com https://www.kubebox.com we are working on affordable Kubernetes clusters for developers and personal usage. It's not as cheap as the mentioned solution but also comes with more memory and CPU power. Around $36 for 8GB and maybe less for just a 4GB node, not yet decided. Master is fully managed and hosted by us. What do you think?

gizzlon 8y ago

Or you can push your webapp to Appengine standard and it's 0$ per month =) Appengine requires you to learn some new stuff, but so does k8. Not the right choice for all apps, obviously, but something to compare this setup to.

pawurb 8y ago

Dokku is a viable infrastructure solution for side projects, even if you don't have much dev ops exp https://pawelurbanek.com/rails-heroku-dokku-migration https://pawelurbanek.com/rails-heroku-dokku-migration

parsadotsh 8y ago

While I don't have experience with large production systems, for hobby projects I found Nomad to be much easier to start with. Even the K8s distros that were supposedly "minimal" took a huge amount of time to setup in a reproducible manner. Plus Nomad's fantastic integration with consul makes it more than just a scheduler and a more complete alternative to Kubernetes.

danieldk 8y ago

I use NixOS, which allows you to define machines declaratively. To the points that he raises: How do you deploy your application? Just rsync it to the server? Define an overlay for your application. For a typical application, this is just a few lines of code with stdenv. What about dependencies? If you use python or ruby you're going to have to install them on the server. Do you intend to just run the commands manually? If you have an overlay for your applications, you can specify dependencies in the derivation of your application. Even though these are not web apps (I primarily work on local applications), I have derivations for some of my applications, so that I can install them on every machine with little effort. Two examples: https://github.com/danieldk/nix-home/blob/master/overlays/30-alpinocorpus.nix https://github.com/danieldk/nix-home/blob/master/overlays/30... https://github.com/danieldk/nix-home/blob/master/overlays/20-finalfrontier.nix https://github.com/danieldk/nix-home/blob/master/overlays/20... If I want to develop one of the applications, say alpinocorpus, I can just drop in a shell with all the dependencies (cmake, boost, Berkeley DB XML, etc.) available: nix-shell '<nixpkgs>' -A alpinocorpus When you exit the shell, you are again in your usual environment. Another option is to create a default.nix file in a project directory and use direnv to automatically load the environment when you switch to a directory. How are you going to run the application? Will you simply start the binary in the background and nohup it? You can define (systemd) services in a NixOS configuration. How will you handle running multiple applications with different domain names or http paths? (you'll probably need to setup haproxy or nginx) NixOS has support for defining nginx virtual hosts. You can also enable Let's Encrypt and Nix will set up systemd timers to automatically renew the certificates. E.g. here is a configuration of one of my machines with a bunch of virtual hosts with Let's Encrypt: https://github.com/danieldk/nixos-config/blob/master/machines/castle.nix https://github.com/danieldk/nixos-config/blob/master/machine... Suppose you update your application. How do you rollout the change? Stop the service, deploy the code, restart the service? How do you avoid downtime? nixos-rebuild switch --upgrade Or with nixops you can deploy and update machines remotely. What if you screw up the deployment? Any way to rollback? (Symlink a folder...? this simple script isn't sounding so simple anymore) Yes, every change creates a new generation. You can always switch into or boot into an older generation as long as it isn't garbage collected. And this works for the whole system, including kernel updates, etc. Does your application use other services like redis? How do you configure those services? Declaratively ;). Adding services.redis.enable = true; to your NixOS configuration does the trick. As a bonus, with nixops you can deploy a machine configurations on real-hardware, VPS, Virtual Box, KVM, etc.

nathan_f77 8y ago

No-one has mentioned Convox yet: https://convox.com https://convox.com Kubernetes is great, but Convox is way easier to set up. It's much closer to having Heroku in your own AWS account. It also leverages a lot of AWS services (ECS, RDS, etc.) instead of re-inventing the wheel. The only downside is that it only supports AWS. I'd recommend Convox for personal projects, but it can be a bit expensive because you have to run a minimum of 3 EC2 instances.

bdcravens 8y ago
In the past I've had similar success with Cloud 66.

nickjj 8y ago

I don't think you're being fair with your comparisons to DO when it comes to price. The f1 micro $5/month Google servers are SO much worse. Those "always free" f1 micros have 600mb of RAM, do not have SSDs and according to some tests[0] you only get 20% of 1 vCPU. There's also some pretty severe write limitations. That's a really big difference vs DO's 1gb of RAM, SSD and 100% of 1 vCPU. The performance difference alone with SSD vs no-SSD is massive, especially on a database and while most web apps aren't CPU bound, 20% of 1 vCPU is bound to run into problems beyond a simple hello world that you set up in your example. Also what happens if your app doesn't even fit in that 600mb? On DO you could get a 600mb app + 100mb OS + 150mb DB + whatever else running on that $5/month server since it has 1gb of ram. In a real world application I wouldn't be surprised if that cluster performs way worse than a single DO server in the $5/month range. In some cases your app wouldn't even be able to run. [0]: https://www.opsdash.com/blog/google-cloud-f1-micro.html https://www.opsdash.com/blog/google-cloud-f1-micro.html

minimaxir 8y ago
This is part of the reason why I use a similar setup as OP, but with a single n1-standard-1 instance (1 vCPU/3.75 GB RAM) instead ($7.30/mo w/ preemptible). The 3-node requirement of GKE is not a hard requirement apparently. Of course, that's mostly for running Jobs; a single node + preemptible with a production app is not a great idea.

ksajadi 8y ago

In my experience, having a reliable, configurable and flexible enough pipeline to generate the needed configuration files for Kubernetes makes a lot of difference in the adoption. Tools like Helm or Cloud 66 Skycap are some examples but I'm sure there are more.

bcoughlan 8y ago

I know a mid-sized company who have a team of 3 people dedicated to managing their Kubernetes clusters. For a small company that can't afford the overhead of that, it's an extra layer of risk and complexity. Amazon EKS launched last month in the EU west region, which I was excited about. But each cluster is $150 per month (20c per hour) before you even add any servers. To have a dev, qa and production cluster we're looking at $450 per month before the cost of EC2 instances to actually run the application. Our EC2 costs for backend services are about $800 per month, so it instantly negates the selling point that we can use reduce costs by using resources more efficiently. Our current deployment setup is groups of Docker services deployed to a set of EC2 instances via Docker Compose with a load balancer in front of each server group. Having isolated server groups means that outages are isolated to particular areas. I don't know how long we'd be down if there was an issue with the Kubernetes cluster. If we introduced Kubernetes it would be left in the hands of a couple of developers to manage, which is the opposite of the DevOps culture we strive for. We stay cloud-agnostic so it's easy for a developer to spin up a stack of dependent services with Docker Compose (Nginx, Redis, RabbitMQ etc.) when working on changes to a service. For a developer to run services locally with Kubernetes, they need to set up MiniKube. Many corporate environments have tight integration with Windows and our developers run Linux in a VM. I experimented with getting MiniKube to run in a VM but found it hard to get going with the vm-driver=none option, and it's something I definitely didn't want to add to our onboarding process. Kubernetes has a serious hype train at the moment reminiscent of the MongoDB days. My inner conspiracy theorist says that this is because Google has a vested interest in preventing vendor lock-in on cloud providers. It solves a Google-sized problem and introduces some complexity with the promise of removing more complexity than it adds. For small to mid sized companies, however, it adds far more complexity than it removes.

StavrosK 8y ago
To be fair, you don't need three clusters, you only need one and you can namespace things.
- bcoughlan 8y ago
  Maybe I'm oldschool but I'd be quite wary about mixing anything in production, qa and dev environments.

maxhallinan 8y ago

I tried out Kubernetes (managed with kops) for a personal project. I had a single websocket endpoint. I wanted to terminate SSL at the load balancer. Found this to be extremely complicated and the troubleshooting resulted in a ~$100 AWS bill.

djhworld 8y ago

I looked at Kubernetes for personal stuff but the overhead was too much for something I'd rather just set and forget. My current solution is to just run docker on some raspberry pi's and have scripts to replace the container when I do an update, with volumes regularly backed up to S3. Works fine, although you don't get the clustering capability (I might try docker swarm one day but for now it's fine)

IshKebab 8y ago

This is so over-complicated for a personal project.

nova22033 8y ago

How are there only 2-3 references to minikube in the comments?

chuckdries 8y ago

I prefer simple tools. - Heroku is cheap for personal projects. - A single Digital Ocean droplet where you manage everything yourself, is cheap for personal projects. - If you're working in node, Glitch.com is literally free node hosting

JAdamMoore 8y ago

Stop trying to make Kubernetes cool. Guess what? People don't need to scale everything, or even better, most people don't need to learn or use Kubernetes, like, ever. Get over it. You made a tool you thought would scale, it didn't, because not everything needs to scale.

xaduha 8y ago

What kind of personal projects need a cluster? I'm pretty sure docker-compose + swarm can satisfy 99% percent of your needs.

rbanffy 8y ago

It really depends on your personal project. Kubernetes is a complicated, complex beast. Even on managed K8s on a cloud provider, you still need to learn a lot of stuff that has absolutely nothing to do with what you intend to accomplish (unless your personal project is learning to use K8s. Most of my personal projects get deployed on Google's App Engine. It's easy, simple and unbelievably cheap. There was a fantastic presentation at the last IO about riding as much as you can on free tiers: https://www.youtube.com/watch?v=N2OG1w6bGFo https://www.youtube.com/watch?v=N2OG1w6bGFo

betoh5 8y ago

For those who wanted to try this on GKE don't forget to open port 80 on your node firewall. https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/ https://kubernetes.io/docs/tasks/access-application-cluster/...

GordonS 8y ago

I need to host a web API and Postgres database in a high availability configuration. I only need a single instance each of the web API and database, but uptime is crucial. I'd like to use containers, partially because it gives an easily reproducible environment for running locally. For such a small setup, what's the recommended way of handling this, such that I get automatic failover and can easily do zero-downtime updates? I did look a bit into k8s, but quickly came to the conclusion that it's far too complex for this. Oh, I'm also constrained to using the Azure cloud, if it matters.

edpichler 8y ago

My personal conclusion using Kubernetes for personal projects: I think it's very good when you have lots of deployments, iterate fastand also a CDI environment. For this, K8s is excellent.

dekhn 8y ago

I really wanted k8s to be a nice replacement for everything I could enjoy about borg. After using it for a while, I found it far more cumbersome to work with, and the architecture to be confusing. I'm quite comfortable with complicated environments, typically all I really want is a small cluster running my long-running infrastructure, and expansion to spot instance worker nodes to meet burst capability. Just setting that up in k8s, keeping it up to date, and monitoring the various components that fail because you configured just-not-enough-RAM for them became tiresome.

jhabdas 8y ago

That guy's contention is written directly above a Website with a copyright dated 2017 and they personally admitted they couldn't remember how to deploy their own site. Was it possible they couldn't do it because it was too complex? My contention is the answer to that question is yes.

cdoxsey 8y ago
If you're curious, I used google app engine and google recently changed the build tools for go on app engine. I have a lot of go source code examples on my blog which aren't intended to be compiled by app engine, but with the recent changes I couldn't get it to deploy. After a couple of hours I decided I was fed up and tried k8s. Its a bit silly, but sadly that's often how software goes, at least in my experience. (also strictly speaking I couldn't do a static site cause there are some other things I run on here that are more than just html) I fixed the copyright date :)

vlucas 8y ago

The issue is not cost. The issue is conceptual overhead and sheer complexity.

mschaef 8y ago

> You don't have to learn systemd; you don't have to know what runlevels are or whether it was groupadd or addgroup; you don't have to format a disk, or learn how to use ps, or, God help you, vim. All this stuff is important and useful and none of it is going away. I have a great deal of respect for sysadmins who can code-golf their way around a unix environment. But wouldn't it be great if developers could productively provision infrastructure without having to know all of this? No. At least for small applications, Unix already has functional, established, and documented mechanisms for the vast majority of how they need to be managed. (Windows does too.) At the end of the day, I think people would be far better off learning and using those, until there is a specific documented need for some feature unique to some more powerful platform. Otherwise, you're continually paying a complexity cost for features you might never use. Going back to the author's original list of 7 questions, I run a few personal apps in AWS, and here's how I answered: > How do you deploy your application? Just rsync it to the server? I scp a JAR file to a staging directory and run an install script. The install script is pretty much standard across the handful of apps I run. > What about dependencies? If you use python or ruby you're going to have to install them on the server. Do you intend to just run the commands manually? The JAR file is an uberjar, so the dependencies are included. > How are you going to run the application? Will you simply start the binary in the background and nohup it? That's probably not great, so if you go the service route, do you need to learn systemd? I learned systemd. It gives me easy service management and a standardized way to ensure the process comes up at server startup. The service script is pretty much standard across the handful of apps I run. > How will you handle running multiple applications with different domain names or http paths? (you'll probably need to setup haproxy or nginx) I installed nginx. Took about 30 minutes. > Suppose you update your application. How do you rollout the change? Stop the service, deploy the code, restart the service? Pretty much. > How do you avoid downtime? I don't. Many small apps can tolerate small downtime windows, including the ones I run, so I don't need to worry about continual operation. If I did have to worry about that, I'd have additional things to worry about in my application design. > What if you screw up the deployment? Any way to rollback? (Symlink a folder...? this simple script isn't sounding so simple anymore) Select the previous jar file, which I will have retained. Schema migrations complicate rollback, so I keep them to a minimum, test a bunch locally, and make sure I have backups of the database taken immediately before a deployment. (So that I can roll back if absolutely necessary.) > Does your application use other services like redis? How do you configure those services? Persistence handled through an embedded SQL database, and a library I've written to managed schema migrations. For nginx and git, I have source code snapshots of the configuration, but that's only needed if I rebuild the server. (Once in over five years.) Generally, for small apps, I try to keep the dependency footprint as small as possible. (Which is pretty much the entire gist of this comment, now that I think about it.)

humbleMouse 8y ago

I'm going to say it again, just like in every kubernetes thread. People should check out openshift. It's built by redhat and is an abstraction layer above k8's. I love it.

nqzero 8y ago

great post - thank you for convincing me not to spend another moment considering using k8s ! /s

rb808 8y ago

This is great. I always thought k8s was perfect for personal projects but too expensive, I'm going to have to double down now.

cdelsolar 8y ago

I set up a single-node cluster for a personal project on DO and it took me some time, but it's up and running and has been fine for over a year. I used a decently beefy instance ($20/monthly on DO) because the Kubernetes processes are actually too resource intensive for the $5 instance. It runs several processes (the main Python app and a helper Go-based API), automatic SSL renewal, a daily cleanup Cronjob, etc. I really like being able to push a completely new container whenever I push to master - this is done with CircleCI, which builds the image, pushes to Docker Hub, and just does the kubectl apply. I couldn't think of any other way to do it that wouldn't result in some downtime if I had to change or upgrade packages. I wrote about it here a bit: https://hackernoon.com/lessons-learned-from-moving-my-side-project-to-kubernetes-c28161a16c69 https://hackernoon.com/lessons-learned-from-moving-my-side-p...

Rotareti 8y ago
> I couldn't think of any other way to do it that wouldn't result in some downtime if I had to change or upgrade packages. https://github.com/helm/helm https://github.com/helm/helm ? You push the new version of your app as a helm chart to your helm repo and then run: helm upgrade my-app
- cdelsolar 8y ago
  no - I meant, outside of Kubernetes, in the past (sorry, my wording in my post was a bit confusing). My app used to just be something I would ssh into and then do a git pull, but whenever I had to upgrade packages it would be tricky. Then I tried Docker with docker-compose and I had similar issues with environment variable changes. Kubernetes is great; all I do is kubectl apply -f new_config.yml and everything just works without downtime. I only very recently learned about Helm when doing automatic SSL upgrading, but it looks cool too. In general I really don't think k8s is the monster people make it out to be. I did all this on the side over the course of a few weeks, a few hours here and there. I know I'm only scratching the surface of what is possible, but everything just works fine, and my .yaml files are pretty self-explanatory.

Tehchops 8y ago

I feel like Kubernetes solves a lot of problems. I don't feel like any of them have to do with personal hosting/blogs. If you really want to go simple/cheap for that, why not eschewing compute entirely and go serverless/static?

nkassis 8y ago
I tend to agree with this. Keeping thing as simple as possible to start is ideal. Also out of the box docker offers a lot of functionality for single machine and development environments that might be good enough. I've seen kubernetes make more sense when you need to grow to multiple machines and are trying to optimize resource utilization.

alexnewman 8y ago

I've never met someone who didn't love k8s unless they already knew how to run machines

sngz 8y ago

"Out of college I spent 5 years working in the Windows ecosystem. I can tell you my first job at a startup using linux was not an easy transition." more interested in how he manage to do that transition. been struggling with that part myself when competing with other candidates who have years of linux experience.

Draiken 8y ago

People keep saying k8s is so complex and the learning curve is steep but I honestly struggle to see it. The only hard part about k8s is setting up the cluster itself. Using it is so easy it boggles my mind how can someone say it's hard. It's a simple API with an amazing CLI that immediately solves dozens of issues you would have sooner or later. Take for example a very simple rails application. I'll keep it simple with 3 routes: manual, automated and with k8s (assuming you're not building the cluster itself). Manually setting up would require you to: - Create a VPS - SSH into the VPS - Install at least Ruby, NGINX (or some webserver), the DB and all Rails dependencies - If you want easy deployments you'd probably use something like Capistrano and have to set it up - If you want the system to be resilient you will have to add some sort of process monitoring and automatic startup This is very easy to get wrong. One missing step and you're having to manually SSH into the machine and trying to figure out what went wrong. An automated setup would essentially do everything you did on the manual setup, but you'd have to write that into something reproducible like Ansible, Chef, Puppet or similar. Every single one of these tools has a gotcha and a learning curve and that's on top of all the base knowledge of setting it all up by hand. Finally with k8s (assuming we're using Google's GKE offering): - Dockerize the app (which is extremely simple with tons of tutorials) - Download the gcloud CLI (alternatively it can all be done through the website if that's your thing) - Create the cluster - Either create some manual yaml files or just use kubectl to create a deployment for: rails, nginx and the database (again, there are millions of examples of this even on the official docs). - Create an ingress and point my DNS to the ingress' IP All of these methods have learning curves and are in many ways complex. However I don't think Kubernetes is a lot different than the others. It's just a different way of doing the same. * Instead of manually finding out how to setup load balancing and reverse proxying on NGINX, you learn how to use the k8s services. * Instead of setting up Capistrano (or any other deployment tool) you learn how a k8s Deployment and rollouts work * Instead of figuring out how to setup L7 load balancing you learn to use k8s Ingress On top of all that with GKE (or any other managed k8s solution, I presume) you'd automatically have: logging, metrics, scaling, rolling updates and much, much more. People are resistant to change and it's definitely not an "one size fits all" situation, but as soon as you need anything beyond an amateur website, k8s provides the best ROI. I've had to do this on all of these scenarios and I can confidently say Kubernetes changed the game. I got a 10 year old Rails application with tons of dependencies from an old dedicated server setup to Kubernetes on a few days (on Kubernetes 1.1 at the time, which was a lot harder) and I'm not some kind of genius. If I can do it, any developer can.

module0000 8y ago

>> You don't have to learn systemd; you don't have to know what runlevels are or whether it was groupadd or addgroup; you don't have to format a disk, or learn how to use ps, or, God help you, vim That's a quote from the article. The thought that someone with that mindset is responsible for anything more than a network-connected waffle iron is terrifying. This article advocates for willful ignorance of Unix/Linux, because you don't need any of those things if you know k8s. That said, k8s is nice for some things. I admin a large deployment at work, and it's relatively painless for developers to be productive with. One of the reasons for that stability isn't k8s itself though - it's the legion of seasoned Unix/Linux pros working alongside me. The reason "it just works" is because under the covers, there are a whole lot of us doing all the boring stuff to hundreds of physical hosts.

Spivak 8y ago
I'm sympathetic to the author's argument, UNIX administration is a full time job on top of whatever system you're using to do deployments. You leverage your ops team at work; is ops-as-a-service really all that different? If one of my devs wants to deploy a rails app I don't expect them to suddenly be a sysadmin or network engineer or infosec expert. I expect their application to be well written but the rest of the stack is up to ops.
blablabla123 8y ago
True, deploying stuff in an actual Unix environment while ignoring need of these tools is the perfect recipe for unstable setups. Although I guess for personal projects it could actually work as long as the load is low enough and when there is no data involved that needs to be protected. ;)

erulabs 8y ago

I had a project about this size on GKE, in an attempt to get a feel for Google hosted Kubernetes product. I went on vacation for two weeks while a container was printing an error (unbeknownst to me). Google wanted to charge me ~700 dollars for StackDriver logging volume (a feature I was not aware was connected to my Kube pod stdout). Google Cloud refused to refund or provide any guide as to how to entirely disable / purge StackDriver, so I'm back on AWS and won't be recommending anyone move to GKE any time soon (it's extremely extremely easy to suddenly more than 100x your bill...) With my hand-built Kube on AWS I have 15gb assigned for my ELK logging cluster, and it costs me nothing because it falls under the free tier. Looking forward to trying out Digital Ocean's product next!

trumpeta 8y ago
to be fair, if you have CloudWatch and send logs there it can get pretty expensive too

fridgamarator 8y ago

+1 for using Crystal

rconti 8y ago

"Kubernetes is free" FTFY