19 ms·

Unironically using Kubernetes for my personal blog

266 points by tate 6y ago

HelloNurse 6y ago
A few lines of Kubernetes configuration ... and a custom Docker image, which might tilt the scales of complexity in favor of starting from a stock installation of whatever server stack serves the blog and adding content, configuration files, SQL scripts etc. more frugally and explicitly. In particular, how is the blog updated? I don't see how replacing and/or backing up a Docker image could be more convenient than copying files through a SFTP client and using other kinds of real clients (administrative web interfaces, SSH, etc.).
hsson 6y ago
I wholeheartedly agree with everything said in this blog post. The real downside to using k8s for a personal blog or hobby project is that it's so damn expensive. I did try rolling my own k8s using k3s and some Raspberry Pis, but that quickly became annoying to maintain and to get up and running in the first place.
lyschoening 6y ago
I mean it's one Kubernetes cluster. What could it cost? 300 dollars?
- waterglassFull 6y ago
  Or 30 bananas
- tail_exchange 6y ago
  It depends on the size of the cluster. If it's just a little blog and you just want it to be as cheap as possible, but still using K8s just for fun, it will cost you around $4/month on GCP (1 f1-micro node).
  - heavyset_go 6y ago
    > 1 f1-micro node That only has 600MB of memory and the minimum memory requirements for a master node is 2GB[1]. [1] https://docs.kublr.com/installation/hardware-recommendation/ https://docs.kublr.com/installation/hardware-recommendation/
    - ex3ndr 6y ago
      Master nodes are not billed by GKE, only worker one. But micro node is not going to work - all memory will be eaten by k8s services.
    - ahmedtd 6y ago
      On GKE, the master nodes are provisioned behind-the-scenes. You get one free zonal master. The f1-micro would be the worker node. It's still a bit of a squeeze, because GKE has system workloads that need to run on user nodes.
- qbasic_forever 6y ago
  A minimal single host cluster runs fine for me on a $10/mo DO droplet with just 4GB RAM. It would probably work on the $5 droplet too if you weren't running much more than a blog.
chomp 6y ago
I mean... I'm happy he's happy, but for instance, I sign up with a budget webhost, run rsync and my blog is deployed. Super low mental burden. There's no arguing that K8s for something like a personal blog is a little overkill. Again, totally not hating on this, we all have our hobbies and I love that the author is super into this.
- taeric 6y ago
  It is mind bending to see how much more complicated than "rsync" deploying a simple static site often gets.
  - kyawzazaw 6y ago
    I just want to write these days so I have been using HEY world. Simple as sending an email!
    - saimiam 6y ago
      Can you tell me a bit more about this? How does it work? Do you have to subscribe to hey.com? What did you like about this approach? Is it a free service? I’m asking because I too made an email->blog post service for myself and I want to see if this is productizable.
      - raihansaputra 6y ago
        Yeah if you have a HEY.com account (paid), you can send an email to world@hey.com and it's published.
        saimiam 6y ago
        Thanks for the answer. Do you also use this feature to blog?
        kyawzazaw 6y ago
samcat116 6y ago
This is something I see a lot of people ignoring about Kubernetes. There are two completely different sides to it: the developer experience, and the sysadmin/devops/sre/what-have-you experience. This post completely ignores that second part, which is arguably much harder. You can ignore large swaths of those topics by using managed K8s platforms, but the pricing for those means that running a blog like this might run you $50-$80/mo
- tw04 6y ago
  Exactly what I was thinking. So you have yaml file that makes it easy to deploy your blog with k8s and you don't understand why people would tell you that's silly. Meanwhile you're spending $50+/month for something you could run on a $2.50/month VM from digital ocean or vultr. And oh by the way, there aren't a dozen moving parts that you don't fully understand that could create security vulnerabilities if misconfigured.
  - mrkeen 6y ago
    I'm on a free plan on Google cloud. I'm currently using docker-compose, but I'll probably switch to minikube when I get some free time.
- bryanlarsen 6y ago
  A single node Kubernetes "cluster" on DigitalOcean is $10/month. Adding a loadbalancer is another $15, but you can avoid that by using a hostNetwork ingress (https://stackoverflow.com/a/60726977/91365 https://stackoverflow.com/a/60726977/91365) Single node Kubernetes is pretty silly on the surface, but it is useful in some cases.
  - mamon 6y ago
    Nothing silly about it. Yes, with one node you don't have High Availability, but you get other benefits, like a standardized way to manage configuration.
  - khalilravanna 6y ago
ibrahimcesar 6y ago
With some IaC tools this would be done with 10 lines
cloogshicer 6y ago
> After spending a couple hours learning the key concepts through the official tutorial (...) That, and the ongoing complexity you're buying into, is exactly the problem.
vinceguidry 6y ago
Kubernetes is... fun. It's fun by making things that used to require hundreds of hours to figure out, only require dozens. This can give superpowers to the right kind of engineer. I spent a bunch of time hand-rolling my own cluster with it's own Wireguard network, on personal hardware, with one node in the cloud because I thought I needed one. What's neat about it is, you generally solve one problem, and once it's solved, it stays solved. Your solution is in a yaml file somewhere or in a command line option that you've persisted to a script or whatever. And everything accumulates! If you manage to get a storage cluster installed, boom, all of a sudden you have a ton of options available to you. Once the cluster is stable and you are comfortable bringing it up and tearing it down so that everything's working right, you can start pulling parts of the k8s infrastructure in, need a DNS server? Run it on your cluster! K8s wants a docker container registry, you really don't want to run that on your cluster in the beginning, but once your cluster's secure, why not! K8s starts eating everything in your life like the effing Borg and it's great! If you're the kinda cat that will spend 80 hours on a Factorio world, then dive into the mods and make the tweaks you really wish the authors would make, and you have enough admin experience to do general troubleshooting of complex systems, I can't recommend Kubernetes enough for the sheer fun factor. The downside if you want to do anything serious with it is, actually the same as running your own infrastructure always is and was, network connectivity. Hardware is cheap. Last mile network connectivity, isn't. The solution to that, is, of course, colocation.
- emodendroket 6y ago
  You can always run it in the cloud and at least theoretically be able to switch providers and take your cluster with you.
  - vinceguidry 6y ago
    Expensive. I've got all these machines sitting here at home, why not use those?
    - camjohnson26 6y ago
      Hard to beat the uptime of a cloud provider with on prem, if you literally mean “at home” even harder, my Comcast goes out a few times a day. Even in offices you don’t benefit from the economies of scale and redundancies in the power and network systems that cloud providers have.
aequitas 6y ago
I might be looking in the wrong places, but what I'm really missing is just an AAS where you deploy your pods and pay just for the CPU, memory, disk and ingress. No managing a cluster, no baseline cost, just usage.
- bryanlarsen 6y ago
  GKE AutoPilot gives that. There's a cluster management baseline fee, but the first one is rebated.
- SwiftyBug 6y ago
  You mean like AWS Lambda, but using your own pods?
- Jarwain 6y ago
  Azure Container instances kinda do this, but you have to orchestrate scaling yourself Azure App Services also fill this niche, but only works for web-based services. If you have a job queue and a worker reading jobs off the queue but otherwise not exposing a web endpoint, it doesn't seem to work quite as well and my (limited) attempts at faking an endpoint haven't appeared to pan out :d
nrmitchi 6y ago
Kubernetes for running a blog (or multiple) is fantastic... if you are already using Kubernetes for other things. If it's for just the blog, and the only goal is to run the blog, then it is (almost definitely) overkill. If you already have a cluster up, or have a bunch of other projects already running on the cluster, then Kubernetes is likely the easiest way to run a blog.
- paxys 6y ago
  The reverse is true as well though. If you learn Kubernetes and set up a cluster to host a blog, you'll find that it is now trivial to deploy a hundred other things to it.
  - nrmitchi 6y ago
    For sure. > and the only goal is to run the blog If part of the end of of "run by blog on Kubernetes" is "I want to use this as a learning experience to run other things on Kubernetes afterwards" it's definitely a worth-while exercise. > you'll find that it is now trivial to deploy a hundred other things to it. I mean, sort of. Starting with a blog is a good introduction and gives an on ramp to running other services, but I wouldn't say it is sufficient experience to call running hundreds of other services "trivial".
uberduper 6y ago
It really doesn't take much experience with kubernetes before it becomes fairly easy. And then using it for trivial things like this isn't a big deal. My home "server" runs plex, sab, mongo, unifi, and various other things in a single node k8s with a local zfs volume provisioner. The previous revision of this server I'd switched to using docker for everything and was annoyed with the upgrade process with just docker alone. With k8s, I just use :latest for most images and an upgrades happen every time I restart a pod or reboot the machine. I've been working with k8s since petSets and so this is all NBD to me. Linux had a learning curve too and ya'll got over it.
- tenacious_tuna 6y ago
  A friend and I have recently stood up homelabs for funsies. Mine's a cluster of x64 boxes, his is a small pile of raspi4s. He went for a k8s cluster off the bat--having worked with cloud stuff for a while, he had the familiarity to hit the ground running. I was immediately overcome by all the abstractions. I had no idea where to look to figure things out, and was mostly relying on advice from my friend. I dind't know what an ingress controller was, much less how to configure one--but I knew I wanted each 'service' to have its own IP I could route to from my network. Overall it felt like I had SO MUCH TO LEARN at *every point* it was difficult to get even close to my actual goals (CI/CD for a personal site + some hosted game servers). I eventually went with the philosophy of "build something now, and move towards perfection later" / "don't let best be the enemy of good" and begain spooling up LXCs and VMs to do the work I needed, planning to move things into k8s later when I better understood the actual things I wanted to move. (Plus then I got some satisfaction out of actually accomplishing the goals I wanted, instead of just banging my head on k8s documentation and learning all the abstractions.) As an example, I've not used docker in any meaningful capacity. For anything. No idea how to make a docker image. To k8s the CI for my docker site, I needed to know how to : 1. install the dependencies, which requires compiling a plugin for pandoc, which requires installing haskell and cabal. this is expensive, so I'd prefer to get the pre-res set up once... but that doesn't seem to be how docker works? Do I need an image repository? can I use DockerHub? I've seen HN talk about how docker is trying to monetize, should I run my own repository? Can I do that on my cluster? I'll need an ingress controller to route to it... I don't even know what that is. 2. I need some way to pass the built website files to the contianer I actually want to host them. I think that means I need an NFS share of somekind to store the files, so one coantainer can load them and another can read them. Do I hos tthat on my NAS? I could put an NFS share in the cluster, maybe. No idea how to get Docker to mount one, or k8s to host one. All the examples I seem to find deal more with connecting to remote services on a host than mounting local storage. is it even local storage? 3. Everyone says infrastructure as code is good, so I guess I'll follow this flux tutorial--only to find out the one I followed is out of date, and I should follow their NEW one. But they still assume I know way more baout k8s than I actually do, but still, I'll spend the few hours to get this operational, so then in theory everything else I deploy can be IaC'd, which is just good practice. At this point I'm so many layers of abstraction deep, I have no idea what I'm actually doing or how concepts relate to each other, and I'm no closer to actually having my goal. So last night I spent 2 hours spinning up VMs on my cluster and installing dependencies, configuring an nginx proxy, and now I actually have my personal blog self-hosted and updatable. Way more "progress" than the 10ish hours I've sunk into building a k8s cluster already. There's something to be said for limiting the number of abstractions you're dealing with.
tombert 6y ago
I don't really blog anymore but I do run a cluster in my basement for personal projects, so I totally get the appeal. I use Docker Swarm, not k8s, which I personally felt was way simpler to set up, but there's something kind of cool that the system I built is truly scalable to thousands of nodes if I really felt like it. Out of curiosity, what do most people here who have done such a thing (with k8s or docker swarm or otherwise) use for storage? I tried briefly to use Gluster-fs but had really bad problems with performance (most likely because I set it up wrong). Right now, I just have a ZFS Raid on one of the nodes, and have an NFS share on that.
- denysvitali 6y ago
  I either use Longhorn (https://longhorn.io/ https://longhorn.io/) or TopoLVM (https://github.com/topolvm/topolvm https://github.com/topolvm/topolvm), depending on what I need. Rook (https://rook.io/ https://rook.io/) is also cool.
- tene 6y ago
  I've had a great experience with https://rook.io/ https://rook.io/ managing Ceph for Kubernetes in production.
chmod775 6y ago
The author seems to think the only real alternatives are docker or whatever. So here's a way to host your personal blog if you don't want to over engineer it: 1. Have a git repo with your nginx/whatever config files. 2. Have a VPS running debian. 3. apt-get install nginx git ... git clone ... ln ... # create symbolic links to your nginx/whatever config in your git repo systemctl restart nginx ... 4. You're done. Create a cron job to automatically pull the latest changes from your git repo if you want. The above steps should take most people around 10 minutes. If you need to actually pivot into something that scales easier from there, I recommend following these steps/levels as your scale increases: 1. Create an automatic install script. 2. Use that script to create a .deb package instead of installing directly (optionally create a repository for this). 3. If you want to move to docker or what have you, it's trivial to install a debian package in a container. But let's be real, you never even will have to do any of the above because it's a personal blog, and it'll probably scale to the world population on a $5 VPS, especially if you slap cloudflare in front.
- topkeks 6y ago
  Seems insanely overcomplicated. firebase deploy --only hosting
- andrewzah 6y ago
  These days I see no point in installing services on a VPS directly, other than docker + docker-compose. You could do it in one image with the reverse proxy + static files, or break it out into two images (this is helpful if you run more services on the VPS). As for updates, caddy v1 used to support pulling in from git, but I don't think that got ported to v2. So what I do is build+push a docker image, and have a cron job on my vps to pull+restart my website's container. My preferences go Traefik > Caddy > Nginx, but traefik definitely has a bit of a learning curve. https://github.com/andrewzah/andrewzah-com-source/blob/master/Dockerfile https://github.com/andrewzah/andrewzah-com-source/blob/maste... https://github.com/andrewzah/andrewzah.com-docker/tree/master/services/http https://github.com/andrewzah/andrewzah.com-docker/tree/maste...
  - oftenwrong 6y ago
koeng 6y ago
I also unironically use kubernetes (in particular, k3s) to run my personal blog. I run it on a colocated server I built myself, with the hope to have a couple colocated servers in the future I could network together. Right now, it has proxmox running a few VMs, 3 of which are for k3s, one for hhd backups, and one for a postgres server (which I am lazy, so am running with Dokku. Makes backups easy). The drives all run ZFS mirrors (2 2tb nvme, 2 8tb hdd) Honestly, it's a super comfy setup, and very little maintenance. The one thing I have is a master update script that does the docker image upgrading and rollouts. I make updates all the time and pretty much don't think about it. More energy activation than a PaaS like Dokku, but worth in long run, I think.
- newman314 6y ago
  Do you have a blog post showing how you set things up?
itsdex 6y ago
Next step: unironically creating a Kubernetes Operator for your personal blog https://github.com/dexhorthy/captains-log https://github.com/dexhorthy/captains-log
devilduck 6y ago
This is like bringing an actual gun to a water pistol fight. Way overkill and the type of thing only engineers care about
erichmond 6y ago
Another thing worth calling out, there's nothing wrong with over-engineering solutions in pursuit of knowledge. Especially when it's just a personal project.
crazypython 6y ago
A side note, how do you use Kubernetes to set up HTTPS on wildcard DNS? Is there a service for autoconfiguring the DNS and getting the certificate without a forward proxy?
- qbasic_forever 6y ago
  cert-manager supports the Lets Encrypt DNS challenge, which supports wildcard certs: https://cert-manager.io/docs/ https://cert-manager.io/docs/ It's a little bit of one time setup and then easy-peezy SSL automatically for all your services. Do note that you need a DNS provider with a good API to programmatically control it (the DNS challenge requires setting a special txt record that Lets Encrypt verifies). Read the docs to learn more, but most of the big ones (AWS Route 53, Cloudflare, etc.) should be fine.
- kryptn 6y ago
  > Is there a service for autoconfiguring the DNS There is! I use external-dns. [1] I haven't actually set up a Let's Encrypt wildcard cert, but I'm pretty certain cert-manager [2] supports it. I don't think you need a proxy if you use the DNS01 challenges. [1] https://github.com/kubernetes-sigs/external-dns/ https://github.com/kubernetes-sigs/external-dns/ [2] https://cert-manager.io/docs/ https://cert-manager.io/docs/
crazypython 6y ago
On the topic of "takes too long to learn" can someone recommend a good tutorial on basic Kubernetes concepts?
- CSSer 6y ago
  I think OP put this just as good or better than I could've in their blog post. > I think Kubernetes has fallen into the Vim/Haskell trap. People will try it for 10 minutes to an hour then get fed up. The point where you start to grok stuff just happens too late for most people to stick with it. Those become a vocal minority proclaiming it as too complex for humans to understand, and scare off people that haven’t tried it at all. I've had good luck with the tutorials and various help articles in the official docs[0]. [0]:https://kubernetes.io/docs/tutorials/kubernetes-basics/ https://kubernetes.io/docs/tutorials/kubernetes-basics/
- qbasic_forever 6y ago
  Kubernetes The Hard Way if you want to start from first principles and learn all about low level tools and setup. In practice you're probably going to use a cloud provider or tool to automate all of this stuff, but it's useful to understand the core infrastructure like TLS certs, etcd, etc. Otherwise the official docs are quite good and worth starting there. I'd be wary of buying books that are more than a year or two old--the k8s space moves fast and they regularly have major updates once or twice a year. Books and docs can get a bit stale or out of date with current best practices.
- pas 6y ago
  The trick is that to have a real chance of administering a k8s cluster you need Linux kernel knowledge (namespaces to understand docker, a bit of iptables, VXLAN or something similar to understand CNI/overlay network, how the container network namespace is connected via veth pipe, chroot/bind mounts, a very little bit of SElinux/apparmor/seccomp), systemd to see how it all starts, etcd to know about the consistency/consensus layer.. so far this is almost identical for all container orchestrator/scheduler platforms (nomad, docker swarm). Then there are the specific k8s concepts: pod (containers that share a network and PID namespace), service (a name, a TCP and/or UDP port number, and the name/selector of the target pod), ingress (HTTP routing, basically haproxy/nginx/envoy or other layer 7 stuff), LoadBalancer (the magic API that connects your cluster to the outside world, allocates IP addresses for Services/Ingresses). Then there are the background parts that take the YAML and convert it into actual running stuff: apiserver (this is the central info hub, kubectl and kubelets and controllers all talk to this basically), kubelet is the actual agent on each node, controllers have control loops and those actually calculate the necessary actions/operations to achieve what the YAML declared. Then there are the subsystems: storage (persistent volume claims and PVs), settings (secrets - see vault too - and configmaps, and various special/custom resources like a TLS certificate, these are stored in etcd, through the apiserver of course), CNI networking flavors and knobs, an endless list of kubelet and apiserver command line arguments, PCI passthrough for GPUs or HSMs or whatever. And of course there's RBAC, security, admission webhooks, see how nowadays Authz and Authn is done through a "lookaside" OpenId Connect service (some Ingress Controllers support these out of the box).
godot 6y ago
[Deleted]
- oauea 6y ago
  Sure, pay some other people to make your own worries go away. This post is clearly not for you.
oauea 6y ago
I am unironically using Kubernetes for my personal home server. Thanks to https://k3s.io/ https://k3s.io/ this is really easy to do, great fun and extremely useful. I have a git repo containing all my helm charts & docker files, testing & deploying changes is absolutely trivial now. And it's great to have everything version controlled. Previously I used to use Ansible, but you quickly run into issues which make you want containerization: Conflicting library/tool versions, packages that pollute too much of the system, port conflicts, hassle of keeping the playbook idempotent, etc. So while docker-compose would also do fine, having kubernetes to manage the ingress' routing system is rather practical. And the same goes for the other bits and bobs of infrastructure offers you if you're already using it. It's just very convenient. I've been doing this for a few years now, and am now up to 14 different apps running on my single home machine in Kubernetes, ranging from Home Assistant to PostgreSQL to Plex. Also it's just good experience. I also use Kubernetes for work, and this has made me noticeably more proficient.
pnathan 6y ago
I run GKE for some small apps. I also use AWS S3 hosting for my personal blog. The cost differences are... non trivial to the point of a bad joke, if we were comparing ability to reliably ship plaintext over the wire. But I'm not. I host a database and webapps on the k8s cluster, without adding extra EC2 nodes, RDS costs, or wrestling with AWS Lambda limitations. I can also confidently say that having something approximating a stable web app demands doing a lot of serious thinking, and "a single server running Apache on Digital Ocean" does not cover that case sufficiently. You need to tolerate failure, failover, load balancing, bin-packing, etc. I used to run a small autoscaling group on EC2 for my own systems; the dang thing would fail to come up on one node very frequently and so a number of the queries would fail. I eventually burnt it to the ground and redid it. I've never had that hassle in k8s. Its designed to succeed, in a way the "box of parts" approach doesn't. Boxes of parts are useful. For a complexity-sensitive & thoughtful infrastructure engineer, having something like the old Synapse/Nerve[1] system with your apps distributed across some 5-20 machines with a monitor lease to spawn new ones on failure would probably approximate Kubernetes for a few years, until you have to do something fancypants. You've still reimplemented part of Kubernetes, though... The other angle is, boxes of parts can go in wildly weird directions.... if you need it. Looking at some infrastructure these days professionally, the question is - when do we move to Kubernetes. It's not interesting or useful to the company to be maintaining our own thing or own strange path. The only questions are around the path - how much rework needs to happen and how much building in k8s needs to happen to get there. GKE is a very good starting point for k8s. Strong recommend. n.b. With respect to the cost. I consider this a professional investment / professional development expense. Spending $100-$200/month on a software engineer's salary is a reasonable return for being able to readily say I have experience in a current topic. Also I can run my own apps. :) https://github.com/airbnb/nerve https://github.com/airbnb/nerve
proxysna 6y ago
My 3 droplets run my Rust, Quake, gitea, registry, wireguard & pihole combo, traefik, grafana + prometheus with hashicorp nomad & consul. So yeah running an orchestrator for trivial tasks is actually fun, this is a good exercise and just in general a fun thing to do.
jvalencia 6y ago
I currently have a k3d cluster running locally with an entire product stack locally. It' nice to think that I can simply move that over to a full cluster at any time. There isn't even a change in tooling as the kubernetes context is the only change needed.
simo7 6y ago
Well I wish he expanded a bit more on the “Just do X, that’s so much simpler” section. He's on AWS, he could have gotten all those benefits with ElasticBeanstalk or ECS even. Plus no yaml files but actual IaaS (terraform etc.) and much better integrations with the other AWS services. I'd personally still use EC2 for a blog, but if you're looking for convenience/battery-included type of thing...yes I'd argue you're better off just doing X instead.
- zug_zug 6y ago
  Yeah... It's like software hipsterism, whatever is weird/hard is "cool," but the moment it become mainstream suddenly a new technology/language needs to made instead. 2 years from now everybody and their grandma will understand kubernetes or have a simple gui for it, and the hipsters will move on to something new and exclusive.
spion 6y ago
YAML is the biggest flaw of Kubernetes, so I'm quite exited that cdk8s is progressing very nicely https://github.com/cdk8s-team/cdk8s https://github.com/cdk8s-team/cdk8s There are other solutions that are potentially better (e.g. Dhall) but cdk8s seems to have momentum and sense for tackling the practical stuff (integrates easily with cdk, library with simplified constructs cdk8s-plus, import and convert existing stuff easily etc)
- kube-system 6y ago
  k8s also has a first-party typescript API.
- bryanlarsen 6y ago
  We use and like https://jkcfg.github.io/#/ https://jkcfg.github.io/#/
- arcticfox 6y ago
  I do all of my k8s work through Pulumi using Typescript, and it as an absolute joy. And it's not limited to k8s, which is great.
systemvoltage 6y ago
Can someone who is familiar with Kubernetes please explain in plain words why we use Kubernetes? I’ve read a lot about it but I always read some bullshit non-answer such as “It’s an orchestration platform for containers”. What does that even mean? Even this blogpost doesn’t explain what and why’s of Kubernetes. I have a docker container. I deploy them on vm. I use load balancer to split traffic. Could you please walk me through what problem Kubernetes would solve here?
- pickledish 6y ago
  After using k8s for about a year at work now, the way I understand it is that there’s a really large gap in abstraction between “a bare metal machine or VM with a shell prompt” and “a service that handles requests with some Python code”, and kubernetes helps fill that gap with some sane defaults and useful abstractions for people who run services. If you run your docker app on kubernetes, you get a lot of things for free with the platform (rolling no-downtime deployments, service discovery, auto scaling) that you’d have to set up manually if you were running your service on (say) EC2 instances instead. It can be a headache to learn sometimes, but ultimately saves a lot of effort if your use case fits!
- __blockcipher__ 6y ago
  Kubernetes does a lot of different things but one of the more important is resource management. You spin up a fleet of Kubernetes nodes which can have totally heterogenous (differing) amounts of compute/memory/disk, and have the scheduler make intelligent decisions about where to stick containers based off your (optionally specified) list of resource requirements. So it solves the bin-packing problem automatically rather than having to manually map out an efficient way to use your infra. It doesn’t reduce the complexity per say - the interactions can get complicated if you’re using stuff like node taints/tolerations instead of the more computationally simple “App X needs this much RAM but beyond that I don’t care where it lives” If I find a bunch of raspberry PIs in my basement and want to have them join my fleet, I just do that and even if my fleet varies from 256 core CPU boxes with huge raid arrays all the way down to tiny raspberry PIs, the scheduling just works. Note here the broader pattern of abstracting away the physical hardware; it’s a really important concept to grok.
unknown2374 6y ago
50 lines for configuration of a blog is not what I'd have personally called "simple". Granted GUIs (e.g. AWS console as mentioned in the blog) are not as easy as an config file, the comparison should be against other blog deployment programs, like hugo/jekyll/ghost, not.. the AWS console.
- turtlebits 6y ago
  That includes 2 load balanced instances of his blog + domain + automatic HTTPs
nanis 6y ago
> I’m not exposed to that complexity Some of us think editing YAML files is complex enough.
revskill 6y ago
What clicks me about Kubernetes is not about orchestration or zero downtime or scaling,... I can just use Kubernetes on 1 node for all of my services. What matters is the declarative approach of Kubernetes. It's like the first time I learnt about React: declarative renderering/deployment based on state !
- heavyset_go 6y ago
  Here's the thing that gets me: Docker Compose and Docker Swarm can offer declarative infrastructure and only take up under 50MB of resident memory. I need 2GB minimum for a K8s master node, 2GB minimum for monitoring, and then 700MB minimum for each worker node[1]. I just can't justify running K8s for a blog when there are less expensive and complex offerings out there. [1] https://docs.kublr.com/installation/hardware-recommendation/ https://docs.kublr.com/installation/hardware-recommendation/
  - turtlebits 6y ago
    Try K3s. I run it on 3x of OVH's cheapest VPS ($3.50/mo) and works great. I have Postgres, Hasura, Keycloak, Riemann, Riemann dashboard and Digdag running- the cluster has been up over a year with no issues.
    - heavyset_go 6y ago
      Can I use multiple machines as nodes for K3s? I think the last time I looked at it, clusters were limited to one machine, but that was a long time ago.
      - revskill 6y ago
        Yes, you can easily join multi master nodes or worker nodes with k3s.
        heavyset_go 6y ago
        Awesome, thanks.
platty 6y ago
I just connect Netlify to my repo, and boom blog is deployed.
anyfoo 6y ago
Let's see what happens in a few years. There are a bunch of blogs/content out there that have been abandoned for a long while, but you can still read the information that was put there in the past. The simpler the system serving the pages is (down to static HTML files), the more likely it is that the website stays up functional. A blog that is built on Kubernetes sounds like it will be dropped pretty quickly once interesting in writing for it wanes.
- astrange 6y ago
  Static HTML pages might be the only safe way to keep a blog running; my experience running Wordpress is if you leave it up long enough someone will drive by, hack it, and Dreamhost will block your site.
kklemon 6y ago
Am I the only one still using Docker Compose?
- wiredfool 6y ago
  No, But I'm looking at what to do to streamline deploys and provisioning across 10's of vms, each running docker-compose. For one or two services on a machine, it's pretty good. Past that... not so much.
  - nikisweeting 6y ago
    I run docker-compose inside supervisord. Surprisingly it works great, they play well together and I highly recommend it for managing multiple compose projects per machine. [program:some-docker-project] command=docker-compose up directory=/opt/some-docker-project ... [program:some-other-docker-project] command=docker-compose up directory=/opt/some-other-docker-project ... This style of setup has served many millions of visitors across dozens of projects in production at our company over the last ~3 years. It's like a poor-mans kubernetes, it handles autorestarting failed projects, we handle ingress with cloudflare argo tunnels in docker, and deployments are easy (just git pull && supervisorctl restart all).
    - wiredfool 6y ago
      I'm more at the level of one Docker-compose per VM, and I've got it loading with some os too. Service or similar. But there are at least a few problems with that. 1) Want Ci where push to github builds the image and deploys. 2) Binpacking time. We're at the point where that's an issue 3) Host machines -- There's too much on them and we need to standardize and streamline.
callamdelaney 6y ago
Okay but when Kubernetes breaks it's usually none-trivial to understand why. I've spent more time than I'd like diagnosing Kubernetes issues with networking and orchestration which could have been spent on building features.
bostonvaulter2 6y ago
> For example, the entire deployment configuration for this blog is contained in this yml file Isn't that not quite correct? If I'm reading this correct (which I may very well not be), isn't this a reference to a Dockerfile: image: marcusbuffett/blog:latest Which then might have lots of other complexity contained within. I wouldn't call that self-contained, I think that's overselling it.
robertlagrant 6y ago
> I think Kubernetes has fallen into the Vim/Haskell trap There was a period where Git was at this point as well, but it seems as though everyone's got over it and decided the learning curve is always worth it.
hardwaresofton 6y ago
Been running my blog on Kubernetes since 2017[0] and it's been great. Kubernetes is a force multiplier and the natural evolution in deployment methodology. It could be tighter/cleaner but it's extremely good for what it provides, but you have to know why you might need the things it's providing for it to seem worth it. If you'd like to try kubernetes and get to understand it and feel comfortable with it I'd recommend: - Work through kubernetes the hard way[1] (ignore/replace the GCP-specific things, if you don't know enough about what the non-GCP corrolary to a GCP-thing is, that's a bit of knowledge you need to fill) - Set up your own cluster from scratch (no kubeadm, no alternate distros, etc), use the simplest options you can find at first (ex. flannel for CNI) - Set up ingress (NGINX ingress is a good place to start) - Run some simple unsecured but diverse workloads (static sites on NGINX, Wordpress, etc), figure out why you might pick a StatefulSet versus DaemonSet. At this point, use the hostPath/local volumes just to avoid trying to grok volume complexity. - Install a useful cluster tool ("addon") like cert-manager[2] from scratch so you can see Kubernetes manage something you'd normally solve with systemd timers and `certbot` (or your reverse proxy would do for you if you're running caddy or traefik) - Start putting your YAML in source control get familiar with either kustomize, helm, or both (you could also just use Make + envsubst like I did for a while[3]). It's at this point that it should click that all you need to get back to a certain state of your cluster is to get a machine, do basic hardening (ufw, etc), install kubernetes, and run "make" in this repo (excluding things like DNS entries, etc). Now things are probably getting fun, because you can have the distant cousin of immutable infrastructure; repeatable infrastructure. - Tear down your cluster, set it up again with kubeadm (note that kubeadm actually has a file-driven configuration option[3]), run your yaml from source control and confirm that all the workloads you had in place are back up and secured. - (optional) Tear down your cluster, try rebuilding it with k3s[4] or k0s[5] - Start looking around and seeing what your options are and the ecosystem that exists -- digging deeper into the interfaces that make Kubernetes tick, for example volume management (Container Storage Interface) by deploying Rook[6] or OpenEBS[7]. Note that one of the best things that can happen to you during this process is something going wrong. Every time something goes wrong, and you go back to fundamentals to fix it. Being able to reason about kubernetes ad hoc (ex. if DNS is correct but you can't reach port 80/443, what should you check first?) is the key to feeling comfortable with it, and while failures and downtime will happen, you shouldn't feel just complete despair/confusion. Normally, once things are stable kubernetes is quite worry-free. I've made a guide like this before, I'll see if I can find it. [EDIT] Found only one post[8] [0]: https://vadosware.io/post/fresh-dedicated-server-to-single-node-kubernetes-cluster-on-coreos/ https://vadosware.io/post/fresh-dedicated-server-to-single-n... [1]: https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way [2]: https://github.com/jetstack/cert-manager https://github.com/jetstack/cert-manager [3]: https://www.vadosware.io/post/using-makefiles-and-envsubst-as-an-alternative-to-helm-and-ksonnet/ https://www.vadosware.io/post/using-makefiles-and-envsubst-a... [4]: https://k3s.io/ https://k3s.io/ [5]: https://docs.k0sproject.io https://docs.k0sproject.io [6]: https://rook.io/docs https://rook.io/docs [7]: https://docs.openebs.io/ https://docs.openebs.io/ [8]: https://news.ycombinator.com/item?id=22117684 https://news.ycombinator.com/item?id=22117684
flemhans 6y ago
Often, when people discuss how easy or hard Kubernetes is, they mean running it on a managed platform like GKE. Running it on your own metal is supposedly what requires that team of engineers.
- verdverm 6y ago
  Setting up and running Kubernetes binaries across many VMs is not hard. The hard part is managing all of the addons, extras, and post setup tasks (roles, quotas, etc) to make (and keep) it production worthy. Operational experience is still rare
lmm 6y ago
The problem I have with Kubernetes is the Docker part. I get why people who are running Python need to use Docker for all their deployments (because anything is better than Python package management), but I've got well-behaved programs in a language that has decent package management, so I just want to run a bunch of them on different hosts. Is there an alternative that lets me orchestrate deploying a bunch of processes on a bunch of servers without having to interpose Docker in between? Anyone using Nomad?
ed_balls 6y ago
I had bad experience with k8s. The learning curve is step. A few days ago k8s decided to run a cron job just after the deploy (it was scheduled for Sunday). Why? it could be a bug (https://github.com/kubernetes/kubernetes/issues/63371 https://github.com/kubernetes/kubernetes/issues/63371) I'm not sure how to even debug this. > A cron job creates a job object about once per execution time of its schedule. We say "about" because there are certain circumstances where two jobs might be created, or no job might be created. We attempt to make these rare, but do not completely prevent them. Therefore, jobs should be idempotent Is there a way to make a cron job idempotent?
dimitar 6y ago
The real competition in terms of price and convenience is a VPS. So to approximate this with Kubernetes you have to avoid the big cloud providers and things like kubeadm as that would mean at least two cores and the resulting price tag (even if you use things like burstable). Using k3s is a nice option. Must avoid load balancer and often static IPs. I wonder why there aren't vendors who are selling k8s namespaces directly with a quota (reinvented shared hosting in k8s).

266 points by tate 6y ago

A few lines of Kubernetes configuration ... and a custom Docker image, which might tilt the scales of complexity in favor of starting from a stock installation of whatever server stack serves the blog and adding content, configuration files, SQL scripts etc. more frugally and explicitly. In particular, how is the blog updated? I don't see how replacing and/or backing up a Docker image could be more convenient than copying files through a SFTP client and using other kinds of real clients (administrative web interfaces, SSH, etc.).

hsson 6y ago

I wholeheartedly agree with everything said in this blog post. The real downside to using k8s for a personal blog or hobby project is that it's so damn expensive. I did try rolling my own k8s using k3s and some Raspberry Pis, but that quickly became annoying to maintain and to get up and running in the first place.

lyschoening 6y ago

I mean it's one Kubernetes cluster. What could it cost? 300 dollars?

waterglassFull 6y ago
Or 30 bananas
tail_exchange 6y ago
It depends on the size of the cluster. If it's just a little blog and you just want it to be as cheap as possible, but still using K8s just for fun, it will cost you around $4/month on GCP (1 f1-micro node).
- heavyset_go 6y ago
  > 1 f1-micro node That only has 600MB of memory and the minimum memory requirements for a master node is 2GB[1]. [1] https://docs.kublr.com/installation/hardware-recommendation/ https://docs.kublr.com/installation/hardware-recommendation/
  - ex3ndr 6y ago
    Master nodes are not billed by GKE, only worker one. But micro node is not going to work - all memory will be eaten by k8s services.
  - ahmedtd 6y ago
    On GKE, the master nodes are provisioned behind-the-scenes. You get one free zonal master. The f1-micro would be the worker node. It's still a bit of a squeeze, because GKE has system workloads that need to run on user nodes.
qbasic_forever 6y ago
A minimal single host cluster runs fine for me on a $10/mo DO droplet with just 4GB RAM. It would probably work on the $5 droplet too if you weren't running much more than a blog.

chomp 6y ago

I mean... I'm happy he's happy, but for instance, I sign up with a budget webhost, run rsync and my blog is deployed. Super low mental burden. There's no arguing that K8s for something like a personal blog is a little overkill. Again, totally not hating on this, we all have our hobbies and I love that the author is super into this.

taeric 6y ago
It is mind bending to see how much more complicated than "rsync" deploying a simple static site often gets.
- kyawzazaw 6y ago
  I just want to write these days so I have been using HEY world. Simple as sending an email!
  - saimiam 6y ago
    Can you tell me a bit more about this? How does it work? Do you have to subscribe to hey.com? What did you like about this approach? Is it a free service? I’m asking because I too made an email->blog post service for myself and I want to see if this is productizable.
    - raihansaputra 6y ago
      Yeah if you have a HEY.com account (paid), you can send an email to world@hey.com and it's published.
      - saimiam 6y ago
        Thanks for the answer. Do you also use this feature to blog?
        kyawzazaw 6y ago

samcat116 6y ago

This is something I see a lot of people ignoring about Kubernetes. There are two completely different sides to it: the developer experience, and the sysadmin/devops/sre/what-have-you experience. This post completely ignores that second part, which is arguably much harder. You can ignore large swaths of those topics by using managed K8s platforms, but the pricing for those means that running a blog like this might run you $50-$80/mo

tw04 6y ago
Exactly what I was thinking. So you have yaml file that makes it easy to deploy your blog with k8s and you don't understand why people would tell you that's silly. Meanwhile you're spending $50+/month for something you could run on a $2.50/month VM from digital ocean or vultr. And oh by the way, there aren't a dozen moving parts that you don't fully understand that could create security vulnerabilities if misconfigured.
- mrkeen 6y ago
  I'm on a free plan on Google cloud. I'm currently using docker-compose, but I'll probably switch to minikube when I get some free time.
bryanlarsen 6y ago
A single node Kubernetes "cluster" on DigitalOcean is $10/month. Adding a loadbalancer is another $15, but you can avoid that by using a hostNetwork ingress (https://stackoverflow.com/a/60726977/91365 https://stackoverflow.com/a/60726977/91365) Single node Kubernetes is pretty silly on the surface, but it is useful in some cases.
- mamon 6y ago
  Nothing silly about it. Yes, with one node you don't have High Availability, but you get other benefits, like a standardized way to manage configuration.
- khalilravanna 6y ago

ibrahimcesar 6y ago

With some IaC tools this would be done with 10 lines

cloogshicer 6y ago

> After spending a couple hours learning the key concepts through the official tutorial (...) That, and the ongoing complexity you're buying into, is exactly the problem.

vinceguidry 6y ago

Kubernetes is... fun. It's fun by making things that used to require hundreds of hours to figure out, only require dozens. This can give superpowers to the right kind of engineer. I spent a bunch of time hand-rolling my own cluster with it's own Wireguard network, on personal hardware, with one node in the cloud because I thought I needed one. What's neat about it is, you generally solve one problem, and once it's solved, it stays solved. Your solution is in a yaml file somewhere or in a command line option that you've persisted to a script or whatever. And everything accumulates! If you manage to get a storage cluster installed, boom, all of a sudden you have a ton of options available to you. Once the cluster is stable and you are comfortable bringing it up and tearing it down so that everything's working right, you can start pulling parts of the k8s infrastructure in, need a DNS server? Run it on your cluster! K8s wants a docker container registry, you really don't want to run that on your cluster in the beginning, but once your cluster's secure, why not! K8s starts eating everything in your life like the effing Borg and it's great! If you're the kinda cat that will spend 80 hours on a Factorio world, then dive into the mods and make the tweaks you really wish the authors would make, and you have enough admin experience to do general troubleshooting of complex systems, I can't recommend Kubernetes enough for the sheer fun factor. The downside if you want to do anything serious with it is, actually the same as running your own infrastructure always is and was, network connectivity. Hardware is cheap. Last mile network connectivity, isn't. The solution to that, is, of course, colocation.

emodendroket 6y ago
You can always run it in the cloud and at least theoretically be able to switch providers and take your cluster with you.
- vinceguidry 6y ago
  Expensive. I've got all these machines sitting here at home, why not use those?
  - camjohnson26 6y ago
    Hard to beat the uptime of a cloud provider with on prem, if you literally mean “at home” even harder, my Comcast goes out a few times a day. Even in offices you don’t benefit from the economies of scale and redundancies in the power and network systems that cloud providers have.

aequitas 6y ago

I might be looking in the wrong places, but what I'm really missing is just an AAS where you deploy your pods and pay just for the CPU, memory, disk and ingress. No managing a cluster, no baseline cost, just usage.

bryanlarsen 6y ago
GKE AutoPilot gives that. There's a cluster management baseline fee, but the first one is rebated.
SwiftyBug 6y ago
You mean like AWS Lambda, but using your own pods?
Jarwain 6y ago
Azure Container instances kinda do this, but you have to orchestrate scaling yourself Azure App Services also fill this niche, but only works for web-based services. If you have a job queue and a worker reading jobs off the queue but otherwise not exposing a web endpoint, it doesn't seem to work quite as well and my (limited) attempts at faking an endpoint haven't appeared to pan out :d

nrmitchi 6y ago

Kubernetes for running a blog (or multiple) is fantastic... if you are already using Kubernetes for other things. If it's for just the blog, and the only goal is to run the blog, then it is (almost definitely) overkill. If you already have a cluster up, or have a bunch of other projects already running on the cluster, then Kubernetes is likely the easiest way to run a blog.

paxys 6y ago
The reverse is true as well though. If you learn Kubernetes and set up a cluster to host a blog, you'll find that it is now trivial to deploy a hundred other things to it.
- nrmitchi 6y ago
  For sure. > and the only goal is to run the blog If part of the end of of "run by blog on Kubernetes" is "I want to use this as a learning experience to run other things on Kubernetes afterwards" it's definitely a worth-while exercise. > you'll find that it is now trivial to deploy a hundred other things to it. I mean, sort of. Starting with a blog is a good introduction and gives an on ramp to running other services, but I wouldn't say it is sufficient experience to call running hundreds of other services "trivial".

uberduper 6y ago

It really doesn't take much experience with kubernetes before it becomes fairly easy. And then using it for trivial things like this isn't a big deal. My home "server" runs plex, sab, mongo, unifi, and various other things in a single node k8s with a local zfs volume provisioner. The previous revision of this server I'd switched to using docker for everything and was annoyed with the upgrade process with just docker alone. With k8s, I just use :latest for most images and an upgrades happen every time I restart a pod or reboot the machine. I've been working with k8s since petSets and so this is all NBD to me. Linux had a learning curve too and ya'll got over it.

tenacious_tuna 6y ago
A friend and I have recently stood up homelabs for funsies. Mine's a cluster of x64 boxes, his is a small pile of raspi4s. He went for a k8s cluster off the bat--having worked with cloud stuff for a while, he had the familiarity to hit the ground running. I was immediately overcome by all the abstractions. I had no idea where to look to figure things out, and was mostly relying on advice from my friend. I dind't know what an ingress controller was, much less how to configure one--but I knew I wanted each 'service' to have its own IP I could route to from my network. Overall it felt like I had SO MUCH TO LEARN at *every point* it was difficult to get even close to my actual goals (CI/CD for a personal site + some hosted game servers). I eventually went with the philosophy of "build something now, and move towards perfection later" / "don't let best be the enemy of good" and begain spooling up LXCs and VMs to do the work I needed, planning to move things into k8s later when I better understood the actual things I wanted to move. (Plus then I got some satisfaction out of actually accomplishing the goals I wanted, instead of just banging my head on k8s documentation and learning all the abstractions.) As an example, I've not used docker in any meaningful capacity. For anything. No idea how to make a docker image. To k8s the CI for my docker site, I needed to know how to : 1. install the dependencies, which requires compiling a plugin for pandoc, which requires installing haskell and cabal. this is expensive, so I'd prefer to get the pre-res set up once... but that doesn't seem to be how docker works? Do I need an image repository? can I use DockerHub? I've seen HN talk about how docker is trying to monetize, should I run my own repository? Can I do that on my cluster? I'll need an ingress controller to route to it... I don't even know what that is. 2. I need some way to pass the built website files to the contianer I actually want to host them. I think that means I need an NFS share of somekind to store the files, so one coantainer can load them and another can read them. Do I hos tthat on my NAS? I could put an NFS share in the cluster, maybe. No idea how to get Docker to mount one, or k8s to host one. All the examples I seem to find deal more with connecting to remote services on a host than mounting local storage. is it even local storage? 3. Everyone says infrastructure as code is good, so I guess I'll follow this flux tutorial--only to find out the one I followed is out of date, and I should follow their NEW one. But they still assume I know way more baout k8s than I actually do, but still, I'll spend the few hours to get this operational, so then in theory everything else I deploy can be IaC'd, which is just good practice. At this point I'm so many layers of abstraction deep, I have no idea what I'm actually doing or how concepts relate to each other, and I'm no closer to actually having my goal. So last night I spent 2 hours spinning up VMs on my cluster and installing dependencies, configuring an nginx proxy, and now I actually have my personal blog self-hosted and updatable. Way more "progress" than the 10ish hours I've sunk into building a k8s cluster already. There's something to be said for limiting the number of abstractions you're dealing with.

tombert 6y ago

I don't really blog anymore but I do run a cluster in my basement for personal projects, so I totally get the appeal. I use Docker Swarm, not k8s, which I personally felt was way simpler to set up, but there's something kind of cool that the system I built is truly scalable to thousands of nodes if I really felt like it. Out of curiosity, what do most people here who have done such a thing (with k8s or docker swarm or otherwise) use for storage? I tried briefly to use Gluster-fs but had really bad problems with performance (most likely because I set it up wrong). Right now, I just have a ZFS Raid on one of the nodes, and have an NFS share on that.

denysvitali 6y ago
I either use Longhorn (https://longhorn.io/ https://longhorn.io/) or TopoLVM (https://github.com/topolvm/topolvm https://github.com/topolvm/topolvm), depending on what I need. Rook (https://rook.io/ https://rook.io/) is also cool.
tene 6y ago
I've had a great experience with https://rook.io/ https://rook.io/ managing Ceph for Kubernetes in production.

chmod775 6y ago

The author seems to think the only real alternatives are docker or whatever. So here's a way to host your personal blog if you don't want to over engineer it: 1. Have a git repo with your nginx/whatever config files. 2. Have a VPS running debian. 3. apt-get install nginx git ... git clone ... ln ... # create symbolic links to your nginx/whatever config in your git repo systemctl restart nginx ... 4. You're done. Create a cron job to automatically pull the latest changes from your git repo if you want. The above steps should take most people around 10 minutes. If you need to actually pivot into something that scales easier from there, I recommend following these steps/levels as your scale increases: 1. Create an automatic install script. 2. Use that script to create a .deb package instead of installing directly (optionally create a repository for this). 3. If you want to move to docker or what have you, it's trivial to install a debian package in a container. But let's be real, you never even will have to do any of the above because it's a personal blog, and it'll probably scale to the world population on a $5 VPS, especially if you slap cloudflare in front.

topkeks 6y ago
Seems insanely overcomplicated. firebase deploy --only hosting
andrewzah 6y ago
These days I see no point in installing services on a VPS directly, other than docker + docker-compose. You could do it in one image with the reverse proxy + static files, or break it out into two images (this is helpful if you run more services on the VPS). As for updates, caddy v1 used to support pulling in from git, but I don't think that got ported to v2. So what I do is build+push a docker image, and have a cron job on my vps to pull+restart my website's container. My preferences go Traefik > Caddy > Nginx, but traefik definitely has a bit of a learning curve. https://github.com/andrewzah/andrewzah-com-source/blob/master/Dockerfile https://github.com/andrewzah/andrewzah-com-source/blob/maste... https://github.com/andrewzah/andrewzah.com-docker/tree/master/services/http https://github.com/andrewzah/andrewzah.com-docker/tree/maste...
- oftenwrong 6y ago

koeng 6y ago

I also unironically use kubernetes (in particular, k3s) to run my personal blog. I run it on a colocated server I built myself, with the hope to have a couple colocated servers in the future I could network together. Right now, it has proxmox running a few VMs, 3 of which are for k3s, one for hhd backups, and one for a postgres server (which I am lazy, so am running with Dokku. Makes backups easy). The drives all run ZFS mirrors (2 2tb nvme, 2 8tb hdd) Honestly, it's a super comfy setup, and very little maintenance. The one thing I have is a master update script that does the docker image upgrading and rollouts. I make updates all the time and pretty much don't think about it. More energy activation than a PaaS like Dokku, but worth in long run, I think.

newman314 6y ago
Do you have a blog post showing how you set things up?

itsdex 6y ago

Next step: unironically creating a Kubernetes Operator for your personal blog https://github.com/dexhorthy/captains-log https://github.com/dexhorthy/captains-log

devilduck 6y ago

This is like bringing an actual gun to a water pistol fight. Way overkill and the type of thing only engineers care about

erichmond 6y ago

Another thing worth calling out, there's nothing wrong with over-engineering solutions in pursuit of knowledge. Especially when it's just a personal project.

crazypython 6y ago

A side note, how do you use Kubernetes to set up HTTPS on wildcard DNS? Is there a service for autoconfiguring the DNS and getting the certificate without a forward proxy?

qbasic_forever 6y ago
cert-manager supports the Lets Encrypt DNS challenge, which supports wildcard certs: https://cert-manager.io/docs/ https://cert-manager.io/docs/ It's a little bit of one time setup and then easy-peezy SSL automatically for all your services. Do note that you need a DNS provider with a good API to programmatically control it (the DNS challenge requires setting a special txt record that Lets Encrypt verifies). Read the docs to learn more, but most of the big ones (AWS Route 53, Cloudflare, etc.) should be fine.
kryptn 6y ago
> Is there a service for autoconfiguring the DNS There is! I use external-dns. [1] I haven't actually set up a Let's Encrypt wildcard cert, but I'm pretty certain cert-manager [2] supports it. I don't think you need a proxy if you use the DNS01 challenges. [1] https://github.com/kubernetes-sigs/external-dns/ https://github.com/kubernetes-sigs/external-dns/ [2] https://cert-manager.io/docs/ https://cert-manager.io/docs/

crazypython 6y ago

On the topic of "takes too long to learn" can someone recommend a good tutorial on basic Kubernetes concepts?

CSSer 6y ago
I think OP put this just as good or better than I could've in their blog post. > I think Kubernetes has fallen into the Vim/Haskell trap. People will try it for 10 minutes to an hour then get fed up. The point where you start to grok stuff just happens too late for most people to stick with it. Those become a vocal minority proclaiming it as too complex for humans to understand, and scare off people that haven’t tried it at all. I've had good luck with the tutorials and various help articles in the official docs[0]. [0]:https://kubernetes.io/docs/tutorials/kubernetes-basics/ https://kubernetes.io/docs/tutorials/kubernetes-basics/
qbasic_forever 6y ago
Kubernetes The Hard Way if you want to start from first principles and learn all about low level tools and setup. In practice you're probably going to use a cloud provider or tool to automate all of this stuff, but it's useful to understand the core infrastructure like TLS certs, etcd, etc. Otherwise the official docs are quite good and worth starting there. I'd be wary of buying books that are more than a year or two old--the k8s space moves fast and they regularly have major updates once or twice a year. Books and docs can get a bit stale or out of date with current best practices.
pas 6y ago
The trick is that to have a real chance of administering a k8s cluster you need Linux kernel knowledge (namespaces to understand docker, a bit of iptables, VXLAN or something similar to understand CNI/overlay network, how the container network namespace is connected via veth pipe, chroot/bind mounts, a very little bit of SElinux/apparmor/seccomp), systemd to see how it all starts, etcd to know about the consistency/consensus layer.. so far this is almost identical for all container orchestrator/scheduler platforms (nomad, docker swarm). Then there are the specific k8s concepts: pod (containers that share a network and PID namespace), service (a name, a TCP and/or UDP port number, and the name/selector of the target pod), ingress (HTTP routing, basically haproxy/nginx/envoy or other layer 7 stuff), LoadBalancer (the magic API that connects your cluster to the outside world, allocates IP addresses for Services/Ingresses). Then there are the background parts that take the YAML and convert it into actual running stuff: apiserver (this is the central info hub, kubectl and kubelets and controllers all talk to this basically), kubelet is the actual agent on each node, controllers have control loops and those actually calculate the necessary actions/operations to achieve what the YAML declared. Then there are the subsystems: storage (persistent volume claims and PVs), settings (secrets - see vault too - and configmaps, and various special/custom resources like a TLS certificate, these are stored in etcd, through the apiserver of course), CNI networking flavors and knobs, an endless list of kubelet and apiserver command line arguments, PCI passthrough for GPUs or HSMs or whatever. And of course there's RBAC, security, admission webhooks, see how nowadays Authz and Authn is done through a "lookaside" OpenId Connect service (some Ingress Controllers support these out of the box).

godot 6y ago

[Deleted]

oauea 6y ago
Sure, pay some other people to make your own worries go away. This post is clearly not for you.

oauea 6y ago

I am unironically using Kubernetes for my personal home server. Thanks to https://k3s.io/ https://k3s.io/ this is really easy to do, great fun and extremely useful. I have a git repo containing all my helm charts & docker files, testing & deploying changes is absolutely trivial now. And it's great to have everything version controlled. Previously I used to use Ansible, but you quickly run into issues which make you want containerization: Conflicting library/tool versions, packages that pollute too much of the system, port conflicts, hassle of keeping the playbook idempotent, etc. So while docker-compose would also do fine, having kubernetes to manage the ingress' routing system is rather practical. And the same goes for the other bits and bobs of infrastructure offers you if you're already using it. It's just very convenient. I've been doing this for a few years now, and am now up to 14 different apps running on my single home machine in Kubernetes, ranging from Home Assistant to PostgreSQL to Plex. Also it's just good experience. I also use Kubernetes for work, and this has made me noticeably more proficient.

pnathan 6y ago

I run GKE for some small apps. I also use AWS S3 hosting for my personal blog. The cost differences are... non trivial to the point of a bad joke, if we were comparing ability to reliably ship plaintext over the wire. But I'm not. I host a database and webapps on the k8s cluster, without adding extra EC2 nodes, RDS costs, or wrestling with AWS Lambda limitations. I can also confidently say that having something approximating a stable web app demands doing a lot of serious thinking, and "a single server running Apache on Digital Ocean" does not cover that case sufficiently. You need to tolerate failure, failover, load balancing, bin-packing, etc. I used to run a small autoscaling group on EC2 for my own systems; the dang thing would fail to come up on one node very frequently and so a number of the queries would fail. I eventually burnt it to the ground and redid it. I've never had that hassle in k8s. Its designed to succeed, in a way the "box of parts" approach doesn't. Boxes of parts are useful. For a complexity-sensitive & thoughtful infrastructure engineer, having something like the old Synapse/Nerve[1] system with your apps distributed across some 5-20 machines with a monitor lease to spawn new ones on failure would probably approximate Kubernetes for a few years, until you have to do something fancypants. You've still reimplemented part of Kubernetes, though... The other angle is, boxes of parts can go in wildly weird directions.... if you need it. Looking at some infrastructure these days professionally, the question is - when do we move to Kubernetes. It's not interesting or useful to the company to be maintaining our own thing or own strange path. The only questions are around the path - how much rework needs to happen and how much building in k8s needs to happen to get there. GKE is a very good starting point for k8s. Strong recommend. n.b. With respect to the cost. I consider this a professional investment / professional development expense. Spending $100-$200/month on a software engineer's salary is a reasonable return for being able to readily say I have experience in a current topic. Also I can run my own apps. :) https://github.com/airbnb/nerve https://github.com/airbnb/nerve

proxysna 6y ago

My 3 droplets run my Rust, Quake, gitea, registry, wireguard & pihole combo, traefik, grafana + prometheus with hashicorp nomad & consul. So yeah running an orchestrator for trivial tasks is actually fun, this is a good exercise and just in general a fun thing to do.

jvalencia 6y ago

I currently have a k3d cluster running locally with an entire product stack locally. It' nice to think that I can simply move that over to a full cluster at any time. There isn't even a change in tooling as the kubernetes context is the only change needed.

simo7 6y ago

Well I wish he expanded a bit more on the “Just do X, that’s so much simpler” section. He's on AWS, he could have gotten all those benefits with ElasticBeanstalk or ECS even. Plus no yaml files but actual IaaS (terraform etc.) and much better integrations with the other AWS services. I'd personally still use EC2 for a blog, but if you're looking for convenience/battery-included type of thing...yes I'd argue you're better off just doing X instead.

zug_zug 6y ago
Yeah... It's like software hipsterism, whatever is weird/hard is "cool," but the moment it become mainstream suddenly a new technology/language needs to made instead. 2 years from now everybody and their grandma will understand kubernetes or have a simple gui for it, and the hipsters will move on to something new and exclusive.

spion 6y ago

YAML is the biggest flaw of Kubernetes, so I'm quite exited that cdk8s is progressing very nicely https://github.com/cdk8s-team/cdk8s https://github.com/cdk8s-team/cdk8s There are other solutions that are potentially better (e.g. Dhall) but cdk8s seems to have momentum and sense for tackling the practical stuff (integrates easily with cdk, library with simplified constructs cdk8s-plus, import and convert existing stuff easily etc)

kube-system 6y ago
k8s also has a first-party typescript API.
bryanlarsen 6y ago
We use and like https://jkcfg.github.io/#/ https://jkcfg.github.io/#/
arcticfox 6y ago
I do all of my k8s work through Pulumi using Typescript, and it as an absolute joy. And it's not limited to k8s, which is great.

systemvoltage 6y ago

Can someone who is familiar with Kubernetes please explain in plain words why we use Kubernetes? I’ve read a lot about it but I always read some bullshit non-answer such as “It’s an orchestration platform for containers”. What does that even mean? Even this blogpost doesn’t explain what and why’s of Kubernetes. I have a docker container. I deploy them on vm. I use load balancer to split traffic. Could you please walk me through what problem Kubernetes would solve here?

pickledish 6y ago
After using k8s for about a year at work now, the way I understand it is that there’s a really large gap in abstraction between “a bare metal machine or VM with a shell prompt” and “a service that handles requests with some Python code”, and kubernetes helps fill that gap with some sane defaults and useful abstractions for people who run services. If you run your docker app on kubernetes, you get a lot of things for free with the platform (rolling no-downtime deployments, service discovery, auto scaling) that you’d have to set up manually if you were running your service on (say) EC2 instances instead. It can be a headache to learn sometimes, but ultimately saves a lot of effort if your use case fits!
__blockcipher__ 6y ago
Kubernetes does a lot of different things but one of the more important is resource management. You spin up a fleet of Kubernetes nodes which can have totally heterogenous (differing) amounts of compute/memory/disk, and have the scheduler make intelligent decisions about where to stick containers based off your (optionally specified) list of resource requirements. So it solves the bin-packing problem automatically rather than having to manually map out an efficient way to use your infra. It doesn’t reduce the complexity per say - the interactions can get complicated if you’re using stuff like node taints/tolerations instead of the more computationally simple “App X needs this much RAM but beyond that I don’t care where it lives” If I find a bunch of raspberry PIs in my basement and want to have them join my fleet, I just do that and even if my fleet varies from 256 core CPU boxes with huge raid arrays all the way down to tiny raspberry PIs, the scheduling just works. Note here the broader pattern of abstracting away the physical hardware; it’s a really important concept to grok.

unknown2374 6y ago

50 lines for configuration of a blog is not what I'd have personally called "simple". Granted GUIs (e.g. AWS console as mentioned in the blog) are not as easy as an config file, the comparison should be against other blog deployment programs, like hugo/jekyll/ghost, not.. the AWS console.

turtlebits 6y ago
That includes 2 load balanced instances of his blog + domain + automatic HTTPs

nanis 6y ago

> I’m not exposed to that complexity Some of us think editing YAML files is complex enough.

revskill 6y ago

What clicks me about Kubernetes is not about orchestration or zero downtime or scaling,... I can just use Kubernetes on 1 node for all of my services. What matters is the declarative approach of Kubernetes. It's like the first time I learnt about React: declarative renderering/deployment based on state !

heavyset_go 6y ago
Here's the thing that gets me: Docker Compose and Docker Swarm can offer declarative infrastructure and only take up under 50MB of resident memory. I need 2GB minimum for a K8s master node, 2GB minimum for monitoring, and then 700MB minimum for each worker node[1]. I just can't justify running K8s for a blog when there are less expensive and complex offerings out there. [1] https://docs.kublr.com/installation/hardware-recommendation/ https://docs.kublr.com/installation/hardware-recommendation/
- turtlebits 6y ago
  Try K3s. I run it on 3x of OVH's cheapest VPS ($3.50/mo) and works great. I have Postgres, Hasura, Keycloak, Riemann, Riemann dashboard and Digdag running- the cluster has been up over a year with no issues.
  - heavyset_go 6y ago
    Can I use multiple machines as nodes for K3s? I think the last time I looked at it, clusters were limited to one machine, but that was a long time ago.
    - revskill 6y ago
      Yes, you can easily join multi master nodes or worker nodes with k3s.
      - heavyset_go 6y ago
        Awesome, thanks.

platty 6y ago

I just connect Netlify to my repo, and boom blog is deployed.

anyfoo 6y ago

Let's see what happens in a few years. There are a bunch of blogs/content out there that have been abandoned for a long while, but you can still read the information that was put there in the past. The simpler the system serving the pages is (down to static HTML files), the more likely it is that the website stays up functional. A blog that is built on Kubernetes sounds like it will be dropped pretty quickly once interesting in writing for it wanes.

astrange 6y ago
Static HTML pages might be the only safe way to keep a blog running; my experience running Wordpress is if you leave it up long enough someone will drive by, hack it, and Dreamhost will block your site.

kklemon 6y ago

Am I the only one still using Docker Compose?

wiredfool 6y ago
No, But I'm looking at what to do to streamline deploys and provisioning across 10's of vms, each running docker-compose. For one or two services on a machine, it's pretty good. Past that... not so much.
- nikisweeting 6y ago
  I run docker-compose inside supervisord. Surprisingly it works great, they play well together and I highly recommend it for managing multiple compose projects per machine. [program:some-docker-project] command=docker-compose up directory=/opt/some-docker-project ... [program:some-other-docker-project] command=docker-compose up directory=/opt/some-other-docker-project ... This style of setup has served many millions of visitors across dozens of projects in production at our company over the last ~3 years. It's like a poor-mans kubernetes, it handles autorestarting failed projects, we handle ingress with cloudflare argo tunnels in docker, and deployments are easy (just git pull && supervisorctl restart all).
  - wiredfool 6y ago
    I'm more at the level of one Docker-compose per VM, and I've got it loading with some os too. Service or similar. But there are at least a few problems with that. 1) Want Ci where push to github builds the image and deploys. 2) Binpacking time. We're at the point where that's an issue 3) Host machines -- There's too much on them and we need to standardize and streamline.

callamdelaney 6y ago

Okay but when Kubernetes breaks it's usually none-trivial to understand why. I've spent more time than I'd like diagnosing Kubernetes issues with networking and orchestration which could have been spent on building features.

bostonvaulter2 6y ago

> For example, the entire deployment configuration for this blog is contained in this yml file Isn't that not quite correct? If I'm reading this correct (which I may very well not be), isn't this a reference to a Dockerfile: image: marcusbuffett/blog:latest Which then might have lots of other complexity contained within. I wouldn't call that self-contained, I think that's overselling it.

robertlagrant 6y ago

> I think Kubernetes has fallen into the Vim/Haskell trap There was a period where Git was at this point as well, but it seems as though everyone's got over it and decided the learning curve is always worth it.

hardwaresofton 6y ago

Been running my blog on Kubernetes since 2017[0] and it's been great. Kubernetes is a force multiplier and the natural evolution in deployment methodology. It could be tighter/cleaner but it's extremely good for what it provides, but you have to know why you might need the things it's providing for it to seem worth it. If you'd like to try kubernetes and get to understand it and feel comfortable with it I'd recommend: - Work through kubernetes the hard way[1] (ignore/replace the GCP-specific things, if you don't know enough about what the non-GCP corrolary to a GCP-thing is, that's a bit of knowledge you need to fill) - Set up your own cluster from scratch (no kubeadm, no alternate distros, etc), use the simplest options you can find at first (ex. flannel for CNI) - Set up ingress (NGINX ingress is a good place to start) - Run some simple unsecured but diverse workloads (static sites on NGINX, Wordpress, etc), figure out why you might pick a StatefulSet versus DaemonSet. At this point, use the hostPath/local volumes just to avoid trying to grok volume complexity. - Install a useful cluster tool ("addon") like cert-manager[2] from scratch so you can see Kubernetes manage something you'd normally solve with systemd timers and `certbot` (or your reverse proxy would do for you if you're running caddy or traefik) - Start putting your YAML in source control get familiar with either kustomize, helm, or both (you could also just use Make + envsubst like I did for a while[3]). It's at this point that it should click that all you need to get back to a certain state of your cluster is to get a machine, do basic hardening (ufw, etc), install kubernetes, and run "make" in this repo (excluding things like DNS entries, etc). Now things are probably getting fun, because you can have the distant cousin of immutable infrastructure; repeatable infrastructure. - Tear down your cluster, set it up again with kubeadm (note that kubeadm actually has a file-driven configuration option[3]), run your yaml from source control and confirm that all the workloads you had in place are back up and secured. - (optional) Tear down your cluster, try rebuilding it with k3s[4] or k0s[5] - Start looking around and seeing what your options are and the ecosystem that exists -- digging deeper into the interfaces that make Kubernetes tick, for example volume management (Container Storage Interface) by deploying Rook[6] or OpenEBS[7]. Note that one of the best things that can happen to you during this process is something going wrong. Every time something goes wrong, and you go back to fundamentals to fix it. Being able to reason about kubernetes ad hoc (ex. if DNS is correct but you can't reach port 80/443, what should you check first?) is the key to feeling comfortable with it, and while failures and downtime will happen, you shouldn't feel just complete despair/confusion. Normally, once things are stable kubernetes is quite worry-free. I've made a guide like this before, I'll see if I can find it. [EDIT] Found only one post[8] [0]: https://vadosware.io/post/fresh-dedicated-server-to-single-node-kubernetes-cluster-on-coreos/ https://vadosware.io/post/fresh-dedicated-server-to-single-n... [1]: https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way [2]: https://github.com/jetstack/cert-manager https://github.com/jetstack/cert-manager [3]: https://www.vadosware.io/post/using-makefiles-and-envsubst-as-an-alternative-to-helm-and-ksonnet/ https://www.vadosware.io/post/using-makefiles-and-envsubst-a... [4]: https://k3s.io/ https://k3s.io/ [5]: https://docs.k0sproject.io https://docs.k0sproject.io [6]: https://rook.io/docs https://rook.io/docs [7]: https://docs.openebs.io/ https://docs.openebs.io/ [8]: https://news.ycombinator.com/item?id=22117684 https://news.ycombinator.com/item?id=22117684

flemhans 6y ago

Often, when people discuss how easy or hard Kubernetes is, they mean running it on a managed platform like GKE. Running it on your own metal is supposedly what requires that team of engineers.

verdverm 6y ago
Setting up and running Kubernetes binaries across many VMs is not hard. The hard part is managing all of the addons, extras, and post setup tasks (roles, quotas, etc) to make (and keep) it production worthy. Operational experience is still rare

lmm 6y ago

The problem I have with Kubernetes is the Docker part. I get why people who are running Python need to use Docker for all their deployments (because anything is better than Python package management), but I've got well-behaved programs in a language that has decent package management, so I just want to run a bunch of them on different hosts. Is there an alternative that lets me orchestrate deploying a bunch of processes on a bunch of servers without having to interpose Docker in between? Anyone using Nomad?

ed_balls 6y ago

I had bad experience with k8s. The learning curve is step. A few days ago k8s decided to run a cron job just after the deploy (it was scheduled for Sunday). Why? it could be a bug (https://github.com/kubernetes/kubernetes/issues/63371 https://github.com/kubernetes/kubernetes/issues/63371) I'm not sure how to even debug this. > A cron job creates a job object about once per execution time of its schedule. We say "about" because there are certain circumstances where two jobs might be created, or no job might be created. We attempt to make these rare, but do not completely prevent them. Therefore, jobs should be idempotent Is there a way to make a cron job idempotent?

dimitar 6y ago

The real competition in terms of price and convenience is a VPS. So to approximate this with Kubernetes you have to avoid the big cloud providers and things like kubeadm as that would mean at least two cores and the resulting price tag (even if you use things like burstable). Using k3s is a nice option. Must avoid load balancer and often static IPs. I wonder why there aren't vendors who are selling k8s namespaces directly with a quota (reinvented shared hosting in k8s).