14 ms·

Kubernetes is... fun. It's fun by making things that used to require hundreds of hours to figure out, only require dozens. This can give superpowers to the righ

Kubernetes is... fun. It's fun by making things that used to require hundreds of hours to figure out, only require dozens. This can give superpowers to the right kind of engineer. I spent a bunch of time hand-rolling my own cluster with it's own Wireguard network, on personal hardware, with one node in the cloud because I thought I needed one. What's neat about it is, you generally solve one problem, and once it's solved, it stays solved. Your solution is in a yaml file somewhere or in a command line option that you've persisted to a script or whatever. And everything accumulates! If you manage to get a storage cluster installed, boom, all of a sudden you have a ton of options available to you. Once the cluster is stable and you are comfortable bringing it up and tearing it down so that everything's working right, you can start pulling parts of the k8s infrastructure in, need a DNS server? Run it on your cluster! K8s wants a docker container registry, you really don't want to run that on your cluster in the beginning, but once your cluster's secure, why not! K8s starts eating everything in your life like the effing Borg and it's great! If you're the kinda cat that will spend 80 hours on a Factorio world, then dive into the mods and make the tweaks you really wish the authors would make, and you have enough admin experience to do general troubleshooting of complex systems, I can't recommend Kubernetes enough for the sheer fun factor. The downside if you want to do anything serious with it is, actually the same as running your own infrastructure always is and was, network connectivity. Hardware is cheap. Last mile network connectivity, isn't. The solution to that, is, of course, colocation.

emodendroket 6y ago
You can always run it in the cloud and at least theoretically be able to switch providers and take your cluster with you.
- vinceguidry 6y ago
  Expensive. I've got all these machines sitting here at home, why not use those?
  - camjohnson26 6y ago
    Hard to beat the uptime of a cloud provider with on prem, if you literally mean “at home” even harder, my Comcast goes out a few times a day. Even in offices you don’t benefit from the economies of scale and redundancies in the power and network systems that cloud providers have.
    - vinceguidry 6y ago
      If you're really trying to serve the outside world, then you want colocation. Cloud providers are way more expensive.
      - emodendroket 6y ago
        Depends on what you’re doing and at what scale. Especially for smaller orgs it’s really hard to duplicate the level of service the big boys can provide yourself.
        vinceguidry 6y ago
        It's certainly not for everybody.
        vinceguidry 6y ago
        I just thought of another reason. Rolling your own cluster forces you to understand what Kubernetes is at a basic level. If you never get that experience, and just roll with cloud services, you aren't really getting that and will tend to get mystified by errors which, you'd have a much better idea of where they're at if you've rolled your own. Kubernetes is very complicated, but the complexity can be managed if you approach it from the bottom up. Using a cloud service robs you of that ability.
        emodendroket 6y ago
        The pitch for Kubernetes is kind of supposed to be “infrastructure as a service,” isn’t it?
majormajor 6y ago
> What's neat about it is, you generally solve one problem, and once it's solved, it stays solved. Your solution is in a yaml file somewhere or in a command line option that you've persisted to a script or whatever. And everything accumulates! If you manage to get a storage cluster installed, boom, all of a sudden you have a ton of options available to you. Can you expand on how this is different from traditional networking? I used to run a FreeBSD router with a ZFS storage setup and various other stuff on my home network, and it wasn't like I had to keep tinkering with it. Once I had network storage, I could use it from lots of different things, etc. But eventually you need software patches, updates, etc, and that was where the ongoing pain would sometimes crop up. Is that so different here? From some coworkers who are closer to the cluster, keeping up with K8s version changes doesn't seem like a small effort.
- oauea 6y ago
  Now smash that machine to bits with a hammer. How long do you need to get the exact same system back up?
  - majormajor 6y ago
    Let's assume I have backups for both my kube configs and my standalone machine... and spare parts for the cluster(s). I'm trying to understand better the maintenance story, because my impression is that kube itself has spec changes, and I have to stay on top of all the dockerfiles I'm using to keep up with patches there?
    - oauea 6y ago
      People generally keep their kubernetes spec files in Git. Be it either the kubernetes .yaml files they apply directly, or helm charts. These files can idempotently be re-applied to a cluster to restore, or upgrade, the running applications. When you lose your cluster and you restore to a new cluster, you will of course initially deploy the same version of kubernetes as you were using previously. Then once you choose to upgrade to a new version of kubernetes, it can happen that some of the APIs you were using have now been deprecated/removed. This is a very slow process, so you had plenty of time to upgrade ahead of time before being forced to. But let's say you ignore all that, and have chosen to upgrade to a new kubernetes major version and are now forced to upgrade all your yamls. This happens rarely, and recently only because a few BETA APIs have become STABLE and people are now expected to use the STABLE versions. So you go ahead and make those few changes, re-run your deploy and you're done.
smoe 6y ago
I think one big big aspect, in “use the right tool for the job”, that is often overlooked, is how good you already are wielding said tool. I use Terraform to manage the few personal cloud resources I have. It is overkill, but I already learned it on the job so it ended up being quicker than setting stuff up by hand. If not, I wouldn’t have bothered learning it. At least in this context. Same when building an MVP, I’m not going to pick some shiny tools I have never used before, but tools I know I can get the job done with. Sure, I do like learning new things and tinker around with them, but it is not always the right time to do so, and the list of things I’d like to get good at, engineering related or not already is long a enough for a lifetime so one needs to prioritize.
- andrewstuart2 6y ago
  Not only how well do you use the tool, but how useful is it to learn the tool? Once you can use it well, what else can you do with it? I recently saw a video via twitter that mocked using kubernetes for your blog by showing someone making a sandwich with woodworking tools. Overkill, yeah. But if you stretch the metaphor to the way it translates in k8s, if you get good at making a sandwich in 5 minutes with the tools, you're fairly close to also stamping out dressers and end tables in 5 minutes. At the very least you can make them, which you can't say of a bread knife. And once you do it a few times you can do it with incredible speed.
turtlebits 6y ago
I really like K3s. Comes with an ingress controller out of the box which makes it easier to get up and running. Add Let's Encrypt, and its pretty trivial to stand up new services on their own subdomains.
- helloguillecl 6y ago
  I'm struggling to easily configure new subdomains that easily using GKE. The managed certificates system in GCloud is straight user-friendly hostile. I'll have a look at that.
- lemarchr 6y ago
  Also worth noting that it's a lot less resource hungry than the official Kubernetes distro. Kubernetes documentation recommends 2 CPUs and 2GB of RAM per node just to have enough left over to actually run any workloads. k3s is specifically designed for resource constrained environments (e.g. Edge, IoT, CI/CD) and can get away with as little as 0.05 CPU and 256M RAM (on worker nodes). I don't recommend this as a way to save money on infrastructure but it does open the door to further use cases.
gravypod 6y ago
> like the effing Borg and it's great! Don't know if that was an intended nod but I liked it :) https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43438.pdf https://storage.googleapis.com/pub-tools-public-publication-...
johnsoft 6y ago
>Once the cluster is stable and you are comfortable bringing it up and tearing it down >K8s wants a docker container registry, you really don't want to run that on your cluster in the beginning, but once your cluster's secure, why not! I'm betraying my ignorance here, but how does this work? If you're running the registry in your cluster, and you tear down your cluster (and the registry with it), how do you rebuild the cluster without being able to pull images?
- hardwaresofton 6y ago
  Usually the important images are hosted somewhere else (ex. you're probably not hosting the controller-manager which is normally pulled from k8s.gcr.io), but for the ones that were hosted in your cluster, the pods will basically fail (and start to back off) until your local registry comes up and you'll see pull related errors. The rest of the pods that don't reference images from your cluster-local registry will start up just fine. Assuming that you rebuild your cluster and restore your registry from backup (or simply reconnect it to some object store that it was connected to before, like S3), your registry pod (in a Deployment/Stateful/DaemonSet) will come up, and then the cluster will be able to connect to it, and the pods that were failing to start will start succeeding and you're back where you started.

emodendroket 6y ago

You can always run it in the cloud and at least theoretically be able to switch providers and take your cluster with you.

vinceguidry 6y ago
Expensive. I've got all these machines sitting here at home, why not use those?
- camjohnson26 6y ago
  Hard to beat the uptime of a cloud provider with on prem, if you literally mean “at home” even harder, my Comcast goes out a few times a day. Even in offices you don’t benefit from the economies of scale and redundancies in the power and network systems that cloud providers have.
  - vinceguidry 6y ago
    If you're really trying to serve the outside world, then you want colocation. Cloud providers are way more expensive.
    - emodendroket 6y ago
      Depends on what you’re doing and at what scale. Especially for smaller orgs it’s really hard to duplicate the level of service the big boys can provide yourself.
      - vinceguidry 6y ago
        It's certainly not for everybody.
      - vinceguidry 6y ago
        I just thought of another reason. Rolling your own cluster forces you to understand what Kubernetes is at a basic level. If you never get that experience, and just roll with cloud services, you aren't really getting that and will tend to get mystified by errors which, you'd have a much better idea of where they're at if you've rolled your own. Kubernetes is very complicated, but the complexity can be managed if you approach it from the bottom up. Using a cloud service robs you of that ability.
        emodendroket 6y ago
        The pitch for Kubernetes is kind of supposed to be “infrastructure as a service,” isn’t it?

majormajor 6y ago

> What's neat about it is, you generally solve one problem, and once it's solved, it stays solved. Your solution is in a yaml file somewhere or in a command line option that you've persisted to a script or whatever. And everything accumulates! If you manage to get a storage cluster installed, boom, all of a sudden you have a ton of options available to you. Can you expand on how this is different from traditional networking? I used to run a FreeBSD router with a ZFS storage setup and various other stuff on my home network, and it wasn't like I had to keep tinkering with it. Once I had network storage, I could use it from lots of different things, etc. But eventually you need software patches, updates, etc, and that was where the ongoing pain would sometimes crop up. Is that so different here? From some coworkers who are closer to the cluster, keeping up with K8s version changes doesn't seem like a small effort.

oauea 6y ago
Now smash that machine to bits with a hammer. How long do you need to get the exact same system back up?
- majormajor 6y ago
  Let's assume I have backups for both my kube configs and my standalone machine... and spare parts for the cluster(s). I'm trying to understand better the maintenance story, because my impression is that kube itself has spec changes, and I have to stay on top of all the dockerfiles I'm using to keep up with patches there?
  - oauea 6y ago
    People generally keep their kubernetes spec files in Git. Be it either the kubernetes .yaml files they apply directly, or helm charts. These files can idempotently be re-applied to a cluster to restore, or upgrade, the running applications. When you lose your cluster and you restore to a new cluster, you will of course initially deploy the same version of kubernetes as you were using previously. Then once you choose to upgrade to a new version of kubernetes, it can happen that some of the APIs you were using have now been deprecated/removed. This is a very slow process, so you had plenty of time to upgrade ahead of time before being forced to. But let's say you ignore all that, and have chosen to upgrade to a new kubernetes major version and are now forced to upgrade all your yamls. This happens rarely, and recently only because a few BETA APIs have become STABLE and people are now expected to use the STABLE versions. So you go ahead and make those few changes, re-run your deploy and you're done.

smoe 6y ago

I think one big big aspect, in “use the right tool for the job”, that is often overlooked, is how good you already are wielding said tool. I use Terraform to manage the few personal cloud resources I have. It is overkill, but I already learned it on the job so it ended up being quicker than setting stuff up by hand. If not, I wouldn’t have bothered learning it. At least in this context. Same when building an MVP, I’m not going to pick some shiny tools I have never used before, but tools I know I can get the job done with. Sure, I do like learning new things and tinker around with them, but it is not always the right time to do so, and the list of things I’d like to get good at, engineering related or not already is long a enough for a lifetime so one needs to prioritize.

andrewstuart2 6y ago
Not only how well do you use the tool, but how useful is it to learn the tool? Once you can use it well, what else can you do with it? I recently saw a video via twitter that mocked using kubernetes for your blog by showing someone making a sandwich with woodworking tools. Overkill, yeah. But if you stretch the metaphor to the way it translates in k8s, if you get good at making a sandwich in 5 minutes with the tools, you're fairly close to also stamping out dressers and end tables in 5 minutes. At the very least you can make them, which you can't say of a bread knife. And once you do it a few times you can do it with incredible speed.

turtlebits 6y ago

I really like K3s. Comes with an ingress controller out of the box which makes it easier to get up and running. Add Let's Encrypt, and its pretty trivial to stand up new services on their own subdomains.

helloguillecl 6y ago
I'm struggling to easily configure new subdomains that easily using GKE. The managed certificates system in GCloud is straight user-friendly hostile. I'll have a look at that.
lemarchr 6y ago
Also worth noting that it's a lot less resource hungry than the official Kubernetes distro. Kubernetes documentation recommends 2 CPUs and 2GB of RAM per node just to have enough left over to actually run any workloads. k3s is specifically designed for resource constrained environments (e.g. Edge, IoT, CI/CD) and can get away with as little as 0.05 CPU and 256M RAM (on worker nodes). I don't recommend this as a way to save money on infrastructure but it does open the door to further use cases.

gravypod 6y ago

> like the effing Borg and it's great! Don't know if that was an intended nod but I liked it :) https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43438.pdf https://storage.googleapis.com/pub-tools-public-publication-...

johnsoft 6y ago

>Once the cluster is stable and you are comfortable bringing it up and tearing it down >K8s wants a docker container registry, you really don't want to run that on your cluster in the beginning, but once your cluster's secure, why not! I'm betraying my ignorance here, but how does this work? If you're running the registry in your cluster, and you tear down your cluster (and the registry with it), how do you rebuild the cluster without being able to pull images?

hardwaresofton 6y ago
Usually the important images are hosted somewhere else (ex. you're probably not hosting the controller-manager which is normally pulled from k8s.gcr.io), but for the ones that were hosted in your cluster, the pods will basically fail (and start to back off) until your local registry comes up and you'll see pull related errors. The rest of the pods that don't reference images from your cluster-local registry will start up just fine. Assuming that you rebuild your cluster and restore your registry from backup (or simply reconnect it to some object store that it was connected to before, like S3), your registry pod (in a Deployment/Stateful/DaemonSet) will come up, and then the cluster will be able to connect to it, and the pods that were failing to start will start succeeding and you're back where you started.