6 ms·

I enjoy reading about homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. This is just anoth

I enjoy reading about homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. This is just another "how I installed kubernetes" thing only without any real scale behind it. There are other ways to deploy and run things, and it takes people toying around with alternatives for something unique to spring up.

Havoc 5y ago
> homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. I've had great success with Ansible+Proxmox. The proxmox CLI interface is good enough that you can basically do DIY terraform like setup straight against CLI without any agents / plugins etc. (I know declarative vs imperative but hopefully the comparison makes sense) Lately I've been toying with local gitlab-CI building and deploying images directly into the environment so that any changes to the code redeploy it (to prod...this is home after all haha) Considered terraform too but not a big fan of their mystery 3rd party plugin provider system
oceanplexian 5y ago
Yeah I’m with you. I run a rack at home with servers running a few applications orchestrated with a bit of Puppet, it’s 100x more reliable than K8s, I’m not having to do weird hacks to make persistent storage work, edit dozens of YAML files and run a whole host of dodgy control plane services. Sadly, I leaned this the hard way by setting up a 4-node k8s cluster and realizing bare-metal support is abysmal and the maintainers simply don’t care about you if you’re not running a cluster on AWS or GCP. Unfortunately my day job still involves k8s and people give me strange looks when I suggest metal so substantially less complex, cheaper, more performant, and easier to orchestrate and maintain.
- movedx 5y ago
  I'm building my own education platform around getting people into IT and towards a career in Cloud and DevOps. A few months back I built a Nomad based cluster with Consul and was like: wow! This is super cool... but it's also expensive and complex. I'm going to be building out the platform with an ALB, two EC2 Instances in an AutoScaling Group based on an AMI I bake with Ansible and Packer. I'll use Terraform for the entire infrastructure. The release pipeline for content updates (it's an MkDocs/Material based book with videos blended in) will be me updating the AMI and firing off a `terraform apply`. I won't be using Docker. I won't require or need an orchestration engine. It's a static website delivered via a custom web server written in Go (as I need custom "Have you paid to view this content?" logic.) That's about as complex as it gets for now. I'm actually looking forward to implementing it and watching as something so simple hopefully goes on to allow me to build and scale a business.
- throwaway894345 5y ago
  Kubernetes is just young. The story for bare metal is improving all the time. You just have to pick the right components for things like storage and load balancing, which is tedious but no more so than picking the right combination of tools to use to build your own Kubernetes replacement. In time even this pain will go away once someone builds a decent bare metal distro with all of this stuff available out of the box (k3s is trending in that direction).
Axsuul 5y ago
I run mine using HashiStack (Nomad, Vault, Consul), highly recommend going that route over Kubernetes which I have also used.
- movedx 5y ago
  I've come from bare metal, to virtualisation, to Cloud. I then went from AutoScaling Groups and AMIs to K8s and Docker Images. Recently I went from K8s to Nomad+Consul and Docker Images. Now I'm building a small platform to host a business I've started and I'm going back to ASGs with simple EC2 Instances. There's really no need for the features Nomad/K8s offer at almost any scale outside of a few big companies.
  - MonaroVXR 5y ago
    But how do you provision the baremetal?
  - Axsuul 5y ago
    It depends. Orchestrators like Nomad can give you a lot of leverage as a small team. It's not very difficult to manage and can give you lots of redundancy, control, and a path to scalability while utilizing best practices so you don't have to think about it. There are lots of ways to do this and not every app needs this but as you begin to get traction, you either have a choice of outsourcing your devops or bringing it in-house which happens way before you are even a big company. Nomad solves the problem of the latter. Also, I'm curious why you didn't just end up sticking with Nomad now that you know it.
    - movedx 5y ago
      Orchestration engines can give you a lot of leverage if you run a managed solution because all you're doing there is deciding on the size you need and providing a credit card number. This is absolutely a great way of your DevOps to the next level with little effort. If you're going DIY, however, then you're not doing yourself any favours by starting out with an orchestration engine. In the case of Nomad you'll also need Consul for it to be truly useful. That means six EC2 Instances (for example) for a truly production trade system - 3x for Nomad, 3x for Consul. Of course you also need mTLS as well - may as well throw Vault into the mix too, then... that's another three EC2 Instances to support that cluster... 9x EC2 Instances into the DIY solution and all you have is an orchestration engine and you've not even started on the workers yet. That cost so far is USD$30,416.04 for the reference architecture for all nine nodes: https://learn.hashicorp.com/tutorials/consul/reference-architecture https://learn.hashicorp.com/tutorials/consul/reference-archi... On the other hand if you keep it simple and just work with AMIs you bake your self when you need to release something new, you need two EC2 Instances. The cost of the compute resources for this for 12 months: UDSD$330. There's no raft consensus which needs three nodes to be stable (but a single AZ failure can result in split brain, so really you need five AZs), no new technologies to understand, no new vendor to work with or pay capital to, nothing more to support. These things - these orchestration engines - have their place, and that place is outside of 99.9% of businesses (large or small.) As for moving away from Nomad: too expensive and too complex. I ran Consul, Nomad and Vault on the same master instances (which is a risk and against best practice) without mTLS (just TLS) and it took a week to get it all running. None of that includes monitoring, backups, auto-scaling, etc. I believe the work involved with going DIY on the orchestration front is not worth the benefits in the short to medium term. In the long term you'll likely outgrow your own simpler solution and at which point you'll likely have the capital to switch to a managed service anyway /shrug It's all relative, after all.
throwaway894345 5y ago
Personally I don’t use Kubernetes for the scalability, but because I haven’t seen anything else remotely maintainable (and yes, Kubernetes still has lots of room for improvement). I don’t want to build a good chunk of Kubernetes myself just to host my software. It’s nice to be able to Google for solutions to my problems or to ask on various forums/chat rooms (or in many cases, not have the problems in the first place because someone else committed the fixes to Kubernetes long ago).
nullwarp 5y ago
I've used everything under the sun and for homelab use IMO nothing beats using docker-compose and salt. 1. Unless you have crazy dynamic scaling needs, there's literally no point in using k8s or nomad 2. Unless you want RBAC and fine tuned ACL, theres literally no point in using k8s or Nomad I hate this idea that you need a Ceph cluster and a 3 node HA cluster to have a home lab. Not saying don't do it because you want to play/test/learn any of those platforms, just that if you are going to run a server service maybe consider if you are doing it for learning a task orchestration service or just want to run XYZ reliably.
- sgarland 5y ago
  I was settled on Docker-Compose and Ansible for a long time, and while it was indeed quite easy to use once everything was tweaked correctly, what finally got me to move to k8s was HA. Having a single node with nginx, DNS resolution and Unifi controller running on it meant if I ever took the node down down for something, everything broke. Plex being down because I wanted to play with something unrelated wasn't ideal. Agree that Ceph is ridiculously overkill though, I tried. Longhorn is much easier and perfectly fine for distributed storage.
  - Macha 5y ago
    Unifi controller isn't a dependency for your network to continue working. Config is distributed to the devices themselves, the controller is just a UI and collector of data.
- moontear 5y ago
  I also run everything with docker stack(s) currently. Not using salt and haven’t heard of it. Why would I want salt and not ansible or something similar?
  - mafro 5y ago
    Salt (https://saltproject.io/ https://saltproject.io/) was an almost-winner around the time that Ansible blew up. It's pretty irrelevant nowadays since terraform and kubernetes - but still it's a very very good tool. I'd suggest if you already know Ansible then don't change. If you have no idea which is which then try them both and let us know :)
discardedrefuse 5y ago
I'm currently building out my single-node "homelab" with coreos + podman + systemd. Most of the config is pulled from 1 ignition file during coreos installation. Its been a learning experience and but also very lightweight.
marginalia_nu 5y ago
My "homelab" is just debian stable, with a moratorium against containerization. Systemd keeps the services going. Updates are manual. It's a bit incredible how clean and easy such a set-up can be. Like there's almost nothing to say. It's extremely reliable and just keeps ticking. I actually did use to have a bunch of virtualization and kubernetes and crap, but got rid of it because it ate literally half the systems' resources, in a death by a thousand containers-type of way. There was also a lot of just jank, stuff was constantly breaking and I was always trying to piece together why from half a dozen out-of-date tutorials. Felt like herding cats.
- sgarland 5y ago
  I've transitioned from a single server running services on baremetal Debian, to running services with Docker on Debian, to running services with Docker on Debian VMs under Proxmox (aka Debian with a repo), to running services on a three-node Kubernetes cluster using k3os (I was using Talos, but it doesn't have Longhorn support yet, and Rook/Ceph was a nightmare). The baremetal server was the easiest to set up, obviously, but the hardest to maintain. Where did I put that random config? Wait, why did this break when I upgraded X? Switching to Docker (and later Compose) wasn't that difficult, and made maintaining services much easier. Going to k8s has been challenging, mostly because of how I'm doing it - a controlplane and worker node on each physical node in Proxmox. Writing the YAML is, well YAML. I'm mostly looking forward to not having to keep my VMs updated. I automated a lot of that (VMs are Packer templates) with Ansible, but it's still annoying. Upgrading k8s nodes is a lot easier, IMO.
- ryandrake 5y ago
  Yea I think I missed the whole web-complexity bus and am probably out of touch, but I still don’t get the use case for docker and containers and kubernets and and orchestration and all that stuff, just for a simple home setup. I serve a tiny web site, email, backups, a NAS and a few other internet services for my family, and my “stack” is vanilla Debian Stable. Maybe I don’t know what I don’t know, but my setup works for me and I don’t really have any problems maintaining it so I figure why add all the complexity? It always feels weird to see threads and threads of people talking about dozens of software programs I’ve never even heard of, let alone used. Maybe I’m living in the past but to me a “stack” is: OS, server, database, application. Like LAMP. Wonder when this changed! It makes me curious about what kinds of stuff people do in their home networks that I never even considered doing.
Macha 5y ago
My setup I just have ansible deploy to a couple of hosts running Arch. Generally I just use OS packages and have ansible deploy appropriate configs then for some software where that's more hassle I use podman rootless containers. Such software includes e.g. Graylog, which wants particular versions of Elasticsearch and mongo, or postgres which requires a manual data migration for annual version upgrades k8s and nomad are on my "maybe eventually later" list though.

Havoc 5y ago

> homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. I've had great success with Ansible+Proxmox. The proxmox CLI interface is good enough that you can basically do DIY terraform like setup straight against CLI without any agents / plugins etc. (I know declarative vs imperative but hopefully the comparison makes sense) Lately I've been toying with local gitlab-CI building and deploying images directly into the environment so that any changes to the code redeploy it (to prod...this is home after all haha) Considered terraform too but not a big fan of their mystery 3rd party plugin provider system

oceanplexian 5y ago

Yeah I’m with you. I run a rack at home with servers running a few applications orchestrated with a bit of Puppet, it’s 100x more reliable than K8s, I’m not having to do weird hacks to make persistent storage work, edit dozens of YAML files and run a whole host of dodgy control plane services. Sadly, I leaned this the hard way by setting up a 4-node k8s cluster and realizing bare-metal support is abysmal and the maintainers simply don’t care about you if you’re not running a cluster on AWS or GCP. Unfortunately my day job still involves k8s and people give me strange looks when I suggest metal so substantially less complex, cheaper, more performant, and easier to orchestrate and maintain.

movedx 5y ago
I'm building my own education platform around getting people into IT and towards a career in Cloud and DevOps. A few months back I built a Nomad based cluster with Consul and was like: wow! This is super cool... but it's also expensive and complex. I'm going to be building out the platform with an ALB, two EC2 Instances in an AutoScaling Group based on an AMI I bake with Ansible and Packer. I'll use Terraform for the entire infrastructure. The release pipeline for content updates (it's an MkDocs/Material based book with videos blended in) will be me updating the AMI and firing off a `terraform apply`. I won't be using Docker. I won't require or need an orchestration engine. It's a static website delivered via a custom web server written in Go (as I need custom "Have you paid to view this content?" logic.) That's about as complex as it gets for now. I'm actually looking forward to implementing it and watching as something so simple hopefully goes on to allow me to build and scale a business.
throwaway894345 5y ago
Kubernetes is just young. The story for bare metal is improving all the time. You just have to pick the right components for things like storage and load balancing, which is tedious but no more so than picking the right combination of tools to use to build your own Kubernetes replacement. In time even this pain will go away once someone builds a decent bare metal distro with all of this stuff available out of the box (k3s is trending in that direction).

Axsuul 5y ago

I run mine using HashiStack (Nomad, Vault, Consul), highly recommend going that route over Kubernetes which I have also used.

movedx 5y ago
I've come from bare metal, to virtualisation, to Cloud. I then went from AutoScaling Groups and AMIs to K8s and Docker Images. Recently I went from K8s to Nomad+Consul and Docker Images. Now I'm building a small platform to host a business I've started and I'm going back to ASGs with simple EC2 Instances. There's really no need for the features Nomad/K8s offer at almost any scale outside of a few big companies.
- MonaroVXR 5y ago
  But how do you provision the baremetal?
- Axsuul 5y ago
  It depends. Orchestrators like Nomad can give you a lot of leverage as a small team. It's not very difficult to manage and can give you lots of redundancy, control, and a path to scalability while utilizing best practices so you don't have to think about it. There are lots of ways to do this and not every app needs this but as you begin to get traction, you either have a choice of outsourcing your devops or bringing it in-house which happens way before you are even a big company. Nomad solves the problem of the latter. Also, I'm curious why you didn't just end up sticking with Nomad now that you know it.
  - movedx 5y ago
    Orchestration engines can give you a lot of leverage if you run a managed solution because all you're doing there is deciding on the size you need and providing a credit card number. This is absolutely a great way of your DevOps to the next level with little effort. If you're going DIY, however, then you're not doing yourself any favours by starting out with an orchestration engine. In the case of Nomad you'll also need Consul for it to be truly useful. That means six EC2 Instances (for example) for a truly production trade system - 3x for Nomad, 3x for Consul. Of course you also need mTLS as well - may as well throw Vault into the mix too, then... that's another three EC2 Instances to support that cluster... 9x EC2 Instances into the DIY solution and all you have is an orchestration engine and you've not even started on the workers yet. That cost so far is USD$30,416.04 for the reference architecture for all nine nodes: https://learn.hashicorp.com/tutorials/consul/reference-architecture https://learn.hashicorp.com/tutorials/consul/reference-archi... On the other hand if you keep it simple and just work with AMIs you bake your self when you need to release something new, you need two EC2 Instances. The cost of the compute resources for this for 12 months: UDSD$330. There's no raft consensus which needs three nodes to be stable (but a single AZ failure can result in split brain, so really you need five AZs), no new technologies to understand, no new vendor to work with or pay capital to, nothing more to support. These things - these orchestration engines - have their place, and that place is outside of 99.9% of businesses (large or small.) As for moving away from Nomad: too expensive and too complex. I ran Consul, Nomad and Vault on the same master instances (which is a risk and against best practice) without mTLS (just TLS) and it took a week to get it all running. None of that includes monitoring, backups, auto-scaling, etc. I believe the work involved with going DIY on the orchestration front is not worth the benefits in the short to medium term. In the long term you'll likely outgrow your own simpler solution and at which point you'll likely have the capital to switch to a managed service anyway /shrug It's all relative, after all.

throwaway894345 5y ago

Personally I don’t use Kubernetes for the scalability, but because I haven’t seen anything else remotely maintainable (and yes, Kubernetes still has lots of room for improvement). I don’t want to build a good chunk of Kubernetes myself just to host my software. It’s nice to be able to Google for solutions to my problems or to ask on various forums/chat rooms (or in many cases, not have the problems in the first place because someone else committed the fixes to Kubernetes long ago).

nullwarp 5y ago

I've used everything under the sun and for homelab use IMO nothing beats using docker-compose and salt. 1. Unless you have crazy dynamic scaling needs, there's literally no point in using k8s or nomad 2. Unless you want RBAC and fine tuned ACL, theres literally no point in using k8s or Nomad I hate this idea that you need a Ceph cluster and a 3 node HA cluster to have a home lab. Not saying don't do it because you want to play/test/learn any of those platforms, just that if you are going to run a server service maybe consider if you are doing it for learning a task orchestration service or just want to run XYZ reliably.

sgarland 5y ago
I was settled on Docker-Compose and Ansible for a long time, and while it was indeed quite easy to use once everything was tweaked correctly, what finally got me to move to k8s was HA. Having a single node with nginx, DNS resolution and Unifi controller running on it meant if I ever took the node down down for something, everything broke. Plex being down because I wanted to play with something unrelated wasn't ideal. Agree that Ceph is ridiculously overkill though, I tried. Longhorn is much easier and perfectly fine for distributed storage.
- Macha 5y ago
  Unifi controller isn't a dependency for your network to continue working. Config is distributed to the devices themselves, the controller is just a UI and collector of data.
moontear 5y ago
I also run everything with docker stack(s) currently. Not using salt and haven’t heard of it. Why would I want salt and not ansible or something similar?
- mafro 5y ago
  Salt (https://saltproject.io/ https://saltproject.io/) was an almost-winner around the time that Ansible blew up. It's pretty irrelevant nowadays since terraform and kubernetes - but still it's a very very good tool. I'd suggest if you already know Ansible then don't change. If you have no idea which is which then try them both and let us know :)

discardedrefuse 5y ago

I'm currently building out my single-node "homelab" with coreos + podman + systemd. Most of the config is pulled from 1 ignition file during coreos installation. Its been a learning experience and but also very lightweight.

marginalia_nu 5y ago

My "homelab" is just debian stable, with a moratorium against containerization. Systemd keeps the services going. Updates are manual. It's a bit incredible how clean and easy such a set-up can be. Like there's almost nothing to say. It's extremely reliable and just keeps ticking. I actually did use to have a bunch of virtualization and kubernetes and crap, but got rid of it because it ate literally half the systems' resources, in a death by a thousand containers-type of way. There was also a lot of just jank, stuff was constantly breaking and I was always trying to piece together why from half a dozen out-of-date tutorials. Felt like herding cats.

sgarland 5y ago
I've transitioned from a single server running services on baremetal Debian, to running services with Docker on Debian, to running services with Docker on Debian VMs under Proxmox (aka Debian with a repo), to running services on a three-node Kubernetes cluster using k3os (I was using Talos, but it doesn't have Longhorn support yet, and Rook/Ceph was a nightmare). The baremetal server was the easiest to set up, obviously, but the hardest to maintain. Where did I put that random config? Wait, why did this break when I upgraded X? Switching to Docker (and later Compose) wasn't that difficult, and made maintaining services much easier. Going to k8s has been challenging, mostly because of how I'm doing it - a controlplane and worker node on each physical node in Proxmox. Writing the YAML is, well YAML. I'm mostly looking forward to not having to keep my VMs updated. I automated a lot of that (VMs are Packer templates) with Ansible, but it's still annoying. Upgrading k8s nodes is a lot easier, IMO.
ryandrake 5y ago
Yea I think I missed the whole web-complexity bus and am probably out of touch, but I still don’t get the use case for docker and containers and kubernets and and orchestration and all that stuff, just for a simple home setup. I serve a tiny web site, email, backups, a NAS and a few other internet services for my family, and my “stack” is vanilla Debian Stable. Maybe I don’t know what I don’t know, but my setup works for me and I don’t really have any problems maintaining it so I figure why add all the complexity? It always feels weird to see threads and threads of people talking about dozens of software programs I’ve never even heard of, let alone used. Maybe I’m living in the past but to me a “stack” is: OS, server, database, application. Like LAMP. Wonder when this changed! It makes me curious about what kinds of stuff people do in their home networks that I never even considered doing.

Macha 5y ago

My setup I just have ansible deploy to a couple of hosts running Arch. Generally I just use OS packages and have ansible deploy appropriate configs then for some software where that's more hassle I use podman rootless containers. Such software includes e.g. Graylog, which wants particular versions of Elasticsearch and mongo, or postgres which requires a manual data migration for annual version upgrades k8s and nomad are on my "maybe eventually later" list though.