3 ms·

Yep, mine is and I'm sure some others' are as well. Truly overkill but it's a fun hobby project.

__turbobrew__ 2y ago
Awesome
ninkendo 2y ago
I’m trying to resist the urge to move all my homelab setup to kubernetes too, mainly also because I don’t want to have to remember every dumb thing I did to customize my server and want everything to be deployable from a git repo, in case I need to rebuild it, etc. But I’ve had the same Linux box for over 15 years now (through various hardware changes, I’ve kept the same Ubuntu install since circa 2008, using do-release-upgrade for major updates and making sure to keep up with security updates) and I’m not sure it’s really worth it to optimize for easy recovery from a fresh install. I back up my homedir and etc and some other important data, and if I had to rebuild my OS I’m sure I’d do it far cleaner this time and wouldn’t want to fully do things the way I had them anyway. Even as a hobby it just doesn’t seem worth it to move away from the “pets, not cattle” model of servers when there’s just one or two of them (technically my router is separate but it’s a very simple router so there’s not much to do there)
- wilted-iris 2y ago
  Absolutely, I think pets are great at home. Personally, I find docker compose to be the sweet spot for reproducibility and ease of use. There's some bespoke setup, but I can go from backups and fresh hardware to running apps in a couple of hours. I switched to k8s entirely because I wanted to learn to use it, and I don't think I'd recommend it for anything else.
  - xena 2y ago
    For what it's worth, my homelab machines are pets.
- nucleardog 2y ago
  I moved all my stuff to k3s for exactly the reasons it seems you’re describing. I’m not saying “do it” but… it was worth it for me. I had a ton of stuff that was important to me running in VMs on a Proxmox server. I had backups, but if anything had happened to that it would still be a nightmare. Most of it had been there long enough that my technical knowledge and knowledge of how I had set it up was rusty to nonexistent. Which meant that I was afraid to touch it. Which made me get rustier… It was starting to be a constant source of stress and anxiety. Now I’ve got single repository that has everything that isn’t “actual user data”. I can (and have tested) taking a couple machines, PXE booting them, and within 15 minutes with no interaction getting to a fresh cluster. Then redeploying all my stuff takes me a couple hours (mostly due to slow internet) with, again, no interaction besides hitting “go” on a bunch of things. And god help me if I forget how Kubernetes works because then I’m probably out of a job too. I switched to running on four Lenovo ThinkCentre Tiny PCs (like $40 each I think) as opposed to a single large box to provide some redundancy as well. Makes things like hardware failures not as much of an issue, upgrades less risky, etc. I’m not afraid to do things with it anymore. I never worry that a piece of hardware failing is going to suddenly and unexpectedly mean a lost weekend. It’s made it all fun again instead of stressful and like work.
johntash 2y ago
What do you use for distributed storage (if anything)? Storage has always been my biggest headache when trying to make geographically-redundant clusters of any kind.
- __turbobrew__ 2y ago
  Rook/Ceph is the standard solution
  - johntash 2y ago
    I like ceph in some cases, but ceph hasn't worked very well IME with multi-region clusters or just higher-than-lan-speed latency. I never tried the actual multi-site support, but for my own homelab use-case - I don't have enough servers in each region to make their own ceph cluster.
    - __turbobrew__ 2y ago
      I don’t think any strongly consistent posix data store will work well in a multi region setup. Each write needs to be propagated before the application is unblocked. When you have tens of milliseconds of site to site latency this means every IO request is going to have worse performance than spinning rust. Object storage can work better in this regard because the IO chunks tend to be larger and therefore the round trip time is traversed less per bit written. Another alternative is that you run your regions in active passive setup so there is only one active region at a time and when the active region is changed all in process state in the previous active region can be sync to the new active region. Writes only need to be synchronously written within the active region. Honestly, if you can do that you can make the big bucks at FAANG. I personally just run a single home NAS which backs up to S3. Very strong durability in that S3 is never going to lose my data at the same time my NAS goes bad, but not great availability in that I will need to rebuild the NAS and pull down files from S3. When it comes to my home tech strong durability is a must but availability, not much so.

Awesome

I’m trying to resist the urge to move all my homelab setup to kubernetes too, mainly also because I don’t want to have to remember every dumb thing I did to customize my server and want everything to be deployable from a git repo, in case I need to rebuild it, etc. But I’ve had the same Linux box for over 15 years now (through various hardware changes, I’ve kept the same Ubuntu install since circa 2008, using do-release-upgrade for major updates and making sure to keep up with security updates) and I’m not sure it’s really worth it to optimize for easy recovery from a fresh install. I back up my homedir and etc and some other important data, and if I had to rebuild my OS I’m sure I’d do it far cleaner this time and wouldn’t want to fully do things the way I had them anyway. Even as a hobby it just doesn’t seem worth it to move away from the “pets, not cattle” model of servers when there’s just one or two of them (technically my router is separate but it’s a very simple router so there’s not much to do there)

wilted-iris 2y ago
Absolutely, I think pets are great at home. Personally, I find docker compose to be the sweet spot for reproducibility and ease of use. There's some bespoke setup, but I can go from backups and fresh hardware to running apps in a couple of hours. I switched to k8s entirely because I wanted to learn to use it, and I don't think I'd recommend it for anything else.
- xena 2y ago
  For what it's worth, my homelab machines are pets.
nucleardog 2y ago
I moved all my stuff to k3s for exactly the reasons it seems you’re describing. I’m not saying “do it” but… it was worth it for me. I had a ton of stuff that was important to me running in VMs on a Proxmox server. I had backups, but if anything had happened to that it would still be a nightmare. Most of it had been there long enough that my technical knowledge and knowledge of how I had set it up was rusty to nonexistent. Which meant that I was afraid to touch it. Which made me get rustier… It was starting to be a constant source of stress and anxiety. Now I’ve got single repository that has everything that isn’t “actual user data”. I can (and have tested) taking a couple machines, PXE booting them, and within 15 minutes with no interaction getting to a fresh cluster. Then redeploying all my stuff takes me a couple hours (mostly due to slow internet) with, again, no interaction besides hitting “go” on a bunch of things. And god help me if I forget how Kubernetes works because then I’m probably out of a job too. I switched to running on four Lenovo ThinkCentre Tiny PCs (like $40 each I think) as opposed to a single large box to provide some redundancy as well. Makes things like hardware failures not as much of an issue, upgrades less risky, etc. I’m not afraid to do things with it anymore. I never worry that a piece of hardware failing is going to suddenly and unexpectedly mean a lost weekend. It’s made it all fun again instead of stressful and like work.

johntash 2y ago

What do you use for distributed storage (if anything)? Storage has always been my biggest headache when trying to make geographically-redundant clusters of any kind.

__turbobrew__ 2y ago
Rook/Ceph is the standard solution
- johntash 2y ago
  I like ceph in some cases, but ceph hasn't worked very well IME with multi-region clusters or just higher-than-lan-speed latency. I never tried the actual multi-site support, but for my own homelab use-case - I don't have enough servers in each region to make their own ceph cluster.
  - __turbobrew__ 2y ago
    I don’t think any strongly consistent posix data store will work well in a multi region setup. Each write needs to be propagated before the application is unblocked. When you have tens of milliseconds of site to site latency this means every IO request is going to have worse performance than spinning rust. Object storage can work better in this regard because the IO chunks tend to be larger and therefore the round trip time is traversed less per bit written. Another alternative is that you run your regions in active passive setup so there is only one active region at a time and when the active region is changed all in process state in the previous active region can be sync to the new active region. Writes only need to be synchronously written within the active region. Honestly, if you can do that you can make the big bucks at FAANG. I personally just run a single home NAS which backs up to S3. Very strong durability in that S3 is never going to lose my data at the same time my NAS goes bad, but not great availability in that I will need to rebuild the NAS and pull down files from S3. When it comes to my home tech strong durability is a must but availability, not much so.