18 ms·

My self-hosting infrastructure, fully automated

716 points by rmbryan 5y ago

endofreach 5y ago
Now all we need is robots hooked into a Ci/CD pipeline that turn on computers and initiate the process. And a few more years to automate computer production, let the computers find bugs & optimize computers, and we have a full circle of life. Good first version, i am excited for the beta!
JamesAdir 5y ago
For people who self host with like setups or even simpler - how you guarantee network access? where I live internet might be shaky at times, and for that I can't access my home setup while away. How do you deal with that?
- killingtime74 5y ago
  I use ZeroTier. Works well with flaky internet
- Macha 5y ago
  Regarding my server connection: I can't help you there, my ISP has only had one incident of downtime I've noticed in the seven years of being with them. Regarding my client connection while away from home: If my client doesn't have a connection, cloud services don't work any better. But a lot of apps (e.g. bitwarden, some jellyfin clients, etc) are smart enough to work offline to some extent via operating on cashed resources.
deleted 5y ago
[deleted]
ianunruh 5y ago
It's sad to see so many people that are dismissive of this setup, because they have some preconceived notions about Kubernetes and other tooling that's used. What has happened to curiosity? Of course nothing about this is "necessary", but the fun is in trial and error. In my experience running Kubernetes in my homelab and colo, every issue has been a learning opportunity, and was usually because of my own mistakes.
josephd79 5y ago
bookmarked. Really cool.
simonw 5y ago
This is very cool! Have you considered something like Tailscale so you can securely access it from outside your home? I've been thinking about spinning up my own home server that way, seeing as Tailscale makes it easy to securely access it from my phone when I'm out and about.
- chaxor 5y ago
  Isn't headscale a clearly better option, since it removes the need to trust and depend on external sources? It the same software essentially, but if we're talking about self hosting, headscale is just inherently better, since it actually is self hosted.
  - egberts1 5y ago
    ¡Dos Mios! 129 Go dependencies just for a Headscale control server?! I think I will wait until a C variant appears after they've settled.
    - chaxor 5y ago
      Normally I would prefer a go implementation given the option (especially since this is network focused), but I was surprised to see that the go implementation of wiregaurd (not headscale, just the vanilla wiregaurd-go) is considerably slower than the rust or C version. The rust version (not that I'm that rust guy) may be the optimum for maintainability/readability and speed.
- rhizome 5y ago
  Honest question: why not SSH?
  - peterbraden 5y ago
    They solve different problems. Tailscale basically allows you to ignore any NAT topologies separating you from your machines. You can have a pseudo local network of your machines behind a VPN, allowing SSH to any machine with zero routing issues. And it’s incredibly seamless. I’m a big fan.
quocanh 5y ago
I love this! Do you experiment with your tech stack? Swapping things in and out to see which apps are best?
- khuedoan 5y ago
  (Repo owner here) Yes, you can see the visualized history of that here https://www.reddit.com/r/homelab/comments/s9bfht/the_evolution_of_my_homelab_over_15_years_from_a/ https://www.reddit.com/r/homelab/comments/s9bfht/the_evoluti... I think the OP discovered my homelab through that post.
unixhero 5y ago
Extremely impressive
turtlebits 5y ago
This is cool, but are you doing this as learning experience? For a homelab it seems severely overkill. I understand it all being self-hosted, but it could be drastically simpler if you adopt a few SaaS products (using free-tier)
- b33j0r 5y ago
  Personally I dig it. I recently did something similar, and felt dirty not automating everything, every step of the way. I knew I’d want to change the LAN architecture and the services, and everything; but it was definitely intended as a learning experience for me. Kinda looking forward to doing it over again with my new/refreshed skillset. Automated everything.
- yosito 5y ago
  I have mixed feelings. There is some over-complication, but I wouldn't call infrastructure without backups complete.
- khuedoan 5y ago
  (Repo owner here) Yes, I'm mostly doing this as a learning exercise; there's still a lot of work to be done before I can rely on it to host my services. > For a homelab it seems severely overkill Isn't that the point of homelab? ;)
- pdonis 5y ago
  > it could be drastically simpler if you adopt a few SaaS products (using free-tier) For some people (including me), the risk involved in any SaaS product suddenly either dropping or imposing unworkable restrictions on free tier is high enough to make the extra work involved in self-hosting worth it. (Granted, my current self-hosting setup is a lot simpler than the one described in this article, but even if mine were more complex I would still say the same thing.)
- citizenpaul 5y ago
  I've been burned too mamy times. Once a product realizes that x% of its free tier uses only feature X they suddenly put that feature behind a paywall. Now I"ve got to research and setup an alternative or pay 20$ for some minor convenice I set it up for. Most free tiers requires a card on file. I shudder to think at having to check dozens or hundreds of such services regularly to make sure they have not sent out an email about how we are now charging you. Then I don't notice for 6 months and find out I spend $150 on something I could have setup myself in less than an hour that will never charge me. I have had hundreds of such products subscribed at work. At least every year or so an entire day gets burnt updating credit card info from logging in and looking up the bespoke way every single site requires the updates to be done for payments. The services that suddenly shut their doors and again require me to research and setup an alternative. Services get acquired all the time then switch to pay only and they have your CC on file already for easy billing.
extinctpotato 5y ago
This is a very good example of how you can save yourself the mental hurdle of remembering how to configure something etc. I can only suspect how much time, trial and error this must've taken. This is my main issue with IaC. The concept really lends itself to any kind of modern infra, however I'm really put off by the sheer amount of time it takes me to whip out a bunch of Ansible playbooks and helper scripts, and on top of that make sure that all of them are idempotent. Maybe I'm doing something wrong and this should be easy?
- artdigital 5y ago
  It’s a good habit to always use tools like terraform for cloud or Ansible/Salt/Puppet for machines instead of directly doing something. Especially cloud setups that just run containers are relatively easy to get idempotent with terraform
  - MonaroVXR 5y ago
    Still need to install the machine with the "hand" or with Satellite or Anaconda.
  - danuker 5y ago
    No. It never gets a habit. It is torture, and you have to look up the commands every time you do it, because you do it infrequently enough not to learn by heart.
    - xyzzy_plugh 5y ago
      Huh? Compared to... running commands infrequently to administer servers? IaC is strictly and comically better.
      - necovek 5y ago
        Commands you use every day in the CLI? In your editor? I can trivially enter Emacs and modify Apache config when it breaks backwards compat between distribution upgrades, but when it's infrastructure as code...
616c 5y ago
Pretty sweet. I have been called a devops person by others around me (and I am hesitant to proudly identify as one) because this homelab is more impressive and modular than a lot of stuff I have seen colleagues and I put together for professional stuff. Well done. I was not aware of the Cloudflare solution. Is this something someone can use, _with_ their Cloudflare Access offering, for personal dev/lab envs without breaking the bank?
- altano 5y ago
  Cloudflare Tunnels (formerly Argo Tunnels) are a free complement to Cloudflare Access. It's an easy way to expose internal servers to the public internet, which you can then lock down with Cloudflare Access.
- LilBytes 5y ago
  I set up Tailscale a little over a week ago, it boggled my mind how easy it was. I'm using it for personal use at the moment and I'm considering changing to a paid user to my friends and family can access Emby over the internet. I did have a few dramas getting in to work for my LxC environments but nothing a quick Google resolved for me.
  - 616c 5y ago
    I've been thinking about it but I wanted to set it up on a BSD VPN appliance, and I was not sure that was easy for pfSense in an easy installable way, then expose some NUC systems. Are you just running it on a system behind a firewall/router/NAT-ed network device or on a terminating device itself? But, as usual, far behind on my personal projects ...
    - LilBytes 5y ago
      Nope. It runs on every client that I want to remotely access. This includes a Proxmox host, a few LxC environments, home PC, work laptop and on my phone. I did have some issues getting it running on WSL2 but as long as I can get to the Windows host running it (and I can), that's fine for me. I didn't need to change anything on my network equipment which is why I was so amazed at the ease of it's use. I do have a static IPv4 at home but from what I understood that had absolutely no bearing on my ease of installation. No port forwarding, no changes to my Meraki MX gateway, no nothing. It was essentially plug and play. Edit: looks like BSD is supported too. Though I don't have a BSD environment to test it on. https://tailscale.com/kb/1074/connect-to-your-nas/?q=Bsd https://tailscale.com/kb/1074/connect-to-your-nas/?q=Bsd
whalesalad 5y ago
Bravo, nice work! I am certainly going to use this as a model for my personal setup. Right now I have a few different hosts running different hypervisors, but would like to consolidate on kube and an iaas tool.
goodpoint 5y ago
Deploying services is 1% of the work. Maintaining them, fixing bugs and bottlenecks is the real works. And you cannot do when you deploy tools worth millions of lines of code. Complexity matters. Those popular products make sense only if you have a 20 engineers is your team or you don't care about reliability.
- Gigachad 5y ago
  This is downvoted but I think it is true, self hosted services don't get nearly as much love as they need. It takes a huge amount of effort, monitoring, and knowledge on the specific tools to be able to host them securely and detect intrusions. I attempted self hosting for years but decided it ultimately wasn't worth it and that I have very little ability to prevent against attacks or detect that they had happened.
  - goodpoint 5y ago
    Not my point. Self-host plenty of stuff but choose *simple* tools, not behemots. Also it's important to use software packaged in distros like Debian in order to receive security updates for many years.
- spaniard89277 5y ago
  Maybe they want to practice. I installed proxmox and I'm pretty much set up, but my needs are pretty huble, just a couple of VMs and maybe containers in the near future.
Lamad123 5y ago
I unaccidentally remembered Gilfoyle's famed "what do I do?!"
throwaway894345 5y ago
This is great. Would love to see something like this packaged as a “homelab cluster distro”!
erulabs 5y ago
Awesome! hajimari is neat, haven't seen that before. I'm building a startup that's trying to put a good chunk of this in a box for folks to buy and plug-n-play self-hosting (as much as that is possible). Since you're running k3s already, you might checkout our templates at https://kubesail.com/templates https://kubesail.com/templates for more home-hosting apps :)
- khuedoan 5y ago
  (Repo owner here) The templates look cool, are those Helm charts underneath? Also I believe the tag filter is malfunctioning: https://kubesail.com/templates-by-tag/Media https://kubesail.com/templates-by-tag/Media
  - erulabs 5y ago
    Ack, will look at that bug! No, the templates are currently plain YAML with a handle-bars style variable replacement (and a couple other neat tricks). We do support Helm charts tho, I just haven't had time to add them to the "Templates" view. There are loads of good Helm charts here: https://kubesail.com/helm/k8s-at-home https://kubesail.com/helm/k8s-at-home
akkartik 5y ago
The question I have whenever I see something like this is what the upgrade process looks like. How much time do you spend per month upgrading dependencies, how many mailing lists you're subscribed to, etc. When the log4j thing blew up, how long did it take you to gain confidence that you were no longer impacted?
- zenlot 5y ago
  Doesn't really matter, as long as it's fun. Some people play games, so people enjoy re-configuring their setup (I do enjoy it too). It's not a business / prod env setting.
  - akkartik 5y ago
    Yeah, that's totally fine. I ask in case someone has thought about it, so I can learn from them. Doing just the fun parts of self-hosting will not get you to infrastructure.
- throwaway894345 5y ago
  Probably no worse than the alternatives. Just about everything is going to be easy to upgrade except possibly the Kubernetes masters (and even that isn’t so bad if you can spare a bit of downtime).
  - akkartik 5y ago
    For sure no worse than the alternatives. I also don't care about downtime. But I'm looking for something that minimizes the overheads over not self-hosting. I'd love to prove Moxie Marlinspike wrong that "people don't want to run their own servers, and never will." (https://moxie.org/2022/01/07/web3-first-impressions.html https://moxie.org/2022/01/07/web3-first-impressions.html) This is the key bottleneck in getting people to run their own servers.
    - throwaway894345 5y ago
      For sure. There’s not much of an economical argument for self-hosting—if you’re doing it, it’s almost implied that you’re doing it for fun. Although having run a small (but really scaleable c/o k8s) homelab for <$2/month, I’m not sure I’d save much effort versus using some PaaS or cloud provider now that I know what I’m doing. Like I’d need a real load balancer and a few other things so we’re talking < ~$100/month to productionize.
gigel82 5y ago
This needs a companion guide about how to set up the host machine. Which Linux distro to choose, how to set it up / harden it, nftables / firewall, public key login, etc. If anyone has one handy, I'd appreciate a link.
- Art9681 5y ago
  Several people in this thread stating they are working on solutions to assist others learning modern cloud/devops stacks. I’ll throw my name in the hat too. I have a single node home lab running enterprise cloud/devops software stack. I have been working on a project to deploy on-prem cloud (OpenStack) in disconnected environment. I’ve had to learn how to build and host all of the infrastructure required to do this without the luxury of internet access. I realized the same thing can be simulated very cheap in a modern computer with enough cores and memory to run the stack as a single bare metal host. My build cost $1300 2 years ago but the knowledge gained has paid me back 5 figures worth of raises in my career in 16 months. Having my own personal CloudBox has allowed me to experiment and fail fast. I am ahead in experience and knowledge than the rest of my team as a result. I have a tool they do not. I realize it would be better if we all had a tool like this. So that’s the pitch, a single node “cloud in a box’, the CloudBox for IT students or professionals to learn any aspect of IT. Now I just need lots more time to actually turn my prototype into a product.
- indigodaddy 5y ago
  Doesn’t the ansible portion do all that? Believe he uses Rocky Linux.
- dmayle 5y ago
  You are unlikely to find what you're looking for. I'm going to dig into this article a bit, but what you're talking about is inherently hard, and delivers tremendous value to businesses. There are a lot of people doing it for money, and someone builds something that starts to actually work, they get acquired, or they take their project and make it enterprise-y (because that's where the funding is). There's flatcar and k3os and fedora coreos and talos and lokomotive. There are maybe a dozen others as well, but those are the ones I know something about. The real problem is that the orchestration of PXE boot, DHCP, name services, server registration, cluster bootstrapping, while simultaneously building a light distribution that makes the right calls on persistence, security, etc. is just *really hard*. I took at a stab at it myself (failed startup) and have a custom distribution based on alpine, but the amount of work to go from there to everything is so large that it's tough to take on if you're small (and there is the constant desire to go enterprise because of the money)
walrus01 5y ago
my main objection to this is choosing host machines that don't meet the criteria of anything like serious server-grade hardware. if this is for a home lab where any one of the services run on it are not actually going to affect you if it goes belly up? or the whole host machine? sure, okay, but that's self hosting a home lab, not self-hosting actual infrastructure... clearly the hardware shown in the image is meant to be small, not noisy, and not take up a lot of space, and operate in somebody's house. but the people I know who seriously self host all their stuff have it on something like a few old Dell R620 1U servers with 1+1 dual hot swap power supplies, RAID-1 for the operating system boot, RAID-1 or RAID-6 (or something like mdadm raid6) for storage drives (again all hot swap capable), etc. note that I am not talking about a lot of money here, you can buy the hardware I described in the preceding paragraph for $300 on eBay and then add your own choice of SSDs. and not in a house, but in something like a small to medium sized ISP's colocation environment, with UPS, generator backed power, etc. also attached to network equipment and a DIA circuit that's considerably more serious than a tp-link unmanaged switch attached to a residential internet service.
- Spooky23 5y ago
  All overkill. If you want something more reliable, get some late model HP or Dell thin clients and put Debian on them. Usually you can pick them up for <$75. I setup something like this to control some farm equipment when I was in college 20 years ago. The farmers’ son called me a few months ago because they were having an issue with the equipment. Turned out oily dust coated the thing and got inside - I had him pop the device open (old Wyse box running Slackware iirc) clean up the goop and hit it with rubbing alcohol. He dried it and everything is working again.
divbzero 5y ago
I was happy to see that this project describes self-hosting in the fullest sense, from hardware on up.
ekianjo 5y ago
You don't need Kubernetes to do self-hosting. Completely overkill.
- walrus01 5y ago
  I would rather choose something like xgp-ng as a hypervisor for a home lab environment such a this, instead of a custom one-off kubernetes environment. Or just a total DIY xen or KVM hypervisor on debian or centos approach. But with considerably more RAM in the host machine than 16GB (like 64-128GB). the kubernetes sort of makes sense if the purpose is also for the person who owns/operates it to test kubernetes things, learn kubernetes, or use it as a home lab of some larger similar environment that they have elsewhere.
  - moondev 5y ago
    > Or just a total DIY xen or KVM hypervisor on debian or centos approach. Or just install kubevirt on the existing cluster and manage kvm virtual machines with k8s https://github.com/kubevirt/kubevirt https://github.com/kubevirt/kubevirt
- the_duke 5y ago
  You don't need it. But Kubernetes exists for a reason and makes a lot of things easier. If you are familiar with the ecosystem, k3s is a great foundation for self-hosted setups. Especially thanks to the many helm charts and operators that exist.
- z3t4 5y ago
  Talking about overkill, he also use 4 physical machines each with a 4 core CPU. But its not about what you need, its about what you want to learn. So if you want to learn about cluster/orchestrating i think its warranted.
- dspillett 5y ago
  Overkill probably, but useful practise for when working on something of a scale where it will make a significant difference. And they might just like playing with these things. A home lab does not have to be DayJob practical!
candiddevmike 5y ago
I enjoy reading about homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. This is just another "how I installed kubernetes" thing only without any real scale behind it. There are other ways to deploy and run things, and it takes people toying around with alternatives for something unique to spring up.
- Havoc 5y ago
  > homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. I've had great success with Ansible+Proxmox. The proxmox CLI interface is good enough that you can basically do DIY terraform like setup straight against CLI without any agents / plugins etc. (I know declarative vs imperative but hopefully the comparison makes sense) Lately I've been toying with local gitlab-CI building and deploying images directly into the environment so that any changes to the code redeploy it (to prod...this is home after all haha) Considered terraform too but not a big fan of their mystery 3rd party plugin provider system
- oceanplexian 5y ago
  Yeah I’m with you. I run a rack at home with servers running a few applications orchestrated with a bit of Puppet, it’s 100x more reliable than K8s, I’m not having to do weird hacks to make persistent storage work, edit dozens of YAML files and run a whole host of dodgy control plane services. Sadly, I leaned this the hard way by setting up a 4-node k8s cluster and realizing bare-metal support is abysmal and the maintainers simply don’t care about you if you’re not running a cluster on AWS or GCP. Unfortunately my day job still involves k8s and people give me strange looks when I suggest metal so substantially less complex, cheaper, more performant, and easier to orchestrate and maintain.
  - movedx 5y ago
    I'm building my own education platform around getting people into IT and towards a career in Cloud and DevOps. A few months back I built a Nomad based cluster with Consul and was like: wow! This is super cool... but it's also expensive and complex. I'm going to be building out the platform with an ALB, two EC2 Instances in an AutoScaling Group based on an AMI I bake with Ansible and Packer. I'll use Terraform for the entire infrastructure. The release pipeline for content updates (it's an MkDocs/Material based book with videos blended in) will be me updating the AMI and firing off a `terraform apply`. I won't be using Docker. I won't require or need an orchestration engine. It's a static website delivered via a custom web server written in Go (as I need custom "Have you paid to view this content?" logic.) That's about as complex as it gets for now. I'm actually looking forward to implementing it and watching as something so simple hopefully goes on to allow me to build and scale a business.
khuedoan 5y ago
Repo owner here, I just created this account, I'm a long time HN lurker. I was surprised to find this on Hacker News, I wanted to wait until the stable release before posting on HN, but thank you for posting :) This project is still in alpha stage, but please feel free to critique; I'd appreciate it. Edit 1: After reading some of the comments, I want to clarify a few things: - Because it is currently in the alpha stage, I do not host anything important on it. - This is also my learning environment, I use Kubernetes in my day job. - Yes it is overkill ;) Edit 2: Wording
- sandGorgon 5y ago
  do you use k3s in production as well ?
  - khuedoan 5y ago
    Is production == my day job? If so we don't, we use managed Kubernetes (we used to manage our own clusters but they will be decommissioned soon).
    - coopreme 5y ago
      Can you go into the reasons you went to managed vs self owned K8S?
- jmakov 5y ago
  Thanks for the project! Once deployed, how do you keep everything up to date?
  - khuedoan 5y ago
    I'm currently bumping the versions manually (via a commit), but I plan to automate that with system upgrade controller [1] and Dependabot [2] (or similar) [1]: https://github.com/rancher/system-upgrade-controller https://github.com/rancher/system-upgrade-controller [2]: https://github.com/dependabot https://github.com/dependabot
- 5y ago
deleted 5y ago
[deleted]
mikesabbagh 5y ago
it amazes me that hacker news run on a single node. All this work is nice and beautiful, the problem will come when you try to update different components.
- christophilus 5y ago
  > it amazes me that hacker news run on a single node. It amazes me how much of the internet doesn’t run on a single node, when it probably could.
  - jaimex2 5y ago
    Why would anyone run a simple elegant site when you can get buried in js dependencies and plumbing stacks for months?
qbasic_forever 5y ago
Very cool, this is like a perfect little kubernetes development environment. The combo of tekton + gitea + k8s is really nice and just a shame it takes bodging all the lego pieces together manually right now. I wish there were a one click exe install like Docker for Desktop that gave people an entire environment like this on their desktop. Like a little private github but so much more powerful. If you're looking for some nice stuff to develop on an environment like this, check out VS Code and Google's cloud code addon: https://marketplace.visualstudio.com/items?itemName=GoogleCloudTools.cloudcode https://marketplace.visualstudio.com/items?itemName=GoogleCl... It's not totally tied to their GCE cloud offerings and is actually just a VS Code integration of their Skaffold tool (a fantastic inner dev loop tool for k8s). It works great with minikube or any other k8s cluster, presumably this one too. You can very quickly get up and running with code in your IDE now running and debugging on your k8s environment.
- khuedoan 5y ago
  (Repo owner here) Glad someone mentioned it, I do have one: https://homelab.khuedoan.com/try_on_a_vm https://homelab.khuedoan.com/try_on_a_vm Although it's still incomplete and not one click yet, that's the direction I'm heading in: anyone can try my homelab on their PC, and if they want to use it on their real hardware, they can continue with the full install.
mrslave 5y ago
I've been eyeing off OpenStack for sometime for a similar use case, but I must admit it is an uninformed opinion. What am I missing between the two solutions (OpenStack vs this specific custom solution)?
cyfex 5y ago
I always found the hurdle with self-hosting to be maintaining, not the initial setup. Things like upgrading, in order to keep getting security fixes, and verifying everything works after the upgrade, are what has taken the most effort and time in the past for me. This looks like a great setup by the author, but difficult to maintain in the long run without significant time investment.
- input_sh 5y ago
  Docker and docker-compose make maintaining stuff dead simple to me. In like 95% of the cases it's just a matter of changing the version in a YAML file and running one command, and in the remaining 5% of the time I'm not concerned at all with some downtime because I'm (mostly) the only user.
  - BrandoElFollito 5y ago
    This. I used to host directly on my server and discovered docker a few years ago. It was a relief. Coupled with watchtower the updates are automatic except for the two services I really rely upon. I used to be on Ubuntu for ages and moved to Arch a few days ago - my server only runs docker today. I still want to keep access to a shell do it is Arch and not Rancher or something (I did not research much the bare metal hypervisors). My maintenance is minutes every month if everything works fine, up to a max of an hour when Home Assistant broke things once two years ago. DRP from bare metal is an hour. Adding a service is a few minutes.
  - toshk 5y ago
    What do you use for serving docker-compose?
    - g_p 5y ago
      Docker-compose doesn't need "served" per-se - it lets you run some containers, and handles some lightweight orchestration around them, so that you have groups of containers for a service (i.e. a Wordpress service compromises a mariadb container and a Wordpress container). The compose file handles port bindings/mappings. If you then want to put a reverse proxy in front of the docker containers (you almost invariably will), then you can look at different options like caddy, Traefik, nginx etc. I, for one, like to be old-fashioned and have my docker containers' ports bound to localhost, then manually maintain my own "outside of docker" instance of nginx as a reverse proxy that uses these as upstreams. That's not the most "container-first" way of working, but it worked for me. Caddy can do similar. Traefik is more integrated with docker and the docker ecosystem, but that might do what you need better.
kstenerud 5y ago
This is definitely cool, but it also highlights a huge problem we have with software nowadays. There are 19 stacks in this repository. 19 pieces of software that require their own maintenance, JUST TO RUN YOUR APPLICATIONS! The amount of extra work required just to host the software that views your pictures, plays your videos, and allows chat with people is absolutely insane. I'm not talking about this particular repo; it's just indicative of how complicated things have become that you must do the equivalent of building and maintaining your own operating system just to get even simple things done. And I belive that it's unnecessarily complicated (once again, not this repo, but rather the state of affairs in general). We're at an inflection point in the industry, and haven't come out the other side yet.
- legends2k 5y ago
  Isn't it how the general software philosophy is? A composition of decoupled, do-one-thing-well pieces working together to achieve a larger task?
  - kstenerud 5y ago
    Yes, but after the initial monoliths of mainframes, we had a golden era of operating systems, where all of the plumbing and infrastructure to run your applications was maintained by someone else (Microsoft, Apple, Commodore, Atari, SGI, Sun, Palm, Redhat, etc). Now we've come full circle back to the bad old days where you need an entire team of dedicated people and arcane knowledge just to run your application software again. I suspect that this will continue until we reach a point where the big players out of necessity come to an agreement for a kind of "distributed POSIX" (I mean in spirit, not actual POSIX). These are exciting times living on the edge of a paradigm shift, but also frustrating!
    - theK 5y ago
      That’s actually very insightful. The OS era you describe was mostly local with not so many things happening in online interactions and way less devices collaborating to create for example your images folder. Now that we mostly have the server-client model ingrained in every online activity there is an obvious complexity centralisation on the server side. Might be interesting to see how web3 and other decentralised web movements might factor in to this!
jaimex2 5y ago
I miss LAMP :(
- midasuni 5y ago
  I use LAMP. WHy would I miss it?
- rambambram 5y ago
  It's still there, also waiting for you my friend.
surfsvammel 5y ago
How much energy does this consume? I have done some home lab going on over the years, and find that trying to do more on less W has been the most fun.
- khuedoan 5y ago
  (Author here) Yep that's my goal too, hence the small form factor PCs. It costs me around 2-5$/month depending on how much I play with it.
nyellin 5y ago
To everyone saying that Kubernetes is unnecessary, try implementing autoscaling, service discovery, secrets management, and autohealing in a vendor independent way without it. Of course none of that is necessary for a self hosted home lab, but neither is gitops. This is a very nice example of how to set stuff up properly. OP, I would love to see Robusta (https://robusta.dev https://robusta.dev) as part of this too. It's definitely in line with your vision of automating everything, as it let's you automate the response to alerts and other events in your cluster. (Disclaimer: I'm one of the maintainers)
- dijit 5y ago
  Oh. I’ve actually done that. It’s not as hard or complicated as you claim. This was before kubernetes, arguably learning kubernetes is harder than implementing auto discovery and auto healing because you have to learn a lot about cluster semantics, the various network providers, service accounts, role based access control, integrations with secrets manager or the very disparate implementations of custom operators which may operate in your environment. Kubernetes is a platform, I don’t doubt it solves a lot of problems. But it’s a complicated wee beastie, and I say that as a person who did it the easy way and used GKE on Google Cloud
  - nyellin 5y ago
    I don't know how you implemented it, but the big innovation of kubernetes was doing this with declarative inputs and using control loops for reconciliation instead of edge triggered changes. It might not matter on a small scale, but on a large scale that's what makes it robust. It's also not trivial to implement a distributed version of this in house at scale. At the very least you need something like etcd at the core
    - dijit 5y ago
      The way we did it is that we registered to a centralised “name” service which was an in memory database. Failure to register to the name service leads to registration towards the next name service in the list, we ran three- so there were two spares, the library would then keep trying to register to the first name service, this would be our reconciliation loop. When you request a service you request “traits” through the name service. Not “trivial” but even though it was implemented in C++ on windows it was much less complicated than kubernetes discovery systems, which there are many ways to integrate (env-var, dns, api) and some should or must be disabled for security reasons (eg. The environment variables that display all services) or can change depending on RBAC and service account. We deployed about 300,000 or so services with this infrastructure and it’s been running in production since 2015. (Original product release was supposed to be 2013 but due to the client changes it was delayed, not due to the backend. This is Video Games unfortunately). I’m not like, “hard” on these opinions, but people talk about kubernetes as if it’s the only way of solving these problems, when it can be the case that you have to learn a lot about kubernetes before you can actually do what you need to do; if you already understand distributed systems then it’s a bigger effort to understand kubernetes than would be to implement a distributed system that solves your needs from scratch
AdrianoKF 5y ago
I recently wanted to give my current RPi4 home server a GitOps makeover, migrating services (Home Assistant, Nextcloud and others) to k3s in the process. What has been an obstacle is the availability of officially maintained Docker images for some of the components I've been wanting to use - afaict neither Argo CD nor Rook have official armv7/aarch64 images (though it seems Argo will release one soon). Until then, I'll hold off on that pet project until I get my hands on a reasonably priced x86 SFF PC (the ThinkCentre M700 Tiny from TFA looks interesting!).
- yebyen 5y ago
  FWIW, FluxCD has full armv7 and aarch64 support (it definitely works on a Raspberry Pi 4, the earlier models with 1GB of RAM too, although depending on the size of your git repos that may not be enough memory, 4GB-8GB certainly is for Flux.) How many Rpi4 servers are we talking? I have never run Rook before (though I ran Ceph, when it was called deis-store way back before it was cool) and I always remember that you needed at least 3-5 servers with at least one dedicated disk for OSD per each to make it work well, if at all! I'm looking at synology-csi right now for my homelab, and wondering if there is hope that I can stop using local-path-storage this way, it looks a little bit outdated, so I'm afraid that may keep me from running the latest version of Kubernetes on it... unless I hack the support into it myself :gasp: surely that's gonna void the warranty
  - yebyen 5y ago
    (I did wind up getting synology-csi to work!)
- 3np 5y ago
  For various reasons I started building all container images myself. In most cases when arm64 images aren't provided, they will build just fine anyway. Running you own registry is super simple. https://docs.docker.com/registry/deploying/ https://docs.docker.com/registry/deploying/
iamgopal 5y ago
Is there similar repo for SAAS apps ? Not too fancy, just Django and a db.
comprev 5y ago
I use something very bare bones as the host OS, say Alpine, and install Docker with a handful of other tools for basic sysadmin tasks (curl, etc.). For a single server deployment docker-compose is very useful.
gorgoiler 5y ago
I wonder how the fan noise is on those M700s. These little Lenovo machines are very pleasant. My only wish with my M93 is that the temperature would stay below 50’C and keep it 100% silent.
nodesocket 5y ago
What do you use to provision Kubernetes persistent volumes on bare metal? I’m looking at open-ebs (https://openebs.io/ https://openebs.io/). Also, when you bump the image tag in a git commit for a given helm chart, how does that get deployed? Is it automatic, or do you manually run helm upgrade commands?
- khuedoan 5y ago
  I'm currently using Longhorn for storage, but if I find some reasonably priced HDDs, I may add or switch to Rook. When you make a change in git, it is automatically deployed without the need for human intervention.
  - nodesocket 5y ago
    Can you elaborate a bit more on how the automatic helm upgrades are performed?

716 points by rmbryan 5y ago

Now all we need is robots hooked into a Ci/CD pipeline that turn on computers and initiate the process. And a few more years to automate computer production, let the computers find bugs & optimize computers, and we have a full circle of life. Good first version, i am excited for the beta!

JamesAdir 5y ago

For people who self host with like setups or even simpler - how you guarantee network access? where I live internet might be shaky at times, and for that I can't access my home setup while away. How do you deal with that?

killingtime74 5y ago
I use ZeroTier. Works well with flaky internet
Macha 5y ago
Regarding my server connection: I can't help you there, my ISP has only had one incident of downtime I've noticed in the seven years of being with them. Regarding my client connection while away from home: If my client doesn't have a connection, cloud services don't work any better. But a lot of apps (e.g. bitwarden, some jellyfin clients, etc) are smart enough to work offline to some extent via operating on cashed resources.

deleted 5y ago

[deleted]

ianunruh 5y ago

It's sad to see so many people that are dismissive of this setup, because they have some preconceived notions about Kubernetes and other tooling that's used. What has happened to curiosity? Of course nothing about this is "necessary", but the fun is in trial and error. In my experience running Kubernetes in my homelab and colo, every issue has been a learning opportunity, and was usually because of my own mistakes.

josephd79 5y ago

bookmarked. Really cool.

simonw 5y ago

This is very cool! Have you considered something like Tailscale so you can securely access it from outside your home? I've been thinking about spinning up my own home server that way, seeing as Tailscale makes it easy to securely access it from my phone when I'm out and about.

chaxor 5y ago
Isn't headscale a clearly better option, since it removes the need to trust and depend on external sources? It the same software essentially, but if we're talking about self hosting, headscale is just inherently better, since it actually is self hosted.
- egberts1 5y ago
  ¡Dos Mios! 129 Go dependencies just for a Headscale control server?! I think I will wait until a C variant appears after they've settled.
  - chaxor 5y ago
    Normally I would prefer a go implementation given the option (especially since this is network focused), but I was surprised to see that the go implementation of wiregaurd (not headscale, just the vanilla wiregaurd-go) is considerably slower than the rust or C version. The rust version (not that I'm that rust guy) may be the optimum for maintainability/readability and speed.
rhizome 5y ago
Honest question: why not SSH?
- peterbraden 5y ago
  They solve different problems. Tailscale basically allows you to ignore any NAT topologies separating you from your machines. You can have a pseudo local network of your machines behind a VPN, allowing SSH to any machine with zero routing issues. And it’s incredibly seamless. I’m a big fan.

quocanh 5y ago

I love this! Do you experiment with your tech stack? Swapping things in and out to see which apps are best?

khuedoan 5y ago
(Repo owner here) Yes, you can see the visualized history of that here https://www.reddit.com/r/homelab/comments/s9bfht/the_evolution_of_my_homelab_over_15_years_from_a/ https://www.reddit.com/r/homelab/comments/s9bfht/the_evoluti... I think the OP discovered my homelab through that post.

unixhero 5y ago

Extremely impressive

turtlebits 5y ago

This is cool, but are you doing this as learning experience? For a homelab it seems severely overkill. I understand it all being self-hosted, but it could be drastically simpler if you adopt a few SaaS products (using free-tier)

b33j0r 5y ago
Personally I dig it. I recently did something similar, and felt dirty not automating everything, every step of the way. I knew I’d want to change the LAN architecture and the services, and everything; but it was definitely intended as a learning experience for me. Kinda looking forward to doing it over again with my new/refreshed skillset. Automated everything.
yosito 5y ago
I have mixed feelings. There is some over-complication, but I wouldn't call infrastructure without backups complete.
khuedoan 5y ago
(Repo owner here) Yes, I'm mostly doing this as a learning exercise; there's still a lot of work to be done before I can rely on it to host my services. > For a homelab it seems severely overkill Isn't that the point of homelab? ;)
pdonis 5y ago
> it could be drastically simpler if you adopt a few SaaS products (using free-tier) For some people (including me), the risk involved in any SaaS product suddenly either dropping or imposing unworkable restrictions on free tier is high enough to make the extra work involved in self-hosting worth it. (Granted, my current self-hosting setup is a lot simpler than the one described in this article, but even if mine were more complex I would still say the same thing.)
citizenpaul 5y ago
I've been burned too mamy times. Once a product realizes that x% of its free tier uses only feature X they suddenly put that feature behind a paywall. Now I"ve got to research and setup an alternative or pay 20$ for some minor convenice I set it up for. Most free tiers requires a card on file. I shudder to think at having to check dozens or hundreds of such services regularly to make sure they have not sent out an email about how we are now charging you. Then I don't notice for 6 months and find out I spend $150 on something I could have setup myself in less than an hour that will never charge me. I have had hundreds of such products subscribed at work. At least every year or so an entire day gets burnt updating credit card info from logging in and looking up the bespoke way every single site requires the updates to be done for payments. The services that suddenly shut their doors and again require me to research and setup an alternative. Services get acquired all the time then switch to pay only and they have your CC on file already for easy billing.

extinctpotato 5y ago

This is a very good example of how you can save yourself the mental hurdle of remembering how to configure something etc. I can only suspect how much time, trial and error this must've taken. This is my main issue with IaC. The concept really lends itself to any kind of modern infra, however I'm really put off by the sheer amount of time it takes me to whip out a bunch of Ansible playbooks and helper scripts, and on top of that make sure that all of them are idempotent. Maybe I'm doing something wrong and this should be easy?

artdigital 5y ago
It’s a good habit to always use tools like terraform for cloud or Ansible/Salt/Puppet for machines instead of directly doing something. Especially cloud setups that just run containers are relatively easy to get idempotent with terraform
- MonaroVXR 5y ago
  Still need to install the machine with the "hand" or with Satellite or Anaconda.
- danuker 5y ago
  No. It never gets a habit. It is torture, and you have to look up the commands every time you do it, because you do it infrequently enough not to learn by heart.
  - xyzzy_plugh 5y ago
    Huh? Compared to... running commands infrequently to administer servers? IaC is strictly and comically better.
    - necovek 5y ago
      Commands you use every day in the CLI? In your editor? I can trivially enter Emacs and modify Apache config when it breaks backwards compat between distribution upgrades, but when it's infrastructure as code...

616c 5y ago

Pretty sweet. I have been called a devops person by others around me (and I am hesitant to proudly identify as one) because this homelab is more impressive and modular than a lot of stuff I have seen colleagues and I put together for professional stuff. Well done. I was not aware of the Cloudflare solution. Is this something someone can use, _with_ their Cloudflare Access offering, for personal dev/lab envs without breaking the bank?

altano 5y ago
Cloudflare Tunnels (formerly Argo Tunnels) are a free complement to Cloudflare Access. It's an easy way to expose internal servers to the public internet, which you can then lock down with Cloudflare Access.
LilBytes 5y ago
I set up Tailscale a little over a week ago, it boggled my mind how easy it was. I'm using it for personal use at the moment and I'm considering changing to a paid user to my friends and family can access Emby over the internet. I did have a few dramas getting in to work for my LxC environments but nothing a quick Google resolved for me.
- 616c 5y ago
  I've been thinking about it but I wanted to set it up on a BSD VPN appliance, and I was not sure that was easy for pfSense in an easy installable way, then expose some NUC systems. Are you just running it on a system behind a firewall/router/NAT-ed network device or on a terminating device itself? But, as usual, far behind on my personal projects ...
  - LilBytes 5y ago
    Nope. It runs on every client that I want to remotely access. This includes a Proxmox host, a few LxC environments, home PC, work laptop and on my phone. I did have some issues getting it running on WSL2 but as long as I can get to the Windows host running it (and I can), that's fine for me. I didn't need to change anything on my network equipment which is why I was so amazed at the ease of it's use. I do have a static IPv4 at home but from what I understood that had absolutely no bearing on my ease of installation. No port forwarding, no changes to my Meraki MX gateway, no nothing. It was essentially plug and play. Edit: looks like BSD is supported too. Though I don't have a BSD environment to test it on. https://tailscale.com/kb/1074/connect-to-your-nas/?q=Bsd https://tailscale.com/kb/1074/connect-to-your-nas/?q=Bsd

whalesalad 5y ago

Bravo, nice work! I am certainly going to use this as a model for my personal setup. Right now I have a few different hosts running different hypervisors, but would like to consolidate on kube and an iaas tool.

goodpoint 5y ago

Deploying services is 1% of the work. Maintaining them, fixing bugs and bottlenecks is the real works. And you cannot do when you deploy tools worth millions of lines of code. Complexity matters. Those popular products make sense only if you have a 20 engineers is your team or you don't care about reliability.

Gigachad 5y ago
This is downvoted but I think it is true, self hosted services don't get nearly as much love as they need. It takes a huge amount of effort, monitoring, and knowledge on the specific tools to be able to host them securely and detect intrusions. I attempted self hosting for years but decided it ultimately wasn't worth it and that I have very little ability to prevent against attacks or detect that they had happened.
- goodpoint 5y ago
  Not my point. Self-host plenty of stuff but choose *simple* tools, not behemots. Also it's important to use software packaged in distros like Debian in order to receive security updates for many years.
spaniard89277 5y ago
Maybe they want to practice. I installed proxmox and I'm pretty much set up, but my needs are pretty huble, just a couple of VMs and maybe containers in the near future.

Lamad123 5y ago

I unaccidentally remembered Gilfoyle's famed "what do I do?!"

throwaway894345 5y ago

This is great. Would love to see something like this packaged as a “homelab cluster distro”!

erulabs 5y ago

Awesome! hajimari is neat, haven't seen that before. I'm building a startup that's trying to put a good chunk of this in a box for folks to buy and plug-n-play self-hosting (as much as that is possible). Since you're running k3s already, you might checkout our templates at https://kubesail.com/templates https://kubesail.com/templates for more home-hosting apps :)

khuedoan 5y ago
(Repo owner here) The templates look cool, are those Helm charts underneath? Also I believe the tag filter is malfunctioning: https://kubesail.com/templates-by-tag/Media https://kubesail.com/templates-by-tag/Media
- erulabs 5y ago
  Ack, will look at that bug! No, the templates are currently plain YAML with a handle-bars style variable replacement (and a couple other neat tricks). We do support Helm charts tho, I just haven't had time to add them to the "Templates" view. There are loads of good Helm charts here: https://kubesail.com/helm/k8s-at-home https://kubesail.com/helm/k8s-at-home

akkartik 5y ago

The question I have whenever I see something like this is what the upgrade process looks like. How much time do you spend per month upgrading dependencies, how many mailing lists you're subscribed to, etc. When the log4j thing blew up, how long did it take you to gain confidence that you were no longer impacted?

zenlot 5y ago
Doesn't really matter, as long as it's fun. Some people play games, so people enjoy re-configuring their setup (I do enjoy it too). It's not a business / prod env setting.
- akkartik 5y ago
  Yeah, that's totally fine. I ask in case someone has thought about it, so I can learn from them. Doing just the fun parts of self-hosting will not get you to infrastructure.
throwaway894345 5y ago
Probably no worse than the alternatives. Just about everything is going to be easy to upgrade except possibly the Kubernetes masters (and even that isn’t so bad if you can spare a bit of downtime).
- akkartik 5y ago
  For sure no worse than the alternatives. I also don't care about downtime. But I'm looking for something that minimizes the overheads over not self-hosting. I'd love to prove Moxie Marlinspike wrong that "people don't want to run their own servers, and never will." (https://moxie.org/2022/01/07/web3-first-impressions.html https://moxie.org/2022/01/07/web3-first-impressions.html) This is the key bottleneck in getting people to run their own servers.
  - throwaway894345 5y ago
    For sure. There’s not much of an economical argument for self-hosting—if you’re doing it, it’s almost implied that you’re doing it for fun. Although having run a small (but really scaleable c/o k8s) homelab for <$2/month, I’m not sure I’d save much effort versus using some PaaS or cloud provider now that I know what I’m doing. Like I’d need a real load balancer and a few other things so we’re talking < ~$100/month to productionize.

gigel82 5y ago

This needs a companion guide about how to set up the host machine. Which Linux distro to choose, how to set it up / harden it, nftables / firewall, public key login, etc. If anyone has one handy, I'd appreciate a link.

Art9681 5y ago
Several people in this thread stating they are working on solutions to assist others learning modern cloud/devops stacks. I’ll throw my name in the hat too. I have a single node home lab running enterprise cloud/devops software stack. I have been working on a project to deploy on-prem cloud (OpenStack) in disconnected environment. I’ve had to learn how to build and host all of the infrastructure required to do this without the luxury of internet access. I realized the same thing can be simulated very cheap in a modern computer with enough cores and memory to run the stack as a single bare metal host. My build cost $1300 2 years ago but the knowledge gained has paid me back 5 figures worth of raises in my career in 16 months. Having my own personal CloudBox has allowed me to experiment and fail fast. I am ahead in experience and knowledge than the rest of my team as a result. I have a tool they do not. I realize it would be better if we all had a tool like this. So that’s the pitch, a single node “cloud in a box’, the CloudBox for IT students or professionals to learn any aspect of IT. Now I just need lots more time to actually turn my prototype into a product.
indigodaddy 5y ago
Doesn’t the ansible portion do all that? Believe he uses Rocky Linux.
dmayle 5y ago
You are unlikely to find what you're looking for. I'm going to dig into this article a bit, but what you're talking about is inherently hard, and delivers tremendous value to businesses. There are a lot of people doing it for money, and someone builds something that starts to actually work, they get acquired, or they take their project and make it enterprise-y (because that's where the funding is). There's flatcar and k3os and fedora coreos and talos and lokomotive. There are maybe a dozen others as well, but those are the ones I know something about. The real problem is that the orchestration of PXE boot, DHCP, name services, server registration, cluster bootstrapping, while simultaneously building a light distribution that makes the right calls on persistence, security, etc. is just *really hard*. I took at a stab at it myself (failed startup) and have a custom distribution based on alpine, but the amount of work to go from there to everything is so large that it's tough to take on if you're small (and there is the constant desire to go enterprise because of the money)

walrus01 5y ago

my main objection to this is choosing host machines that don't meet the criteria of anything like serious server-grade hardware. if this is for a home lab where any one of the services run on it are not actually going to affect you if it goes belly up? or the whole host machine? sure, okay, but that's self hosting a home lab, not self-hosting actual infrastructure... clearly the hardware shown in the image is meant to be small, not noisy, and not take up a lot of space, and operate in somebody's house. but the people I know who seriously self host all their stuff have it on something like a few old Dell R620 1U servers with 1+1 dual hot swap power supplies, RAID-1 for the operating system boot, RAID-1 or RAID-6 (or something like mdadm raid6) for storage drives (again all hot swap capable), etc. note that I am not talking about a lot of money here, you can buy the hardware I described in the preceding paragraph for $300 on eBay and then add your own choice of SSDs. and not in a house, but in something like a small to medium sized ISP's colocation environment, with UPS, generator backed power, etc. also attached to network equipment and a DIA circuit that's considerably more serious than a tp-link unmanaged switch attached to a residential internet service.

Spooky23 5y ago
All overkill. If you want something more reliable, get some late model HP or Dell thin clients and put Debian on them. Usually you can pick them up for <$75. I setup something like this to control some farm equipment when I was in college 20 years ago. The farmers’ son called me a few months ago because they were having an issue with the equipment. Turned out oily dust coated the thing and got inside - I had him pop the device open (old Wyse box running Slackware iirc) clean up the goop and hit it with rubbing alcohol. He dried it and everything is working again.

divbzero 5y ago

I was happy to see that this project describes self-hosting in the fullest sense, from hardware on up.

ekianjo 5y ago

You don't need Kubernetes to do self-hosting. Completely overkill.

walrus01 5y ago
I would rather choose something like xgp-ng as a hypervisor for a home lab environment such a this, instead of a custom one-off kubernetes environment. Or just a total DIY xen or KVM hypervisor on debian or centos approach. But with considerably more RAM in the host machine than 16GB (like 64-128GB). the kubernetes sort of makes sense if the purpose is also for the person who owns/operates it to test kubernetes things, learn kubernetes, or use it as a home lab of some larger similar environment that they have elsewhere.
- moondev 5y ago
  > Or just a total DIY xen or KVM hypervisor on debian or centos approach. Or just install kubevirt on the existing cluster and manage kvm virtual machines with k8s https://github.com/kubevirt/kubevirt https://github.com/kubevirt/kubevirt
the_duke 5y ago
You don't need it. But Kubernetes exists for a reason and makes a lot of things easier. If you are familiar with the ecosystem, k3s is a great foundation for self-hosted setups. Especially thanks to the many helm charts and operators that exist.
z3t4 5y ago
Talking about overkill, he also use 4 physical machines each with a 4 core CPU. But its not about what you need, its about what you want to learn. So if you want to learn about cluster/orchestrating i think its warranted.
dspillett 5y ago
Overkill probably, but useful practise for when working on something of a scale where it will make a significant difference. And they might just like playing with these things. A home lab does not have to be DayJob practical!

candiddevmike 5y ago

I enjoy reading about homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. This is just another "how I installed kubernetes" thing only without any real scale behind it. There are other ways to deploy and run things, and it takes people toying around with alternatives for something unique to spring up.

Havoc 5y ago
> homelabs that aren't using kubernetes or some other scheduler and how they're doing it/what the full stack looks like. I've had great success with Ansible+Proxmox. The proxmox CLI interface is good enough that you can basically do DIY terraform like setup straight against CLI without any agents / plugins etc. (I know declarative vs imperative but hopefully the comparison makes sense) Lately I've been toying with local gitlab-CI building and deploying images directly into the environment so that any changes to the code redeploy it (to prod...this is home after all haha) Considered terraform too but not a big fan of their mystery 3rd party plugin provider system
oceanplexian 5y ago
Yeah I’m with you. I run a rack at home with servers running a few applications orchestrated with a bit of Puppet, it’s 100x more reliable than K8s, I’m not having to do weird hacks to make persistent storage work, edit dozens of YAML files and run a whole host of dodgy control plane services. Sadly, I leaned this the hard way by setting up a 4-node k8s cluster and realizing bare-metal support is abysmal and the maintainers simply don’t care about you if you’re not running a cluster on AWS or GCP. Unfortunately my day job still involves k8s and people give me strange looks when I suggest metal so substantially less complex, cheaper, more performant, and easier to orchestrate and maintain.
- movedx 5y ago
  I'm building my own education platform around getting people into IT and towards a career in Cloud and DevOps. A few months back I built a Nomad based cluster with Consul and was like: wow! This is super cool... but it's also expensive and complex. I'm going to be building out the platform with an ALB, two EC2 Instances in an AutoScaling Group based on an AMI I bake with Ansible and Packer. I'll use Terraform for the entire infrastructure. The release pipeline for content updates (it's an MkDocs/Material based book with videos blended in) will be me updating the AMI and firing off a `terraform apply`. I won't be using Docker. I won't require or need an orchestration engine. It's a static website delivered via a custom web server written in Go (as I need custom "Have you paid to view this content?" logic.) That's about as complex as it gets for now. I'm actually looking forward to implementing it and watching as something so simple hopefully goes on to allow me to build and scale a business.

khuedoan 5y ago

Repo owner here, I just created this account, I'm a long time HN lurker. I was surprised to find this on Hacker News, I wanted to wait until the stable release before posting on HN, but thank you for posting :) This project is still in alpha stage, but please feel free to critique; I'd appreciate it. Edit 1: After reading some of the comments, I want to clarify a few things: - Because it is currently in the alpha stage, I do not host anything important on it. - This is also my learning environment, I use Kubernetes in my day job. - Yes it is overkill ;) Edit 2: Wording

sandGorgon 5y ago
do you use k3s in production as well ?
- khuedoan 5y ago
  Is production == my day job? If so we don't, we use managed Kubernetes (we used to manage our own clusters but they will be decommissioned soon).
  - coopreme 5y ago
    Can you go into the reasons you went to managed vs self owned K8S?
jmakov 5y ago
Thanks for the project! Once deployed, how do you keep everything up to date?
- khuedoan 5y ago
  I'm currently bumping the versions manually (via a commit), but I plan to automate that with system upgrade controller [1] and Dependabot [2] (or similar) [1]: https://github.com/rancher/system-upgrade-controller https://github.com/rancher/system-upgrade-controller [2]: https://github.com/dependabot https://github.com/dependabot
5y ago

deleted 5y ago

[deleted]

mikesabbagh 5y ago

it amazes me that hacker news run on a single node. All this work is nice and beautiful, the problem will come when you try to update different components.

christophilus 5y ago
> it amazes me that hacker news run on a single node. It amazes me how much of the internet doesn’t run on a single node, when it probably could.
- jaimex2 5y ago
  Why would anyone run a simple elegant site when you can get buried in js dependencies and plumbing stacks for months?

qbasic_forever 5y ago

Very cool, this is like a perfect little kubernetes development environment. The combo of tekton + gitea + k8s is really nice and just a shame it takes bodging all the lego pieces together manually right now. I wish there were a one click exe install like Docker for Desktop that gave people an entire environment like this on their desktop. Like a little private github but so much more powerful. If you're looking for some nice stuff to develop on an environment like this, check out VS Code and Google's cloud code addon: https://marketplace.visualstudio.com/items?itemName=GoogleCloudTools.cloudcode https://marketplace.visualstudio.com/items?itemName=GoogleCl... It's not totally tied to their GCE cloud offerings and is actually just a VS Code integration of their Skaffold tool (a fantastic inner dev loop tool for k8s). It works great with minikube or any other k8s cluster, presumably this one too. You can very quickly get up and running with code in your IDE now running and debugging on your k8s environment.

khuedoan 5y ago
(Repo owner here) Glad someone mentioned it, I do have one: https://homelab.khuedoan.com/try_on_a_vm https://homelab.khuedoan.com/try_on_a_vm Although it's still incomplete and not one click yet, that's the direction I'm heading in: anyone can try my homelab on their PC, and if they want to use it on their real hardware, they can continue with the full install.

mrslave 5y ago

I've been eyeing off OpenStack for sometime for a similar use case, but I must admit it is an uninformed opinion. What am I missing between the two solutions (OpenStack vs this specific custom solution)?

cyfex 5y ago

I always found the hurdle with self-hosting to be maintaining, not the initial setup. Things like upgrading, in order to keep getting security fixes, and verifying everything works after the upgrade, are what has taken the most effort and time in the past for me. This looks like a great setup by the author, but difficult to maintain in the long run without significant time investment.

input_sh 5y ago
Docker and docker-compose make maintaining stuff dead simple to me. In like 95% of the cases it's just a matter of changing the version in a YAML file and running one command, and in the remaining 5% of the time I'm not concerned at all with some downtime because I'm (mostly) the only user.
- BrandoElFollito 5y ago
  This. I used to host directly on my server and discovered docker a few years ago. It was a relief. Coupled with watchtower the updates are automatic except for the two services I really rely upon. I used to be on Ubuntu for ages and moved to Arch a few days ago - my server only runs docker today. I still want to keep access to a shell do it is Arch and not Rancher or something (I did not research much the bare metal hypervisors). My maintenance is minutes every month if everything works fine, up to a max of an hour when Home Assistant broke things once two years ago. DRP from bare metal is an hour. Adding a service is a few minutes.
- toshk 5y ago
  What do you use for serving docker-compose?
  - g_p 5y ago
    Docker-compose doesn't need "served" per-se - it lets you run some containers, and handles some lightweight orchestration around them, so that you have groups of containers for a service (i.e. a Wordpress service compromises a mariadb container and a Wordpress container). The compose file handles port bindings/mappings. If you then want to put a reverse proxy in front of the docker containers (you almost invariably will), then you can look at different options like caddy, Traefik, nginx etc. I, for one, like to be old-fashioned and have my docker containers' ports bound to localhost, then manually maintain my own "outside of docker" instance of nginx as a reverse proxy that uses these as upstreams. That's not the most "container-first" way of working, but it worked for me. Caddy can do similar. Traefik is more integrated with docker and the docker ecosystem, but that might do what you need better.

kstenerud 5y ago

This is definitely cool, but it also highlights a huge problem we have with software nowadays. There are 19 stacks in this repository. 19 pieces of software that require their own maintenance, JUST TO RUN YOUR APPLICATIONS! The amount of extra work required just to host the software that views your pictures, plays your videos, and allows chat with people is absolutely insane. I'm not talking about this particular repo; it's just indicative of how complicated things have become that you must do the equivalent of building and maintaining your own operating system just to get even simple things done. And I belive that it's unnecessarily complicated (once again, not this repo, but rather the state of affairs in general). We're at an inflection point in the industry, and haven't come out the other side yet.

legends2k 5y ago
Isn't it how the general software philosophy is? A composition of decoupled, do-one-thing-well pieces working together to achieve a larger task?
- kstenerud 5y ago
  Yes, but after the initial monoliths of mainframes, we had a golden era of operating systems, where all of the plumbing and infrastructure to run your applications was maintained by someone else (Microsoft, Apple, Commodore, Atari, SGI, Sun, Palm, Redhat, etc). Now we've come full circle back to the bad old days where you need an entire team of dedicated people and arcane knowledge just to run your application software again. I suspect that this will continue until we reach a point where the big players out of necessity come to an agreement for a kind of "distributed POSIX" (I mean in spirit, not actual POSIX). These are exciting times living on the edge of a paradigm shift, but also frustrating!
  - theK 5y ago
    That’s actually very insightful. The OS era you describe was mostly local with not so many things happening in online interactions and way less devices collaborating to create for example your images folder. Now that we mostly have the server-client model ingrained in every online activity there is an obvious complexity centralisation on the server side. Might be interesting to see how web3 and other decentralised web movements might factor in to this!

jaimex2 5y ago

I miss LAMP :(

midasuni 5y ago
I use LAMP. WHy would I miss it?
rambambram 5y ago
It's still there, also waiting for you my friend.

surfsvammel 5y ago

How much energy does this consume? I have done some home lab going on over the years, and find that trying to do more on less W has been the most fun.

khuedoan 5y ago
(Author here) Yep that's my goal too, hence the small form factor PCs. It costs me around 2-5$/month depending on how much I play with it.

nyellin 5y ago

To everyone saying that Kubernetes is unnecessary, try implementing autoscaling, service discovery, secrets management, and autohealing in a vendor independent way without it. Of course none of that is necessary for a self hosted home lab, but neither is gitops. This is a very nice example of how to set stuff up properly. OP, I would love to see Robusta (https://robusta.dev https://robusta.dev) as part of this too. It's definitely in line with your vision of automating everything, as it let's you automate the response to alerts and other events in your cluster. (Disclaimer: I'm one of the maintainers)

dijit 5y ago
Oh. I’ve actually done that. It’s not as hard or complicated as you claim. This was before kubernetes, arguably learning kubernetes is harder than implementing auto discovery and auto healing because you have to learn a lot about cluster semantics, the various network providers, service accounts, role based access control, integrations with secrets manager or the very disparate implementations of custom operators which may operate in your environment. Kubernetes is a platform, I don’t doubt it solves a lot of problems. But it’s a complicated wee beastie, and I say that as a person who did it the easy way and used GKE on Google Cloud
- nyellin 5y ago
  I don't know how you implemented it, but the big innovation of kubernetes was doing this with declarative inputs and using control loops for reconciliation instead of edge triggered changes. It might not matter on a small scale, but on a large scale that's what makes it robust. It's also not trivial to implement a distributed version of this in house at scale. At the very least you need something like etcd at the core
  - dijit 5y ago
    The way we did it is that we registered to a centralised “name” service which was an in memory database. Failure to register to the name service leads to registration towards the next name service in the list, we ran three- so there were two spares, the library would then keep trying to register to the first name service, this would be our reconciliation loop. When you request a service you request “traits” through the name service. Not “trivial” but even though it was implemented in C++ on windows it was much less complicated than kubernetes discovery systems, which there are many ways to integrate (env-var, dns, api) and some should or must be disabled for security reasons (eg. The environment variables that display all services) or can change depending on RBAC and service account. We deployed about 300,000 or so services with this infrastructure and it’s been running in production since 2015. (Original product release was supposed to be 2013 but due to the client changes it was delayed, not due to the backend. This is Video Games unfortunately). I’m not like, “hard” on these opinions, but people talk about kubernetes as if it’s the only way of solving these problems, when it can be the case that you have to learn a lot about kubernetes before you can actually do what you need to do; if you already understand distributed systems then it’s a bigger effort to understand kubernetes than would be to implement a distributed system that solves your needs from scratch

AdrianoKF 5y ago

I recently wanted to give my current RPi4 home server a GitOps makeover, migrating services (Home Assistant, Nextcloud and others) to k3s in the process. What has been an obstacle is the availability of officially maintained Docker images for some of the components I've been wanting to use - afaict neither Argo CD nor Rook have official armv7/aarch64 images (though it seems Argo will release one soon). Until then, I'll hold off on that pet project until I get my hands on a reasonably priced x86 SFF PC (the ThinkCentre M700 Tiny from TFA looks interesting!).

yebyen 5y ago
FWIW, FluxCD has full armv7 and aarch64 support (it definitely works on a Raspberry Pi 4, the earlier models with 1GB of RAM too, although depending on the size of your git repos that may not be enough memory, 4GB-8GB certainly is for Flux.) How many Rpi4 servers are we talking? I have never run Rook before (though I ran Ceph, when it was called deis-store way back before it was cool) and I always remember that you needed at least 3-5 servers with at least one dedicated disk for OSD per each to make it work well, if at all! I'm looking at synology-csi right now for my homelab, and wondering if there is hope that I can stop using local-path-storage this way, it looks a little bit outdated, so I'm afraid that may keep me from running the latest version of Kubernetes on it... unless I hack the support into it myself :gasp: surely that's gonna void the warranty
- yebyen 5y ago
  (I did wind up getting synology-csi to work!)
3np 5y ago
For various reasons I started building all container images myself. In most cases when arm64 images aren't provided, they will build just fine anyway. Running you own registry is super simple. https://docs.docker.com/registry/deploying/ https://docs.docker.com/registry/deploying/

iamgopal 5y ago

Is there similar repo for SAAS apps ? Not too fancy, just Django and a db.

comprev 5y ago

I use something very bare bones as the host OS, say Alpine, and install Docker with a handful of other tools for basic sysadmin tasks (curl, etc.). For a single server deployment docker-compose is very useful.

gorgoiler 5y ago

I wonder how the fan noise is on those M700s. These little Lenovo machines are very pleasant. My only wish with my M93 is that the temperature would stay below 50’C and keep it 100% silent.

nodesocket 5y ago

What do you use to provision Kubernetes persistent volumes on bare metal? I’m looking at open-ebs (https://openebs.io/ https://openebs.io/). Also, when you bump the image tag in a git commit for a given helm chart, how does that get deployed? Is it automatic, or do you manually run helm upgrade commands?

khuedoan 5y ago
I'm currently using Longhorn for storage, but if I find some reasonably priced HDDs, I may add or switch to Rook. When you make a change in git, it is automatically deployed without the need for human intervention.
- nodesocket 5y ago
  Can you elaborate a bit more on how the automatic helm upgrades are performed?