11 ms·
Leveling Up My Homelab
- bediger4000 1y agoThat looks and sounds great. Good for you!
- Arn_Thor 1y agoThere are certain tells when anyone with a homelab starts making real money off the work they do on it
- timwis 1y agoWow, a beast. Looking forward to the next post!
- globular-toast 1y agoSeeing the final picture made me think of something: one of my "life hacks" is to not accept cables that are too long. I used to think the longer the better, and just coil it up or something. But long cables just get in the way and collect dust etc. If something is going to be considered permanent, cut that cable down to length. Either buy shorter moulded cables, or learn to wire cables yourself. Too often have I left a cable long "just in case" only for it to get in the way for years to come. For patch cables it's easiest and best to buy moulded cables that fit your rack. For things like power cables (extension leads etc.) it's easiest to wire them yourself (at least, in the UK where our plugs are designed to be wired by anyone).
- wltr 1y agoOn the other hand, I did this, cut my cables, and after I needed to reorganise things slightly, it was very difficult, even a centimetre was at luxury. Also, when I need to move a computer for some reason, there’s no room, at all. These days, I’m trying to leave at least some extra cm (usually that’s an inch or two, depending on their location) for that. I’d do very tight cable cut only for situations when I’m super sure nothing will ever move. Again, even then, I’d rather leave some extra inch, just in case.
- cweagans 1y agoI am very much on board with this line of thinking. Because things are still somewhat in flux, it was much easier to plan for excess cabling and have a place for that cabling to live in the rack so that things can be moved if needed. I'll probably re-cable it with cut-to-length cables in the future. One thing that I haven't found a solution for though: I have a lot of USB and HDMI cable coiled up behind the Beelink boxes (for KVM connectivity). I've found the normal length cables (1', 3', 6', etc), but I haven't been able to find custom length cables for those specific connections. Do you happen to know anywhere I can find those?
- bkummel 1y agoIf you ask me, this is not a Homelab anymore.
- ocharles 1y agoI'm curious how much this costs to run. I.e., how much are you paying for electricity?
- chasely 1y agoNot sure about this one in specific but assuming most of the time the system is idle at ~200W, you’d be looking at ~$25/mo for most states in the US. Peak draw could probably be 2kW for a beefy system so electricity costs could really skyrocket depending on usage patterns.
- cweagans 1y agoPeak draw for the entire cabinet with everything running full bore cannot/will not exceed 1800W (by design).
- cweagans 1y agoI'm currently sitting at < 200W, but I expect that to go up with a higher workload. The SER9s idle at 5-7W, but they can run at 50-60W sustained without thermal throttling. Some reviewers have claimed that they can run at 75-80W sustained for 10-15 minutes, but I think that's pretty unlikely. We have pretty reasonable power rates here (https://www.idahopower.com/accounts-service/understand-your-bill/pricing/idaho-pricing/for-your-home/ https://www.idahopower.com/accounts-service/understand-your-...), so ~$12-20 per month depending on tier.
- parkuman 1y agoI love the astronomical jump in compute. Mac Mini and mini PC not enough? GET 8. Love it! I’m in the middle of my first homelab journey with just a single mini PC with 8GB of RAM. Excited to one day get to this level.
- wltr 1y agoYeah, I started with an obsolete PC a decade ago, and now I’m on a Raspberry Pi 2 and a couple of Orange Pis (zero, 1st gen). Weirdly, it’s enough for me now. I self-host not too many things as of now. So I guess I’d grow at least a couple of Mac minis over time too. However, I’m trying to go slowly there. Funny thing is that despite me admitting that I’m more likely to arrive there too, each time I see a superlab, I wonder ‘what do you guys do with all this?!’
- louwrentius 1y agoIt’s always fun to see how people build their home labs but I went exactly the opposite route. I focused on energy consumption, because of cost and - gasp - wanting to be mindful about it given the current predicament. Anything that needs to be on 24/7 is on a Pi, and anything that consumes more power is turned off by default (remote poweron possible). For me at home there is zero need for redundancy and I use a cluster of four tini-mini-micro 1L PCs for my lab work. There are also turned off by default and are also low-power.
- bionsystem 1y agoThat's what I want to do as well, Pis running a vpn and some wake-on-lan scripts for a couple of old boxes (I will probably run SmartOS on those). Anything to share about your setup ? (blogs/gits that you wrote or followed) I also got some cloud credits from my employer, but a bit paranoïd about putting my data there (although most of it isn't sensitive).
- louwrentius 1y agoNo links but it’s just a Pi5 as a Linux router and three Pi4s for grafana (iot stuff), jellyfin and hosting my blog on solar power.
- baq 1y agoIf you’re thinking about rpi5, reconsider an n100 minipc. Rpi5 is surprisingly expensive, both upfront and tco.
- louwrentius 1y agoI wrote a HN hit about something similar https://louwrentius.com/the-raspberry-pi-5-is-no-match-for-a-tini-mini-micro-pc.html https://louwrentius.com/the-raspberry-pi-5-is-no-match-for-a...
- g-clef 1y agoIt's funny, I started with rPis for the same reason, but I'm about to replace them. I bought 20 rPi 4Bs for my homelab, and I just couldn't get them to do what I needed. I was looking to run a home k8s cluster and the Pis were just not suited to it at all (don't use sd storage for k8s 'cause it'll burn out the card w/writes, booting off usb was unstable even with powered usb hubs, netboot turned into an enormous pain in the neck).
- grim_io 1y agoThis exceeds the computing requirements of most IT related companies, by far. Cool, but nothing a single compute machine wouldn't get done with a bunch of VM's if learning and home workload are the focus. This thing probably idles at a couple hundred watts.
- spicyusername 1y agoYea, but, like owning a car you soup up in your garage, it isn't about what you _need_, it's about what's fun and what's enough to give you something to do in your free time.
- grim_io 1y agoAll good, as long as there is self-awareness that it's not built on a "need", but on a "want" basis. It should be obvious to the reader that this is very much overkill, even for the stated goals of expandability and learning.
- cweagans 1y agoHow do you know if it's overkill without knowing anything about what I am/will be running on it? Certainly, there are projects for which this setup might be appropriately sized, wouldn't you think? Incidentally, this setup does leave me some significant headroom in terms of compute resources, but that's by design.
- grim_io 1y agoPeople don't usually buy this much compute for running a mail server or a media transcoder, so... probably AI? :) If it's inference, don't expect great performance or cost-effectiveness. But if you learn a lot during it, I wish you all the best!
- Aurornis 1y agoIf we're going with the car analogy, this is like buy 8 Miatas to keep in your driveway instead of combing all of the money into a single much faster car. If the goal is to have a lot of something so you can play with many different things, this gets the job done. If the goal is high performance and maximum efficiency for learning or compute, a setup with dozens of smaller computers like this is usually not the optimal choice.
- buran77 1y ago> Storage UniFi UNAS PRO: (replacing the Synology NAS) 7x 8TB Seagate IronWolf With the way Ubiquity has treated their software stack on the network side in the past years (major bugs, regressions, and updates that had to be reissued multiple times), I wouldn't trust them with all my data. Ubiquiti's QA was outsourced to the customers and a NAS is the last place where I want to risk bad updates, no matter how many backups I have.
- tecleandor 1y agoSame with Synology. I would directly jump to a TrueNAS install, or a BSD/Linux install with all the needed software.
- buran77 1y agoSynology has shady business practices more recently and outdated tech stack since forever but I haven't heard anything particularly bad about reliability. For a NAS the safety of the data is the highest priority. Anything that endangers the data isn't just a drawback, or a "con", it's an instant elimination. Right now I wouldn't trust Ubiquiti NAS to store my recycle bin, I need to see a long track record of reliability and a commitment to quality.
- cweagans 1y agoThis is definitely one of the purchasing decisions that I regret. My backups are robust and trustworthy enough that I don't have data loss concerns, but the software is atrocious and the customizability is extremely limited. e.g. I wanted to serve tftp directly from the NAS. I can log in and `apt install tftpd-hpa`, but that package has to be reinstalled every time the NAS updates. I'll be replacing this in the medium term, but I'm not buying more hardware for a little while lol
- NoiseBert69 1y agoI down-leveled my homelab due to energy costs. It now only consists of a Intel n100 with a big SSD and 32GB RAM running Proxmox. These China TopTon-boxed with their 5x Intel i226-IV network cards are great and can be passively cooled. Every night the Proxmox makes a backup onto a Raspberry Pi which runs the Proxmox Backup Server.
- mindrunner 1y agothis. the energy costs have made me downscale to micro pcs
- sgc 1y agoIs that also running a router like opnsense, or do you have a dedicated box for that? Curious about specs for networking gear...
- NoiseBert69 1y agoShabby 10 year old Fritzbox is my home LAN. It plays DHCP-Server, VDSL-Modem and Firewall. Next step will be most likekly a VDSL-modem + something from Ubiquiti as the new Fritz! product portfolio is... a weird mess.
- n4bz0r 1y agoI've read that PBS requires fast (NVME-fast) storage and a decent CPU to handle incremental backups efficiently. What speeds do you get when restoring backups?
- mysteria 1y agoHDDs work fine for me with PBS, and I get regular 120mb/s HDD speeds when restoring backups. Honestly how often do you restore backups anyways?
- prmoustache 1y agoWell you should do that frequently, at least to make sure you have restorable ones.
- MASNeo 1y agoThis is awesome and I wish more of this were happening. Hardware home labs are the best way to learn. I gained most of my Linux/FreeBSD skills at home. It feels with cloud computing a generation of computer scientists kind of missed out on the experience.
- tietjens 1y agoCloud by day, home lab by night.
- tietjens 1y agoEnjoyed this. Lots of large homelabs like this are built in order to stream video or local llm models and that usually leaves me feeling a bit left out because I have been building my own, but have no interest in either of those things. Some services I am interested in are hosting my own RSS feed reader, an ebook library, and a password manager. But I'm always looking for more if there are any suggestions.
- danparsonson 1y agoWell there's loads - Nextcloud/Syncthing to replace Dropbox, Forgejo to replace Github, Joplin on the back of Nextcloud or similar for note taking.... personal wiki, todo list, email with webmail (Roundcube), home video camera management, etc etc Check out https://github.com/awesome-selfhosted/awesome-selfhosted https://github.com/awesome-selfhosted/awesome-selfhosted
- marcuskaz 1y ago> I am interested in are hosting my own RSS feed reader, an ebook library, and a password manager You can do that on a Raspberry Pi Zero for $15, and for $12 you can get a 128gb microsd card, plenty of storage. It'll take up minimal power and fit in an Altoid tin.
- rcarmo 1y agoYou can host all of that on an N100 mini-PC in oh... around 4GB RAM, based on the equivalent footprint of the Docker/LXC combos I use for those.
- Mistletoe 1y agoWhat do people do with homelabs? I’ve always wondered. Is it anything of consequence or is it just like making a roadster in your garage?
- import 1y agoI have two machines and NAS. Running around 50 docker containers. Including home assistant, adguard, wireguard, Vscode server, rss reader, change detection, romm, metube,vaultwarden, immich, caldav carddav server and bunch of custom applications I am working on. (Also hosting some websites). It’s so much fun and helps me to own my data.
- ivanjermakov 1y agoI run a home server but it is a single post-lease Dell mini PC I got for $50. The only times it goes above 5% CPU/GPU usage is when I'm building a project to deploy or transcode video. I'm sure nobody needs this much compute for personal use (24/7), so roadster in a garage is a good analogy.
- deleted 1y ago[deleted]
- tennysont 1y agoFor those not keeping count, total hardware spend is in the 13k-20k USD ballpark, by my count. The thing that I like about this post is that it touches on many of the difficulties of running a homelab with many physical hosts. You might not need all or most of this setup, but at least you have an idea of how this particular design (a decent one!) scales after reading this. - Array of off-the-shell compute servers with console access + networking + power - ArgoCD + GitOps makes a K8 cluster declarative - Talos makes the physical hosts that provide the K8 cluster be declarative - Dedicated machines for storage, Control Plan, and Networking isolate the infrequently-changing stateful parts This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab. I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding) A single machine with a lot of CPU and a LOT of memory/storage is typically what people want---so that projects they're setting up are fast and having lots of old/idling projects is fine. My old recommendation was a mini-ITX with 128 GB of ram and a modern AMD cpu should take most people very far. Perhaps a smaller NUC/Beelink computer if you're not storage hungry. However, keep in mind that a single machine will make it hard to tinker with large parts of the stack. It's harder to play with kernel mod settings if you need to constantly reboot all your projects and possibly nuke your lab. It's harder to test podman vs docker if in involves turning off all your contains. A more complicated home-lab gives you more surface area to tinker with. That's both more fun and makes you a better engineer. Of course, you can get most of this experience for far less money if you budget isn't quite as generous. I personally prefer a digital nomad aesthetic, so I focus on small & simple on-prem hosts paired with cloud stacks. I'm willing to pay a premium on compute to have less commitment and physical footprint. I've been considering setting up a K8 cluster on Hetzner dedicated machines. In my case, that Mini-ITX box is actually a storage-optimized ATX build for backing up my laptop (daily-driver) and media server.
- rcarmo 1y agoThe SER 9 workers in particular feel like they're overly beefy. I run 4-5VMs inside Proxmox in a single equivalent box...
- 1y ago
- thelastgallon 1y agoI'd like to replicate this (a smaller version) but with ECC RAM. It is hard to find mini PCs (or laptops) with ECC RAM support.
- fmajid 1y agoHP and Lenovo have SFF or UCFF workstations with ECC support like the Z2 Mini G9 or P3 Ultra SFF.
- wltr 1y agoSerious question: why would you need one for a home usage? I mean, you can easily reboot it nightly, and nobody will ever notice. Unless your personal server is used by people across the continents, of course. But even then, my SBC reboots within 20 seconds or something. Not really noticeable.
- Havoc 1y agoFancy & pretty high end! Confused as to why 10x nodes but running single control node and no HA? Control nodes can be pretty light - even down to raspberry 4s
- cweagans 1y agoSoon :)
- mysteria 1y agoIt might be fun to have so many machines, but in reality it's simpler and cheaper to virtualize everything on two or three powerful hosts. Considering that you're using a soundproof rack already you might as well go with used rack servers with lots of memory and compute. Those also come with goodies like BMCs, dual PSUs, and ECC. Personally I have two Xeon rack servers running in a Proxmox cluster with a SBC Qdevice. It has more than enough memory and compute for my needs and it also serves as my virtualized router and NAS. The whole setup only takes up 4U of space (servers + switch + modem/qdevice) with a single UPS on the floor, and idle power is around 150W.
- cweagans 1y agoWe may be optimizing for different needs. For instance, while I was able to get a significant amount of extra height, I didn't have a lot of cabinet depth to work with, which is somewhat limiting for traditional server hardware. There are short-depth options out there, but I also wanted at lease some GPU capability. The integrated GPUs in the SER9s are not top of the line by any means, but they're more than capable for what I want to be working on.
- master_crab 1y agoDid i read one control plane node and 8x workers for the K8s cluster? What happens when the control plane flakes? I run 4 thin clients and only one of them is a worker. The rest are untainted control plane nodes in HA.
- cweagans 1y agoYou read that right. Currently, the DR plan is "replace/repair the machine and bring everything back up from a backup". It's not a good plan, but it's also only the short term plan. Longer term, I'll likely add another control plane node or two.
- master_crab 1y agoRun three. Helps with quorum.
- PeterStuer 1y agoHonestly, I have had 'has a homelab' as one of my mental checkboxes for great hires for a long time.
- nathan_douglas 1y agoOh hai Cameron! I'm rebuilding my homelab [1] too, actually, but deprioritizing it while I stave off a wee spot o' burnout. I'm excited to see that you're building on Talos; I am too! I used to use CoreOS back in the day, like 8-9 years ago or smth, on PXE-booted VMWare VMs, and I've always missed that cleanliness. That's a large part of why I'm rebuilding right now - I based everything around Ansible + Terraform, and that's workable of course but working iteratively on a homelab leaves so much cruft around, can lead to incidental complexity, etc etc etc. Anyway, I'm pumped to keep reading! [1] https://clog.goldentooth.net/ https://clog.goldentooth.net/
- cweagans 1y agoHey hey! Good to see you here! Talos is absolutely incredible. There's a learning curve to it, but it's not as steep as it seems. I started with Ansible, but found myself getting really annoyed at the amount of YAML I was shoveling around for basic things, so I ended up writing a series of bash scripts that rsync files out to where they need to go, run actions when a particular file changes, etc. Provisioning scripts are bundled up and piped to the remote machine over SSH. It's been pretty nice. I'm thinking about building that out into a separate project somewhere. I'd love to check out what you're working on! The link seems to be broken though.
- nathan_douglas 1y agoTalos (and Talhelper) seem pretty reasonable so far. Digging Sops too for managing secrets. I was using Ansible Vault before which worked but was weirdly cumbersome to automate (go figure), and Sops seems to Just Work™. > The link seems to be broken though. Yeah, I'm a world-class infra engineer. smdh. Changed how the DNS record was created but didn't push my changes so they were reverted by a scheduled job facepalm Think it's back now...
- cweagans 1y ago> Digging Sops too for managing secrets. I was using Ansible Vault before which worked but was weirdly cumbersome to automate (go figure), and Sops seems to Just Work™. I considered both of those and ended up using the External Secrets Operator + 1Password for my secrets. Maybe not the _best_ solution, but it saved a fair amount of effort on my part. > Think it's back now... It's back! Looking forward to reading!
- evanreichard 1y agoThis year I redid my whole lab as well and prioritized declarative everything so it'd be easy to go from scratch if needed. So I've got nix to create RKE2 images with cloud-init for XCP-ng templates, Terraform with XenOrchestra provider, and ArgoCD for the K8s cluster. I've got two R640's so I can live migrate, and an R720XD with TrueNAS (democratic-csi for K8s persistence). QSFP (40Gb) for TrueNAS / R720XD, and SFP+ (10Gb) for R640's linked to a Brocade ICX 6610. So I can update the hosts, and K8 nodes with 0 downtime. Do I need it? No, but I learned a lot and had / have fun deploying and maintaining it.
- 000ooo000 1y agoHow are you approaching heat management inside the soundproof closet?