12 ms·

Overengineering my homelab so I don't pay cloud providers

253 points by JNRowe 1y ago

tehlike 1y ago
I moved my side project to my garage so I don't have to pay hetzner 600+$ and counting every month
- tracker1 1y ago
  I've got a single server and a /28 IP block on OVH for public facing stuff. Mostly because it's cheaper than the bump from my "home" internet to "business" to be able to use common server ports (blocked on the home service). Works well enough for what I need.
  - tehlike 1y ago
    I use cloudflare tunnel to do most of this, and i am fine with the implications (like CF mitm'ing my website traffic).
- lazyweb 1y ago
  I did the exact opposite. And by that I mean physically moved my homelab into their colo earlier this year. Runs like a charm, costs about 500€ per month total. Sounds like a lot, but I was almost paying the same before - 220€ for power at home, 110€ for a dedicated Hetzner server, 95€ for a secondary internet connection (as not to interfere with the main uplink used for home office by my partner and me). Not having to deal with the extra heat, noise and used up space at home anymore has been worth it as well.
  - tehlike 1y ago
    My storage needs were increasing by the day. Electricity is now a small monthly cost. I have more cores and ram than ever, and can easily expand it. Main machine now runs with 1TB ram and 15TB SSD and other has more than 384G ram. I currently use 3TB ssd storage, and get way more performnce than Hetzner's VMs with ceph ssd disks. I do need redundancy, but it's not something hetzner was giving me anyway, and if my anectode is not a mess up, i actually got database corruption on hetzner that never happened on my own local setup. I'd have colo'ed or used dedicated as it's definitely better than their VMs, but they don't have that in their US datacenters. I am pretty happy with my current setup, I have significantly less down time (few mins a month) than when I was on hetzner - but this is mostly due to my need for more ram at times. I also used this as an excuse to get 56G mellanox fiber switch and get poe cameras etc in a full homelab manner, so it's been fun, on top of being cheaper. Noise is not a concern, I got a sound-proof server rack that's pretty nice. It takes up space, but i have kids, so my garage is near full at times anyways :)
treve 1y ago
If anyone is looking to get started with a homelab at a good price, I can highly recommend checking ebay for a Dell Wyse 5070. They flooded the market for $50 and are likely powerful enough for many needs. They have a M.2 slot that support SATA. The 'extended' version also has space for a small pcie card and has a parallel and 2 serial ports for a blast to the past.
- senectus1 1y ago
  curious to know what you would use this for?
  - denkmoon 1y ago
    local dns, static site hosting, local apt cache, various other network services (unifi controller if you've got those APs for example), remote/headless dev machine (maybe not for kernel or bigcorp java development), or whatever else you want. mail if you want. Anything :) Those little thin clients aren't gonna be fast doing "big" things, but serving up a few dns packets or whatever to your local network is easy work and pretty useful.
  - treve 1y ago
    I use it for media hosting. Backups (connected USB disks), home assistant, syncthing
  - rwyinuse 1y ago
    Even these low-power CPU's are surprisingly capable. As an example of more fancy thing, one could slap in some external storage, install Jellyfin, and run their own local streaming service off such a machine. The CPU is modern enough for efficient hardware transcoding of a stream.
- pandemic_region 1y ago
  What if power consumption is taken into account? Are there any devices in that category that are ok to leave on 24/7 ?
  - 1y ago
czhu12 1y ago
This was exactly a use case I had in mind when building https://canine.sh https://canine.sh -- also uses k3s as a provider, and provides a Heroku-like devex. How to actually reliably expose a homelab to the broader internet is a little tricky, cloudflare tunnels mostly does the trick but can only expose one port at a time, so the set up is somewhat annoying
- BLKNSLVR 1y ago
  I've got basically raw internet coming in to my OPNSense device, although I had to request certain ports to be removed from the ISP's by-default-blocked policy, since I host a mail server - but the ISP is fine with this, they have a form for it, super easy. Some family members are behind CGNAT, and I'm not sure if their ISP has the option to move out from behind that, but since they don't self-host it's probably slightly more secure from outside probes. We're still able to privately share communications via my VPN hub to which they connect, which allows me to remotely troubleshoot minor issues. I haven't looked into cloudflare tunnels, but haven't felt the need.
- vladvasiliu 1y ago
  What do you mean by "one port at a time"? I run cloudflared on one machine, and it proxies one subdomain to one port, and another to a unix socket (could have been a second port, no pb).
timc3 1y ago
The encryption is interesting but I wouldn’t call this over engineered at all, in fact it’s rather basic compared to a lot of homelabs I see people building particularly where people are doing K8s or similar over multiple machines. Also Proxmox was called out as the only choice when that is very much not the case. It is a good choice for sure, but there are others.
BLKNSLVR 1y ago
> But my server could be shut down because of a power outage or another reason. I might be at work or even on holidays when it happens, and even wireguard can’t solve this. A 'power outage' incident doesn't seem to have been mitigated. My homelab has had evolving mitigations: I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime, which got replaced by a dedicated inverter/charger/transfer-switch attached to a big-ass AGM caravan battery (which on a couple of occasions powered through two-to-three hour power outages), and has now been replaced with these recent LiFePo4 battery power station thingies. Of course, it's only a homelab, there's nothing critically important that I'm hosting, but that's not the dang point, I want to beat most of "the things", and I don't like having to check that everything has rebooted properly after a minor power fluctuation (I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet).
- qwertox 1y ago
  > I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime Can you share more about this? I have a APC Back UPS PRO USV 1500VA (BR1500G-GR) and it would be nice to know if this is possible with that one as well.
  - BLKNSLVR 1y ago
    That UPS eventually died, and I'm not sure if it was because it was hooked up to a larger battery than it was designed for, but it's still only 12 volts so I don't think the electronics would notice. What they may notice is extended run-time in the event of a power failure. It was a crude mod. Take the cover off and remove the existing little security alarm battery, use tin snips to cut a hole in the side of the metal UPS cover (this was challenging, it was relatively thick metal, I'd recommend using an angle grinder in an appropriately safe environment far away from the internals of the UPS), and feed the battery cables out through the hole. I probably got some additional cables with appropriately sized terminations to effectively extend the short existing ones (since they were only designed to be used within the device). And then connect it up to a car battery. Cover any exposed metal on the connectors with that shrink rubber tubing or electrical tape. Be very careful with exposed metal around it anywhere, especially touching the RED POSITIVE pole of the battery. Get a battery box - I got one for the big-ass AGM battery. Test it out on a laptop that's had it's battery removed or disconnected that, just in case, you don't care too much about losing. Get a battery charger that can revive a flat battery, and do a full refresh/renew charge on the car battery once a year or after it's had to push through a power outage that may have used more than a few percent of its capacity. Personally, I think it's safer a less hassle to go for a LiFePo4 (LFP) Power Station style device that has UPS capabilities. LFP batteries have 3,000-ish cycle lifetimes, which could be nearly ten years with daily use.
tietjens 1y ago
This is such a great post. I have a small collection of posts for inspiration in creating my homelab and this is getting added to it. Current have a Pi 4 with PiHole and a Beelink. Going to add one or two more machines.
MaKey 1y ago
Your VPS provider likely uses servers with ECC RAM, this home server doesn't. For most people it doesn't seem to matter but for me it does - a home server where I store my data needs to have ECC RAM.
- ninjin 1y ago
  Seconded, but hard to find for small boxes. I have seen in-band ECC on Asus Nucs, but that is as good as it gets from what I can tell.
  - MaKey 1y ago
    There is the new Minisforum N5 Pro, which supports DDR5 ECC RAM. I'm keeping an eye on it.
- mvanbaak 1y ago
  totally agree. That's why my homelab storage server(s) are 2nd hand enterprise machines. They come with ECC.
  - dardeaup 1y ago
    True! Unfortunately, an enterprise machine is more likely to have considerable power draw and quite possibly be much louder. I have a 2013 Apple Mac Pro (trashcan) that uses ECC. They're also cheap, small, and quiet.
- thibaultamartin 1y ago
  I'm not too familiar with it. Why do you want or need ECC RAM for your homelab?
  - MaKey 1y ago
    ECC RAM protects against bit flips (a bit changing to the wrong state). These can be caused by electric or magnetic interference. A pixel of a picture suddenly having the wrong color because of a bit flip is not that bad but some day an important file might end up corrupted because of one. I want to sleep well at night not having to worry about silent data corruption so ECC RAM it is. See here for more information: https://en.wikipedia.org/wiki/ECC_memory https://en.wikipedia.org/wiki/ECC_memory
arjie 1y ago
I have a setup that is perhaps not as robust, but where my primary aim was that I should be able to incrementally encapsulate the parts. https://wiki.roshangeorge.dev/w/One_Quick_Way_To_Host_A_WebApp https://wiki.roshangeorge.dev/w/One_Quick_Way_To_Host_A_WebA... As an example, I use cloudflare tunnel to point to an nginx that reverse proxies all the services, but I could just as well point DNS to that nginx and it would still work. I had to rebuild the entire thing on my home server when I found that the cheap VPS I was using was super over-provisioned ($2/mo for 2 Ryzen 7950 cores? Of course it was) and I had this thing at home anyway, and this served me well for that use-case. When I rebuilt it, I was able to get it running pretty quickly and each piece could be incrementally done: i.e. I could run without cloudflare tunnel and then add it to the mix, I could run without R2 and then switch file storage to R2 because I used FUSE s3fs to mount R2, so on and so forth.
cyprien_g 1y ago
Building a homelab is an awesome way to learn a lot of things. I also used to over-engineer my homelab, but I recently took a more simplistic approach (https://www.cyprien.io/posts/homelab/ https://www.cyprien.io/posts/homelab/), even though it’s probably still over-engineered for most people. I realized that I already do too much of this in my day job, so I don’t want to do it at home anymore.
8fingerlouie 1y ago
While i get the whole homelab thing is exiting and a great learning experience, it's simply not worth the time and effort for the majority of people. You will end up paying much more for your services, along with spending a ton of time maintaining it (and if you don't, you will probably find yourself on the end of a 0-day hack sometime). In Northern/Western Europe, where power costs around €0.3/kWh on average, just the power consumption of a simple 4 bay NAS will cost you almost as much as buying Google Drive / OneDrive / iCloud / Dropbox / Jottacloud / Whatever. A simple Synology 4 bay NAS like a DS923+ with 4 x 4TB Seagate Ironwolf drives will use between 150 kWh and 300 kWh per year (100% idle vs 100% active, so somewhere in between), which will cost you between €45 and €90 per year, and that's power alone. Factoring in the cost of the hardware will probably double that (over a 5 year period). It's cheaper (and easier) to use public cloud, and then use something like Cryptomator (https://cryptomator.org/ https://cryptomator.org/) to encrypt data before uploading it. That way you get the best of both worlds, privacy without any of the sysadm tasks. Edit: I'll just add, as you grow older, you come to realize that time is a finite resource, and while money may seem like it is finite, you can always make more money. Don't spend your time hunched over servers. Spend it doing things you love with people that matter to you. Eventually those people won't be there anymore, and the memories you make with those people will matter far more to you in 20 years, than the €20/month you paid for cloud services.
- guappa 1y ago
  a NAS has like 10000x more storage than google drive and is also way faster locally.
  - brazzy 1y ago
    The premium plan from Google has 2 TB and costs about the same annually as the electricity for the NAS that the GP comment suggested for comparison (at 100% usage). So at the same ONGOING cost (not even counting initial investment), the NAS has 8 times more storage. 16 times if you assume it will be mostly idle. Except if you want high availability with RAID, then you're back to 8 times. And haven't yet thought about backups. All this assuming that you even need that much storage, which most people definitely do not.
crinkly 1y ago
Yeah good luck with that one. I ran a home lab for a number of years. This was a fairly extensive set up - 4 rack mount servers, UPS, ethernet switch etc with LTO backups. Did streaming, email and file storage for the whole family as well as my own experiments. One morning I woke up to a dead node. The DMZ service node. I found this out because my wife had no internet. It was running the NAT and email too. Some swapping of power supplies later and I found the whole thing was a complete brick. Board gone. It's 07:45 and my wife can't check her schedule and I'm still trying to get 3 kids out of the door. At that point I realised I'd fucked up by running a home lab. I didn't have the time or redundancy to have anyone rely on me. I dug the ISP's provided WiFi router out, plugged it in and configured it quickly and got her laptop and phone working on it. Her email was down but she could check calendar etc (on icloud). By the end of the day I'd moved all the family email off to fastmail and fixed everything to talk to the ISP router properly. I spent the next year cleaning up the shit that was on those servers and found out that between us we only had about 300 gig of data worth keeping which was distributed out to individual macbooks and everyone is responsible for backing their own stuff up (time machine makes this easy). Eventually email was moved to icloud as well when domains came along. I burned 7TB of crap, sold all the kit and never ran a home lab again. Then I realised I didn't have to pay for the energy, the hardware or expend the time running it. There are no total outages and no problems if there's a power failure. The backups are simple, cheap and reliable. I don't even have a NAS now - I just bought everyone some Samsung T7 shield disks. I have a huge weight off my shoulders and more free time and money. I didn't learn anything I wouldn't have learned at work anyway.
- plqbfbv 1y ago
  I can relate to this. I still run my own x86 box as a router to have the AP controller, but I'm strongly considering dropping this. I need to update it and patch it, hoping nothing goes wrong in the process. If something breaks I'm the only one that can repair it, and I really don't want to hear my wife screaming at me at 7am when I wake up. Eventually I came to your same conclusion, but I still run a hybrid setup that allows me to keep the router (for now), and a NAS for backup (3-2-1) and some local services. I run a dedicated server from Hetzner for "always on" services, so that the hardware, power redundancy and operational toil are offloaded. I gave up long ago on email: any hosting service will be way better than me doing it - I know I can do it, but is it worth my sanity? Nope.
annoyingnoob 1y ago
I love infrastructure. I run my own services at home too, and have for many years. But to be honest, the older I get the less fun it is to deal with issues that come up from time to time. At some point you'll need to upgrade hardware and software, you get to do the exercise over again. There will always be lessons learned and its get better each time. Its still work.
philip1209 1y ago
This is cool - I have a similar home lab on a Mac Mini [1]. The encryption question is interesting. I don't have disk encryption turned on, because I want the computer to recover from power failure. If power turns off then on, the server would be offline until I decrypt it. How does your "Wake on LAN" work with the encryption? [1] https://github.com/contraptionco/toolbox https://github.com/contraptionco/toolbox
- doubled112 1y ago
  They mention it in the article, in a big yellow note box. You could use an IP KVM, or you could install Dropbear SSH server into the initramfs.
  - thibaultamartin 1y ago
    Author here, indeed I bought a JetKVM. A colleague had one and recommended it warmly! I'm Very happy with it so far, but my usage has been rather basic. I've heard that it might be difficult to get one in the US though. https://jetkvm.com/ https://jetkvm.com/
    - doubled112 1y ago
      I've looked into similar solutions since the mini PC I'm using as a home server doesn't support WoL, but never pulled the trigger. I keep putting it off since it is on a UPS and power outages aren't that frequent. Accessibility isn't too bad since it's under the TV stand.
    - grosswait 1y ago
      PiKVM is another great solution.
st3fan 1y ago
"I used the full disk with LLM, and set up disk encryption" I know we're in the AI hype cycle but I bet you meant LVM there >:-)
- thibaultamartin 1y ago
  Oh lord of the rings, that's embarrassing. Thanks for reporting the typo!
rpcope1 1y ago
I really don't get why people like the Minisforum stuff over alternatives. I've unfortunately been given one, and honestly I'm really unimpressed between crap firmware, no real expandability and just all of the other compromises that come with buying Aliexpress hardware. For the same money you can either pick up a used entry model Dell/HP/Lenovo server (and if they're E3/W/other entry level Xeon, they're usually not terrible on power) or get a good ATX chassis and power it with some off lease Supermicro hardware. Then you don't need to compromise on things like OOB management, hot swap bays, a real SAS card, real 10G nics, ECC ram, etc. etc. Maybe people are just afraid of doing a little bit of putting hardware together? I've seen and have systems that use the above gear that have been going for well over a decade now, with basically no hiccups, and even the old Sandy Bridge era E3 stuff both punches above probably even RPi5 and N100 and doesn't draw more than 30-40W when you don't have spinning disks in there. I'm sure if you avoid AMD and go find a newer T variant Intel chip, you can both have your cake and eat it.
- daymanstep 1y ago
  ECC capable hardware tend to be very power hungry.
  - p12tic 1y ago
    That's just an artifact of Intel disabling ECC on consumer processors. There's no reason for ECC to have significantly higher power consumption. It's just an additional memory chip per stick and a tiny bit of additional logic on CPU side to calculate ECC. If power consumption is the target, ECC is not a problem. I know firsthand that even old Xeon D servers can hit 25W full system idle. On AMD side 4850G has 8 cores and can hit sub 25W full system idle as well.
    - daymanstep 1y ago
      My HP 800 mini idles at 3W
  - Hikikomori 1y ago
    Not always, HP Microserver n54l had support.
jackhalford 1y ago
Why install proxmox on top of debian? Proxmox distributes an iso that basically does the same as you with preseeding. Although recently had to install proxmox as an apt pkg on top of debian happened because the proxmox iso wouldn’t install properly. That actually happened twice. I think I’ll just install debian from now on…
- Havoc 1y ago
  I’ve encountered three separate machines where the proxmox installer fails. Both cli and gui. Debian first gets around that. Never figured out why
kayson 1y ago
After some research, it seems much easier to just back up the Proxmox config (and VM disk images, if they're needed) than to define or deploy Proxmox VMs with OpenTofu or ansible. https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)#_recovery https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pm...
burnt-resistor 1y ago
My home lab: - Lightning protection - 2 2U UPSes with 2 extra 2U battery packs each, temperature and humidity monitoring, and remote management - 1x vSphere 7u3w Enterprise Plus + vCenter, 512 GiB ECC RAM, 16 TiB of RAID10 SSD, 96 thread EPYC server with 4 10 Gb optical NICs and 4x SAS3 ports - 4U JBOD external NAS 330 TiB usable shared mostly over Samba with Time Machine support - 2x Ryzen boxes with 128 GiB of ECC RAM, 100 Gb links to each other, 4 TiB RAID1 SSD, also used for distributed builds (also vSphere) - Additional non-ECC 96 GiB tiny ITX Ryzen Windows lab machine - Misc. non-ECC 128 GiB micro ITX Ryzen for additional distributed build capacity, currently Fedora w/ Podman and Docker - Deciso OPNsense (Business license) router with 10 Gb optical ports, WireGuard, NTP-DHCP-DNS - PoE 4x RPi 5 + SSD Ceph cluster - Ubiquiti U7 Pro XGS APs - Eufy security cameras with Home base - PKI (TLS CA), TOTP 2FA ssh, YK gpg/ssh agent, RustDesk - All boxes except lab, Fedora, and RPis are lights-out manageable and so don't need a KVM To Do: NFS 4.2, LDAP, Krb5, TACACS+, SAML/OpenID (authelia), SNMP, Nagios or Grafana/Prometheus, K8s Cost: $150/month in electricity
- asalahli 1y ago
  What kinds of workloads do you use this infra for, if you don't mind sharing?

253 points by JNRowe 1y ago

tehlike 1y ago

I moved my side project to my garage so I don't have to pay hetzner 600+$ and counting every month

tracker1 1y ago
I've got a single server and a /28 IP block on OVH for public facing stuff. Mostly because it's cheaper than the bump from my "home" internet to "business" to be able to use common server ports (blocked on the home service). Works well enough for what I need.
- tehlike 1y ago
  I use cloudflare tunnel to do most of this, and i am fine with the implications (like CF mitm'ing my website traffic).
lazyweb 1y ago
I did the exact opposite. And by that I mean physically moved my homelab into their colo earlier this year. Runs like a charm, costs about 500€ per month total. Sounds like a lot, but I was almost paying the same before - 220€ for power at home, 110€ for a dedicated Hetzner server, 95€ for a secondary internet connection (as not to interfere with the main uplink used for home office by my partner and me). Not having to deal with the extra heat, noise and used up space at home anymore has been worth it as well.
- tehlike 1y ago
  My storage needs were increasing by the day. Electricity is now a small monthly cost. I have more cores and ram than ever, and can easily expand it. Main machine now runs with 1TB ram and 15TB SSD and other has more than 384G ram. I currently use 3TB ssd storage, and get way more performnce than Hetzner's VMs with ceph ssd disks. I do need redundancy, but it's not something hetzner was giving me anyway, and if my anectode is not a mess up, i actually got database corruption on hetzner that never happened on my own local setup. I'd have colo'ed or used dedicated as it's definitely better than their VMs, but they don't have that in their US datacenters. I am pretty happy with my current setup, I have significantly less down time (few mins a month) than when I was on hetzner - but this is mostly due to my need for more ram at times. I also used this as an excuse to get 56G mellanox fiber switch and get poe cameras etc in a full homelab manner, so it's been fun, on top of being cheaper. Noise is not a concern, I got a sound-proof server rack that's pretty nice. It takes up space, but i have kids, so my garage is near full at times anyways :)

treve 1y ago

If anyone is looking to get started with a homelab at a good price, I can highly recommend checking ebay for a Dell Wyse 5070. They flooded the market for $50 and are likely powerful enough for many needs. They have a M.2 slot that support SATA. The 'extended' version also has space for a small pcie card and has a parallel and 2 serial ports for a blast to the past.

senectus1 1y ago
curious to know what you would use this for?
- denkmoon 1y ago
  local dns, static site hosting, local apt cache, various other network services (unifi controller if you've got those APs for example), remote/headless dev machine (maybe not for kernel or bigcorp java development), or whatever else you want. mail if you want. Anything :) Those little thin clients aren't gonna be fast doing "big" things, but serving up a few dns packets or whatever to your local network is easy work and pretty useful.
- treve 1y ago
  I use it for media hosting. Backups (connected USB disks), home assistant, syncthing
- rwyinuse 1y ago
  Even these low-power CPU's are surprisingly capable. As an example of more fancy thing, one could slap in some external storage, install Jellyfin, and run their own local streaming service off such a machine. The CPU is modern enough for efficient hardware transcoding of a stream.
pandemic_region 1y ago
What if power consumption is taken into account? Are there any devices in that category that are ok to leave on 24/7 ?
- 1y ago

czhu12 1y ago

This was exactly a use case I had in mind when building https://canine.sh https://canine.sh -- also uses k3s as a provider, and provides a Heroku-like devex. How to actually reliably expose a homelab to the broader internet is a little tricky, cloudflare tunnels mostly does the trick but can only expose one port at a time, so the set up is somewhat annoying

BLKNSLVR 1y ago
I've got basically raw internet coming in to my OPNSense device, although I had to request certain ports to be removed from the ISP's by-default-blocked policy, since I host a mail server - but the ISP is fine with this, they have a form for it, super easy. Some family members are behind CGNAT, and I'm not sure if their ISP has the option to move out from behind that, but since they don't self-host it's probably slightly more secure from outside probes. We're still able to privately share communications via my VPN hub to which they connect, which allows me to remotely troubleshoot minor issues. I haven't looked into cloudflare tunnels, but haven't felt the need.
vladvasiliu 1y ago
What do you mean by "one port at a time"? I run cloudflared on one machine, and it proxies one subdomain to one port, and another to a unix socket (could have been a second port, no pb).

timc3 1y ago

The encryption is interesting but I wouldn’t call this over engineered at all, in fact it’s rather basic compared to a lot of homelabs I see people building particularly where people are doing K8s or similar over multiple machines. Also Proxmox was called out as the only choice when that is very much not the case. It is a good choice for sure, but there are others.

BLKNSLVR 1y ago

> But my server could be shut down because of a power outage or another reason. I might be at work or even on holidays when it happens, and even wireguard can’t solve this. A 'power outage' incident doesn't seem to have been mitigated. My homelab has had evolving mitigations: I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime, which got replaced by a dedicated inverter/charger/transfer-switch attached to a big-ass AGM caravan battery (which on a couple of occasions powered through two-to-three hour power outages), and has now been replaced with these recent LiFePo4 battery power station thingies. Of course, it's only a homelab, there's nothing critically important that I'm hosting, but that's not the dang point, I want to beat most of "the things", and I don't like having to check that everything has rebooted properly after a minor power fluctuation (I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet).

qwertox 1y ago
> I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime Can you share more about this? I have a APC Back UPS PRO USV 1500VA (BR1500G-GR) and it would be nice to know if this is possible with that one as well.
- BLKNSLVR 1y ago
  That UPS eventually died, and I'm not sure if it was because it was hooked up to a larger battery than it was designed for, but it's still only 12 volts so I don't think the electronics would notice. What they may notice is extended run-time in the event of a power failure. It was a crude mod. Take the cover off and remove the existing little security alarm battery, use tin snips to cut a hole in the side of the metal UPS cover (this was challenging, it was relatively thick metal, I'd recommend using an angle grinder in an appropriately safe environment far away from the internals of the UPS), and feed the battery cables out through the hole. I probably got some additional cables with appropriately sized terminations to effectively extend the short existing ones (since they were only designed to be used within the device). And then connect it up to a car battery. Cover any exposed metal on the connectors with that shrink rubber tubing or electrical tape. Be very careful with exposed metal around it anywhere, especially touching the RED POSITIVE pole of the battery. Get a battery box - I got one for the big-ass AGM battery. Test it out on a laptop that's had it's battery removed or disconnected that, just in case, you don't care too much about losing. Get a battery charger that can revive a flat battery, and do a full refresh/renew charge on the car battery once a year or after it's had to push through a power outage that may have used more than a few percent of its capacity. Personally, I think it's safer a less hassle to go for a LiFePo4 (LFP) Power Station style device that has UPS capabilities. LFP batteries have 3,000-ish cycle lifetimes, which could be nearly ten years with daily use.

tietjens 1y ago

This is such a great post. I have a small collection of posts for inspiration in creating my homelab and this is getting added to it. Current have a Pi 4 with PiHole and a Beelink. Going to add one or two more machines.

MaKey 1y ago

Your VPS provider likely uses servers with ECC RAM, this home server doesn't. For most people it doesn't seem to matter but for me it does - a home server where I store my data needs to have ECC RAM.

ninjin 1y ago
Seconded, but hard to find for small boxes. I have seen in-band ECC on Asus Nucs, but that is as good as it gets from what I can tell.
- MaKey 1y ago
  There is the new Minisforum N5 Pro, which supports DDR5 ECC RAM. I'm keeping an eye on it.
mvanbaak 1y ago
totally agree. That's why my homelab storage server(s) are 2nd hand enterprise machines. They come with ECC.
- dardeaup 1y ago
  True! Unfortunately, an enterprise machine is more likely to have considerable power draw and quite possibly be much louder. I have a 2013 Apple Mac Pro (trashcan) that uses ECC. They're also cheap, small, and quiet.
thibaultamartin 1y ago
I'm not too familiar with it. Why do you want or need ECC RAM for your homelab?
- MaKey 1y ago
  ECC RAM protects against bit flips (a bit changing to the wrong state). These can be caused by electric or magnetic interference. A pixel of a picture suddenly having the wrong color because of a bit flip is not that bad but some day an important file might end up corrupted because of one. I want to sleep well at night not having to worry about silent data corruption so ECC RAM it is. See here for more information: https://en.wikipedia.org/wiki/ECC_memory https://en.wikipedia.org/wiki/ECC_memory

arjie 1y ago

I have a setup that is perhaps not as robust, but where my primary aim was that I should be able to incrementally encapsulate the parts. https://wiki.roshangeorge.dev/w/One_Quick_Way_To_Host_A_WebApp https://wiki.roshangeorge.dev/w/One_Quick_Way_To_Host_A_WebA... As an example, I use cloudflare tunnel to point to an nginx that reverse proxies all the services, but I could just as well point DNS to that nginx and it would still work. I had to rebuild the entire thing on my home server when I found that the cheap VPS I was using was super over-provisioned ($2/mo for 2 Ryzen 7950 cores? Of course it was) and I had this thing at home anyway, and this served me well for that use-case. When I rebuilt it, I was able to get it running pretty quickly and each piece could be incrementally done: i.e. I could run without cloudflare tunnel and then add it to the mix, I could run without R2 and then switch file storage to R2 because I used FUSE s3fs to mount R2, so on and so forth.

cyprien_g 1y ago

Building a homelab is an awesome way to learn a lot of things. I also used to over-engineer my homelab, but I recently took a more simplistic approach (https://www.cyprien.io/posts/homelab/ https://www.cyprien.io/posts/homelab/), even though it’s probably still over-engineered for most people. I realized that I already do too much of this in my day job, so I don’t want to do it at home anymore.

8fingerlouie 1y ago

While i get the whole homelab thing is exiting and a great learning experience, it's simply not worth the time and effort for the majority of people. You will end up paying much more for your services, along with spending a ton of time maintaining it (and if you don't, you will probably find yourself on the end of a 0-day hack sometime). In Northern/Western Europe, where power costs around €0.3/kWh on average, just the power consumption of a simple 4 bay NAS will cost you almost as much as buying Google Drive / OneDrive / iCloud / Dropbox / Jottacloud / Whatever. A simple Synology 4 bay NAS like a DS923+ with 4 x 4TB Seagate Ironwolf drives will use between 150 kWh and 300 kWh per year (100% idle vs 100% active, so somewhere in between), which will cost you between €45 and €90 per year, and that's power alone. Factoring in the cost of the hardware will probably double that (over a 5 year period). It's cheaper (and easier) to use public cloud, and then use something like Cryptomator (https://cryptomator.org/ https://cryptomator.org/) to encrypt data before uploading it. That way you get the best of both worlds, privacy without any of the sysadm tasks. Edit: I'll just add, as you grow older, you come to realize that time is a finite resource, and while money may seem like it is finite, you can always make more money. Don't spend your time hunched over servers. Spend it doing things you love with people that matter to you. Eventually those people won't be there anymore, and the memories you make with those people will matter far more to you in 20 years, than the €20/month you paid for cloud services.

guappa 1y ago
a NAS has like 10000x more storage than google drive and is also way faster locally.
- brazzy 1y ago
  The premium plan from Google has 2 TB and costs about the same annually as the electricity for the NAS that the GP comment suggested for comparison (at 100% usage). So at the same ONGOING cost (not even counting initial investment), the NAS has 8 times more storage. 16 times if you assume it will be mostly idle. Except if you want high availability with RAID, then you're back to 8 times. And haven't yet thought about backups. All this assuming that you even need that much storage, which most people definitely do not.

crinkly 1y ago

Yeah good luck with that one. I ran a home lab for a number of years. This was a fairly extensive set up - 4 rack mount servers, UPS, ethernet switch etc with LTO backups. Did streaming, email and file storage for the whole family as well as my own experiments. One morning I woke up to a dead node. The DMZ service node. I found this out because my wife had no internet. It was running the NAT and email too. Some swapping of power supplies later and I found the whole thing was a complete brick. Board gone. It's 07:45 and my wife can't check her schedule and I'm still trying to get 3 kids out of the door. At that point I realised I'd fucked up by running a home lab. I didn't have the time or redundancy to have anyone rely on me. I dug the ISP's provided WiFi router out, plugged it in and configured it quickly and got her laptop and phone working on it. Her email was down but she could check calendar etc (on icloud). By the end of the day I'd moved all the family email off to fastmail and fixed everything to talk to the ISP router properly. I spent the next year cleaning up the shit that was on those servers and found out that between us we only had about 300 gig of data worth keeping which was distributed out to individual macbooks and everyone is responsible for backing their own stuff up (time machine makes this easy). Eventually email was moved to icloud as well when domains came along. I burned 7TB of crap, sold all the kit and never ran a home lab again. Then I realised I didn't have to pay for the energy, the hardware or expend the time running it. There are no total outages and no problems if there's a power failure. The backups are simple, cheap and reliable. I don't even have a NAS now - I just bought everyone some Samsung T7 shield disks. I have a huge weight off my shoulders and more free time and money. I didn't learn anything I wouldn't have learned at work anyway.

plqbfbv 1y ago
I can relate to this. I still run my own x86 box as a router to have the AP controller, but I'm strongly considering dropping this. I need to update it and patch it, hoping nothing goes wrong in the process. If something breaks I'm the only one that can repair it, and I really don't want to hear my wife screaming at me at 7am when I wake up. Eventually I came to your same conclusion, but I still run a hybrid setup that allows me to keep the router (for now), and a NAS for backup (3-2-1) and some local services. I run a dedicated server from Hetzner for "always on" services, so that the hardware, power redundancy and operational toil are offloaded. I gave up long ago on email: any hosting service will be way better than me doing it - I know I can do it, but is it worth my sanity? Nope.

annoyingnoob 1y ago

I love infrastructure. I run my own services at home too, and have for many years. But to be honest, the older I get the less fun it is to deal with issues that come up from time to time. At some point you'll need to upgrade hardware and software, you get to do the exercise over again. There will always be lessons learned and its get better each time. Its still work.

philip1209 1y ago

This is cool - I have a similar home lab on a Mac Mini [1]. The encryption question is interesting. I don't have disk encryption turned on, because I want the computer to recover from power failure. If power turns off then on, the server would be offline until I decrypt it. How does your "Wake on LAN" work with the encryption? [1] https://github.com/contraptionco/toolbox https://github.com/contraptionco/toolbox

doubled112 1y ago
They mention it in the article, in a big yellow note box. You could use an IP KVM, or you could install Dropbear SSH server into the initramfs.
- thibaultamartin 1y ago
  Author here, indeed I bought a JetKVM. A colleague had one and recommended it warmly! I'm Very happy with it so far, but my usage has been rather basic. I've heard that it might be difficult to get one in the US though. https://jetkvm.com/ https://jetkvm.com/
  - doubled112 1y ago
    I've looked into similar solutions since the mini PC I'm using as a home server doesn't support WoL, but never pulled the trigger. I keep putting it off since it is on a UPS and power outages aren't that frequent. Accessibility isn't too bad since it's under the TV stand.
  - grosswait 1y ago
    PiKVM is another great solution.

st3fan 1y ago

"I used the full disk with LLM, and set up disk encryption" I know we're in the AI hype cycle but I bet you meant LVM there >:-)

thibaultamartin 1y ago
Oh lord of the rings, that's embarrassing. Thanks for reporting the typo!

rpcope1 1y ago

I really don't get why people like the Minisforum stuff over alternatives. I've unfortunately been given one, and honestly I'm really unimpressed between crap firmware, no real expandability and just all of the other compromises that come with buying Aliexpress hardware. For the same money you can either pick up a used entry model Dell/HP/Lenovo server (and if they're E3/W/other entry level Xeon, they're usually not terrible on power) or get a good ATX chassis and power it with some off lease Supermicro hardware. Then you don't need to compromise on things like OOB management, hot swap bays, a real SAS card, real 10G nics, ECC ram, etc. etc. Maybe people are just afraid of doing a little bit of putting hardware together? I've seen and have systems that use the above gear that have been going for well over a decade now, with basically no hiccups, and even the old Sandy Bridge era E3 stuff both punches above probably even RPi5 and N100 and doesn't draw more than 30-40W when you don't have spinning disks in there. I'm sure if you avoid AMD and go find a newer T variant Intel chip, you can both have your cake and eat it.

daymanstep 1y ago
ECC capable hardware tend to be very power hungry.
- p12tic 1y ago
  That's just an artifact of Intel disabling ECC on consumer processors. There's no reason for ECC to have significantly higher power consumption. It's just an additional memory chip per stick and a tiny bit of additional logic on CPU side to calculate ECC. If power consumption is the target, ECC is not a problem. I know firsthand that even old Xeon D servers can hit 25W full system idle. On AMD side 4850G has 8 cores and can hit sub 25W full system idle as well.
  - daymanstep 1y ago
    My HP 800 mini idles at 3W
- Hikikomori 1y ago
  Not always, HP Microserver n54l had support.

jackhalford 1y ago

Why install proxmox on top of debian? Proxmox distributes an iso that basically does the same as you with preseeding. Although recently had to install proxmox as an apt pkg on top of debian happened because the proxmox iso wouldn’t install properly. That actually happened twice. I think I’ll just install debian from now on…

Havoc 1y ago
I’ve encountered three separate machines where the proxmox installer fails. Both cli and gui. Debian first gets around that. Never figured out why

kayson 1y ago

After some research, it seems much easier to just back up the Proxmox config (and VM disk images, if they're needed) than to define or deploy Proxmox VMs with OpenTofu or ansible. https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)#_recovery https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pm...

burnt-resistor 1y ago

My home lab: - Lightning protection - 2 2U UPSes with 2 extra 2U battery packs each, temperature and humidity monitoring, and remote management - 1x vSphere 7u3w Enterprise Plus + vCenter, 512 GiB ECC RAM, 16 TiB of RAID10 SSD, 96 thread EPYC server with 4 10 Gb optical NICs and 4x SAS3 ports - 4U JBOD external NAS 330 TiB usable shared mostly over Samba with Time Machine support - 2x Ryzen boxes with 128 GiB of ECC RAM, 100 Gb links to each other, 4 TiB RAID1 SSD, also used for distributed builds (also vSphere) - Additional non-ECC 96 GiB tiny ITX Ryzen Windows lab machine - Misc. non-ECC 128 GiB micro ITX Ryzen for additional distributed build capacity, currently Fedora w/ Podman and Docker - Deciso OPNsense (Business license) router with 10 Gb optical ports, WireGuard, NTP-DHCP-DNS - PoE 4x RPi 5 + SSD Ceph cluster - Ubiquiti U7 Pro XGS APs - Eufy security cameras with Home base - PKI (TLS CA), TOTP 2FA ssh, YK gpg/ssh agent, RustDesk - All boxes except lab, Fedora, and RPis are lights-out manageable and so don't need a KVM To Do: NFS 4.2, LDAP, Krb5, TACACS+, SAML/OpenID (authelia), SNMP, Nagios or Grafana/Prometheus, K8s Cost: $150/month in electricity

asalahli 1y ago
What kinds of workloads do you use this infra for, if you don't mind sharing?