8 ms·
I host this blog from my garage
- rektide 5y agoVery nice set up. Thanks for sharing, thanks for submitting. Very great technology to be able to set up & have running, have at your back. A lot of haters, but this is such a vast leap beyond a personal server, to having real cloud infrastructure, to having one and only one control plane for all your systems. Recap of tech choices: Cilium networking, MetalLB load balancing, nginx ingress, Rook/Ceph storage, Prometheus/Grafana monitoring/alerting, Loki logs, Harbor container registry, Flux for GitOps. Also a huge shout out to the Ingress controller comparison spreadsheet[1] the author made! Really nice being able to compare these feature sets. Neat to see a couple folks already working on HTTP3 & QUIC load balancing, who does Maglev routing, who has the best offload capabilities. This definitely brought my interest in Apache APISIX way way up. The author digs at their own infrastructure some, calling it for learning. But to me, I think it's bad/dangerous having a conservative outlook that personal users should have to learn/use inferior/lesser tools (dozens of hours), it's bad to bifurcate the computing world into high and low computing. It took the author a lot of effort I'm sure to cobble together this system, but my hope is, over time, we start to make works like this much more paved road, and we start to make them much more accessible & documented & supported. We still haven't seen personal-use-focused Kubernetes distributions arrive, but I'm hoping that happens. Here's the author's description: > By trying out "flashy" software in a low-stakes environment, I can have some idea of how it will work in production-critical ones, which helps me make better choices. (As an example, my attempts to run Istio in a homelab setting have convinced me that it should be avoided in most circumstances.) Again I think this is undershooting the real value, of using real tools. The hill to climb right now seems big. But imo the effort one's going to invest in selfhosting is probably going to be significant, and I like a lot the plan of shooting for good. [1] https://docs.google.com/spreadsheets/d/191WWNpjJ2za6-nbG4ZoUMXMpUK8KlCIosvQB0f-oq3k/edit#gid=907731238 https://docs.google.com/spreadsheets/d/191WWNpjJ2za6-nbG4ZoU...
- shosti 5y agoThanks, glad you liked it! Minor clarification, I didn’t write the ingress comparison, that’s from https://learnk8s.io https://learnk8s.io (who I’ve been working with recently). They have a bunch of awesome resources on their site.
- treesknees 5y agoHeh, I'm actually more interested in the "garage" piece of the blog which is barely touched on. I also run a home lab, but from my climate-controlled relatively clean basement. I've thought about moving it to the garage, but I'm worried about temperature and dust/dirt ingress. Do you have a sealed cabinet or air filters or any other environmental controls around the servers? Or are you using industrial machines rated for dusty locations? Otherwise a garage actually doesn't seem like a great place to put servers.
- shosti 5y ago(Author here) I’m kind of cruel to my hardware lol (they’re all frankenservers hacked together with random parts from eBay). I’m not doing anything fancy to protect from dust and whatnot, but it’s pretty much always chilly in the garage (and I’m using low power parts) so heating hasn’t been an issue.
- chomp 5y agoI live in the Southern US and it's extremely humid and hot in the summers. Only certain systems are fine with being in the garage. For instance, a Supermicro chassis I put out there began rusting in a few spots after a few months, but my custom system that I built was perfectly fine. I suspect the metals Supermicro uses are treated in some way and then cut, but the edges aren't re-treated, allowing rust to form on all of the edges of the chassis. I never put spinners out there, only SSDs. My bulk storage lives inside my house. I have an HP Sandy Bridge Xeon box that lives out there and it's always super quiet, but I don't push it hard either. Raspberry pis are super okay with being outside. I'd say if you can make a small Pi cluster, it's a good fit. I use an open air rack next to my water heater. I don't do any woodworking or anything, so the dust isn't too bad.
- cma 5y agoThere was a talk from comma.ai with a part on what they did to prevent corrosion running a datacenter in their garage to avoid the nvidia datacenter tax or something: https://www.youtube.com/watch?v=D0JO5dr8cPE&t=15m12s https://www.youtube.com/watch?v=D0JO5dr8cPE&t=15m12s
- superkuh 5y ago>Well, hosting a blog from home is probably not a great idea from a practical perspective. I'm not a hater, but maybe he perceives it as not practical here because of all the fun unnecessary complexity. Keeping something like this going for more than a couple years would require complex sysadmin maintainence for updates (which is fun till it isn't). But if you just install nginx from your system repositories, have your hugo generated .html and media files in your www dir, and forward a port 80 on your router to your server LAN IP:80, it's going to work until your distro stops working without input and without security issues. For most people, for most of the time, it works great. And it's okay if it doesn't work some of the time. Hosting your blog from home, in whatever room, is a great idea and very practical.
- shosti 5y ago(Author here) I mostly worry about security for this. If you have nothing private on your network it's probably fine, but if you have, say, a NAS that isn't using proper authentication (pretty common), an os/nginx vulnerability could end up exposing stuff. Of course there are much simpler ways to lock things down also :)
- scubbo 5y agoThank you for a great article! I recently took the plunge of building-and-hosting a blog too - but, due to security concerns, I took the entirely opposite approach of making it fully cloud-based (Git repos for infra and for content -> AWS CodePipeline, Hugo during CodeBuild -> S3 and CloudFront). This was sadly ironic since I'd mostly wanted to blog about my experiences with homelabbing, but I didn't trust myself to open a port to the outside world. Thanks to your blog I might finally learn Kubernetes and use a Cloudflare tunnel to implement a similar truly-selfhosted blog!
- shosti 5y agoI would say you don't really need Kubernetes for this sort of setup (I already was running all the K8s stuff which is why I went with it, but docker compose or even just running things in systemd without containers would work too). I think the main thing is to have some sort of network isolation (like a separate VLAN or a server that blocks outbound traffic) between stuff that's exposed to the internet and stuff that's private on the network.
- hinkley 5y agoIf you wanted to 'publish' a more interactive website to the internet while still keeping all of the publishing tools safe inside an enclave/intranet somewhere, then you probably need a read-only copy of a database. But any time I look at read-replica configurations, it looks like they expect the replica to contact the master. You can't set up a proper bastion server if you have to have a secret door back into your intranet to replicate data. This has left me stuck on the starting blocks for a couple of pet projects I might like to work on. Anybody know how to solve this problem, short of roll-your-own?
- int0x2e 5y agoHow much latency and inconsistency can you live with? Could you dump full or incremental database backups from your secure enclave, upload them to blobs and ingest on the cloud? Could you publish a change feed of sorts to Kafka and apply changes?
- hinkley 5y agoIt would have to be incremental because a full dump does not scale up. Of course if you scale down you could probably get away with shipping snapshots of a sqlite database, and I understand some people do just that. But it would be nice to have streaming WAL logs with the instigator flipped. Probably I need some sort of middleware to dis-intermediate the pipe.
- teraflop 5y agoWith Postgres, you can do log-shipping replication using whatever mechanism you like to deliver the WAL files, with no direct network connection between the primary and standby servers. It's still kind of roll-your-own, because you need to provide shell scripts to handle archiving, fetching and cleaning up the log files on either end, but it can be done.
- ivank 5y agoWireGuard
- mro_name 5y agowouldn't a raspberry pi do it from within the drawer? Not only https://solar.lowtechmagazine.com https://solar.lowtechmagazine.com does similar, but https://cheapskatesguide.org/articles/raspberry-pi-3-4-web-server.html https://cheapskatesguide.org/articles/raspberry-pi-3-4-web-s... has some numbers. Seems feasible, unless you are the NYT.
- shosti 5y agoYeah that’s what I would do if I was just hosting a blog, I already had the k8s lab set up for other reasons and wanted to get more use out of it.
- mro_name 5y agogot it. Found the solar numbers meanwhile: https://solar.lowtechmagazine.com/2020/01/how-sustainable-is-a-solar-powered-website.html https://solar.lowtechmagazine.com/2020/01/how-sustainable-is...
- probotect0r 5y agoHe clearly says: To stave off the inevitable haters: I know this is over-engineered. That’s kind of the point! By trying out “flashy” software in a low-stakes environment...
- bob446 5y agoImpressive but makes me want to use AWS more than ever
- mkasberg 5y agoIt's unfortunate that there aren't better solutions for securely exposing a part of your home network to the cloud. Imagine if it were easy to set up a secure solution to access your home lab from anywhere - you could use it as your own private cloud. And perhaps then it would also be easy to expose part of that publicly, like a blog.
- charcircuit 5y ago>Imagine if it were easy to set up a secure solution to access your home lab from anywhere It is. You just need to port forward the ports you want exposed to the "cloud" (aka the internet)
- shosti 5y agoI feel like Tailscale gets you pretty close to an “easy private cloud”, it’s just the “public” part that’s still a bit tricky.
- rubatuga 5y agoYou could try our service https://hoppy.network https://hoppy.network It tunnels a public IPv4 and IPv6 (/56) over WireGuard. We have a good IP reputation so you can use it as a mail server. We’re using the service ourselves, so overall pretty reliable.
- heavyset_go 5y agoI do this with Wireguard.
- whoknowswhat11 5y agoI also got a lot of mileage out of a homelab. I experimented with proxmox and ESXi on the homelab front. Now ESXi is at the business. Same thing with different server setups (Dell / HP / SuperMicro). Gives you a good feel for updates, KVM options etc. Same thing with networking - I'm wasting time on Mikrotik and it's pretty fun. It can be handy as a pool of additional resources. I had a very built out homelab server (with new SSD drives etc). Leadtimes got long for some Dell Server configs, we needed something 1-2 days, just grabbed a box from home. With ESXi once the real machine comes in its not that hard to move stuff over. The big win is experimenting with low stakes. I find I'm learning 2-3x faster in that setup. I can try something, reinstall worst case, reboot, KVM, and literally plug in very easily (I have a hardware KVM setup in addition to the iDRAC type stuff).
- midasuni 5y agoIf only he’d hosted it on AWS he too could be offline
- lorenzo95 5y agoI am curious why you use an ingress-nginx deployment. It looks like your blog pod is also running a nginx container, which makes sense. You could just point cloudflared at the service of that pod could you not? Maybe you just wanted an ingress/metallb setup but I guess I was wondering if it adds anything else to the mix. Maybe some security feature? Nice post. I am interested in the flux piece. I noticed that Azure Arc uses fluxcd as well. Going to have to give that(flux) a try.
- shosti 5y agoThe ingress controller is so I can have multiple public apps and configure them with Ingresses (definitely possible to do the overall setup with just cloudflared but I wanted to use built in ingress objects instead of a complicated cloudflared config).
- slickdork 5y agoFun read, thanks. I've always wondered how to securely do such a thing and cloudfare tunnel looks like a good solution. Of course the next thing you gotta solve is that you're running 3 HA clusters, but on one host. Can you replicate a backup to a pi somewhere else in your house that enables on emergency fail over, in case a rat chews through the power cable of the garage server?
- david422 5y agoI used to run my own personal website out of my home. Until I went for a job interview and gave out a link to my website. But the site was down ... and when I got home my cat was chewing on the ethernet wire. So found some cheap virtual hosting that was much more stable.
- MonaroVXR 5y agoDo you have suggestions for cheap hosting in Europe?
- tlamponi 5y agoA OVH or Hetzner VPS can get you a long way for 3.5 to 5€ a month, and they're both quite established and not just a one-man show hoster. Also, you get an actual VM, not just a container, so you can do pretty much everything with it. For network stability I'd prefer OVH, but they only offer you a single IPv6 address, which is just stupid - but depending on your use case that may not be so relevant to you. Hetzners VPS/cloud offering improved a lot in UX in recent year.
- moooo99 5y agoAbsolutely agree on Hetzner. For raw compute Hetzner managed to create a fairly compelling offering. The pricing is very affordable and the performance seems as expected, not as unpredictable as with lots of small hosters
- stefandesu 5y agoA cheap one that I had good experience with is https://contabo.com https://contabo.com.
- nicbou 5y agoMy robot vacuum bumped into the power bar and unplugged the server. It was the most clever state-sponsored cyberattack I've ever witnessed.
- javajosh 5y ago>I self-host... and use Cloudflare Tunnel... This decision protects your home IP address, but it still makes you reliant on something that is not self-hosted. It kind of feels like cheating, but I can't think of a good way to mitigate. What happens if your home ISP gets DDOS'd? What's that like? How do you (or your ISP, or their ISP or ...someone) fix it?
- circularfoyers 5y agoPerhaps having a setup where it has a fallback on Cloudflare Tunnel when you detect you're under a DDOS attack. Could this not be feasible?
- watermelon0 5y ago1.) You probably don't want to expose your own IP address. 2.) If your IP is under DDOS attack, you won't be able to connect to CF tunnel.