16 ms·
I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes
by lrvick 5y ago
I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem.
When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber.
I know where our data lives and no one can access it without a warrant and my explicit knowledge. I also save myself several hundred a month in third party cloud provider fees to host the same services and can reboot upgrade or repair anything whenever I want, but in general no more maintenance than cloud servers . I also never end up with exciting bills when experiments are forgotten about.
You pretty much get all the pros and cons of home ownership. For me it is mostly pros. Also keeps me dogfooding all the same practices I recommend to my clients.
- Osiris 5y agoWhat do you do for off-site backups?
- lrvick 5y agoI threw a NAS in a friends house that is configured to bootup and reverse ssh tunnel to my rack where it then accepts scheduled encrypted duplicity backups. I do not even need to trust my friend as duplicity encrypts all data against a yubikey held pgp keychain before it leaves. The backup NAS could phone home from anywhere with internet access.
- willjp 5y agoThis is very slick. Apparently, I need to make some techie friends. Tarsnap is pretty affordable for off-site backups though!
- whalesalad 5y agoBonus the servers keep the house warmer in the winter :D
- denton-scratch 5y agoYou have a 42U rack to accomodate your family's home-computing needs? How large is your family?
- CRConrad 5y agoThe lower half of it houses a kitchenette.
- rambojazz 5y ago42u for a home rack seems like excessive (unless you run all 3u-4u servers).
- hnthrowaway0315 5y agoThis sounds very interesting. Do you have a blog describing your experience just curious? Thanks for sharing~~
- lrvick 5y agoNo but I founded and help run https://hashbang.sh https://hashbang.sh where we give out free shell services and mentor people in security, privacy, and digital sovereignty.
- 2Gkashmiri 5y agocurious that the souce metions MIT license at top but in the bottom it says "2. Don't use our resources for closed source projects"; was that intentional? my current understanding of MIT license is a "do anything, i don't care, just don't blame me". if i were you and wanted to use "dont use our resources" part, i would set it as AGPL or even SSPL if you are brave enough.
- e-dt 5y agoThere's a difference between what you are allowed to do with the code, which is what the MIT license covers, and what you are allowed to do with a specific hosted platform on which the code is running, which is what that term is part of.
- 2Gkashmiri 5y agooh. Do... you need to specify another license for that? this is interesting.
- DarylZero 5y agoIt's not a license, it's rule #2, where "This _network_ has three rules: [...]"
- DizzyDoo 5y agoWoah, 42u, that's quite something! How many of those slots are you using? What kind of cooling do you have to have for that closet, and how is the volume for the rest of the house?
- thefunnyman 5y agoNot OP but you find a lot of these kinds of setups on Reddit. Check out /r/homelab if you’re interested. Even crazier is /r/homedatacenter where you’ll find dudes with full tape libraries and everything. Super interesting to browse through.
- flakiness 5y agoThanks for sharing this! I tend to forget but there are people who love servers.
- jlcoff 5y agoFWIW, I recently purchased maybe 21U worth of servers for less than $2000. Mostly IBM 2U servers (all dual 6-cores / 12-threads Xeon, most for spare parts), NAS (enough for triple >12TB redundancy), and LTO-4 loaders and drives to go in a rack I picked up for free locally which also came Cisco switches. I'm gonna run my private cloud merging 3 different un-backed up physical computers, and migrate services off Google. That's my second free 42U track, but the other was mostly used as shelf space. I've got a third rack rusting in my backyard which I bought for locally for $200, originally intended to run my former employer test infra, that I brought back home after they laid us off.
- lrvick 5y agoAll of them, but to be fair the bottom half of the rack is mostly multimedia gear that serves the whole house and a 6U gaming PC. I have venting into my ceiling which connects outside but I am mostly using consumer gear like Raspberry pis and Intel NUCs and the rack fully fills the doorway so volume and heat is not a real issue.
- Karrot_Kream 5y agoI'm curious how much you're paying for business fiber. Do you have IP transit in your home, or is it just a business connection?
- Godel_unicode 5y ago100mb FiOS Business is on the order of $70.
- warent 5y agoCenturylink gigabit business fiber is about $140/mo, and a block of 8 static IPv4s is $25/mo
- lrvick 5y ago$300/mo for 300/300 with 5 clean static IPs suitable for nameservers, mail servers, etc.
- Yuioup 5y agoYes but you have talent and a lifetime of experience, plus space for a noisy 42u rack full of servers, but not everybody does...
- m8s 5y agoThey never suggested otherwise. They simply shared their current setup.
- sseagull 5y agoPeople who have been woodworking for decades can own very expensive tools so that they can create very complicated things. People who are experts in cars can own very expensive cars and tools to tune them. People who have been working in music can have very expensive instruments and expensive headphones, microphones, sequencers, etc. We seem to be looking down on experienced "computer experts" and wanting to take their tools away. It's been grinding my gears lately.
- hackthefender 5y agoI don't think that's the point the commenter was making. The analogous situation would be if someone posted that they made their kitchen table from scratch, and the commenter said that it's great but not everyone has a lathe in their basement, so good that places like Ikea exist as well.
- cerved 5y agobruh nobody's looking down on people running they're own metal Server hardware is fun but it's not trivial to manage, buy or run. So when someone talks about how they've managed servers for 2 decades, own a house where they can install a 42 rack and how much better it is than a hosted solution. A lot of people rightly point out that this is hardly feasible for most people
- DarylZero 5y agoPeople don't need a 42U rack to put a server in their closet though. You buy some surplus server on ebay and throw it on a shelf, no big deal.
- c2h5oh 5y agoI remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.
- papito 5y agoLOL. Priceless. Having these skills is very valuable. Us old farts used to do a lot with what today would be called "bootstrapped". Scarcity is no longer a "problem", except that it is. Scarcity keeps you sharp, efficient, on the edge - where you need to be. It's also - cheaper.
- c2h5oh 5y agoWho would have guessed having local, low latency, high iops drives would be better than VM using iSCSI-attached drive for storage, right? ;-)
- dilyevsky 5y agoTbf iscsi and especially it’s future demise - nvmeof can be made quite fast. The problem that in gp2/3, gce-pd cases it’s probably not some native storage but another layer of software-based distributed store which makes stuff slow/expensive
- bushbaba 5y agoBut also less prone to single drive failures. And easily auto expanding storage sizes. Everything is a trade off.
- dilyevsky 5y agoExcept people end up running systems on them that have their own replication built in like kafka, cockroach etc. In all those cases you’d be better off with your node just hard crashing if disk goes bad rather than the latency penalty that inevitably follows the failover
- massysett 5y agoWhat services would you be paying “several hundred a month” for - in what currency? In US dollars my personal cloud services don’t run more than $20 a month.
- reacharavindh 5y agoYou can also pick and choose how far you go with self management. To me, as much fun as it is, I can’t be bothered to run the physical server in my home(not because I can’t, but because I won’t). That’s why my solution is to lease a bare metal server on Hetzner, pay EUR 38/month and host all my personal projects on it. This way I get a reliable network and operation for my server and I still hold reasonable control over my data and what I run.
- sandworm101 5y agoI'm in a similar boat. I had a server in a closet but switched to a NAS a few years ago and haven't looked back. I do run into people who thinks NAS is old hat, that everything should be backed to a cloud somewhere. These are the people who think they are watching 4k because they select that option on youtube. I want to watch 4k without buffering, without artifacts. NAS lets me do that. And when I go away for a few days or weeks, everything is turned off. No data leaks possible when the "spinning rust" isn't spinning.
- dspillett 5y agoI assume/hope you have good, tested, off-site backups if the important data & config… I run my stuff from home too, though it is smaller scale than yours currently. Off-site & soft-offline backups are on encrypted volumes on servers & VMs elsewhere.
- Aeolun 5y agoThe backup server in my home is the offsite backup. The primary data lives in the datacenter.
- dspillett 5y agoUntil recently my home server was not capable in many respects to the outside machines (picked for cheap storage, not CPU/RAM considerations, going with the "redundant array of inexpensive backup locations" method), but I do currently have a variable dedicated server that might see some live workloads moved to it.
- bamboozled 5y ago> I know where our data lives and no one can access it without a warrant and my explicit knowledge As far as you know? Your data is exposed to The Internet so someone could be accessing it.
- lrvick 5y agoYou forget the wonders of TLS when you control both ends.
- bamboozled 5y agoI'm not talking about people snooping your data, I'm talking about a configuration error or exploit which gives people access to your information. Maybe I am naive, but I think the chances of this happening on a cloud based service like Dropbox are going to be a lot lower.
- hackerfromthefu 5y agoIt used to be the running joke that another large commercial database was just found unsecured on AWS! It's not as common in the last year or so which is a good sign!
- hackerfromthefu 5y agoOh, here we go! Just a week ago, thousands of unsecured databases on AWS! https://infosecwriteups.com/how-i-discovered-thousands-of-open-databases-on-aws-764729aa7f32 https://infosecwriteups.com/how-i-discovered-thousands-of-op...
- sjtindell 5y agoJust curious, what data do you worry about a warrant for? Asking to protect my own data.
- lrvick 5y agoWith a lawful warrant they can look at whatever they wish with my explicit consent and knowledge to make sure they do not exceed the law, because I have heard enough first and second hand horror stories to know how abusive investigators can be, particularly from ex investigator friends. I work with a lot of people in the security industry, particularly those in countries or organizations that might not take kindly to those activities. I have a similar responsibility as a journalist to protect my sources, vulnerabilities still under embargo, etc. I however mostly detest how almost every third party SaaS sells out our metrics and data for profit, long term consequences be damned. The less I empower profit maximizing machine learning to manipulate me and my family the better.
- freedomben 5y agoYou can do an amazing amount of work on a raspberry pi these days. I have a couple of R620s that were running several services. I lost a drive and one went down so I stood up a Pi4 (8GB) to quickly get things back up, and I was amazed that it handled a huge piece of the load, much more than I expected. Definitely worth a try for somebody just starting out.
- scubbo 5y agoReading this as my home Wireguard-host has just gone mysteriously unresponsive for the second time in a month[0], rendering all my home-hosted service inaccessible - you're right about the pros _and_ the cons. I'm not shying away from this self-hosted adventure, but I'm also realistic about the downsides. [0] https://raspberrypi.stackexchange.com/questions/135610/connecting-ethernet-cable-to-raspberry-pi-causes-wifi-to-drop-and-ethernet-does https://raspberrypi.stackexchange.com/questions/135610/conne...
- exikyut 5y agoThe TL;DR of [0], for visibility: [ 150.076220] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:468 dev_watchdog+0x308/0x30c [ 150.076255] NETDEV WATCHDOG: eth0 (bcmgenet): transmit queue 1 timed out Essentially Ethernet chucks a wobbly until reboot and it's only possible to connect via Wi-Fi.
- ineedasername 5y ago42u? That's quite a family!
- walrus01 5y agoI have done pretty much the same as you, except that instead of an on-premises 42U rack of stuff, a lot of my personal and family infrastructure lives in a single beefy 1U server (32 cores, 512GB of RAM, set up as a hypervisor) in colocation at a major IX point. Then there's a set of about six or seven $4 per month KVM VM at geographically distributed offsite 3rd party hosting companies for things like secondary authortative nameservers and such. And an offsite backup system that mirrors everything on the system in colocation to a medium sized disk array that lives in the corner of a closet in a family member's basement.
- alexgaribay 5y agoI self-host my stuff as well but with a 2012 Mac Mini running Proxmox. I use it as a means to learn sysadmin things. I get my static IP by using the smallest VPS from vultr with a wireguard tunnel forwarding http/s traffic to a docker container running nginx proxy manager. For those wanting to learn, I highly recommend joining r/homelab and r/selfhosted. Those communities have a lot in common and you can learn a lot.
- pojzon 5y agoWhy not simply use DyDns to your local home server ? Most modern routers have an option to integrate with dynamic dns providers.
- kafkaIncarnate 5y ago42U rack?? What do you have in it? I have a 2U self built SAN and a 2U Dell server. The SAN has 24TB on it and can scale up to 96TB. At most I'd need a backup of that and a replica of the other server for most ideal purposes at home (which are obscene for home use), so seriously what could you possibly need beyond a good virtualization server and lots of storage?
- taubek 5y agoI'm where I'm today because of my love for Linux. In the 90s my go to distribution was Slackware. I had it spread across several dozens of floppy disks. Knowing how to keep your server running and understanding internals is a great skill, but that doesn't mean that progress should stop. Standalone servers are great, but this greatness comes at a price. It takes time to maintain server, it takes time to configure additional services. But at the same time they bring you joy (and frustrations) and much more knowledge and deeper understanding of what goes under the hood.
- BrandoElFollito 5y agoSame here. There is, at least for me, one big serious downside: what happens when I die. My wife is quite concerned about her being left with a web of home automation, hosted emails etc. I understand her and I am trying to find a way out. My current idea is to document how to de-automatize the home and how to deal with emails and fiber access (the main things to worry about). Any ideas are very much welcome
- theK 5y agoAm at the same point. Have been thinking about a variety of options 1. bidirectional pacts with self-hosting friend 2. creating a network and culture of self hosting techies that develop the skills to assist (money and documentation required here but more resilient than no 1) 3. investing in your kids. Technically option 3 should be the best since it also engineers around generations (options 1 and 2 would roughly be locked to a narrower age group). But it also can be a double edge sword, what if they don’t like tech? Or if you overdo it with trying to make them interested in tech and self hosting and it backfires? So yeah, no real solution yet. But I’d subscribe to that newsletter if there was one
- BrandoElFollito 5y ago1 and 2 are interesting but I have some doubts about the feasibility in my region. It is already quite hard to find people interested in automation with Home Assistant, so having someone who would be willing to understand the setup and scripts of someone else would be really tricky. 3 could have been a solution but my kids are not interested in development (if they would, it would have been my first solution)
- kilroy123 5y agoWow that's something I never really considered before. I think the documentation would be a good first start. Maybe some kind of script(s) that could be run that just do all the de-automation?
- BrandoElFollito 5y ago
- dorfsmay 5y ago> when it comes to me and my family I Who will keep maintaining your infra when you die, say you get hit by the infamous bus later today?
- vaylian 5y agoI can't answer for OP. But I don't think that the situation would be any better if the services are hosted in the cloud. Stuff needs to be maintained either way.
- dorfsmay 5y agoNot full services like Gmail pCloud etc... Non-tech people can keep using those services and can access tech support even if OP is no longer available.
- crisper78 5y agoAwesome! Kudos for owning your own data and infra! I don't think systems administration is a lost art, people just wanted to do things easier for less money and staff, and no one can blame them or the industry as a whole. The art is still right there, linux hasn't changed that much in 20+ years, I still have my first discs. It also takes some experience to "feel" your way through systems administration problems, working from the bottom up of course, feeling like its a network problem still has a high rate of success when troubleshooting for me! I try everyday to teach some of these skills to people I work with, I just call it devops, or SRE, or some cyber cloud support position someone makes up, they are all systems admins/engineers to me still. I enjoy watching people learn and apply that knowledge to future problems, getting to the root cause or close to it, and the satisfaction that comes from fixing the issue from start to finish on your own, looking things up is not cheating in systems administration!