5 ms·

This is a great read, but I'd like to hear more about the nitty gritty, what server hardware you use, it seems like you have 3 pis and then a bunch of really ad

by vxNsr 8y ago

This is a great read, but I'd like to hear more about the nitty gritty, what server hardware you use, it seems like you have 3 pis and then a bunch of really advanced enterprise-level stuff for other work... I'd love a guide on what you did to get here. The internet really needs something like that.

hkajhfjl 8y ago
I'm not the OP, but I have been doing this for some years now. Here's how I went about it. I got out of social media in early 2012. Deleted FB and Twitter. I do maintain a presence on LinkedIn, but I never post anything there, nor do I login. It's mostly there because employers expect a LI profile. It's just a bland copy of my resume. I moved my email from Gmail to my own domain from 2013-2015. It took almost two years because I had Gmail since April 2004, a few days after launch. My whole digital life was tied into Gmail, and moving stuff over takes time. I still have that Gmail account, but again, I never log into anything Google, and it simply forwards any email to my real account. I get maybe 1 email a month from Gmail. I host my email with Fastmail, but I've also selfhosted, and tried O365. It's easy to switch, and I keep an offline backup of everything. Now, my hardware and software setup. At home, I have a Synology NAS with ~12 TB of usable storage. This is my primary data store - I have software that automiatically backs up all photos from my Iphone, Time Machine, Veeam, Rysnc, and email backup. I own my own physical server (Dell R720XD) which is colocated in a datacenter in a different country. This server has 96 GB of RAM, 16 TB of storage, and 12 Xeon cores. It usually runs between 12-25 VMs, some of which are production (personal), hosting my websites, git, various projects, and so on. It's connected to the internet through a gigabit connection with a 10 TB monthly bandwidth cap (I never come close to 1 TB!). This costs me ~$85 a month. I don't do BGP, though I could if I paid some more and put in a proper router and got my own IP range.. as it stands, the 8 public IPs I get are enough for me. The colo server is connected via an always-on, site-to-site IPSEC VPN to my home LAN (through my EdgeRouter X). I have symmetric gigabit, so it's practically like having the machine in my house. My NAS backs up to the colo server, and the colo server to the NAS (via rsync). The NAS also backs up to a local USB drive, for quick restores. All my home data lives in four places 1. The local PC/laptop/phone. 2. The NAS 3. The USB Disk 4. The offsite colo server. Likewise, all my colo data lives in the same four places. I can VPN in to the colo server and get connected to my home lan from anywhere in the world. Any sensitive data is encrypted - and anything really sensitive is not stored on a computer. This setup works very well for me and my family, and I control all my data and am not dependent on a cloud provider. I also have a bunch of other servers running in my home, but they aren't part of my core infrastructure. The cost? ~$1200 for the R720. ~$1000 for the NAS. and ~$85 a month for colocation costs.
- dewey 8y ago
  Do you have measures in place to give access to someone trusted in case something happens to you? With a specialized setup like this hosting, for example all family pictures, I’d always be scared of having a low bus factor.
  - hkajhfjl 8y ago
    Yes and no. My spouse can easily access anything family related - pictures, documents, passwords (keepass) from their computer. The NAS and the Colo Server both present as shared drives on the network and they use it almost every day. The USB drive is plug and play and has copies of anything important. However, all my git repos, websites, tech projects etc will probably not survive my passing. I don't really care about that. They will still be backed up and accessible on the NAS, so if someone in my family (I have a few technically adept people in the family) wanted to check it out, my spouse would be able to give them access. Honestly, I don't think anyone would care, but who knows. All passwords and access details are in KeePass and my spouse uses that everyday so knows how to use it, My colo server is paid a year in advance every year, so it would chug along for a few months before the provider shut it down and shipped it back to my house (again, my spouse knows how to get it shipped back). All in all, I think my family would be able to access anything truly valuable without too much trouble, and who cares about the other stuff anyway.
- voltagex_ 8y ago
  Few more questions: * How do you backup Fastmail? * How'd you find your colo provider? * If it's in a different country, how does the latency affect you?
  - walterbell 8y ago
    Ditto on the colo provider question. Which virtualization do you use on server and how do you handle remote admin and security updates for guest VMs and host?
    - hkajhfjl 8y ago
fallenhitokiri 8y ago
OP here - since there seems to be some interest on the details I will try to answer them in as much detail as possible in a longer blog post. I already wrote about the network setup[1] The server is a Dell Precision T5610 (20 physical cores, 64GB memory, 4 Samsung Pro SSDs and quad gigabit NIC additionally to the internal one) and a QNap TS-531X with Seagate IronWolf drives. I would recommend against using PIs at this point, see other comment[2]. Right now all services are running in docker in a dedicated VM on VMWare Workstation. The server also hosts a full work environment I can RDP and SSH into, depending on what I do, when I am traveling only with my iPad Pro (RDP because my last job involved a lot of Java for which I prefer IntelliJ). At some point I should likely migrate to vSphere, but I had the workstation license already and it was quick to setup. [1] https://screamingatmyscreen.com/2018/7/building-our-home-and-office-network/ https://screamingatmyscreen.com/2018/7/building-our-home-and... [2] https://news.ycombinator.com/item?id=17966264 https://news.ycombinator.com/item?id=17966264
- 8fingerlouie 8y ago
  I'm probably beating a dead horse here with Docker Policies available and all, but i personally don't trust Docker in production. Each Docker image runs it's own stack, and each stack has the potential to contain vulnerabilities. Even with services such as Watchtower ( https://hub.docker.com/r/v2tec/watchtower/ https://hub.docker.com/r/v2tec/watchtower/ ), you're still not safe. Some images are abandoned for years, others are only updated once their "final" product is updated, meaning you could be hosting a handful of vulnerable services without even knowing it. I host all my internet facing stuff on FreeBSD in jails, though Linux with LXC would do just as good. I have one stack to update, and once that's updated, everything else is updated as well.
  - fallenhitokiri 8y ago
    Nothing I host is exposed to the Internet but only available locally. When on the road I VPN back home. But generally I agree, for Internet facing services I would likely make a few different decisions than in this case.

by vxNsr 8y ago

hkajhfjl 8y ago

I'm not the OP, but I have been doing this for some years now. Here's how I went about it. I got out of social media in early 2012. Deleted FB and Twitter. I do maintain a presence on LinkedIn, but I never post anything there, nor do I login. It's mostly there because employers expect a LI profile. It's just a bland copy of my resume. I moved my email from Gmail to my own domain from 2013-2015. It took almost two years because I had Gmail since April 2004, a few days after launch. My whole digital life was tied into Gmail, and moving stuff over takes time. I still have that Gmail account, but again, I never log into anything Google, and it simply forwards any email to my real account. I get maybe 1 email a month from Gmail. I host my email with Fastmail, but I've also selfhosted, and tried O365. It's easy to switch, and I keep an offline backup of everything. Now, my hardware and software setup. At home, I have a Synology NAS with ~12 TB of usable storage. This is my primary data store - I have software that automiatically backs up all photos from my Iphone, Time Machine, Veeam, Rysnc, and email backup. I own my own physical server (Dell R720XD) which is colocated in a datacenter in a different country. This server has 96 GB of RAM, 16 TB of storage, and 12 Xeon cores. It usually runs between 12-25 VMs, some of which are production (personal), hosting my websites, git, various projects, and so on. It's connected to the internet through a gigabit connection with a 10 TB monthly bandwidth cap (I never come close to 1 TB!). This costs me ~$85 a month. I don't do BGP, though I could if I paid some more and put in a proper router and got my own IP range.. as it stands, the 8 public IPs I get are enough for me. The colo server is connected via an always-on, site-to-site IPSEC VPN to my home LAN (through my EdgeRouter X). I have symmetric gigabit, so it's practically like having the machine in my house. My NAS backs up to the colo server, and the colo server to the NAS (via rsync). The NAS also backs up to a local USB drive, for quick restores. All my home data lives in four places 1. The local PC/laptop/phone. 2. The NAS 3. The USB Disk 4. The offsite colo server. Likewise, all my colo data lives in the same four places. I can VPN in to the colo server and get connected to my home lan from anywhere in the world. Any sensitive data is encrypted - and anything really sensitive is not stored on a computer. This setup works very well for me and my family, and I control all my data and am not dependent on a cloud provider. I also have a bunch of other servers running in my home, but they aren't part of my core infrastructure. The cost? ~$1200 for the R720. ~$1000 for the NAS. and ~$85 a month for colocation costs.

dewey 8y ago
Do you have measures in place to give access to someone trusted in case something happens to you? With a specialized setup like this hosting, for example all family pictures, I’d always be scared of having a low bus factor.
- hkajhfjl 8y ago
  Yes and no. My spouse can easily access anything family related - pictures, documents, passwords (keepass) from their computer. The NAS and the Colo Server both present as shared drives on the network and they use it almost every day. The USB drive is plug and play and has copies of anything important. However, all my git repos, websites, tech projects etc will probably not survive my passing. I don't really care about that. They will still be backed up and accessible on the NAS, so if someone in my family (I have a few technically adept people in the family) wanted to check it out, my spouse would be able to give them access. Honestly, I don't think anyone would care, but who knows. All passwords and access details are in KeePass and my spouse uses that everyday so knows how to use it, My colo server is paid a year in advance every year, so it would chug along for a few months before the provider shut it down and shipped it back to my house (again, my spouse knows how to get it shipped back). All in all, I think my family would be able to access anything truly valuable without too much trouble, and who cares about the other stuff anyway.
voltagex_ 8y ago
Few more questions: * How do you backup Fastmail? * How'd you find your colo provider? * If it's in a different country, how does the latency affect you?
- walterbell 8y ago
  Ditto on the colo provider question. Which virtualization do you use on server and how do you handle remote admin and security updates for guest VMs and host?
  - hkajhfjl 8y ago

fallenhitokiri 8y ago

OP here - since there seems to be some interest on the details I will try to answer them in as much detail as possible in a longer blog post. I already wrote about the network setup[1] The server is a Dell Precision T5610 (20 physical cores, 64GB memory, 4 Samsung Pro SSDs and quad gigabit NIC additionally to the internal one) and a QNap TS-531X with Seagate IronWolf drives. I would recommend against using PIs at this point, see other comment[2]. Right now all services are running in docker in a dedicated VM on VMWare Workstation. The server also hosts a full work environment I can RDP and SSH into, depending on what I do, when I am traveling only with my iPad Pro (RDP because my last job involved a lot of Java for which I prefer IntelliJ). At some point I should likely migrate to vSphere, but I had the workstation license already and it was quick to setup. [1] https://screamingatmyscreen.com/2018/7/building-our-home-and-office-network/ https://screamingatmyscreen.com/2018/7/building-our-home-and... [2] https://news.ycombinator.com/item?id=17966264 https://news.ycombinator.com/item?id=17966264

8fingerlouie 8y ago
I'm probably beating a dead horse here with Docker Policies available and all, but i personally don't trust Docker in production. Each Docker image runs it's own stack, and each stack has the potential to contain vulnerabilities. Even with services such as Watchtower ( https://hub.docker.com/r/v2tec/watchtower/ https://hub.docker.com/r/v2tec/watchtower/ ), you're still not safe. Some images are abandoned for years, others are only updated once their "final" product is updated, meaning you could be hosting a handful of vulnerable services without even knowing it. I host all my internet facing stuff on FreeBSD in jails, though Linux with LXC would do just as good. I have one stack to update, and once that's updated, everything else is updated as well.
- fallenhitokiri 8y ago
  Nothing I host is exposed to the Internet but only available locally. When on the road I VPN back home. But generally I agree, for Internet facing services I would likely make a few different decisions than in this case.