4 ms·

i started self-hosting a bunch of stuff last month: Pleroma (like Mastodon/Twitter), Matrix (chat), Gitea (like Github) and Jellyfin (like Plex, a media server)

i started self-hosting a bunch of stuff last month: Pleroma (like Mastodon/Twitter), Matrix (chat), Gitea (like Github) and Jellyfin (like Plex, a media server). AFTER i set up the hardware/OS, these each took about 1-2 hours to setup, and it gets faster each time as i get more accustomed to the common parts (nginx, systemd, Lets Encrypt, and whatever containerization you use). today i accidentally nuked everything by not flushing the disk before rebooting and then naively letting fsck try to ‘fix’ it (which just makes things worse since it unlinks every inode it thinks is wrong instead of helping you recover data). now i’m manually dumping blocks and re-linking them in, supplementing whatever’s not recoverable with a 3-day old backup. that’s probably gonna take an entire day to fix up. after this i have to figure out a better backup solution, because it costs me $5 of API requests every time i rclone the system to Backblaze, making frequent backups too expensive. after that, i have to figure out the email part of things. AFAICT it’s pretty much impossible to 100% self-host email because of blacklisting. you have to at least proxy it through a VPS, or something. and in between that i may spin up a DNS server to overcome the part where it takes 60min for any new service to be accessible because of the negative caching common in DNS. no, this stuff is just way too involved for anyone who hasn’t already spent a decade in the CLI. i’m only doing this because i’m a nerd with time on his hands between jobs. self-hosting isn’t gonna catch on this decade. but maybe we can federate, so that you just need to have one friend who cares about this stuff manage the infra and provide it to their peers as a social good. also, i don’t think privacy is the right angle for promoting self-hosting. a good deal of the things that people self-host have a public-facing component (websites; public chatrooms; etc). if privacy is what you seek, then you should strive to live life offline. the larger differentiator for self-housing is control.

spacemanmatt 5y ago
After I've built a server for a purpose, the one thing I want most is a script that does it again. Spending another identical hour on a similar server just makes me sad.
- southerntofu 5y ago
  That's precisely why i started working on https://codeberg.org/southerntofu/ansible-selfhosted https://codeberg.org/southerntofu/ansible-selfhosted Its abstractions are still a bit shaky and a lot can be improved so it's far from ready for the general public, but i still consider it a great step forward because for the supported configurations, all i have to do to setup a new server is: - edit config.yml - run roles/deploy.sh - enjoy I'm happy to answer any questions on why (politics) and how (technics) i'm building this, and i'd be more than thrilled to receive feedback and contribution. In the past week i started working on a test suite so it's easier to contribute.
southerntofu 5y ago
> today i accidentally nuked everything by not flushing the disk before rebooting What do you mean? Did you interrupt the reboot process (eg. repetitive ^C)? Otherwise the OS should flush everything properly. > after this i have to figure out a better backup solution If you have other friends doing selfhosting, giving them a HDD to put in their server so you can rsync your data is a good solution. Also, keeping another local backup is a good solution. Doing both is even better. > AFAICT it’s pretty much impossible to 100% self-host email because of blacklisting It depends. It's almost impossible with some bad-faith providers (google/microsoft) otherwise everything works well. And even with those bad-faith providers, residential IPs usually have better reputation (if only because your ISP likely blocks outgoing TCP 25 by default and you have to ask them to unfilter it) than VPS IPs which all have a long history of abuse. > and in between that i may spin up a DNS server to overcome the part where it takes 60min for any new service to be accessible If you didn't previously query that domain name on your resolver, it will not be cached and will resolve quasi-instantly. The question is how long does your authoritative name server take to apply your changes to the config: if it takes any longer than 10s you should consider switching providers. > but maybe we can federate, so that you just need to have one friend who cares about this stuff manage the infra and provide it to their peers as a social good. Very good point! That's the entire premise behind hosting coops like disroot.org or various members of the libreho.st/chatons.org federations.
- wallacoloo 5y ago
  > What do you mean? Did you interrupt the reboot process (eg. repetitive ^C)? Otherwise the OS should flush everything properly. here’s my best guess: in my setup i have a host running a qemu vm and most of the interesting stuff happens inside the vm. originally that vm image was just 8 GB, but then i got a HDD to dedicate to it. with VM powered off, i partitioned the HDD and then dd’d the VM image onto it. then i booted the VM via KVM passthrough of /dev/sdb… it booted fine; i ran ‘df’ and noticed that i forgot to resize the fs to the HDD, so i ran resize2fs. 3 days later, i `shutdown` the VM and then `reboot`d the host. the host didn’t actually come back: the power light and activity lights were off. after 5 minutes of this i power-cycled at the wall socket. host came back up. vm wouldn’t boot. ran fsck from the rescue shell. now it booted, but no services were operational. since i couldn’t login to the vm (ssh broken + password logins had long been deactivated), i shutdown the vm and mounted its fs on the host. ‘df’ showed that the host thought the fs was only 8 GB in capacity. i don’t think it was outright disk corruption, because the poweroff wasn’t that messy (but i come from btrfs, which has handled like 20 power faults on me w/ zero issue: idk how solid EXT4 is to these things). my best guess is that somewhere along the way, the changes from resize2fs didn’t actually make it to disk, or were overrided with stale in-memory values. maybe when i updated the guest’s kernel some post-upgrade script did something to push the old fs size to disk somewhere. or maybe the host had the old 8 GB fs size cached and flushed that during shutdown/start. unfortunately i’m not sure i’ll ever know.

spacemanmatt 5y ago

After I've built a server for a purpose, the one thing I want most is a script that does it again. Spending another identical hour on a similar server just makes me sad.

southerntofu 5y ago
That's precisely why i started working on https://codeberg.org/southerntofu/ansible-selfhosted https://codeberg.org/southerntofu/ansible-selfhosted Its abstractions are still a bit shaky and a lot can be improved so it's far from ready for the general public, but i still consider it a great step forward because for the supported configurations, all i have to do to setup a new server is: - edit config.yml - run roles/deploy.sh - enjoy I'm happy to answer any questions on why (politics) and how (technics) i'm building this, and i'd be more than thrilled to receive feedback and contribution. In the past week i started working on a test suite so it's easier to contribute.

southerntofu 5y ago

> today i accidentally nuked everything by not flushing the disk before rebooting What do you mean? Did you interrupt the reboot process (eg. repetitive ^C)? Otherwise the OS should flush everything properly. > after this i have to figure out a better backup solution If you have other friends doing selfhosting, giving them a HDD to put in their server so you can rsync your data is a good solution. Also, keeping another local backup is a good solution. Doing both is even better. > AFAICT it’s pretty much impossible to 100% self-host email because of blacklisting It depends. It's almost impossible with some bad-faith providers (google/microsoft) otherwise everything works well. And even with those bad-faith providers, residential IPs usually have better reputation (if only because your ISP likely blocks outgoing TCP 25 by default and you have to ask them to unfilter it) than VPS IPs which all have a long history of abuse. > and in between that i may spin up a DNS server to overcome the part where it takes 60min for any new service to be accessible If you didn't previously query that domain name on your resolver, it will not be cached and will resolve quasi-instantly. The question is how long does your authoritative name server take to apply your changes to the config: if it takes any longer than 10s you should consider switching providers. > but maybe we can federate, so that you just need to have one friend who cares about this stuff manage the infra and provide it to their peers as a social good. Very good point! That's the entire premise behind hosting coops like disroot.org or various members of the libreho.st/chatons.org federations.

wallacoloo 5y ago
> What do you mean? Did you interrupt the reboot process (eg. repetitive ^C)? Otherwise the OS should flush everything properly. here’s my best guess: in my setup i have a host running a qemu vm and most of the interesting stuff happens inside the vm. originally that vm image was just 8 GB, but then i got a HDD to dedicate to it. with VM powered off, i partitioned the HDD and then dd’d the VM image onto it. then i booted the VM via KVM passthrough of /dev/sdb… it booted fine; i ran ‘df’ and noticed that i forgot to resize the fs to the HDD, so i ran resize2fs. 3 days later, i `shutdown` the VM and then `reboot`d the host. the host didn’t actually come back: the power light and activity lights were off. after 5 minutes of this i power-cycled at the wall socket. host came back up. vm wouldn’t boot. ran fsck from the rescue shell. now it booted, but no services were operational. since i couldn’t login to the vm (ssh broken + password logins had long been deactivated), i shutdown the vm and mounted its fs on the host. ‘df’ showed that the host thought the fs was only 8 GB in capacity. i don’t think it was outright disk corruption, because the poweroff wasn’t that messy (but i come from btrfs, which has handled like 20 power faults on me w/ zero issue: idk how solid EXT4 is to these things). my best guess is that somewhere along the way, the changes from resize2fs didn’t actually make it to disk, or were overrided with stale in-memory values. maybe when i updated the guest’s kernel some post-upgrade script did something to push the old fs size to disk somewhere. or maybe the host had the old 8 GB fs size cached and flushed that during shutdown/start. unfortunately i’m not sure i’ll ever know.