6 ms·

Speaking of a multipurpose home server, how do you guys compartmentalize it so that one faulty or vulnerable app doesn't take the whole thing down? Docker/cont

by blfr 4y ago

Speaking of a multipurpose home server, how do you guys compartmentalize it so that one faulty or vulnerable app doesn't take the whole thing down? Docker/containers used to not be hardened enough. Are they now? Virtualization/VMs used to be the answer but it adds both performance and management overhead. Is there a good system here? Or something else entirely? Like old school separate users.

melony 4y ago
I miss sandstorm.
- ryukafalz 4y ago
  I still use Sandstorm! Some of the apps are a bit outdated but the security model means that mostly doesn’t matter. The WordPress Sandstorm app is slow enough at rebuilding the static side of our large site that I’ve been meaning to try forking it or building my own though. But Sandstorm itself has been great.
- ocdtrekkie 4y ago
  It's still here and we're still working on it! It's 300th release just rolled out. I'm personally working on packaging three different apps right now.
kayson 4y ago
Docker is the de-facto standard in the community now (and, to a lesser extent, alternatives like LXC or podman). The daemon should be run rootless if possible, or the containers rootless if not. You can still use VMs, and some use that as an additional layer of isolation because they're virtualizing anyways (performance overhead is really negligible). I've been self-hosting on my home server for at least 5 years now, and I think I've only seen two or three vulnerabilities across all the services I know about, none of which were ever really exploitable.
- scrozart 4y ago
  Have you tried using kubernetes to manage your containers? Wondering if the extra level of complexity is worth it for a home server.
  - khimaros 4y ago
    it isn't
  - bongobingo1 4y ago
    Kubernetes alone recommends at least 1gb of ram just for itself IIRC, so that may push it out of some home servers such as RPIs or smaller nucs depending on the actual service load.
    - robertlagrant 4y ago
      K3s is half that. Still quite a lot, but not as much!
    - prmoustache 4y ago
      Not with k3s. But still, for a single or less than 3 machines and/or in a single location I don't see the point.
  - adra 4y ago
    I've dabbled, but really docker is way easier than k8s uses until you start moving into multi-server workloads
  - vineyardmike 4y ago
    100% not worth it. If you need multi-host for some reason (beyond “I want it” - and you don’t) then try docker swarm. It’s your home environment. You want it to be easy. You want to use the tools you run not maintain them. If you want to learn k8 for professional growth, learn it separately from a home server. Your home server can be more pet than cattle.
    - 4y ago
xxpor 4y ago
I use proxmox, which is more or less a VM and workflow manager on top of KVM. The overhead on something like an RPi would be ridiculous, but on modern x86 hardware with an IOMMU (VT-d in Intel speak, AMD-Vi for AMD), the overhead of passing through HW is, for homelab purposes, essentially 0. A lot more expensive, but the organization and extensibility is well worth it. I have anything that I expose directly to the internet on a separate VM from my "internal" services. If I were super paranoid, I'd expose them to separate VLANs, and then use my FW to control network traffic. The Intel 82599 can enforce different vlans on different VFs with SR-IOV. I have a VM that runs flatcar for docker for things that are too hard to set up otherwise, but I vastly prefer NixOS for most things.
- float4 4y ago
  > If I were super paranoid, I'd expose them to separate VLANs, and then use my FW to control network traffic This is exactly what I did initially, but it was indeed a bit of a pain to manage. Eventually I went with something in between, by first compartmentalizing services and then putting them in separate VMs with separate VLANs: 0. Router / FW. 1. WireGuard / reverse proxy. 2. Personal, e.g. file storage, backups. 3. Hosting. My personal site is reverse proxied through Cloudflare and only their IP ranges are whitelisted. 4. Compute, i.e. stuff I want to compile / develop / run on my server. Handy if I want to run a heavy simulation overnight or need more disk space / RAM / CPU power than my M1 MB Air has available. 5. Services. This runs many small tools / services that don't need access to my RAID pool or anything like that. If this gets infected I wouldn't really care. 6. VPN. This VM can only access the internet through a VPN. Doesn't have anything installed ATM, but has been used in the past for urlwatch and torrenting. 7. Test. This is where I try out new software before actually installing it on the correct VM. Once I've concluded testing I rollback this VM to a clean install. It takes a weekend to install Proxmox and set up the VMs / VLANs, but after that it easy to use.
- lostlogin 4y ago
  Another option is the free tier of ESXi. It works well, but having tested Proxmox recently, I really liked it.
- blibble 4y ago
dontlaugh 4y ago
I use Unraid, which manages storage for you and lets you run Docker containers for apps.
Spivak 4y ago
Containers are fine for this unless you reach the popularity where you are attracting dedicated attackers. Use userns-remap. Run the docker daemon rootless if you want but don’t stress about it. Set up auth to the docker socket. Don’t bother with running the processes in the container as not uid 0, with remap it’s effort for little gain. Now breaking containment means having a local privesc on your Linux distro or breaking the auth on the docker socket. Like that’s plenty for drive by attackers.
sekh60 4y ago
I use one VM per component. The overhead is pretty minimal and VMs I think are still more secure than containers. Maybe I am just a tech dinosaur though. I run my VMs on OpenStack for the networking flexibility, and use Ceph for block and file system.
onehair 4y ago
I use docker containers with separate dedicated users with just enough permissions for their purpose. For example my media server user can't touch anything other than the media files and isn't part of sudo.
ocdtrekkie 4y ago
You're looking for Sandstorm containers. They are much more hardened and purpose-built for self-hosting. To my knowledge, nobody's ever reported a container escape that affects Sandstorm.
the_third_wave 4y ago
ProxMox running containers wherever possible - which is nearly everywhere except for when you need to run different OSs (Windows, Android, etc.). Even the router runs in a container with all the other containers connecting to it through bridges. These bridges are assigned VLANs which are brought out tagged on one of the Ethernet ports which connects to a managed switch which takes care of untagging to specific ports and/or trunking VLANs to the different buildings on the farm.
NexRebular 4y ago
SmartOS with zones. Mostly native but some LX thrown in for too linux-specific software.
LaputanMachine 4y ago
I create a separate user for each app, and use the systemd exec configuration [1] for sandboxing [2]. Some apps only get read-only access to their own files, and no Internet access, for example (along with many other restrictions). I have some systemd drop-in units that I frequently reuse. For standard services, I use Apparmor with the default `apparmor-profiles`, as well as fail2ban with some additional firewall rules. [1]: https://man.archlinux.org/man/systemd.exec.5 https://man.archlinux.org/man/systemd.exec.5 [2]: https://wiki.archlinux.org/title/User:NetSysFire/systemd_sandboxing https://wiki.archlinux.org/title/User:NetSysFire/systemd_san...
Havoc 4y ago
VMs and LXC and occasionally some docker. Modern hardware is powerful enough that overhead is a non-issue for most of these apps Generally your just serving a single user - you - so even potato grade gear is fine
kiririn 4y ago
> Docker/containers used to not be hardened enough. Are they now? I don’t think they ever will be. At least once a year there is a kernel bug where root in a non-root container/namespace can be elevated to root on the host
Melatonic 4y ago
VM's do not actually add that much overhead (depending on the workload - GPU's are notoroiously hard to share). And what most people do not realize is that something like VMware or another hypervisor is also very good at managing things like RAM across many VM's. In many cases you can overprovision (meaning you can have VM's that are technically "assigned" a total amount of RAM or CPU that is more than you even have physically) and still have great performance. The key is always to install the hypervisor on bare metal (dont run VMware on top of Windows or try to host a server where the "base" os is OSX or something).

by blfr 4y ago

melony 4y ago

I miss sandstorm.

ryukafalz 4y ago
I still use Sandstorm! Some of the apps are a bit outdated but the security model means that mostly doesn’t matter. The WordPress Sandstorm app is slow enough at rebuilding the static side of our large site that I’ve been meaning to try forking it or building my own though. But Sandstorm itself has been great.
ocdtrekkie 4y ago
It's still here and we're still working on it! It's 300th release just rolled out. I'm personally working on packaging three different apps right now.

kayson 4y ago

Docker is the de-facto standard in the community now (and, to a lesser extent, alternatives like LXC or podman). The daemon should be run rootless if possible, or the containers rootless if not. You can still use VMs, and some use that as an additional layer of isolation because they're virtualizing anyways (performance overhead is really negligible). I've been self-hosting on my home server for at least 5 years now, and I think I've only seen two or three vulnerabilities across all the services I know about, none of which were ever really exploitable.

scrozart 4y ago
Have you tried using kubernetes to manage your containers? Wondering if the extra level of complexity is worth it for a home server.
- khimaros 4y ago
  it isn't
- bongobingo1 4y ago
  Kubernetes alone recommends at least 1gb of ram just for itself IIRC, so that may push it out of some home servers such as RPIs or smaller nucs depending on the actual service load.
  - robertlagrant 4y ago
    K3s is half that. Still quite a lot, but not as much!
  - prmoustache 4y ago
    Not with k3s. But still, for a single or less than 3 machines and/or in a single location I don't see the point.
- adra 4y ago
  I've dabbled, but really docker is way easier than k8s uses until you start moving into multi-server workloads
- vineyardmike 4y ago
  100% not worth it. If you need multi-host for some reason (beyond “I want it” - and you don’t) then try docker swarm. It’s your home environment. You want it to be easy. You want to use the tools you run not maintain them. If you want to learn k8 for professional growth, learn it separately from a home server. Your home server can be more pet than cattle.
  - 4y ago

xxpor 4y ago

I use proxmox, which is more or less a VM and workflow manager on top of KVM. The overhead on something like an RPi would be ridiculous, but on modern x86 hardware with an IOMMU (VT-d in Intel speak, AMD-Vi for AMD), the overhead of passing through HW is, for homelab purposes, essentially 0. A lot more expensive, but the organization and extensibility is well worth it. I have anything that I expose directly to the internet on a separate VM from my "internal" services. If I were super paranoid, I'd expose them to separate VLANs, and then use my FW to control network traffic. The Intel 82599 can enforce different vlans on different VFs with SR-IOV. I have a VM that runs flatcar for docker for things that are too hard to set up otherwise, but I vastly prefer NixOS for most things.

float4 4y ago
> If I were super paranoid, I'd expose them to separate VLANs, and then use my FW to control network traffic This is exactly what I did initially, but it was indeed a bit of a pain to manage. Eventually I went with something in between, by first compartmentalizing services and then putting them in separate VMs with separate VLANs: 0. Router / FW. 1. WireGuard / reverse proxy. 2. Personal, e.g. file storage, backups. 3. Hosting. My personal site is reverse proxied through Cloudflare and only their IP ranges are whitelisted. 4. Compute, i.e. stuff I want to compile / develop / run on my server. Handy if I want to run a heavy simulation overnight or need more disk space / RAM / CPU power than my M1 MB Air has available. 5. Services. This runs many small tools / services that don't need access to my RAID pool or anything like that. If this gets infected I wouldn't really care. 6. VPN. This VM can only access the internet through a VPN. Doesn't have anything installed ATM, but has been used in the past for urlwatch and torrenting. 7. Test. This is where I try out new software before actually installing it on the correct VM. Once I've concluded testing I rollback this VM to a clean install. It takes a weekend to install Proxmox and set up the VMs / VLANs, but after that it easy to use.
lostlogin 4y ago
Another option is the free tier of ESXi. It works well, but having tested Proxmox recently, I really liked it.
blibble 4y ago

dontlaugh 4y ago

I use Unraid, which manages storage for you and lets you run Docker containers for apps.

Spivak 4y ago

Containers are fine for this unless you reach the popularity where you are attracting dedicated attackers. Use userns-remap. Run the docker daemon rootless if you want but don’t stress about it. Set up auth to the docker socket. Don’t bother with running the processes in the container as not uid 0, with remap it’s effort for little gain. Now breaking containment means having a local privesc on your Linux distro or breaking the auth on the docker socket. Like that’s plenty for drive by attackers.

sekh60 4y ago

I use one VM per component. The overhead is pretty minimal and VMs I think are still more secure than containers. Maybe I am just a tech dinosaur though. I run my VMs on OpenStack for the networking flexibility, and use Ceph for block and file system.

onehair 4y ago

I use docker containers with separate dedicated users with just enough permissions for their purpose. For example my media server user can't touch anything other than the media files and isn't part of sudo.

ocdtrekkie 4y ago

You're looking for Sandstorm containers. They are much more hardened and purpose-built for self-hosting. To my knowledge, nobody's ever reported a container escape that affects Sandstorm.

the_third_wave 4y ago

ProxMox running containers wherever possible - which is nearly everywhere except for when you need to run different OSs (Windows, Android, etc.). Even the router runs in a container with all the other containers connecting to it through bridges. These bridges are assigned VLANs which are brought out tagged on one of the Ethernet ports which connects to a managed switch which takes care of untagging to specific ports and/or trunking VLANs to the different buildings on the farm.

NexRebular 4y ago

SmartOS with zones. Mostly native but some LX thrown in for too linux-specific software.

LaputanMachine 4y ago

I create a separate user for each app, and use the systemd exec configuration [1] for sandboxing [2]. Some apps only get read-only access to their own files, and no Internet access, for example (along with many other restrictions). I have some systemd drop-in units that I frequently reuse. For standard services, I use Apparmor with the default `apparmor-profiles`, as well as fail2ban with some additional firewall rules. [1]: https://man.archlinux.org/man/systemd.exec.5 https://man.archlinux.org/man/systemd.exec.5 [2]: https://wiki.archlinux.org/title/User:NetSysFire/systemd_sandboxing https://wiki.archlinux.org/title/User:NetSysFire/systemd_san...

Havoc 4y ago

VMs and LXC and occasionally some docker. Modern hardware is powerful enough that overhead is a non-issue for most of these apps Generally your just serving a single user - you - so even potato grade gear is fine

kiririn 4y ago

> Docker/containers used to not be hardened enough. Are they now? I don’t think they ever will be. At least once a year there is a kernel bug where root in a non-root container/namespace can be elevated to root on the host

Melatonic 4y ago

VM's do not actually add that much overhead (depending on the workload - GPU's are notoroiously hard to share). And what most people do not realize is that something like VMware or another hypervisor is also very good at managing things like RAM across many VM's. In many cases you can overprovision (meaning you can have VM's that are technically "assigned" a total amount of RAM or CPU that is more than you even have physically) and still have great performance. The key is always to install the hypervisor on bare metal (dont run VMware on top of Windows or try to host a server where the "base" os is OSX or something).