4 ms·

I feel the author presents a false dichotomy. I would argue that production servers should be immutable appliances. They should not have mutable root file-syst

by lrvick 5y ago

I feel the author presents a false dichotomy. I would argue that production servers should be immutable appliances. They should not have mutable root file-systems, they should not have package managers, they should not have ssh or require direct modification. Those are security accidents waiting to happen. I do not babysit my servers in my home rack. I have a company to run and better things to do just like the author, but that is no reason to give up freedom, privacy, and security. I like my servers to be as reliable and low maintenance as a refrigerator. They are not pets or a garden. Try container optimized appliance operating systems like Talos OS or application specific firmware-like images like Homeassistant provides and you can generally set up services with automatic updates and forget about them for extended periods of time, knowing they will auto-rollback and notify you if manual intervention is ever required. If they fully fail you can just restore a new stack in minutes from a git commit and your last backup.

edgyquant 5y ago
Interesting. Are there any OS’ that implement such a structure? Sounds kind of like chromeos if it was geared towards servers.
- jleedev 5y ago
  Container Optimized OS is quite literally “We let the Chromium OS folks make a server OS.” https://cloud.google.com/container-optimized-os/ https://cloud.google.com/container-optimized-os/
- politician 5y ago
  Nix?
  - edgyquant 5y ago
    I don’t think nix has an immutable filesystem, servers etc. Im sure you could build an imagine like that but you could do that with any distro.
    - elikoga 5y ago
    - alberth 5y ago
      Nanobsd has immutable file system. Mostly used for routers. https://docs.freebsd.org/en/articles/nanobsd/ https://docs.freebsd.org/en/articles/nanobsd/
  - BFLpL0QNek 5y ago
    This is a great blog post on how to do this on nix, https://grahamc.com/blog/erase-your-darlings https://grahamc.com/blog/erase-your-darlings . My Nix machine is stateless apart from a few directories such as home that I want to remain across reboots. When I shutdown/restart the server anything not in a mount I’ve explicitly configured to be persisted, gets erased. The initial setup was a bit of work but the end result works flawlessly.
- mmh0000 5y ago
  There are several options, one of the most popular Linux options right now is Fedora Silverblue[0] "The toolbox is using container technology to bring back your familiar tools and development environment on top of the immutable Silverblue base, for the best of both worlds." [0] https://silverblue.fedoraproject.org/ https://silverblue.fedoraproject.org/
- lrvick 5y ago
  There is Flatcar Linux which is literally based on ChromeOS for standalone systems. For high availability Talos Linux is the way to go with their k8s-appliance-os approach, IMO.
tyingq 5y ago
There is still some work there for the use cases he's mentioned (git hosts, backups, web servers). Rotating https certs, adding mutable volumes for backups, ssh (you say you don't need it, but it's pretty useful if you're running a git server), and so on. So, yes, they can do much of what you're describing, but it doesn't always work out of the box...there's work involved.
- lrvick 5y ago
  I run ssh for git inside a "from scratch" container with the git binary running unprivileged as an application. There is no need to interact with the host operating system. Backups can be handled by a backup container. If the host OS is stateless, there is nothing to backup. All you care about are data volumes, which you can then use any automated container daemon backup solution for. A very well solved problem. For rotating https certs, there are endless letsencrypt containers that automate that for you too. The point is you do the work for what you need up front, once, and then you don't need much work to keep it maintained.
- kelnos 5y ago
  I think the point there is that all of what you've mentioned are one-time "costs". You set them up in the beginning (and presumably the parent is managing this stuff in Ansible or Chef or whatever, so if the box irreparably dies, it's not hard to provision a new one) and then you don't need to worry about them. (The one recurring thing is TLS cert rotation, but for a home/personal server I'd presume most people would just use Let's Encrypt via certbot, which will auto-renew via cron.)
- prmoustache 5y ago
  Most of my services are at home in a dedicated lan but I still have one vps server that is used as a reverse proxy and which forward everything via a tunnel. The OS itself doesn't need much maintenance. I have set up auto updates and auto reboot on a regular basis. And you know what? Distros do their job very well and it just works. We are talking home service, I can deal with (less than) 5 minutes of downtime once in a week. I use rhel derivated distro which means major disruptive upgrade are only dealt with once every n years. I may choose an immutable distro in the future when support ends though. Rotating https certs? Certbot makes automatic rotation easy. ssh? What is the deal? You set it up basically once. Add a bit of port knocking if you want more silent logs. To add a bit of security, set up crowdsec, fail2ban. Again, these tools do not require a lot of maintenance. Most of the burden of maintenance comes from the management of the apps/services you want to have live. PaaS won't really help you much with that. SaaS tend to be expensive or lock you, they help if the service you want to maintain is complex. I don't think that is the case for the services described in the OP: static websites, git and backups. There are traps though. For example should you run containers to easily maintain certain services or use the distro packages? You can argue that containers whom upstream project provide image for allows you to easily upgrade any service. I can switch from any postgresql version in a snap for instance. However while it is easy to track and set up auto update a major version using a tag, you might want to be more cautious with major versions bumps. And this keeping informed with the various upstream project, know when and why to do major upgrade, what are the impact and potentially maintenance task of any major release upgrade. This is the hardest part. Sometimes it might be best forgetting about using upstream container for some services and sticking to the version supported by your long term maintenance distro that does the job of backporting any security patch to have that peace of mind. You can still run it as a container, but using the distro packages. That is what I do for databases for example.
deleted 5y ago
[deleted]

by lrvick 5y ago

edgyquant 5y ago

Interesting. Are there any OS’ that implement such a structure? Sounds kind of like chromeos if it was geared towards servers.

jleedev 5y ago
Container Optimized OS is quite literally “We let the Chromium OS folks make a server OS.” https://cloud.google.com/container-optimized-os/ https://cloud.google.com/container-optimized-os/
politician 5y ago
Nix?
- edgyquant 5y ago
  I don’t think nix has an immutable filesystem, servers etc. Im sure you could build an imagine like that but you could do that with any distro.
  - elikoga 5y ago
  - alberth 5y ago
    Nanobsd has immutable file system. Mostly used for routers. https://docs.freebsd.org/en/articles/nanobsd/ https://docs.freebsd.org/en/articles/nanobsd/
- BFLpL0QNek 5y ago
  This is a great blog post on how to do this on nix, https://grahamc.com/blog/erase-your-darlings https://grahamc.com/blog/erase-your-darlings . My Nix machine is stateless apart from a few directories such as home that I want to remain across reboots. When I shutdown/restart the server anything not in a mount I’ve explicitly configured to be persisted, gets erased. The initial setup was a bit of work but the end result works flawlessly.
mmh0000 5y ago
There are several options, one of the most popular Linux options right now is Fedora Silverblue[0] "The toolbox is using container technology to bring back your familiar tools and development environment on top of the immutable Silverblue base, for the best of both worlds." [0] https://silverblue.fedoraproject.org/ https://silverblue.fedoraproject.org/
lrvick 5y ago
There is Flatcar Linux which is literally based on ChromeOS for standalone systems. For high availability Talos Linux is the way to go with their k8s-appliance-os approach, IMO.

tyingq 5y ago

There is still some work there for the use cases he's mentioned (git hosts, backups, web servers). Rotating https certs, adding mutable volumes for backups, ssh (you say you don't need it, but it's pretty useful if you're running a git server), and so on. So, yes, they can do much of what you're describing, but it doesn't always work out of the box...there's work involved.

lrvick 5y ago
I run ssh for git inside a "from scratch" container with the git binary running unprivileged as an application. There is no need to interact with the host operating system. Backups can be handled by a backup container. If the host OS is stateless, there is nothing to backup. All you care about are data volumes, which you can then use any automated container daemon backup solution for. A very well solved problem. For rotating https certs, there are endless letsencrypt containers that automate that for you too. The point is you do the work for what you need up front, once, and then you don't need much work to keep it maintained.
kelnos 5y ago
I think the point there is that all of what you've mentioned are one-time "costs". You set them up in the beginning (and presumably the parent is managing this stuff in Ansible or Chef or whatever, so if the box irreparably dies, it's not hard to provision a new one) and then you don't need to worry about them. (The one recurring thing is TLS cert rotation, but for a home/personal server I'd presume most people would just use Let's Encrypt via certbot, which will auto-renew via cron.)
prmoustache 5y ago
Most of my services are at home in a dedicated lan but I still have one vps server that is used as a reverse proxy and which forward everything via a tunnel. The OS itself doesn't need much maintenance. I have set up auto updates and auto reboot on a regular basis. And you know what? Distros do their job very well and it just works. We are talking home service, I can deal with (less than) 5 minutes of downtime once in a week. I use rhel derivated distro which means major disruptive upgrade are only dealt with once every n years. I may choose an immutable distro in the future when support ends though. Rotating https certs? Certbot makes automatic rotation easy. ssh? What is the deal? You set it up basically once. Add a bit of port knocking if you want more silent logs. To add a bit of security, set up crowdsec, fail2ban. Again, these tools do not require a lot of maintenance. Most of the burden of maintenance comes from the management of the apps/services you want to have live. PaaS won't really help you much with that. SaaS tend to be expensive or lock you, they help if the service you want to maintain is complex. I don't think that is the case for the services described in the OP: static websites, git and backups. There are traps though. For example should you run containers to easily maintain certain services or use the distro packages? You can argue that containers whom upstream project provide image for allows you to easily upgrade any service. I can switch from any postgresql version in a snap for instance. However while it is easy to track and set up auto update a major version using a tag, you might want to be more cautious with major versions bumps. And this keeping informed with the various upstream project, know when and why to do major upgrade, what are the impact and potentially maintenance task of any major release upgrade. This is the hardest part. Sometimes it might be best forgetting about using upstream container for some services and sticking to the version supported by your long term maintenance distro that does the job of backporting any security patch to have that peace of mind. You can still run it as a container, but using the distro packages. That is what I do for databases for example.

deleted 5y ago

[deleted]