4 ms·

This appears to suffer from the same mistake as many of these things do in this space: it focuses on making it really easy to run lots of software, but has a ve

by raggi 3y ago

This appears to suffer from the same mistake as many of these things do in this space: it focuses on making it really easy to run lots of software, but has a very poor story when it comes to making the data and time you put in safe across upgrades and issues. The only documented page on backing up requires taking the entire system down, and there appears to be no guidance or provision for safely handling software upgrades. This sets people up for the worst kind of self-hosting failure: they get all excited about setting up a bunch of potentially really useful applications, invest time and data into them, then get burned really badly when it all comes crashing down in an upgrade or hardware failure, with improper preparation. This is how people move back to saas and never look back again, it's utterly critical to get right, and completely missing here.

yonixw 3y ago
That's exactly how I feel. Not to mention that I am always looking for how to monitor the system, and there is no uniform standard, if it is possible to monitor at all. As if there is a hidden message that apart from monitoring how much disk space is left or CPU everything else is irrelevant. But for such VMs that share many processes, it is exactly the opposite! I must know when there is a problem who exactly is responsible!
everforward 3y ago
I'm working on something similar, and the crux of that issue is configurability vs automation. I.e. it's very easy to make backups for a system that users can't configure at all. You just ship the image with some rsync commands or something and done. Once you start letting people edit the config files, you get into a spot where now you basically need to be able to parse the config files to read file paths. That often means making version-specific parsers for configuration options that are introduced or removed in some versions, or have differing defaults (i.e. in 1.2 if they don't set "storage_path" it's at /x/, but in 1.3 if they don't set it, it defaults to /y/). That gets to be a lot of work. Then it gets even worse when the users can edit the Docker config for the images, because all bets are off at that point. The user could do all kinds of weird, fucky shit that infinitely loops naive backup scripts because of host volume mounts or have one of the mounts actually be part of NFS mount on the host with like 200ms of latency so backups just hang for forever and etc. It's just begging for an infinite series of bugs from people who did something weird with their config and ended up backing up their whole drive instead of the Docker folder, or removing their whole root drive because they mount their host FS root in the backups folder and it got deleted by the "old backup cleanup" script, or who knows what. At some point, it's easier to just make your own setup where you define the limitations of that setup than it is to use someone else's setup but have to find the limitations on your own.
- apitman 3y ago
  The one thing you have working on your favor is that the more likely someone is to want to customize, the more likely they are to understand they have to be responsible for backups and updates. Is your project public? I'm working on one as well and would love to see what you have cooking.
  - everforward 3y ago
    It's not at the moment, but I wouldn't be opposed to chucking you some access if you wanted to look (or talk about joining, I'm solo at the moment). The rough gist is that it's for hosting dedicated servers for video games, with an emphasis on having an easier to use UI than most providers combined with a usable API/CLI for advanced users. Ie. Instead of editing config files as regular text files, I want to offer a view with drop downs for each config option, a description of what each value does. (with some hidden for orchestration reasons, like the port it listens on) Architecture-wise, it's a 3-tier app. HTTP API frontend talks to gRPC backend talks to Postgres. gRPC in that middle tier because I want to move the control plane for the agents into gRPC. Currently, the control plane consists of generating an Ansible inventory and executing it, and Ansible handles downloading the game if it doesn't exist, making dirs, SFTPing configs, install a Prometheus agent, etc. It's not anything I intend to hyperscale, but I've always been the dedicated server guy among my friends, so I figured I might as well make a little thing out of it. I'd be happy with a small ARR.
- megous 3y ago
  > Once you start letting people edit the config files, That way lies madness... What can be done instead is to provide your own unified (some schema validated) configuration options, and generate the app specific config files from your source of truth. Then you can know what the user can configure and how to back everything up (and how to do a lot of other things in automated fashion). And you also have a safe upgrade path if format of any underlaying config changes.
  - everforward 3y ago
    You can, but I find it very tedious. O2M or M2M relationships become cumbersome. Not because they're hard to use, but just because there's so fucking many of them. I opted to store the app and version, and have a function that retrieves a config parser for it. I just didn't want to manage 300 tables just to get the schema to work. That's also partially because I'm using Ent in Go, so it generates a struct for each table. It'd make my auto-complete useless, and that's not a price I'm willing to pay on a personal project lol
ocdtrekkie 3y ago
This is one of the reasons I groan every time there's a new selfhosting thing that uses Docker. It's easy to start because all the wiring is there and everyone already makes their apps available with a Dockerfile, but it's not a remotely good solution for selfhosting and people should stop trying to use it that way.
- apitman 3y ago
  Unfortunately Sandstorm's approach, ie rewrite all the apps to fit a specific ecosystem, hasn't taken off either. I think docker with a little extra tooling for managing updates and backups is likely to be the solution eventually.
  - ocdtrekkie 3y ago
    The problem is the Docker solution will never ever work and yet people are wasting thousands of person-hours trying, over and over in fact in dozens of redundant non-solutions. Sure Sandstorm or something like it will take a lot more effort, but at least it's moving in the correct direction. Just because Docker selfhosting has "taken off" doesn't mean it isn't still a dead end. EDIT: Also, to be clear, Docker is also a "specific ecosystem". It's very popular, it doesn't make it any more general than anything else.
    - EraYaN 3y ago
      Why would backups and restores never work though? The only thing you need to do is save your volumes/mounts essentially? The rest is a bunch of yaml in some docker-compose file for example. How is that "will never ever work"?
      - ocdtrekkie 3y ago
        The issue is that there is a lot more to a home server than updates and backups, and even doing those correctly automatically is... pretty hard if you don't have very strong opinions about how apps need to support them. So to give you an idea, Sandstorm requires: An app always must recover from forced termination (the only way it shuts apps down!) and that a given app carry all of the code it needs to update its data from any previous version. If you don't do these things, your homelabber will inevitably be fixing stuff themselves in various apps because their server lost power or the app developer demanded some bespoke intermediate inane update steps, or you missed a few versions while your server was off. What percentage of apps made available via Docker meets those requirements? Probably a lot less than you'd think. (Most of them that do easily work on Sandstorm!) But if you want to make selfhosting work for non-sysadmins (or sysadmins who don't want to do a second job when they get home from their main job), you have to do these things. Another thing is that you basically have to have security right because your users and even your server admin probably aren't experts, and definitely aren't paying for an enterprise-class firewall. So a big part of Sandstorm's opinionated app design is around making it fundamentally impossible for app vulnerabilities to exist. Things like proxying everything through the platform, isolating individual documents into their own app instances, handling authentication and authorization, etc. all become core to making apps secure enough to not have to worry about them that much. This is especially important because a lot of selfhosted apps get abandoned and unmaintained, but selfhosters still want to use them. Of course, all these things fundamentally require changes to the app your random Docker container shipper can't insist on, and that's before we get even into performance concerns, which are key since selfhosters tend to use old or cheap hardware.
stavros 3y ago
I wrote and use Harbormaster (https://harbormaster.readthedocs.io/ https://harbormaster.readthedocs.io/) for this use case. It doesn't have a UI, but it basically only needs a Compose file to run an app, and all data (for all apps) lives in a data/ directory that you can back up at your leisure. Everything is auto updated from a single YAML specification (which I usually pin), so the process to update something is "change the version in the config, commit, and the server will pick the change up and deploy it". It's just a thin layer over "git pull; docker compose restart", but I love it.

by raggi 3y ago

yonixw 3y ago

That's exactly how I feel. Not to mention that I am always looking for how to monitor the system, and there is no uniform standard, if it is possible to monitor at all. As if there is a hidden message that apart from monitoring how much disk space is left or CPU everything else is irrelevant. But for such VMs that share many processes, it is exactly the opposite! I must know when there is a problem who exactly is responsible!

everforward 3y ago

I'm working on something similar, and the crux of that issue is configurability vs automation. I.e. it's very easy to make backups for a system that users can't configure at all. You just ship the image with some rsync commands or something and done. Once you start letting people edit the config files, you get into a spot where now you basically need to be able to parse the config files to read file paths. That often means making version-specific parsers for configuration options that are introduced or removed in some versions, or have differing defaults (i.e. in 1.2 if they don't set "storage_path" it's at /x/, but in 1.3 if they don't set it, it defaults to /y/). That gets to be a lot of work. Then it gets even worse when the users can edit the Docker config for the images, because all bets are off at that point. The user could do all kinds of weird, fucky shit that infinitely loops naive backup scripts because of host volume mounts or have one of the mounts actually be part of NFS mount on the host with like 200ms of latency so backups just hang for forever and etc. It's just begging for an infinite series of bugs from people who did something weird with their config and ended up backing up their whole drive instead of the Docker folder, or removing their whole root drive because they mount their host FS root in the backups folder and it got deleted by the "old backup cleanup" script, or who knows what. At some point, it's easier to just make your own setup where you define the limitations of that setup than it is to use someone else's setup but have to find the limitations on your own.

apitman 3y ago
The one thing you have working on your favor is that the more likely someone is to want to customize, the more likely they are to understand they have to be responsible for backups and updates. Is your project public? I'm working on one as well and would love to see what you have cooking.
- everforward 3y ago
  It's not at the moment, but I wouldn't be opposed to chucking you some access if you wanted to look (or talk about joining, I'm solo at the moment). The rough gist is that it's for hosting dedicated servers for video games, with an emphasis on having an easier to use UI than most providers combined with a usable API/CLI for advanced users. Ie. Instead of editing config files as regular text files, I want to offer a view with drop downs for each config option, a description of what each value does. (with some hidden for orchestration reasons, like the port it listens on) Architecture-wise, it's a 3-tier app. HTTP API frontend talks to gRPC backend talks to Postgres. gRPC in that middle tier because I want to move the control plane for the agents into gRPC. Currently, the control plane consists of generating an Ansible inventory and executing it, and Ansible handles downloading the game if it doesn't exist, making dirs, SFTPing configs, install a Prometheus agent, etc. It's not anything I intend to hyperscale, but I've always been the dedicated server guy among my friends, so I figured I might as well make a little thing out of it. I'd be happy with a small ARR.
megous 3y ago
> Once you start letting people edit the config files, That way lies madness... What can be done instead is to provide your own unified (some schema validated) configuration options, and generate the app specific config files from your source of truth. Then you can know what the user can configure and how to back everything up (and how to do a lot of other things in automated fashion). And you also have a safe upgrade path if format of any underlaying config changes.
- everforward 3y ago
  You can, but I find it very tedious. O2M or M2M relationships become cumbersome. Not because they're hard to use, but just because there's so fucking many of them. I opted to store the app and version, and have a function that retrieves a config parser for it. I just didn't want to manage 300 tables just to get the schema to work. That's also partially because I'm using Ent in Go, so it generates a struct for each table. It'd make my auto-complete useless, and that's not a price I'm willing to pay on a personal project lol

ocdtrekkie 3y ago

This is one of the reasons I groan every time there's a new selfhosting thing that uses Docker. It's easy to start because all the wiring is there and everyone already makes their apps available with a Dockerfile, but it's not a remotely good solution for selfhosting and people should stop trying to use it that way.

apitman 3y ago
Unfortunately Sandstorm's approach, ie rewrite all the apps to fit a specific ecosystem, hasn't taken off either. I think docker with a little extra tooling for managing updates and backups is likely to be the solution eventually.
- ocdtrekkie 3y ago
  The problem is the Docker solution will never ever work and yet people are wasting thousands of person-hours trying, over and over in fact in dozens of redundant non-solutions. Sure Sandstorm or something like it will take a lot more effort, but at least it's moving in the correct direction. Just because Docker selfhosting has "taken off" doesn't mean it isn't still a dead end. EDIT: Also, to be clear, Docker is also a "specific ecosystem". It's very popular, it doesn't make it any more general than anything else.
  - EraYaN 3y ago
    Why would backups and restores never work though? The only thing you need to do is save your volumes/mounts essentially? The rest is a bunch of yaml in some docker-compose file for example. How is that "will never ever work"?
    - ocdtrekkie 3y ago
      The issue is that there is a lot more to a home server than updates and backups, and even doing those correctly automatically is... pretty hard if you don't have very strong opinions about how apps need to support them. So to give you an idea, Sandstorm requires: An app always must recover from forced termination (the only way it shuts apps down!) and that a given app carry all of the code it needs to update its data from any previous version. If you don't do these things, your homelabber will inevitably be fixing stuff themselves in various apps because their server lost power or the app developer demanded some bespoke intermediate inane update steps, or you missed a few versions while your server was off. What percentage of apps made available via Docker meets those requirements? Probably a lot less than you'd think. (Most of them that do easily work on Sandstorm!) But if you want to make selfhosting work for non-sysadmins (or sysadmins who don't want to do a second job when they get home from their main job), you have to do these things. Another thing is that you basically have to have security right because your users and even your server admin probably aren't experts, and definitely aren't paying for an enterprise-class firewall. So a big part of Sandstorm's opinionated app design is around making it fundamentally impossible for app vulnerabilities to exist. Things like proxying everything through the platform, isolating individual documents into their own app instances, handling authentication and authorization, etc. all become core to making apps secure enough to not have to worry about them that much. This is especially important because a lot of selfhosted apps get abandoned and unmaintained, but selfhosters still want to use them. Of course, all these things fundamentally require changes to the app your random Docker container shipper can't insist on, and that's before we get even into performance concerns, which are key since selfhosters tend to use old or cheap hardware.

stavros 3y ago

I wrote and use Harbormaster (https://harbormaster.readthedocs.io/ https://harbormaster.readthedocs.io/) for this use case. It doesn't have a UI, but it basically only needs a Compose file to run an app, and all data (for all apps) lives in a data/ directory that you can back up at your leisure. Everything is auto updated from a single YAML specification (which I usually pin), so the process to update something is "change the version in the config, commit, and the server will pick the change up and deploy it". It's just a thin layer over "git pull; docker compose restart", but I love it.