12 ms·

Self-hosting is something that we should be constantly iterating on making easier; it's really the path forward for privacy centric folks. The main challenges a

by kodah 5y ago

Self-hosting is something that we should be constantly iterating on making easier; it's really the path forward for privacy centric folks. The main challenges are managing workload scheduling (SystemD is complicated for a layperson). Networking is another challenge; for instance, if you wanted all or part of these services to remain offline or on a Mesh VPN there's a lot of knowledge required. There's some projects trying to tackle the workload orchestration piece; CasaOS (https://www.casaos.io/ https://www.casaos.io/) being one of my favorites but there's also Portainer (https://portainer.io https://portainer.io). TailScale and and ZeroTier are great for Mesh VPN networking, where you may need to run some workloads in the cloud but want them networked with your home applications (or just to keep them offline). They also allow you to access applications running on a home server that doesn't have a static IP. Cloudflare Access is okay; I haven't tried it because it deviates from the mesh VPN model significantly.

Hendrikto 5y ago
> SystemD is complicated for a layperson Is it? It has clean and logical abstractions, and consistency. Services depending in each other isn‘t complex or difficult to understand. I suspect that a nice GUI would make systemd quite usable for non-expert users. BTW: It‘s called ”systemd“: > Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. [0] [0]: https://www.freedesktop.org/wiki/Software/systemd/#spelling https://www.freedesktop.org/wiki/Software/systemd/#spelling
- kodah 5y ago
  > Is it? It has clean and logical abstractions, and consistency. Services depending in each other isn‘t complex or difficult to understand. For a technologist or engineer, yes. For a layperson, no. The average consumer who desires privacy is probably neither a technologist or engineer, so the longterm target is something that just works. Laypeople also aren't going to entertain the kind of pedantry that is systemd vs systemD vs System D vs SystemD so making systems that abstract further away from those communities is beneficial. Edit: Thank you for your correction, as a systems engineer, but I couldn't help but highlight this is a big hurdle even in the Linux communities that I've been a part of as desktop Linux as gained wider adoption by laypeople.
  - nobody9999 5y ago
    >For a technologist or engineer, yes. For a layperson, no. The average consumer who desires privacy is probably neither a technologist or engineer, so the longterm target is something that just works. In comparison to system V initd startup files, systemd unit files are, arguably, less complicated. I'd say the "complexity" of systemd unit files is mostly irrelevant to end users. For a relatively non-technical user, implementing whatever application/service one might want to use should be as simple as installing the relevant package(s) and dependency(ies) via existing, well managed package management systems. That said, too many developers encourage self-hosting, but don't provide appropriate packages and defaults for most popular distributions. If developers spent just a little more time creating buildable packages (supporting the creation of binary and source .rpm, .deb, etc. packages) with sane defaults/startup files could make the inclusion of such apps into the standard/extras repositories of a broad swathe of Linux distributions much simpler and, for the non-technical user, easy to install and configure. Matrix Synapse[0] and Diaspora[1] both come to mind in this respect. Installation and configuration of these platforms requires the installation of several software development frameworks and separate (from the standard system package managers, e.g., DNF, apt, dpkg, etc.) package management tools for the language dependencies. Requiring installation of software dev environments and building the software/databases/admin tools for such "self-hosted" solutions just confuses non-technical users. As a professional with decades of Unix/Linux implementation and management experience, I find implementing such platforms simple enough. Just read the docs, install the dependencies and compile/install/configure the software. For a non-technical person, that's likely a non-starter unless there's a UI that will do so automagically. Fortunately, there is such a UI for most Linux/Unix distributions -- it's called the system package manager. Unless and until developers provide distribution developers/maintainers with appropriate packageable sources (or even separate repositories with binaries!) to be added to the default repositories, self hosting many apps will only be the purview of technical users. This annoys me. A lot. Not because I, personally, mind a complicated set up process for such applications, but because it limits the ability of both Linux/Unix distributions and self-hosted applications/platforms to be used more broadly by non-technical users. Especially with tools like Diaspora, Matrix/Synapse and others which have the potential to overturn centralized hell holes like Twitter, Facebook, Instagram, WhatsApp, etc. It's been at least five years since I first installed a Diaspora pod and a year since I installed Synapse and a STUN server. In both cases, had I not been a long-time user/manager/implementor of Unix/Linux and associated sw dev environments, the install would have been nightmarish. For both platforms, installation pretty much requires knowledge of software development tools and practices, as well as more than a passing familiarity with Unix/Linux shells and environments. I can't imagine my 64 year-old sister in-law (a reasonably well educated and smart cookie with decent problem-solving skills) taking the time to learn how to use git, clang/gxx or even docker to install this "self hostable" stuff. That should be the target audience for such self hosted tools, not devs and other technical people. Taking the time to make one's application/platform easily installable/configurable (and building from git repos and/or Docker-compose aren't "easy" for non-technical folks) by non-technical end users could make a huge difference in this space. [0] https://matrix.org/docs/projects/server/synapse/ https://matrix.org/docs/projects/server/synapse/ [1] https://en.wikipedia.org/wiki/Diaspora_(social_network) https://en.wikipedia.org/wiki/Diaspora_(social_network)
- GrayShade 5y ago
  I swear, writing it as SystemD isa shibboleth of systemd haters.
  - kodah 5y ago
    For the record, I actually like and use it. I'm just at work and didn't put much thought on how to spell it. I also didn't really expect someone to care that much in a general, high-level discussion.
  - cozzyd 5y ago
    they are all in favor of SystemE
    - bqmjjx0kac 5y ago
      More like SystemSh
      - cozzyd 5y ago
        SystemŞ?
- spiffytech 5y ago
  Technologists have a very skewed idea of what's complicated vs easy with computers. Things we think are absolutely trivial are often insurmountable hurdles for laypeople. (This can, of course, happen if you put a technologist outside their element, too)
- teekert 5y ago
  What lay person does anything with systemd though? I have all my services in a docker-compose.yaml... Sure, I remember the days before systemd, I remember upstart, Gentoo's rc.conf. I still think it's useful I can find my way trough the internals of a Linux box, but for me all that stuff is far in the past. This is how it goes nowadays: Install the system in 20 min, clone the infra as code, put the data back, start the infrastructure... Where does the init system still play a role?
- 0xdeadb00f 5y ago
  I'm certainly not a layperson, but systemd frequently confuses me. I want to edit a service to harden it for example. Oh, wait I shouldn't edit it directly with vi? Because it gets overwritten by package updates. Okay, makes sense, I need to use systemctl edit instead. But that opens a file that has everything commented out. Do I uncomment the [Unit] heading? What do I need to keep and where do I add my additions? I recall there being a comment at the start of this file, but unless I'm misremembering it doesn't answer that. All I ask of it to do one thing - start something.service after other.service. yet it just refuses to order them this way. Why? I have no idea. I also have no idea where to start debugging a problem like this. There's a billion ways to try and do this after all: do I add Before=something to other.service? Do I add After=other to something.service? Both? Wants=something?
  - bmn__ 5y ago
    > it gets overwritten by package updates This doesn't happen. The package manager installs the new configuration under a different name so that you do not lose your changes and can merge them easily.
    - johnny22 5y ago
      what they are saying is that they edited the file in /usr/lib , which definitely would get overwritten. You're supossed to copy it into /etc/systemd/ for the appropriate service type.
      - v_p_n_p_v 5y ago
        I think you all proved the point that this system is too complicated for anyone outside of a small group of professional IT people.
        lmns 5y ago
        Are you implying that getting init script customisations overwritten by package managers isn't a problem with non-systemd init managers? I have lost track how often that happened with sysvinit, because the "logic" how to treat customisations was usually handled by the package manager and they messed it up regularly. systemd has a standard way to handle customisations. As long as you put everything you do in /etc/systemd/system, everything is fine. It's simple and works across distributions.
- zepearl 5y ago
  > Services depending in each other isn‘t complex or difficult to understand. It is for me with Systemd - I had to spend hours (on two different occasions, if I remember correctly on Debian & Linux Mint) trying to understand how to set a dependency against an NFS filesystem mount so that a DB would not be started before that, and to make that work reliably => Systemd's docs & behaviour (& special distro settings related to systemD?) weren't that great for me.
- hsn915 5y ago
  Comments like yours are a great example of why Linux has a hard time being user friendly. You take something that's deeply technical but easy to understand for yourself, and somehow generalize it to everyone. "If it's easy for me, it is easy for everyone", without noticing that perhaps your expertise plays a big role in making it easy. The better question is why should a layperson ever need to know about systemd in the first place. I'm a technical person and I don't know about systemd, nor have I ever needed it. When designing products for end users who are not technical, the overriding design goal should be easiness and reliability. Easy: does not require special knowledge or expertise to operate. For someone who just wants to self host, having to learn about systemd, by definition makes the product/system not easy. Reliability: does the system implode in unexpected way based on the slight variations in the environment? If it implodes as such then it is fragile. If it does not, then it is robust and reliable. For the end user, what matters is that the system is easy and reliable. If the system is easy but not reliable, the user is effectively forced to become an expert in order to fix the system when it breaks. Thus, if a system is not reliable, then it doesn't matter that it's "easy". Simple/Complicated only concerns the developers. It's important to keep systems as simple as possible under the hood, because a system that is simple is easier to make reliable than a system that is complicated. But for the end user it does not matter directly. Simplicity is about how many parts the system is composed of and how these parts interact. If there are many many parts that all have to interact with each other, then the system is complex. Maybe once someone learns about systemd they can find it conceptually simple. But that's a moot point. The point is: they should not even have to learn about its existence. A system where editing text files can make or break the system, is not a reliable system. It's easy to make mistakes in text files. Specially for users without expertise. Imagine yourself a windows user. You edit a text file. Restart the machine, and now it doesn't boot, or boots in a special text-only mode. (This is not unheard of on linux based systems).
  - klodolph 5y ago
    Systemd is easy compared to init scripts. Maybe that’s the comparison point. I know that’s my comparison point, since I wrangled init scripts for a couple decades before systemd, and when systemd came out, suddenly it felt a million times easier to run services. > A system where editing text files can make or break the system, is not a reliable system. It's easy to make mistakes in text files. Specially for users without expertise. I don’t think that’s a salvageable notion of “reliability”. I could bork Mac OS or Windows easily enough by editing text files, if I really wanted to.
- takinola 5y ago
  If you know what systemd is, you are, by definition, not a layperson.
  - Hendrikto 5y ago
    End users would not need to know. They could just click the activate button in a GUI.
- bavila 5y ago
  > It has clean and logical abstractions, and consistency. Sorry, but an actual layperson would already be lost upon reaching the word "abstraction".
  - Hendrikto 5y ago
    I am talking to technical people here. You would not expose that terminology to end users.
- vineyardmike 5y ago
  If someone says it’s complicated, then yes it’s complicated. It may not be complicated for you, but by definition someone finding it complicated makes it complicated.
- rtpg 5y ago
  I loved using LingonX on Mac, one of the things I'd really love to do is make that but for systemd stuff. I think the abstractions are good, but there's a lot of terms that are needed just for the basic "Spin up this process on boot" (one might say that explicit is better than implicit, but I dunno).
- friendzis 5y ago
  > Services depending in each other isn‘t complex or difficult to understand. I am not an expert here, but to the best of my knowledge systemd has no concept of inter-host dependencies, making it effectively useless for anything distributed. For example classic lb+app+db cannot be directly controlled by systemd.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
Havoc 5y ago
Quite surprised at seeing CasaOS mentioned so often here. It's quite a young project & best as I can tell it was sorta a sideproject of the guys sitting on their hands while trying to ship Zimaboard kickstarter hardware during a ship shortage. Good for them that it is seeing traction :)
fknorangesite 5y ago
> Self-hosting is something that we should be constantly iterating on making easier I'm pretty sure that's exactly what we did and ended up where we are today. Any sufficiently-advanced self-hosting is indistinguishable from AWS? I'm not sure how joking I am.
- deleted 5y ago
  [deleted]
- chaxor 5y ago
  You don't really have control of any of the hardware on AWS, and therefore they can track everything you do. (If they say they don't, you just have to trust them - there's never a real way to verify) If you're fine with that, then OK - but the public has been shown time and again this doesn't always end happily. So if they leak all of your life's passwords in plaintext, you have to be ok with that. Which is exactly why OP pointed out this is where we should be headed if we care about privacy.
gz5 5y ago
tailscale is strong for network-centric use cases. openziti is strong for app-centric use cases - put the (programmable, zero trust) network into your self-hosted app (via SDKs for various languages), rather than putting the app on the network. https://openziti.github.io/ https://openziti.github.io/ (quick starts) https://github.com/openziti https://github.com/openziti disclosure: founder of company selling saas on top of openziti
- pbronez 5y ago
  That’s pretty cool. So I could use Ziti two write a client-server app where the server is only accessible/visible to clients running Ziti with appropriate keys?
  - gz5 5y ago
    yep, literally shut down all inbound firewall ports and link listeners. keys are bootstrapped and you can add your own CA if desired (RFC 7030). https://ziti.dev/blog/bootstrapping-trust-part-5-bootstrapping-trust/ https://ziti.dev/blog/bootstrapping-trust-part-5-bootstrappi...
lumost 5y ago
Having started my career in hosting, I would suggest that this world is unlikely to come back except for exceptionally small applications with minimal business impact. What does self-hosting provide which end-end encryption does not? Self-hosting means: - Needing to know how to configure your linux host across firewalls, upgrades, backups. - Negotiating contracts with network service providers. While verifying that you have the right kind of optic on the network line drop. - Thinking through the order of operations on every remote hands request, and idiot proofing them so that no one accidentally unplugs your DB. - Making sure that you have sufficient cold spares that a server loss doesn't nuke your business for 6-12 weeks depending on how the hardware manufacturers view your business. - Building your own monitoring, notifications, and deployment tools using both open source and in-house tools. - Building expertise in all of your custom tools. - A 6-20 week lead time to provision a build server. - Paying for all of your hardware for 3-5 years, regardless of whether you will actually need it. - Over-provisioning memory or CPU to make up for the fact that you can't get hardware fast enough. - Getting paged in the middle of the night because the hardware is over-provisioned and something gets overwhelmed or a physical machine died. - Dealing with the fact that an overworked systems engineer or developer is never making any component the best. And everything you touch will just passably work. - Everyone will have their own opinions on how something should be done, and every decision will have long term consequences. Get ready for physical vs virtual debates till the heat death of the universe.
- moreira 5y ago
  I think there might be a vast gap between what what the article is talking about, and what you're suggesting. Somebody self-hosting their project management app on a LAMP server on some random web hosting company is one thing. What you're talking about is something else entirely. And yes, having a web hosting company manage your server isn't "real" self hosting, for all the reasons you described, but it's a far cry from dumping all your data with a big data-guzzling giant. Your data isn't calling home, isn't being used by a company, it's sitting on your server under your control, and only you manage it. I think that's their main gripe.
  - lumost 5y ago
karmakaze 5y ago
It's pretty easy to write a unit file for a service and install/use it. A layperson could easily follow a guide with just a few of the most common cases.
- Isthatablackgsd 5y ago
  As a layperson, no I don't. You will be surprised how many of us couldn't easily follow and understands guides for self-hosting. It took me a while to set it up since the documentation expects me to know everything. Which that is fair because self hosting are technical and requires some knowledge to set it up properly. The challenge is that where I can learn this knowledge in condensed manner? There probably are something out there but they are not centralized enough to help laypeople to run it. Let alone trying to learn how to keep it secured.
- asoneth 5y ago
  These may seem like very easy tasks for you. If you truly believe that a layperson can "easily follow" a technical guide or that a guide with the most common cases is sufficient to maintain a webserver... then the only thing I can say is that your experience with both laypeople and webservers is worlds apart from mine.
- wmf 5y ago
  This thread is sad. You don't need to create a unit file to install Fortnite. Why set your sights so low?
kovac 5y ago
For laypeople self-hosting is out of the question for now. I'd say the more immediate problem is that even for competent engineers this is a difficult task with all the artificial restrictions put in place in the name of security, anti-fraud, etc.
- deleted 5y ago
  [deleted]
ClumsyPilot 5y ago
Most consumer platforms don't have functional automatic backups, so this is a pie in the sky at the moment. Even for a proffesionsl, self hosting is kind of time consuming
teleforce 5y ago
Totally agreed, for the sake of humanity we should strive to make self-hosting as easy and as seamless as possible. But why stop at self-hosting? Beyond self-hosting, it could be extended to local-first paradigm meaning that there's a choice to have a scalable on demand auxiliary cloud based for handling bursty access demands if you need it. In addition, you can have extra backup for the peace of mind. I'm currently working on realiable solutions (physical wireless and application layers) to extend the local-first system to be automatically secured even you have intermittent Internet outage unlike TailScale and ZeroTier [2]. This system will be invaluable where Internet connection is not reliable due to weather, harsh environment, war, unreliable power provider or lousy ISPs [3]. [1] Local-First Software: You Own Your Data, in spite of the Cloud: https://martin.kleppmann.com/papers/local-first.pdf https://martin.kleppmann.com/papers/local-first.pdf [2] Internet outage: https://en.wikipedia.org/wiki/Internet_outage https://en.wikipedia.org/wiki/Internet_outage [3] NIST Helps Next-Generation Cell Technology See Past the Greenery: https://www.nist.gov/news-events/news/2022/01/nist-helps-next-generation-cell-technology-see-past-greenery https://www.nist.gov/news-events/news/2022/01/nist-helps-nex...
- southerntofu 5y ago
  > I'm currently working on realiable solutions (physical wireless and application layers) to extend the local-first system to be automatically secured even you have intermittent Internet outage unlike TailScale and ZeroTier Sorry but i don't understand what this means. Internet outage should not affect your LAN services, no matter what selfhosting distro you use.
  - teleforce 5y ago
    For silo self hosting, Internet outage should not affect your services, but for most of local-first services it's critical to maintain your systems security with necessary credentials. For an example consider a rental car that you take for camping in the countryside. Internet outage that makes your rental car useless is unacceptable.
    - southerntofu 5y ago
Semaphor 5y ago
Huh, ZimaBoard [0] (Hardware SBC project by the CasaOS people) looks super cool. Sadly still on pre-order, but that is almost exactly what I want. [0]: https://www.zimaboard.com/ https://www.zimaboard.com/
pishpash 5y ago
What about the hardware side of this? All this talk about software...
Phlogi 5y ago
Can CasaOS run compose stacks or simply containers?
- Zeik0s 5y ago
  can do both, iirc.
eternityforest 5y ago
The problem is certificates and WAN access, and lack of MDNS on Android. There's basically no way to do anything that doesn't involve some manual setup, aside from developing a new purpose built app, and maintaining it in addition to the product, probably on two platforms. If Mozilla still had FlyWeb things could be plug and play. I have a set of proposals here to bring some of that back: https://github.com/WICG/proposals/issues/43 https://github.com/WICG/proposals/issues/43 And some are considering the Tox protocol, but in general, we have not solved the most basic issue of self hosting. How do I connect to my device in a way that just works, LAN or WAN, without manually serting up the client or registering for a service?
gigatexal 5y ago
This. I think -- speaking for myself mostly -- folks move to cloud for the simplicity a web interface provides. If you like, there's usually a CLI that abstracts the compute and other things. Self hosting -- at least whenever I did it -- was always: start with a VM, install Linux, configure ports, configure security, install a webserver, deal with the security of that, manage conf files, deploy website, etc etc Host a static page on github pages makes all that a ton easier and also free. That's a trite example sure. But when I was at a previous company who did almost everything on premises I couldn't help but think if we had an internal portal/system a la GCP's console or that of Amazon's that could be a way for devs to spin up resources and have it all managed and even be a bit programmatic (no no K8s doesn't solve all of this it's its own bag of crazy) then we'd not need cloud much since we'd not need the almost infinite scale that cloud offers.
swiftcoder 5y ago
The only model I've ever seen this work is the Mac/Windows model. You provide a standard installer for the server (or even distribute it via the app store). The user launches it through the standard graphical app launch model (Finder or Start Menu), the server displays a suitably user-friendly GUI configuration panel, and then minimises itself to the notifications tray. The linux model of "first learn how to use a package manager, edit configuration files by hand, and configure init scripts" is never going to be something that I can comfortably explain to computer users like my parents...
wallacoloo 5y ago
i started self-hosting a bunch of stuff last month: Pleroma (like Mastodon/Twitter), Matrix (chat), Gitea (like Github) and Jellyfin (like Plex, a media server). AFTER i set up the hardware/OS, these each took about 1-2 hours to setup, and it gets faster each time as i get more accustomed to the common parts (nginx, systemd, Lets Encrypt, and whatever containerization you use). today i accidentally nuked everything by not flushing the disk before rebooting and then naively letting fsck try to ‘fix’ it (which just makes things worse since it unlinks every inode it thinks is wrong instead of helping you recover data). now i’m manually dumping blocks and re-linking them in, supplementing whatever’s not recoverable with a 3-day old backup. that’s probably gonna take an entire day to fix up. after this i have to figure out a better backup solution, because it costs me $5 of API requests every time i rclone the system to Backblaze, making frequent backups too expensive. after that, i have to figure out the email part of things. AFAICT it’s pretty much impossible to 100% self-host email because of blacklisting. you have to at least proxy it through a VPS, or something. and in between that i may spin up a DNS server to overcome the part where it takes 60min for any new service to be accessible because of the negative caching common in DNS. no, this stuff is just way too involved for anyone who hasn’t already spent a decade in the CLI. i’m only doing this because i’m a nerd with time on his hands between jobs. self-hosting isn’t gonna catch on this decade. but maybe we can federate, so that you just need to have one friend who cares about this stuff manage the infra and provide it to their peers as a social good. also, i don’t think privacy is the right angle for promoting self-hosting. a good deal of the things that people self-host have a public-facing component (websites; public chatrooms; etc). if privacy is what you seek, then you should strive to live life offline. the larger differentiator for self-housing is control.
- spacemanmatt 5y ago
  After I've built a server for a purpose, the one thing I want most is a script that does it again. Spending another identical hour on a similar server just makes me sad.
  - southerntofu 5y ago
    That's precisely why i started working on https://codeberg.org/southerntofu/ansible-selfhosted https://codeberg.org/southerntofu/ansible-selfhosted Its abstractions are still a bit shaky and a lot can be improved so it's far from ready for the general public, but i still consider it a great step forward because for the supported configurations, all i have to do to setup a new server is: - edit config.yml - run roles/deploy.sh - enjoy I'm happy to answer any questions on why (politics) and how (technics) i'm building this, and i'd be more than thrilled to receive feedback and contribution. In the past week i started working on a test suite so it's easier to contribute.
raxxorrax 5y ago
You can still self-host and use external resources to manage network and system security. You would keep full control over the machine this way. Having professionals sensibly partitioning different resources in respective subnets is still one of the most valuable defense mechanisms against many threats.
jdrc 5y ago
There was a time when everyone and their brother were self hosting: napster, kazaa, hotline etc. Why has this trend stalled for 20 years
ddaalluu2 5y ago
First off it's systemd not systemD. And it's not complicated, in fact it's simpler than writing shell scripts. Portainer is a bad docker clone but more often than not doesn't work with docker scripts. But containerization is the wrong way anyway. If you can't setup a certain service on your machine you should not expose that host to the internet. That might sound arrogant but only for the inexperienced mind. Packaging everything in a docker package doesn't remove the burden of responsibility of operating an internet-exposed host. Idk what you're even trying to say. It feels like you're just throwing buzzwords out there without knowing what they mean or what they're for. If you want VPN wireguard is your choice. If you need a gatekeeper/firewall opnsense or openwrt.

by kodah 5y ago

Hendrikto 5y ago

> SystemD is complicated for a layperson Is it? It has clean and logical abstractions, and consistency. Services depending in each other isn‘t complex or difficult to understand. I suspect that a nice GUI would make systemd quite usable for non-expert users. BTW: It‘s called ”systemd“: > Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. [0] [0]: https://www.freedesktop.org/wiki/Software/systemd/#spelling https://www.freedesktop.org/wiki/Software/systemd/#spelling

kodah 5y ago
> Is it? It has clean and logical abstractions, and consistency. Services depending in each other isn‘t complex or difficult to understand. For a technologist or engineer, yes. For a layperson, no. The average consumer who desires privacy is probably neither a technologist or engineer, so the longterm target is something that just works. Laypeople also aren't going to entertain the kind of pedantry that is systemd vs systemD vs System D vs SystemD so making systems that abstract further away from those communities is beneficial. Edit: Thank you for your correction, as a systems engineer, but I couldn't help but highlight this is a big hurdle even in the Linux communities that I've been a part of as desktop Linux as gained wider adoption by laypeople.
- nobody9999 5y ago
  >For a technologist or engineer, yes. For a layperson, no. The average consumer who desires privacy is probably neither a technologist or engineer, so the longterm target is something that just works. In comparison to system V initd startup files, systemd unit files are, arguably, less complicated. I'd say the "complexity" of systemd unit files is mostly irrelevant to end users. For a relatively non-technical user, implementing whatever application/service one might want to use should be as simple as installing the relevant package(s) and dependency(ies) via existing, well managed package management systems. That said, too many developers encourage self-hosting, but don't provide appropriate packages and defaults for most popular distributions. If developers spent just a little more time creating buildable packages (supporting the creation of binary and source .rpm, .deb, etc. packages) with sane defaults/startup files could make the inclusion of such apps into the standard/extras repositories of a broad swathe of Linux distributions much simpler and, for the non-technical user, easy to install and configure. Matrix Synapse[0] and Diaspora[1] both come to mind in this respect. Installation and configuration of these platforms requires the installation of several software development frameworks and separate (from the standard system package managers, e.g., DNF, apt, dpkg, etc.) package management tools for the language dependencies. Requiring installation of software dev environments and building the software/databases/admin tools for such "self-hosted" solutions just confuses non-technical users. As a professional with decades of Unix/Linux implementation and management experience, I find implementing such platforms simple enough. Just read the docs, install the dependencies and compile/install/configure the software. For a non-technical person, that's likely a non-starter unless there's a UI that will do so automagically. Fortunately, there is such a UI for most Linux/Unix distributions -- it's called the system package manager. Unless and until developers provide distribution developers/maintainers with appropriate packageable sources (or even separate repositories with binaries!) to be added to the default repositories, self hosting many apps will only be the purview of technical users. This annoys me. A lot. Not because I, personally, mind a complicated set up process for such applications, but because it limits the ability of both Linux/Unix distributions and self-hosted applications/platforms to be used more broadly by non-technical users. Especially with tools like Diaspora, Matrix/Synapse and others which have the potential to overturn centralized hell holes like Twitter, Facebook, Instagram, WhatsApp, etc. It's been at least five years since I first installed a Diaspora pod and a year since I installed Synapse and a STUN server. In both cases, had I not been a long-time user/manager/implementor of Unix/Linux and associated sw dev environments, the install would have been nightmarish. For both platforms, installation pretty much requires knowledge of software development tools and practices, as well as more than a passing familiarity with Unix/Linux shells and environments. I can't imagine my 64 year-old sister in-law (a reasonably well educated and smart cookie with decent problem-solving skills) taking the time to learn how to use git, clang/gxx or even docker to install this "self hostable" stuff. That should be the target audience for such self hosted tools, not devs and other technical people. Taking the time to make one's application/platform easily installable/configurable (and building from git repos and/or Docker-compose aren't "easy" for non-technical folks) by non-technical end users could make a huge difference in this space. [0] https://matrix.org/docs/projects/server/synapse/ https://matrix.org/docs/projects/server/synapse/ [1] https://en.wikipedia.org/wiki/Diaspora_(social_network) https://en.wikipedia.org/wiki/Diaspora_(social_network)
GrayShade 5y ago
I swear, writing it as SystemD isa shibboleth of systemd haters.
- kodah 5y ago
  For the record, I actually like and use it. I'm just at work and didn't put much thought on how to spell it. I also didn't really expect someone to care that much in a general, high-level discussion.
- cozzyd 5y ago
  they are all in favor of SystemE
  - bqmjjx0kac 5y ago
    More like SystemSh
    - cozzyd 5y ago
      SystemŞ?
spiffytech 5y ago
Technologists have a very skewed idea of what's complicated vs easy with computers. Things we think are absolutely trivial are often insurmountable hurdles for laypeople. (This can, of course, happen if you put a technologist outside their element, too)
teekert 5y ago
What lay person does anything with systemd though? I have all my services in a docker-compose.yaml... Sure, I remember the days before systemd, I remember upstart, Gentoo's rc.conf. I still think it's useful I can find my way trough the internals of a Linux box, but for me all that stuff is far in the past. This is how it goes nowadays: Install the system in 20 min, clone the infra as code, put the data back, start the infrastructure... Where does the init system still play a role?
0xdeadb00f 5y ago
I'm certainly not a layperson, but systemd frequently confuses me. I want to edit a service to harden it for example. Oh, wait I shouldn't edit it directly with vi? Because it gets overwritten by package updates. Okay, makes sense, I need to use systemctl edit instead. But that opens a file that has everything commented out. Do I uncomment the [Unit] heading? What do I need to keep and where do I add my additions? I recall there being a comment at the start of this file, but unless I'm misremembering it doesn't answer that. All I ask of it to do one thing - start something.service after other.service. yet it just refuses to order them this way. Why? I have no idea. I also have no idea where to start debugging a problem like this. There's a billion ways to try and do this after all: do I add Before=something to other.service? Do I add After=other to something.service? Both? Wants=something?
- bmn__ 5y ago
  > it gets overwritten by package updates This doesn't happen. The package manager installs the new configuration under a different name so that you do not lose your changes and can merge them easily.
  - johnny22 5y ago
    what they are saying is that they edited the file in /usr/lib , which definitely would get overwritten. You're supossed to copy it into /etc/systemd/ for the appropriate service type.
    - v_p_n_p_v 5y ago
      I think you all proved the point that this system is too complicated for anyone outside of a small group of professional IT people.
      - lmns 5y ago
        Are you implying that getting init script customisations overwritten by package managers isn't a problem with non-systemd init managers? I have lost track how often that happened with sysvinit, because the "logic" how to treat customisations was usually handled by the package manager and they messed it up regularly. systemd has a standard way to handle customisations. As long as you put everything you do in /etc/systemd/system, everything is fine. It's simple and works across distributions.
zepearl 5y ago
> Services depending in each other isn‘t complex or difficult to understand. It is for me with Systemd - I had to spend hours (on two different occasions, if I remember correctly on Debian & Linux Mint) trying to understand how to set a dependency against an NFS filesystem mount so that a DB would not be started before that, and to make that work reliably => Systemd's docs & behaviour (& special distro settings related to systemD?) weren't that great for me.
hsn915 5y ago
Comments like yours are a great example of why Linux has a hard time being user friendly. You take something that's deeply technical but easy to understand for yourself, and somehow generalize it to everyone. "If it's easy for me, it is easy for everyone", without noticing that perhaps your expertise plays a big role in making it easy. The better question is why should a layperson ever need to know about systemd in the first place. I'm a technical person and I don't know about systemd, nor have I ever needed it. When designing products for end users who are not technical, the overriding design goal should be easiness and reliability. Easy: does not require special knowledge or expertise to operate. For someone who just wants to self host, having to learn about systemd, by definition makes the product/system not easy. Reliability: does the system implode in unexpected way based on the slight variations in the environment? If it implodes as such then it is fragile. If it does not, then it is robust and reliable. For the end user, what matters is that the system is easy and reliable. If the system is easy but not reliable, the user is effectively forced to become an expert in order to fix the system when it breaks. Thus, if a system is not reliable, then it doesn't matter that it's "easy". Simple/Complicated only concerns the developers. It's important to keep systems as simple as possible under the hood, because a system that is simple is easier to make reliable than a system that is complicated. But for the end user it does not matter directly. Simplicity is about how many parts the system is composed of and how these parts interact. If there are many many parts that all have to interact with each other, then the system is complex. Maybe once someone learns about systemd they can find it conceptually simple. But that's a moot point. The point is: they should not even have to learn about its existence. A system where editing text files can make or break the system, is not a reliable system. It's easy to make mistakes in text files. Specially for users without expertise. Imagine yourself a windows user. You edit a text file. Restart the machine, and now it doesn't boot, or boots in a special text-only mode. (This is not unheard of on linux based systems).
- klodolph 5y ago
  Systemd is easy compared to init scripts. Maybe that’s the comparison point. I know that’s my comparison point, since I wrangled init scripts for a couple decades before systemd, and when systemd came out, suddenly it felt a million times easier to run services. > A system where editing text files can make or break the system, is not a reliable system. It's easy to make mistakes in text files. Specially for users without expertise. I don’t think that’s a salvageable notion of “reliability”. I could bork Mac OS or Windows easily enough by editing text files, if I really wanted to.
takinola 5y ago
If you know what systemd is, you are, by definition, not a layperson.
- Hendrikto 5y ago
  End users would not need to know. They could just click the activate button in a GUI.
bavila 5y ago
> It has clean and logical abstractions, and consistency. Sorry, but an actual layperson would already be lost upon reaching the word "abstraction".
- Hendrikto 5y ago
  I am talking to technical people here. You would not expose that terminology to end users.
vineyardmike 5y ago
If someone says it’s complicated, then yes it’s complicated. It may not be complicated for you, but by definition someone finding it complicated makes it complicated.
rtpg 5y ago
I loved using LingonX on Mac, one of the things I'd really love to do is make that but for systemd stuff. I think the abstractions are good, but there's a lot of terms that are needed just for the basic "Spin up this process on boot" (one might say that explicit is better than implicit, but I dunno).
friendzis 5y ago
> Services depending in each other isn‘t complex or difficult to understand. I am not an expert here, but to the best of my knowledge systemd has no concept of inter-host dependencies, making it effectively useless for anything distributed. For example classic lb+app+db cannot be directly controlled by systemd.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

Havoc 5y ago

Quite surprised at seeing CasaOS mentioned so often here. It's quite a young project & best as I can tell it was sorta a sideproject of the guys sitting on their hands while trying to ship Zimaboard kickstarter hardware during a ship shortage. Good for them that it is seeing traction :)

fknorangesite 5y ago

> Self-hosting is something that we should be constantly iterating on making easier I'm pretty sure that's exactly what we did and ended up where we are today. Any sufficiently-advanced self-hosting is indistinguishable from AWS? I'm not sure how joking I am.

deleted 5y ago
[deleted]
chaxor 5y ago
You don't really have control of any of the hardware on AWS, and therefore they can track everything you do. (If they say they don't, you just have to trust them - there's never a real way to verify) If you're fine with that, then OK - but the public has been shown time and again this doesn't always end happily. So if they leak all of your life's passwords in plaintext, you have to be ok with that. Which is exactly why OP pointed out this is where we should be headed if we care about privacy.

gz5 5y ago

tailscale is strong for network-centric use cases. openziti is strong for app-centric use cases - put the (programmable, zero trust) network into your self-hosted app (via SDKs for various languages), rather than putting the app on the network. https://openziti.github.io/ https://openziti.github.io/ (quick starts) https://github.com/openziti https://github.com/openziti disclosure: founder of company selling saas on top of openziti

pbronez 5y ago
That’s pretty cool. So I could use Ziti two write a client-server app where the server is only accessible/visible to clients running Ziti with appropriate keys?
- gz5 5y ago
  yep, literally shut down all inbound firewall ports and link listeners. keys are bootstrapped and you can add your own CA if desired (RFC 7030). https://ziti.dev/blog/bootstrapping-trust-part-5-bootstrapping-trust/ https://ziti.dev/blog/bootstrapping-trust-part-5-bootstrappi...

lumost 5y ago

Having started my career in hosting, I would suggest that this world is unlikely to come back except for exceptionally small applications with minimal business impact. What does self-hosting provide which end-end encryption does not? Self-hosting means: - Needing to know how to configure your linux host across firewalls, upgrades, backups. - Negotiating contracts with network service providers. While verifying that you have the right kind of optic on the network line drop. - Thinking through the order of operations on every remote hands request, and idiot proofing them so that no one accidentally unplugs your DB. - Making sure that you have sufficient cold spares that a server loss doesn't nuke your business for 6-12 weeks depending on how the hardware manufacturers view your business. - Building your own monitoring, notifications, and deployment tools using both open source and in-house tools. - Building expertise in all of your custom tools. - A 6-20 week lead time to provision a build server. - Paying for all of your hardware for 3-5 years, regardless of whether you will actually need it. - Over-provisioning memory or CPU to make up for the fact that you can't get hardware fast enough. - Getting paged in the middle of the night because the hardware is over-provisioned and something gets overwhelmed or a physical machine died. - Dealing with the fact that an overworked systems engineer or developer is never making any component the best. And everything you touch will just passably work. - Everyone will have their own opinions on how something should be done, and every decision will have long term consequences. Get ready for physical vs virtual debates till the heat death of the universe.

moreira 5y ago
I think there might be a vast gap between what what the article is talking about, and what you're suggesting. Somebody self-hosting their project management app on a LAMP server on some random web hosting company is one thing. What you're talking about is something else entirely. And yes, having a web hosting company manage your server isn't "real" self hosting, for all the reasons you described, but it's a far cry from dumping all your data with a big data-guzzling giant. Your data isn't calling home, isn't being used by a company, it's sitting on your server under your control, and only you manage it. I think that's their main gripe.
- lumost 5y ago

karmakaze 5y ago

It's pretty easy to write a unit file for a service and install/use it. A layperson could easily follow a guide with just a few of the most common cases.

Isthatablackgsd 5y ago
As a layperson, no I don't. You will be surprised how many of us couldn't easily follow and understands guides for self-hosting. It took me a while to set it up since the documentation expects me to know everything. Which that is fair because self hosting are technical and requires some knowledge to set it up properly. The challenge is that where I can learn this knowledge in condensed manner? There probably are something out there but they are not centralized enough to help laypeople to run it. Let alone trying to learn how to keep it secured.
asoneth 5y ago
These may seem like very easy tasks for you. If you truly believe that a layperson can "easily follow" a technical guide or that a guide with the most common cases is sufficient to maintain a webserver... then the only thing I can say is that your experience with both laypeople and webservers is worlds apart from mine.
wmf 5y ago
This thread is sad. You don't need to create a unit file to install Fortnite. Why set your sights so low?

kovac 5y ago

For laypeople self-hosting is out of the question for now. I'd say the more immediate problem is that even for competent engineers this is a difficult task with all the artificial restrictions put in place in the name of security, anti-fraud, etc.

deleted 5y ago
[deleted]

ClumsyPilot 5y ago

Most consumer platforms don't have functional automatic backups, so this is a pie in the sky at the moment. Even for a proffesionsl, self hosting is kind of time consuming

teleforce 5y ago

Totally agreed, for the sake of humanity we should strive to make self-hosting as easy and as seamless as possible. But why stop at self-hosting? Beyond self-hosting, it could be extended to local-first paradigm meaning that there's a choice to have a scalable on demand auxiliary cloud based for handling bursty access demands if you need it. In addition, you can have extra backup for the peace of mind. I'm currently working on realiable solutions (physical wireless and application layers) to extend the local-first system to be automatically secured even you have intermittent Internet outage unlike TailScale and ZeroTier [2]. This system will be invaluable where Internet connection is not reliable due to weather, harsh environment, war, unreliable power provider or lousy ISPs [3]. [1] Local-First Software: You Own Your Data, in spite of the Cloud: https://martin.kleppmann.com/papers/local-first.pdf https://martin.kleppmann.com/papers/local-first.pdf [2] Internet outage: https://en.wikipedia.org/wiki/Internet_outage https://en.wikipedia.org/wiki/Internet_outage [3] NIST Helps Next-Generation Cell Technology See Past the Greenery: https://www.nist.gov/news-events/news/2022/01/nist-helps-next-generation-cell-technology-see-past-greenery https://www.nist.gov/news-events/news/2022/01/nist-helps-nex...

southerntofu 5y ago
> I'm currently working on realiable solutions (physical wireless and application layers) to extend the local-first system to be automatically secured even you have intermittent Internet outage unlike TailScale and ZeroTier Sorry but i don't understand what this means. Internet outage should not affect your LAN services, no matter what selfhosting distro you use.
- teleforce 5y ago
  For silo self hosting, Internet outage should not affect your services, but for most of local-first services it's critical to maintain your systems security with necessary credentials. For an example consider a rental car that you take for camping in the countryside. Internet outage that makes your rental car useless is unacceptable.
  - southerntofu 5y ago

Semaphor 5y ago

Huh, ZimaBoard [0] (Hardware SBC project by the CasaOS people) looks super cool. Sadly still on pre-order, but that is almost exactly what I want. [0]: https://www.zimaboard.com/ https://www.zimaboard.com/

pishpash 5y ago

What about the hardware side of this? All this talk about software...

Phlogi 5y ago

Can CasaOS run compose stacks or simply containers?

Zeik0s 5y ago
can do both, iirc.

eternityforest 5y ago

The problem is certificates and WAN access, and lack of MDNS on Android. There's basically no way to do anything that doesn't involve some manual setup, aside from developing a new purpose built app, and maintaining it in addition to the product, probably on two platforms. If Mozilla still had FlyWeb things could be plug and play. I have a set of proposals here to bring some of that back: https://github.com/WICG/proposals/issues/43 https://github.com/WICG/proposals/issues/43 And some are considering the Tox protocol, but in general, we have not solved the most basic issue of self hosting. How do I connect to my device in a way that just works, LAN or WAN, without manually serting up the client or registering for a service?

gigatexal 5y ago

This. I think -- speaking for myself mostly -- folks move to cloud for the simplicity a web interface provides. If you like, there's usually a CLI that abstracts the compute and other things. Self hosting -- at least whenever I did it -- was always: start with a VM, install Linux, configure ports, configure security, install a webserver, deal with the security of that, manage conf files, deploy website, etc etc Host a static page on github pages makes all that a ton easier and also free. That's a trite example sure. But when I was at a previous company who did almost everything on premises I couldn't help but think if we had an internal portal/system a la GCP's console or that of Amazon's that could be a way for devs to spin up resources and have it all managed and even be a bit programmatic (no no K8s doesn't solve all of this it's its own bag of crazy) then we'd not need cloud much since we'd not need the almost infinite scale that cloud offers.

swiftcoder 5y ago

The only model I've ever seen this work is the Mac/Windows model. You provide a standard installer for the server (or even distribute it via the app store). The user launches it through the standard graphical app launch model (Finder or Start Menu), the server displays a suitably user-friendly GUI configuration panel, and then minimises itself to the notifications tray. The linux model of "first learn how to use a package manager, edit configuration files by hand, and configure init scripts" is never going to be something that I can comfortably explain to computer users like my parents...

wallacoloo 5y ago

i started self-hosting a bunch of stuff last month: Pleroma (like Mastodon/Twitter), Matrix (chat), Gitea (like Github) and Jellyfin (like Plex, a media server). AFTER i set up the hardware/OS, these each took about 1-2 hours to setup, and it gets faster each time as i get more accustomed to the common parts (nginx, systemd, Lets Encrypt, and whatever containerization you use). today i accidentally nuked everything by not flushing the disk before rebooting and then naively letting fsck try to ‘fix’ it (which just makes things worse since it unlinks every inode it thinks is wrong instead of helping you recover data). now i’m manually dumping blocks and re-linking them in, supplementing whatever’s not recoverable with a 3-day old backup. that’s probably gonna take an entire day to fix up. after this i have to figure out a better backup solution, because it costs me $5 of API requests every time i rclone the system to Backblaze, making frequent backups too expensive. after that, i have to figure out the email part of things. AFAICT it’s pretty much impossible to 100% self-host email because of blacklisting. you have to at least proxy it through a VPS, or something. and in between that i may spin up a DNS server to overcome the part where it takes 60min for any new service to be accessible because of the negative caching common in DNS. no, this stuff is just way too involved for anyone who hasn’t already spent a decade in the CLI. i’m only doing this because i’m a nerd with time on his hands between jobs. self-hosting isn’t gonna catch on this decade. but maybe we can federate, so that you just need to have one friend who cares about this stuff manage the infra and provide it to their peers as a social good. also, i don’t think privacy is the right angle for promoting self-hosting. a good deal of the things that people self-host have a public-facing component (websites; public chatrooms; etc). if privacy is what you seek, then you should strive to live life offline. the larger differentiator for self-housing is control.

spacemanmatt 5y ago
After I've built a server for a purpose, the one thing I want most is a script that does it again. Spending another identical hour on a similar server just makes me sad.
- southerntofu 5y ago
  That's precisely why i started working on https://codeberg.org/southerntofu/ansible-selfhosted https://codeberg.org/southerntofu/ansible-selfhosted Its abstractions are still a bit shaky and a lot can be improved so it's far from ready for the general public, but i still consider it a great step forward because for the supported configurations, all i have to do to setup a new server is: - edit config.yml - run roles/deploy.sh - enjoy I'm happy to answer any questions on why (politics) and how (technics) i'm building this, and i'd be more than thrilled to receive feedback and contribution. In the past week i started working on a test suite so it's easier to contribute.

raxxorrax 5y ago

You can still self-host and use external resources to manage network and system security. You would keep full control over the machine this way. Having professionals sensibly partitioning different resources in respective subnets is still one of the most valuable defense mechanisms against many threats.

jdrc 5y ago

There was a time when everyone and their brother were self hosting: napster, kazaa, hotline etc. Why has this trend stalled for 20 years

ddaalluu2 5y ago

First off it's systemd not systemD. And it's not complicated, in fact it's simpler than writing shell scripts. Portainer is a bad docker clone but more often than not doesn't work with docker scripts. But containerization is the wrong way anyway. If you can't setup a certain service on your machine you should not expose that host to the internet. That might sound arrogant but only for the inexperienced mind. Packaging everything in a docker package doesn't remove the burden of responsibility of operating an internet-exposed host. Idk what you're even trying to say. It feels like you're just throwing buzzwords out there without knowing what they mean or what they're for. If you want VPN wireguard is your choice. If you need a gatekeeper/firewall opnsense or openwrt.