13 ms·

Sovereign: Ansible playbooks to build and maintain your own private cloud

556 points by c0restraint 7y ago

navaati 7y ago
For my fellow HNers, this is "private cloud" in the meaning of Owncloud, not of Openstack.
- choward 7y ago
  I hear about Nextcloud a lot more than I hear about Owncloud. Does anyone know why this project uses Owncloud instead of Nextcloud?
  - crashbunny 7y ago
    This project started before Nextcloud existed, I don't know why it hasn't switched, though.
    - nemoniac 7y ago
      Also Nextcloud puts a LOT more effort into publicity than Owncloud.
    - AdmiralAsshat 7y ago
      NextCloud was started as a fork of OwnCloud by a bunch of OwnCloud devs that were unhappy with the direction the product was taking. You can think of it in much the same way as OpenOffice vs LibreOffice: devs fork to make a new product, the "original" product stagnates and is mostly used for rent-seeking. The downside of both is that, to my ears, both "OpenOffice" and "OwnCloud" better signify to outsiders what the product accomplishes, while "LibreOffice" and "NextCloud" really don't, unless you're already familiar with the product or product history.
  - judge2020 7y ago
    Based on how long this project has existed, I suspect they chose owncloud before nextcloud was forked and changing it hasn't happened/some people don't want to change their already working installations.
  - deleted 7y ago
    [deleted]
  - Tepix 7y ago
    I think there's an open issue for the switch. Just need a good pull request by a volunteer...
  - gramakri 7y ago
    For some history: ownCloud is a very old stable product. 2-3 years back it got forked into Nexcloud because the founder of ownCloud had some disagreement about ownCloud's direction. Both the products are backed by companies and both are doing quite well. I would say Nextcloud goes more and more into expanding it's use case and thus makes it product more extensible via plugins. This can be good or bad depending on how you look at it. Plugins go unmaintained/incompatible over time and are a constant source of pain when upgrading. Wordpress gets away with this because it has a massive community. ownCloud on the other hand has decided to double down on it's roots of file sharing/syncing. I heard they rewrote their stack from PHP to Go now and the frontend is now React.
OJFord 7y ago
Isn't this just duplicating effort that's probably already been done, in many cases by the first-party maintainers, in Dockerfiles? I don't mean to start 'Docker vs. Ansible', I just wonder why if you wanted a quick way to setup a single-server 'own private cloud' you wouldn't just go with what already exists, and list the images you want in a docker-compose.yaml file? (Which would additionally set you up for 'scaling' if you had any concern that you might be able to save some cash with a two or three smaller servers than one big one by the time you'd installed everything you want.)
- _jal 7y ago
  Isn't docker just duplicating effort that's already been done in LXC or jails?
  - OJFord 7y ago
    Are LXC and jails polularly used by maintainers of projects like owncloud, readlater, and such that Sovereign installs? I explicitly said that I wasn't making it about Docker vs. Ansible - I don't care - I just mean that Docker is very often used by first party maintainers (and if not by someone else) to package these services, so use what's there; if it had happened to have been something else that took off in that way, an Ansible playbook say, then that, but it's Dockerfiles that are in that position.
- thinkmassive 7y ago
  Not everyone wants Docker installed on their servers. It’s not a requirement with this playbook. Furthermore, if you do want to use containers, there are tools like ansible-bender[1] that use Ansible to build container images. [1] https://github.com/ansible-community/ansible-bender https://github.com/ansible-community/ansible-bender (edited the link to point to the ansible-community repo)
  - OJFord 7y ago
    But I'd have thought if you care that Docker isn't installed on your servers, you're probably not running this anyway? It seems to me that the target demographic is people that just want the least effort minimal faff way of getting some services up and running for personal non-production use. And for that it was my suggestion that many of the services probably already provide a Dockerfile upstream, so the easiest thing to do would be to install docker-compose, list the images, and `up`.
cs702 7y ago
Does anyone here on HN have experience using Sovereign in a team setting? I have a few questions: * Mobile contact and calendar syncing: How well and reliably does it work? * Calendar group features: how well do they work? * Setup and maintenance: how much hassle is involved?
- ses1984 7y ago
  Doing that stuff is hard. Sovereign doesn't solve all your operational problems. I think it's suitable for personal use. I wouldn't run it in a production setting without thoroughly understanding all parts of the stack. I would say it's good for personal use or to demonstrate what ansible is capable of.
choward 7y ago
One of the most important things when managing data is not losing it. Does this have a solution for doing backups built in or do you come up with your own solution like using something your web host provides?
- tzumby 7y ago
  I haven't used this myself but it looks like it uses nightly encrypted backups using Tarsnap
- nemoniac 7y ago
  Did you miss the bit about tarsnap?
  - choward 7y ago
    Apparently I did. Oops.
say_it_as_it_is 7y ago
Could anyone describe what this means without the marketing cloud-speak bullshit?
- jj11837 7y ago
  Seriously, this technology isn't new.
  - slowhand09 7y ago
    True, but its new to some people. Probably 1/3 of the articles posted on HN are several years old, but still new to some people.
- apetresc 7y ago
  It's abundantly clear in the README. Every single component is spelled out. Did you only read the title or something?
- ainiriand 7y ago
  You really have a very good username.
  - say_it_as_it_is 7y ago
    living up to it as best I can
- Ohn0 7y ago
  It's using ansible to setup and configure a bunch of open source applications for common "cloud" apps like email, files, rss, etc.
slovette 7y ago
Is there a benefit to doing this over something like Cloudron(1)? I see this being for people that just want things to work without much of the effort to make it so. If that’s the case, a simple web UI that treats all the little solutions as “apps” in a way makes sense. Not plugging here, just curious to the practical everyday differences. 1. https://cloudron.io https://cloudron.io
- ocdtrekkie 7y ago
  Cloudron.io seems pretty nice, speaking as someone who uses/works on a different platform (Sandstorm.io). Cloudron is going to cost you money if you want the benefits of automatic updates and the like, which is a downside for some (I think paying for good products/services is worthwhile), but I absolutely think making management of a ton of apps simple like a phone is the key to self-hosted online services. One of the biggest features of a good self-hosting platform should be unified authentication and identity, so you aren't managing your account on a dozen different apps. I've always been impressed by Cloudron's well-maintained app library and constant march of major feature improvements to the platform.
  - rlander 7y ago
    Just wanted to mention that Sandstorm is awesome. For some reason, despite being around for years, I only found out about it recently. I really think that it should be way more popular than it currently is. Lots of apps are outdated, so I’m planning on investing some personal time to contribute to it.
    - ocdtrekkie 7y ago
      By all means, hit us up on the mailing list or in IRC if you have any questions or need any help!
jlkuester7 7y ago
This is a pretty cool setup! I have been tinkering in the self-hosting world for awhile now and I would say my biggest piece of advice is to learn Docker. (Not trying to shill for Docker here or start a Docker vs. Ansible flame war.) In my personal experience, Docker images (and docker-compose orchestrations) are more pervasive than Ansible playbooks and when it comes to self-hosting for personal use, hosting all my services in Docker has made it much easier to deploy/maintain them. (Not speaking to business use cases since in that situation you should really have a deep familiarity with your tech stack and be able to roll-your-own Ansible playbook/Dockerfiles.) Also, there are so many great FLOSS alternatives to Google Apps. This repo contains some, but here are some of my favorites: * Drive/Calendar/Photos/Keep: - https://nextcloud.com/ (I prefer this over OwnCloud) * Docs/Sheets/Slides - https://www.onlyoffice.com/ * Mail - https://mailu.io/ (basically a Docker-based deployment of Postfix/Dovecot/etc) * Hangouts - Server - https://matrix.org/ - Client - https://about.riot.im/ - (I prefer Matrix.org over Jabber/XMPP) * G+/Twitter - https://joinmastodon.org/
- tvanantwerp 7y ago
  Are there any resources you recommend for learning to use Docker specifically for self-hosted services in a home lab setting?
  - jlkuester7 7y ago
    IMHO the official Docker documentation is great! Their "getting started" page is a good overview of the Docker basics. https://docs.docker.com/get-started/ https://docs.docker.com/get-started/ The CLI --help is pretty useful too. I have also found Docker's in-depth documentation to be very helpful when trying to figure out the more complex features. I got started by just grabbing a $5 DigitalOcean droplet (can get them with Docker pre-installed) and then played around trying to setup a simple app. (I think it was RocketChat.) https://rocket.chat/docs/installation/docker-containers/ https://rocket.chat/docs/installation/docker-containers/
  - orionhasyou 7y ago
    https://www.linuxserver.io/ https://www.linuxserver.io/ not a resource but docker containers you might want to run in your homelab. checkout awesome docker on github or katacoda
dguido 7y ago
There are so many servers and apps being installed by Sovereign that I'm certain few would be able to keep it secure (https://github.com/sovereign/sovereign/wiki/Software-used-by-Sovereign https://github.com/sovereign/sovereign/wiki/Software-used-by...). The big win for the cloud is that you're paying a fraction of the cost for access to a, typically, enormous security and operations team. If you want to build software like this that allows people to self-host, you need to scale down what you deploy to what a single person can reasonably manage. This isn't it. Fun todo: Install this somewhere, nmap it for open ports, then ask "How many of these services had a remotely exploitable CVE in the last year?" "If one of these services had one tomorrow, would I know to patch it and take action faster than someone would takeover my box?" I don't see any containment mechanisms on any of these services beyond what's included by default so a compromise of one service likely leads to total compromise of the entire box. I had to think about this a lot with AlgoVPN (https://github.com/trailofbits/algo https://github.com/trailofbits/algo), and we built a system with no out-of-the-box remote administration, strong isolation between services with AppArmor, CPU accounting, and privilege reductions, and limited third party dependencies and software. You can't count on a full-time, expert system administrator.
- PaulRobinson 7y ago
  fail2ban and rkhunter are in the kit, and that offsets some of the issues: you get some assurance and protection right there out of the box. You can also comment out the bits you don't want from https://github.com/sovereign/sovereign/blob/master/site.yml https://github.com/sovereign/sovereign/blob/master/site.yml before you run the top level playbook.
  - sneak 7y ago
    fail2ban is security theater. Turn off password-based ssh authentication, use keys only, and you’re done. You don’t need additional software for it. For extra security, bind ssh to localhost only and run a tor hidden service on the machine for accessing it.
- wpietri 7y ago
  This touches on a problem I've been thinking about a lot. AWS, etc, have solved the problem of hardware operation. Containerization is doing the same for OS operation. I think the next thing is app operation. For some things, I'm happy to use SaaS providers, where they are responsible for the whole stack. For others, I'm happy to use apps, where they just provide the code, and I provide the platform. But for a number of things, I want something in between: I provide storage and compute, they provide code and operations. Bitwarden for me is a good example. They're a password manager who provides their backend as a docker container anybody can run. I like that, as I don't really want them to have my data, and if they go out of business, I don't want to be cut off from my passwords. But I won't run the backend myself, because I don't have the time and expertise necessary to make sure it stays secure. Another good example is photo hosting. I would rather keep all my photos on space I control. But I also need modern, maintained software for syncing, serving, and controlling access to photos and related data. I'm happy to pay somebody to make and maintain that software, but not nearly as happy if that means that at any point they might shut down and take my data with them. I suspect we're headed toward a future where people like Synology and Digital Ocean sell storage+compute, and then other companies sell and maintain user-selected software that runs on those environments. Basically, some sort of app store for servers. But I'd love to see this happen in an open, nonproprietary way, as the drawbacks of Apple's and Google's app stores have become pretty clear.
gramakri 7y ago
For those in the market to run a private cloud, please try https://cloudron.io https://cloudron.io . Our motivation is to make it simple to selfhost apps . The main advantage is that we take care of automatic updates across all the apps we package. Happy to answer any questions. Disclaimer: I am the co-founder
- jlgaddis 7y ago
  Is it really self-hosting if you're running it on someone's cloud?
  - wpietri 7y ago
    Definitely. I mean, you could play an infinite regress game. Do you own the hardware? Do you own the cage the hardware is in? Do you own the building that the cage is in, and the land that the building is on? And then we can go toward owning the power company and the connections to anybody your servers talk to. But in practice, self-hosting is about control. If what you're running it on is a commodity cloud instance that you could get from a half-dozen providers, then any one cloud provider has very little leverage over you.
    - Tepix 7y ago
      No. If you have dedicated hardware (rented or owned) and full disk encryption you have decent control over your data. On a virtual server you have no control and no privacy.
      - wpietri 7y ago
        Depends on what you mean by control. It sounds like you're worried about different downside risks than I am.
  - gramakri 7y ago
    I think like many other terms like 'cloud', 'private', 'start up' there is a lot of gray area :). It's up to you where you draw the line. For me, self-hosting means running software in a manner where I have control of the data/application code and the server. With that definition, running software on EC2/DO/Linode is self-hosting. When I self-host using these servers, I know what the server is running and where the data resides. Also, I think there are other similar popular terms. For those who run in their own premises, the term is on-premise. For those running it home, usually they call it home lab/NAS/home server. Self-hosting to me encompasses all this. Also, self-hosting doesn't necessarily mean just open source. There are some amazing closed apps out there that you can self-host - emby, confluence, teamspeak to name a few. Two of my favorite spots - https://github.com/awesome-selfhosted/awesome-selfhosted https://github.com/awesome-selfhosted/awesome-selfhosted and https://www.reddit.com/r/selfhosted/ https://www.reddit.com/r/selfhosted/
djsumdog 7y ago
I wrote something similar that's custom for my personal infrastructure: https://github.com/sumdog/bee2 https://github.com/sumdog/bee2 There are some blog posts in the README that go into how I built a lot of it. A lot of it is specialized for me though. I have a ton of rspec/tests but I don't have a real config schema or entirely useful error messages. I might add some in the future. Looking at the list in this, I'd advice against nextCloud(ownCloud). I recently setup their official Docker containers and the web piece works alright, but their F-droid app continually crashes and I had to uninstall it and the nextcloud-client in Gentoo's package manager segfaults at home and refused to build at work. I've read other stories of data loss with nextcloud. It might be better now but my initial experiences made me use syncthing. Syncthing does use relays if you're behind a NAT, but if you have openvpn setup, you can also force it to use a direct IP address as well. If you're thinking if self hosting and have the time, I'd suggest building it yourself; borrowing (and properly accrediting/licensing) other open source projects, their ansible scripts and containers and such. You learn a whole lot about why this tooling is so complex.
TheFiend7 7y ago
This looks super cool. Though somewhat offtopic, this line absolutely cracked me up. >A VPS (or bare-metal server if you wanna ball hard). I can appreciate a sense of humor.
mindslight 7y ago
On the general topic of Ansible and personal infrastructure: Every time I attempt to use Ansible (or its kin) to manage my own network, it feels overly obtuse and ultimately unhelpful. Its gains seem to be rooted in configuring a large number of identical servers, and isn't geared for a handful of hosts with some commonalities and some differences. Writing playbooks feels like a still-imperative wrapper around shell commands, just in a bespoke and verbose YAML syntax. Instead I am using my own script that runs a tree of files through a template engine, drops them on each host being configured, and then runs triggers based on what has changed. This seems utterly simplistic, lacks polish, eschews common practices, etc. But the overall configuration seems straightforwardly grokkable compared to the heavy tools.
- asokoloski 7y ago
  I had a similar reaction, after trying out Ansible at my last job. We ended up switching to fabric, which is all in Python. It was mostly good but had some awkward warts, which it seems that they've mostly addressed in fabric2. Anyway, it might be worth a look, based on my understanding of your use case.
  - mindslight 7y ago
    Fabric seems more down to earth, but doesn't itself solve the problem of actually defining the configuration of each host. Having said that, I am to the point where it would be really nice if my ssh pushes ran in parallel, which is one of those robust niceties you give up by going your own way. So I'll have to revisit Fabric because it would be complementary - thanks for the reminder!
- bcrosby95 7y ago
  I never really saw Ansible as a heavy tool. But a place I worked at in the past used cfengine, so my barometer for this is probably a bit different.
  - mindslight 7y ago
    You had a point about error checking that I think is pertinent to my not being in Ansible's niche. If a service fails to start on a single host, it's because I'm already tinkering with that configuration and hence will notice the failure. Whereas with tens of highly similar hosts, you want to find out about exceptions sooner. I'm actually getting to this point with some common things (apt upgrade), hence looking into Ansible again recently. Overall my goal is to write very few on-host configuration scripts, and prefer overwriting files. For example, most triggers are simply service restarts, which can also be performed by a full reboot.
Tepix 7y ago
Wow, didn't expect to see Sovereign at the top of HN today! I'm one of the project contributors. If this project piques your interest, please consider contributing! We could really need more helping hands. Ansible is easy to learn and most (not all!) problems due to new versions are easy to fix. Also, if you only want to use a fraction of what Sovereign has to offer to reduce your server's attack surface, that's easy! Just follow the instructions.
fak3r 7y ago
I've used this project off and on for years, and it's always worked perfectly. I'd have an infosec conference to go to, I'd setup a host with wireguard, give my friends the cert, we'd all tunnel out though that, then tear it down after the con. Total cost a few dollars (most are $5/month)
crmrc114 7y ago
Dumb question maybe... but why would you not just configure this yourself on a single virt/host? Most of these services would take less than a day to configure. So many questions on why this is a good thing. Like, there are countless ways to configure your MTA and spam filtering- if you are going to have to dig through this config.. why not just roll your own? Can someone explain to me why you need ansimble for this? or am I just being stupid and this is like an exercise to show what the toolchain can do?
- tryptophan 7y ago
  If you're the type of person that configures servers on your own time for fun, than you would likely find automating the process 'just because' to be fun as well. You do not need ansible for config of a personal server at all.
  - crmrc114 7y ago
    Oh cool cool, I thought I had a stroke or was missing something here. This is a cool idea in that case.
- apple4ever 7y ago
  Because if you want to switch providers, you’d have to do that all again. Or if your current provider’s VPS just dies (I’ve had it happen) you’d have to do that all again. With Ansible, you just run it again and in a fraction of the time its back up.
Annatar 7y ago
Quit this damn nonsense with Ansible or whatever garbage fashion fad is in vogue these days and finally learn how to make OS packages so you can do configuration management with normal shell scripting inside of them, because that's what it's for. The amount of incompetence and insanity from incompetence has gone too far. Damn it, this is exactly why IT sucks so bad!!!
- davestephens 7y ago
  Maybe you've never worked with someone that sucked at shell scripting, or packaging. Ansible is awesome for enabling people to do reasonably complicated things in a consistent manner, at scale, without having to write all of the boilerplate code to be able to do so. This is forgetting the fact that Ansible is reasonably opinionated, which is great for lowering the barrier to entry and helping devs/admins to be productive quickly. When I just need to Get Shit Done, Ansible is awesome.
  - wtf1234 7y ago
    > lowering the barrier to entry That's the opposite of a good thing. > When I just need to Get Shit Done, Ansible is awesome. "just" is the keyword. "just" instead of caring about long term maintainability and security
    - apple4ever 7y ago
      LOL on your first point. For your second, Ansible is specifically designed for long term maintainability and security.
  - Annatar 7y ago
    I've worked with plenty of people who had no business working in IT, but some of them were smart enough to recognize that they could benefit from education, so I taught them how to correctly and properly program in shell. Others were just insecure assholes who didn't care about working with computers and were in IT just for the money but didn't want anyone to know they're incompetent (even though it was obvious). Those always wanted to needlessly spend money on buying shitty software or using "tools" like "Ansible".
jophde 7y ago
I currently just let my desktop run constantly. It runs Windows in a KMS for games and sleeping breaks it so I never even suspend. It only seems to use about $10/month in power. I have been considering using DDNS through my router and Cloudflare and trying to create a iOS/Android app that will automatically upload my photos to the my DDNS for storage on my desktop. It feels a little crazy but the idea of syncing my photos to my own machine with no middle man is comforting. DDNS seems like it's a local too good to be true for solving the dynamic IP problem. I'd prefer to have a static IP for my gigabit Internet but sadly Webpass doesn't allow it. Does anyone have experience doing something like this?
- hrjd 7y ago
  Ngrok as alternative to dynamic DNS?
  - jophde 7y ago
    DDNS maps a domain to your current dynamic IP. Its built into most routers. They will push the new ip to a DDNS service when it changes.
ggm 7y ago
Version for a Pi. Version for BSD.
haolez 7y ago
I've used Ansible successfully to turn Windows laptops into Wintendo machines :)
haolez 7y ago
As the CTO of an established company, I cannot imagine a situation where I would prefer to maintain my own infrastructure vs using managed cloud services. If I get locked in on a specific product, it's way cheaper to redesign that around an alternative vendor than it is to maintain a private cloud (Ansible, Kubernetes and friends included). As a nerd, I'd prefer to do things myself, but I have business needs to attend to.
- modoc 7y ago
  Cost. Depending on your scale and needs, running stuff on real hardware you manage can be a lot cheaper than AWS/GCP.
- xref 7y ago
  Just depends on workload. Especially if your biz is bandwidth intensive that can be your largest single line item on the major clouds. Then you gotta pay engineers to find ways to reduce your bandwidth costs and the snake eats its own tail
- apple4ever 7y ago
  Performance, cost, security. For small businesses, it may be a wash (due to ability to hire system engineers), but for medium to large business rolling your own is almost always better.
liotier 7y ago
Nowhere do I see mention of how to update and upgrade this thing after initial deployment... How does it work ?
bilekas 7y ago
Nice resource, will definitely give it a spin, also this is the first time I've ever come accross `Tarsnap` looks really interesting too !
mekster 7y ago
There seems to be a few questionable picks in the readme. * Why pick ownCloud over NextCloud? The former's forum had 139 posts in the last 7 days and the latter's forum had about 1700. Also some of the features in the former product are locked for enterprise only. https://central.owncloud.org/about https://central.owncloud.org/about https://help.nextcloud.com/about https://help.nextcloud.com/about * Tarsnap is a paid online service. You could try restic command to have encrypted backup to remote storages. * cgit is an old project released more than 10 years ago and despite being written by the author of wireguard, we have far better stuff like Gitea (or its fork source Gogs) to have user access control with nice web interface for git project management.
platform 7y ago
I looked at Soverign at the time I was setting up private cloud on a $5 VPS (prgmr.com with 1.5 gb ram) I went with YunoHost. https://yunohost.org/#/apps https://yunohost.org/#/apps I initially tried sovereign, but once I figured out I had to pay for tarsnap backup service, and that it did not have ansible for nginx setup (I needed that experience for work stuff), I went with Yunohost. Sofar I am happy with YunoHost and subscribed to send periodic donation to the project. Overall, though, if you are working with ansible at work, or want to advance in devops field, learning ansible and contributing to Sovereign project would be a good path to take.

556 points by c0restraint 7y ago

navaati 7y ago

For my fellow HNers, this is "private cloud" in the meaning of Owncloud, not of Openstack.

choward 7y ago
I hear about Nextcloud a lot more than I hear about Owncloud. Does anyone know why this project uses Owncloud instead of Nextcloud?
- crashbunny 7y ago
  This project started before Nextcloud existed, I don't know why it hasn't switched, though.
  - nemoniac 7y ago
    Also Nextcloud puts a LOT more effort into publicity than Owncloud.
  - AdmiralAsshat 7y ago
    NextCloud was started as a fork of OwnCloud by a bunch of OwnCloud devs that were unhappy with the direction the product was taking. You can think of it in much the same way as OpenOffice vs LibreOffice: devs fork to make a new product, the "original" product stagnates and is mostly used for rent-seeking. The downside of both is that, to my ears, both "OpenOffice" and "OwnCloud" better signify to outsiders what the product accomplishes, while "LibreOffice" and "NextCloud" really don't, unless you're already familiar with the product or product history.
- judge2020 7y ago
  Based on how long this project has existed, I suspect they chose owncloud before nextcloud was forked and changing it hasn't happened/some people don't want to change their already working installations.
- deleted 7y ago
  [deleted]
- Tepix 7y ago
  I think there's an open issue for the switch. Just need a good pull request by a volunteer...
- gramakri 7y ago
  For some history: ownCloud is a very old stable product. 2-3 years back it got forked into Nexcloud because the founder of ownCloud had some disagreement about ownCloud's direction. Both the products are backed by companies and both are doing quite well. I would say Nextcloud goes more and more into expanding it's use case and thus makes it product more extensible via plugins. This can be good or bad depending on how you look at it. Plugins go unmaintained/incompatible over time and are a constant source of pain when upgrading. Wordpress gets away with this because it has a massive community. ownCloud on the other hand has decided to double down on it's roots of file sharing/syncing. I heard they rewrote their stack from PHP to Go now and the frontend is now React.

OJFord 7y ago

Isn't this just duplicating effort that's probably already been done, in many cases by the first-party maintainers, in Dockerfiles? I don't mean to start 'Docker vs. Ansible', I just wonder why if you wanted a quick way to setup a single-server 'own private cloud' you wouldn't just go with what already exists, and list the images you want in a docker-compose.yaml file? (Which would additionally set you up for 'scaling' if you had any concern that you might be able to save some cash with a two or three smaller servers than one big one by the time you'd installed everything you want.)

_jal 7y ago
Isn't docker just duplicating effort that's already been done in LXC or jails?
- OJFord 7y ago
  Are LXC and jails polularly used by maintainers of projects like owncloud, readlater, and such that Sovereign installs? I explicitly said that I wasn't making it about Docker vs. Ansible - I don't care - I just mean that Docker is very often used by first party maintainers (and if not by someone else) to package these services, so use what's there; if it had happened to have been something else that took off in that way, an Ansible playbook say, then that, but it's Dockerfiles that are in that position.
thinkmassive 7y ago
Not everyone wants Docker installed on their servers. It’s not a requirement with this playbook. Furthermore, if you do want to use containers, there are tools like ansible-bender[1] that use Ansible to build container images. [1] https://github.com/ansible-community/ansible-bender https://github.com/ansible-community/ansible-bender (edited the link to point to the ansible-community repo)
- OJFord 7y ago
  But I'd have thought if you care that Docker isn't installed on your servers, you're probably not running this anyway? It seems to me that the target demographic is people that just want the least effort minimal faff way of getting some services up and running for personal non-production use. And for that it was my suggestion that many of the services probably already provide a Dockerfile upstream, so the easiest thing to do would be to install docker-compose, list the images, and `up`.

cs702 7y ago

Does anyone here on HN have experience using Sovereign in a team setting? I have a few questions: * Mobile contact and calendar syncing: How well and reliably does it work? * Calendar group features: how well do they work? * Setup and maintenance: how much hassle is involved?

ses1984 7y ago
Doing that stuff is hard. Sovereign doesn't solve all your operational problems. I think it's suitable for personal use. I wouldn't run it in a production setting without thoroughly understanding all parts of the stack. I would say it's good for personal use or to demonstrate what ansible is capable of.

choward 7y ago

One of the most important things when managing data is not losing it. Does this have a solution for doing backups built in or do you come up with your own solution like using something your web host provides?

tzumby 7y ago
I haven't used this myself but it looks like it uses nightly encrypted backups using Tarsnap
nemoniac 7y ago
Did you miss the bit about tarsnap?
- choward 7y ago
  Apparently I did. Oops.

say_it_as_it_is 7y ago

Could anyone describe what this means without the marketing cloud-speak bullshit?

jj11837 7y ago
Seriously, this technology isn't new.
- slowhand09 7y ago
  True, but its new to some people. Probably 1/3 of the articles posted on HN are several years old, but still new to some people.
apetresc 7y ago
It's abundantly clear in the README. Every single component is spelled out. Did you only read the title or something?
ainiriand 7y ago
You really have a very good username.
- say_it_as_it_is 7y ago
  living up to it as best I can
Ohn0 7y ago
It's using ansible to setup and configure a bunch of open source applications for common "cloud" apps like email, files, rss, etc.

slovette 7y ago

Is there a benefit to doing this over something like Cloudron(1)? I see this being for people that just want things to work without much of the effort to make it so. If that’s the case, a simple web UI that treats all the little solutions as “apps” in a way makes sense. Not plugging here, just curious to the practical everyday differences. 1. https://cloudron.io https://cloudron.io

ocdtrekkie 7y ago
Cloudron.io seems pretty nice, speaking as someone who uses/works on a different platform (Sandstorm.io). Cloudron is going to cost you money if you want the benefits of automatic updates and the like, which is a downside for some (I think paying for good products/services is worthwhile), but I absolutely think making management of a ton of apps simple like a phone is the key to self-hosted online services. One of the biggest features of a good self-hosting platform should be unified authentication and identity, so you aren't managing your account on a dozen different apps. I've always been impressed by Cloudron's well-maintained app library and constant march of major feature improvements to the platform.
- rlander 7y ago
  Just wanted to mention that Sandstorm is awesome. For some reason, despite being around for years, I only found out about it recently. I really think that it should be way more popular than it currently is. Lots of apps are outdated, so I’m planning on investing some personal time to contribute to it.
  - ocdtrekkie 7y ago
    By all means, hit us up on the mailing list or in IRC if you have any questions or need any help!

jlkuester7 7y ago

This is a pretty cool setup! I have been tinkering in the self-hosting world for awhile now and I would say my biggest piece of advice is to learn Docker. (Not trying to shill for Docker here or start a Docker vs. Ansible flame war.) In my personal experience, Docker images (and docker-compose orchestrations) are more pervasive than Ansible playbooks and when it comes to self-hosting for personal use, hosting all my services in Docker has made it much easier to deploy/maintain them. (Not speaking to business use cases since in that situation you should really have a deep familiarity with your tech stack and be able to roll-your-own Ansible playbook/Dockerfiles.) Also, there are so many great FLOSS alternatives to Google Apps. This repo contains some, but here are some of my favorites: * Drive/Calendar/Photos/Keep: - https://nextcloud.com/ (I prefer this over OwnCloud) * Docs/Sheets/Slides - https://www.onlyoffice.com/ * Mail - https://mailu.io/ (basically a Docker-based deployment of Postfix/Dovecot/etc) * Hangouts - Server - https://matrix.org/ - Client - https://about.riot.im/ - (I prefer Matrix.org over Jabber/XMPP) * G+/Twitter - https://joinmastodon.org/

tvanantwerp 7y ago
Are there any resources you recommend for learning to use Docker specifically for self-hosted services in a home lab setting?
- jlkuester7 7y ago
  IMHO the official Docker documentation is great! Their "getting started" page is a good overview of the Docker basics. https://docs.docker.com/get-started/ https://docs.docker.com/get-started/ The CLI --help is pretty useful too. I have also found Docker's in-depth documentation to be very helpful when trying to figure out the more complex features. I got started by just grabbing a $5 DigitalOcean droplet (can get them with Docker pre-installed) and then played around trying to setup a simple app. (I think it was RocketChat.) https://rocket.chat/docs/installation/docker-containers/ https://rocket.chat/docs/installation/docker-containers/
- orionhasyou 7y ago
  https://www.linuxserver.io/ https://www.linuxserver.io/ not a resource but docker containers you might want to run in your homelab. checkout awesome docker on github or katacoda

dguido 7y ago

There are so many servers and apps being installed by Sovereign that I'm certain few would be able to keep it secure (https://github.com/sovereign/sovereign/wiki/Software-used-by-Sovereign https://github.com/sovereign/sovereign/wiki/Software-used-by...). The big win for the cloud is that you're paying a fraction of the cost for access to a, typically, enormous security and operations team. If you want to build software like this that allows people to self-host, you need to scale down what you deploy to what a single person can reasonably manage. This isn't it. Fun todo: Install this somewhere, nmap it for open ports, then ask "How many of these services had a remotely exploitable CVE in the last year?" "If one of these services had one tomorrow, would I know to patch it and take action faster than someone would takeover my box?" I don't see any containment mechanisms on any of these services beyond what's included by default so a compromise of one service likely leads to total compromise of the entire box. I had to think about this a lot with AlgoVPN (https://github.com/trailofbits/algo https://github.com/trailofbits/algo), and we built a system with no out-of-the-box remote administration, strong isolation between services with AppArmor, CPU accounting, and privilege reductions, and limited third party dependencies and software. You can't count on a full-time, expert system administrator.

PaulRobinson 7y ago
fail2ban and rkhunter are in the kit, and that offsets some of the issues: you get some assurance and protection right there out of the box. You can also comment out the bits you don't want from https://github.com/sovereign/sovereign/blob/master/site.yml https://github.com/sovereign/sovereign/blob/master/site.yml before you run the top level playbook.
- sneak 7y ago
  fail2ban is security theater. Turn off password-based ssh authentication, use keys only, and you’re done. You don’t need additional software for it. For extra security, bind ssh to localhost only and run a tor hidden service on the machine for accessing it.
wpietri 7y ago
This touches on a problem I've been thinking about a lot. AWS, etc, have solved the problem of hardware operation. Containerization is doing the same for OS operation. I think the next thing is app operation. For some things, I'm happy to use SaaS providers, where they are responsible for the whole stack. For others, I'm happy to use apps, where they just provide the code, and I provide the platform. But for a number of things, I want something in between: I provide storage and compute, they provide code and operations. Bitwarden for me is a good example. They're a password manager who provides their backend as a docker container anybody can run. I like that, as I don't really want them to have my data, and if they go out of business, I don't want to be cut off from my passwords. But I won't run the backend myself, because I don't have the time and expertise necessary to make sure it stays secure. Another good example is photo hosting. I would rather keep all my photos on space I control. But I also need modern, maintained software for syncing, serving, and controlling access to photos and related data. I'm happy to pay somebody to make and maintain that software, but not nearly as happy if that means that at any point they might shut down and take my data with them. I suspect we're headed toward a future where people like Synology and Digital Ocean sell storage+compute, and then other companies sell and maintain user-selected software that runs on those environments. Basically, some sort of app store for servers. But I'd love to see this happen in an open, nonproprietary way, as the drawbacks of Apple's and Google's app stores have become pretty clear.

gramakri 7y ago

For those in the market to run a private cloud, please try https://cloudron.io https://cloudron.io . Our motivation is to make it simple to selfhost apps . The main advantage is that we take care of automatic updates across all the apps we package. Happy to answer any questions. Disclaimer: I am the co-founder

jlgaddis 7y ago
Is it really self-hosting if you're running it on someone's cloud?
- wpietri 7y ago
  Definitely. I mean, you could play an infinite regress game. Do you own the hardware? Do you own the cage the hardware is in? Do you own the building that the cage is in, and the land that the building is on? And then we can go toward owning the power company and the connections to anybody your servers talk to. But in practice, self-hosting is about control. If what you're running it on is a commodity cloud instance that you could get from a half-dozen providers, then any one cloud provider has very little leverage over you.
  - Tepix 7y ago
    No. If you have dedicated hardware (rented or owned) and full disk encryption you have decent control over your data. On a virtual server you have no control and no privacy.
    - wpietri 7y ago
      Depends on what you mean by control. It sounds like you're worried about different downside risks than I am.
- gramakri 7y ago
  I think like many other terms like 'cloud', 'private', 'start up' there is a lot of gray area :). It's up to you where you draw the line. For me, self-hosting means running software in a manner where I have control of the data/application code and the server. With that definition, running software on EC2/DO/Linode is self-hosting. When I self-host using these servers, I know what the server is running and where the data resides. Also, I think there are other similar popular terms. For those who run in their own premises, the term is on-premise. For those running it home, usually they call it home lab/NAS/home server. Self-hosting to me encompasses all this. Also, self-hosting doesn't necessarily mean just open source. There are some amazing closed apps out there that you can self-host - emby, confluence, teamspeak to name a few. Two of my favorite spots - https://github.com/awesome-selfhosted/awesome-selfhosted https://github.com/awesome-selfhosted/awesome-selfhosted and https://www.reddit.com/r/selfhosted/ https://www.reddit.com/r/selfhosted/

djsumdog 7y ago

I wrote something similar that's custom for my personal infrastructure: https://github.com/sumdog/bee2 https://github.com/sumdog/bee2 There are some blog posts in the README that go into how I built a lot of it. A lot of it is specialized for me though. I have a ton of rspec/tests but I don't have a real config schema or entirely useful error messages. I might add some in the future. Looking at the list in this, I'd advice against nextCloud(ownCloud). I recently setup their official Docker containers and the web piece works alright, but their F-droid app continually crashes and I had to uninstall it and the nextcloud-client in Gentoo's package manager segfaults at home and refused to build at work. I've read other stories of data loss with nextcloud. It might be better now but my initial experiences made me use syncthing. Syncthing does use relays if you're behind a NAT, but if you have openvpn setup, you can also force it to use a direct IP address as well. If you're thinking if self hosting and have the time, I'd suggest building it yourself; borrowing (and properly accrediting/licensing) other open source projects, their ansible scripts and containers and such. You learn a whole lot about why this tooling is so complex.

TheFiend7 7y ago

This looks super cool. Though somewhat offtopic, this line absolutely cracked me up. >A VPS (or bare-metal server if you wanna ball hard). I can appreciate a sense of humor.

mindslight 7y ago

On the general topic of Ansible and personal infrastructure: Every time I attempt to use Ansible (or its kin) to manage my own network, it feels overly obtuse and ultimately unhelpful. Its gains seem to be rooted in configuring a large number of identical servers, and isn't geared for a handful of hosts with some commonalities and some differences. Writing playbooks feels like a still-imperative wrapper around shell commands, just in a bespoke and verbose YAML syntax. Instead I am using my own script that runs a tree of files through a template engine, drops them on each host being configured, and then runs triggers based on what has changed. This seems utterly simplistic, lacks polish, eschews common practices, etc. But the overall configuration seems straightforwardly grokkable compared to the heavy tools.

asokoloski 7y ago
I had a similar reaction, after trying out Ansible at my last job. We ended up switching to fabric, which is all in Python. It was mostly good but had some awkward warts, which it seems that they've mostly addressed in fabric2. Anyway, it might be worth a look, based on my understanding of your use case.
- mindslight 7y ago
  Fabric seems more down to earth, but doesn't itself solve the problem of actually defining the configuration of each host. Having said that, I am to the point where it would be really nice if my ssh pushes ran in parallel, which is one of those robust niceties you give up by going your own way. So I'll have to revisit Fabric because it would be complementary - thanks for the reminder!
bcrosby95 7y ago
I never really saw Ansible as a heavy tool. But a place I worked at in the past used cfengine, so my barometer for this is probably a bit different.
- mindslight 7y ago
  You had a point about error checking that I think is pertinent to my not being in Ansible's niche. If a service fails to start on a single host, it's because I'm already tinkering with that configuration and hence will notice the failure. Whereas with tens of highly similar hosts, you want to find out about exceptions sooner. I'm actually getting to this point with some common things (apt upgrade), hence looking into Ansible again recently. Overall my goal is to write very few on-host configuration scripts, and prefer overwriting files. For example, most triggers are simply service restarts, which can also be performed by a full reboot.

Tepix 7y ago

Wow, didn't expect to see Sovereign at the top of HN today! I'm one of the project contributors. If this project piques your interest, please consider contributing! We could really need more helping hands. Ansible is easy to learn and most (not all!) problems due to new versions are easy to fix. Also, if you only want to use a fraction of what Sovereign has to offer to reduce your server's attack surface, that's easy! Just follow the instructions.

fak3r 7y ago

I've used this project off and on for years, and it's always worked perfectly. I'd have an infosec conference to go to, I'd setup a host with wireguard, give my friends the cert, we'd all tunnel out though that, then tear it down after the con. Total cost a few dollars (most are $5/month)

crmrc114 7y ago

Dumb question maybe... but why would you not just configure this yourself on a single virt/host? Most of these services would take less than a day to configure. So many questions on why this is a good thing. Like, there are countless ways to configure your MTA and spam filtering- if you are going to have to dig through this config.. why not just roll your own? Can someone explain to me why you need ansimble for this? or am I just being stupid and this is like an exercise to show what the toolchain can do?

tryptophan 7y ago
If you're the type of person that configures servers on your own time for fun, than you would likely find automating the process 'just because' to be fun as well. You do not need ansible for config of a personal server at all.
- crmrc114 7y ago
  Oh cool cool, I thought I had a stroke or was missing something here. This is a cool idea in that case.
apple4ever 7y ago
Because if you want to switch providers, you’d have to do that all again. Or if your current provider’s VPS just dies (I’ve had it happen) you’d have to do that all again. With Ansible, you just run it again and in a fraction of the time its back up.

Annatar 7y ago

Quit this damn nonsense with Ansible or whatever garbage fashion fad is in vogue these days and finally learn how to make OS packages so you can do configuration management with normal shell scripting inside of them, because that's what it's for. The amount of incompetence and insanity from incompetence has gone too far. Damn it, this is exactly why IT sucks so bad!!!

davestephens 7y ago
Maybe you've never worked with someone that sucked at shell scripting, or packaging. Ansible is awesome for enabling people to do reasonably complicated things in a consistent manner, at scale, without having to write all of the boilerplate code to be able to do so. This is forgetting the fact that Ansible is reasonably opinionated, which is great for lowering the barrier to entry and helping devs/admins to be productive quickly. When I just need to Get Shit Done, Ansible is awesome.
- wtf1234 7y ago
  > lowering the barrier to entry That's the opposite of a good thing. > When I just need to Get Shit Done, Ansible is awesome. "just" is the keyword. "just" instead of caring about long term maintainability and security
  - apple4ever 7y ago
    LOL on your first point. For your second, Ansible is specifically designed for long term maintainability and security.
- Annatar 7y ago
  I've worked with plenty of people who had no business working in IT, but some of them were smart enough to recognize that they could benefit from education, so I taught them how to correctly and properly program in shell. Others were just insecure assholes who didn't care about working with computers and were in IT just for the money but didn't want anyone to know they're incompetent (even though it was obvious). Those always wanted to needlessly spend money on buying shitty software or using "tools" like "Ansible".

jophde 7y ago

I currently just let my desktop run constantly. It runs Windows in a KMS for games and sleeping breaks it so I never even suspend. It only seems to use about $10/month in power. I have been considering using DDNS through my router and Cloudflare and trying to create a iOS/Android app that will automatically upload my photos to the my DDNS for storage on my desktop. It feels a little crazy but the idea of syncing my photos to my own machine with no middle man is comforting. DDNS seems like it's a local too good to be true for solving the dynamic IP problem. I'd prefer to have a static IP for my gigabit Internet but sadly Webpass doesn't allow it. Does anyone have experience doing something like this?

hrjd 7y ago
Ngrok as alternative to dynamic DNS?
- jophde 7y ago
  DDNS maps a domain to your current dynamic IP. Its built into most routers. They will push the new ip to a DDNS service when it changes.

ggm 7y ago

Version for a Pi. Version for BSD.

haolez 7y ago

I've used Ansible successfully to turn Windows laptops into Wintendo machines :)

haolez 7y ago

As the CTO of an established company, I cannot imagine a situation where I would prefer to maintain my own infrastructure vs using managed cloud services. If I get locked in on a specific product, it's way cheaper to redesign that around an alternative vendor than it is to maintain a private cloud (Ansible, Kubernetes and friends included). As a nerd, I'd prefer to do things myself, but I have business needs to attend to.

modoc 7y ago
Cost. Depending on your scale and needs, running stuff on real hardware you manage can be a lot cheaper than AWS/GCP.
xref 7y ago
Just depends on workload. Especially if your biz is bandwidth intensive that can be your largest single line item on the major clouds. Then you gotta pay engineers to find ways to reduce your bandwidth costs and the snake eats its own tail
apple4ever 7y ago
Performance, cost, security. For small businesses, it may be a wash (due to ability to hire system engineers), but for medium to large business rolling your own is almost always better.

liotier 7y ago

Nowhere do I see mention of how to update and upgrade this thing after initial deployment... How does it work ?

bilekas 7y ago

Nice resource, will definitely give it a spin, also this is the first time I've ever come accross `Tarsnap` looks really interesting too !

mekster 7y ago

There seems to be a few questionable picks in the readme. * Why pick ownCloud over NextCloud? The former's forum had 139 posts in the last 7 days and the latter's forum had about 1700. Also some of the features in the former product are locked for enterprise only. https://central.owncloud.org/about https://central.owncloud.org/about https://help.nextcloud.com/about https://help.nextcloud.com/about * Tarsnap is a paid online service. You could try restic command to have encrypted backup to remote storages. * cgit is an old project released more than 10 years ago and despite being written by the author of wireguard, we have far better stuff like Gitea (or its fork source Gogs) to have user access control with nice web interface for git project management.

platform 7y ago

I looked at Soverign at the time I was setting up private cloud on a $5 VPS (prgmr.com with 1.5 gb ram) I went with YunoHost. https://yunohost.org/#/apps https://yunohost.org/#/apps I initially tried sovereign, but once I figured out I had to pay for tarsnap backup service, and that it did not have ansible for nginx setup (I needed that experience for work stuff), I went with Yunohost. Sofar I am happy with YunoHost and subscribed to send periodic donation to the project. Overall, though, if you are working with ansible at work, or want to advance in devops field, learning ansible and contributing to Sovereign project would be a good path to take.