12 ms·

The diagram alone is more than enough of an argument to dissuade me from giving this a shot right now - it's simply too complicated and too much to manage for t

The diagram alone is more than enough of an argument to dissuade me from giving this a shot right now - it's simply too complicated and too much to manage for the amount of time I can dedicate to it. BUT - I'm really thankful for people who keep posting and sharing these sorts of projects; they're the ones iterating the process for the rest of us who need something a bit more turn-key. I'm excited to see this eventually result in something like the following: - Standard / Easy to update containerized setup. - Out of the box multi-location syncs (e.g. home, VPS, etc.) - Takes 5 minutes to configure/add new locations I want this to be as easy as adding a new AP to my mesh wifi system at home: plug it in, open the app, name the AP, and click "Done". (Edit - formatting)

gsreenivas 6y ago
check out Helm - thehelm.com
merb 6y ago
just use k3s + (restic + velero [backup]) it's soo much easier you can basically install everything with the same tooling and update everything with the same tooling. if something breaks, bam you can just restore the whole cluster with velero (including local volumes)
imwillofficial 6y ago
I used Cloudron.io for most of the above. Check them out. Most polished self hosting experience by far.
- babelfish 6y ago
  And they have a good 'eject' option.
SkipperCat 6y ago
We always have this debate at work. Do we build the system ourselves or do we purchase a product? On prem Prometheus or push everything to DataDog? I'm always a fan of building things myself because I like building things, but my company compares engineering time vs product cost.
clairity 6y ago
i've actually daydreamed about starting a computing appliance company that would make a variety of services plug and play for consumers and small businesses, from email to storage, to networking, to security, and to smart home. it's actually the direction apple is headed, but they're encumbered by the innovator's dilemma, which leaves an opportunity for an upstart. google and facebook are similarly too focused on adtech, while amazon on commerce, to lock up this market yet.
- mikepurvis 6y ago
  It would be fun stuff to build, but I feel like you'd struggle to make money. Google and Amazon can afford to give away the hardware, and they can smuggle their ecosystem into your house as a thermostat or a smart speaker or a phone app, or whatever. Like, how do you persuade the audience of enthusiasts (think: Unifi buyers) to pay for a subscription to managed software they run on their own computers, raspis, whatever? I would probably spend $10/mo on something like that, but much above that and you'd be fighting against the armchair commentary of users who won't appreciate the effort that goes into stability and will basically have a "no wireless, less space than a Nomad, lame" attitude.
  - clairity 6y ago
    there are actually tons of companies in this space already making money (e.g., wyze), but it’s highly fragmented and none have a unified vision or product strategy yet. so yes, they’re vulnerable to the behemoths right now, but those dynamics aren’t locked in yet. it’s mostly tough because of the high upfront capital costs (manufacturing, r&d, and marketing). people still talk fondly about discontinued apple routers and what nest could have been as an independent venture, for example.
    - mikepurvis 6y ago
      Maybe I'm misunderstanding the pitch from the GP, but Wyze seems like it's pretty clearly a hardware + cloud services play, similar to most other IOT ecosystems except maybe Hue. The (optional) monthly cost paid there is for loosened restrictions on an already existing works-anywhere setup— it's an upsell for power users, not a cost of entry. This seems a lot easier to me than on-prem cloud services, either in BYOH form ("but it's just software") or as a packaged appliance ("another hub to install, really?"). I would say that the closest thing to this right now for paid is coming from the storage side— NAS providers like Synology using hardware sales to support a limited ecosystem of "one click" deployable apps. And for free, it's ecosystems like HomeAssistant, which a lot of people just deploy as a fire-and-forget RPi image, but as expected with a free ecosystem, as soon as you get off the ultra-common use cases, you're reading source code to figure out how it works, and wading through a tangle of unmaintained "community" plugins that only do half of what you want.
mikepurvis 6y ago
> The diagram alone is more than enough of an argument to dissuade me from giving this a shot right now - it's simply too complicated and too much to manage for the amount of time I can dedicate to it. Yeah. I have a basic home server and I feel like even with fairly modest needs/desires (Jellyfin, Deluge, Zoneminder, some kind of file syncing, I gave up on photos because my whole family uses Google for that), it's hard to find a reasonable workflow/setup that covers it all. It was basically down to partitioning by VM (proxmox) or partitioned by container (docker), and I went with Docker + Portainer, but I'm not really happy with it; even basic functionality like redeploying a Compose configuration has sat as a feature-ask for three years [1]. Maybe I'm wanting it to be something that it just isn't, and I'd be happier with microk8s and managing the apps as Helm charts. But is that just inviting additional complexity where none is needed? [1]: https://github.com/portainer/portainer/issues/1753 https://github.com/portainer/portainer/issues/1753
- vorpalhex 6y ago
  I used to have a portainer centric setup.. now I just use docker-compose directly. I have my compose split into different files with a makefile to keep things "make start" simple. Highly recommend.
  - mikepurvis 6y ago
    That is helpful to know— I may try to move more in this direction the next time I'm hacking on it.
  - simplecto 6y ago
    This is my setup. Compose and Makefiles Ben running like this for 3 years. No fuss.
- heavyset_go 6y ago
  You can migrate from docker-compose to Swarm very easily, and it'll handle redeployments automatically for you.
belval 6y ago
I think the diagram gives a skewed view of how hard this actually is. I run a very similar setup only my VPS is only a proxy for my home server and it requires very little maintenance. I run everything with docker-compose and I haven't had to work on my setup at all this year and only about 8 hours in 2020 to setup the Wireguard network to replace the ssh tunnels I was using previously for VPS -> server communications. At the end of the day YMMV and use what you are comfortable with, but it's not as crazy undertaking as it sounds.
- divbzero 6y ago
  Yes, and many popular applications are prepackaged as one-click apps by cloud providers like Vultr [1] or Digital Ocean [2]. [1]: https://www.vultr.com/features/one-click-apps/ https://www.vultr.com/features/one-click-apps/ [2]: https://marketplace.digitalocean.com https://marketplace.digitalocean.com You can also enable automatic backups for your servers.
  - deleted 6y ago
    [deleted]
  - Semaphor 6y ago
    Somewhat OT, but never realized how expensive those cloud instances are. For comparison, I pay $4.95/month (billed annually) for a KVM VPS with 2 Ghz, 2 GB RAM, 40gb SSD, 400 GB HDD in the Netherlands. That seems a lot better for selfhosting where you probably want more raw storage than more SSD space.
    - rkachowski 6y ago
      That's pretty awesome, what provider are you with?
      - Semaphor 6y ago
        https://www.prometeus.net/site/terakvm.php https://www.prometeus.net/site/terakvm.php But it looks like I got it on a sale, TERAKVM-400 comes down to $6.94 at normal prices.
turtlebits 6y ago
People like to bash on the complexity of Kubernetes, but this is exactly what it's for. Use a cloud provider or your own automation to create a cluster, then apply a set of configs to bring your services up.
ssivark 6y ago
From what I hear, it’s all pretty nicely containerized/turnkey already. There are even several “meta apps” (Eg: Homelab OS, YUNoHost, etc) which are like the base layer on which many of these services are available as “applications” which have been pre-configured and can be trivially instantiated. Those curious can check out /r/SelfHosted.
lucideer 6y ago
After reading through it all, I think this is more a condemnation of the author's diagram (or at least their decision to put that particular one up-front), than of their process in general, nor the challenge. Breakdown of (my) issues with the diagram: - author's interaction with each device is explicitly included, adding unnecessary noise - "partial" and "full" real-time sync are shown as separate processes, whereas there's no obvious need to differentiate them in such a high-level overview - devices with "partial" and "full" sync (see above) are colour-coded differently; again differentiation unnecessary - including onsite & off-site backups in the same diagram is cool but would probably be nicer living in a dedicated backup diagram for better focus Here's a simplified version of the same diagram: ┌───────────────────┐ ┌───────────┐ ┌───────────────► │ │ nextcloud │ │ │ phone │ │ music │ │ └───────────────────┤ │ videos ◄─────realtime sync├───────────────► │ │ photos │ │ │ laptop │ │ docs │ │ ├───────────────────┤ │ calendar │ │ │ │ ├───────────┤ │ │ │ │ │ ├───────────────► desktop │ │ crm ◄─────┐ │ │ │ │ │ │ │ │ │ ├───────────┤ │ │ ├───────────────────┤ │ │ │ │ │ │ │ analytics ◄─────┤ │ │ │ │ │ │ └───────────────► │ ├───────────┤ │ │ │ │ │ │ │ │ │ web │◄── daily sync────────────────────► │ │ │ │ synology │ ├───────────┤ │ │ │ │ │ │ │ │ │ git ◄─────┤ │ │ │ │ │ │ │ ├───────────┤ │ └───────────────────┘ │ │ │ │ devtools ◄─────┘ │ │ └───────────┘
- alcover 6y ago
m463 6y ago
I think do a little at a time and keep at it. Over time it adds up. At sometime you will hit something interesting: Personal Sovereignty. I've seen other folks hit this in weird ways. My friend started working on cars with his buddy. They finally got to an old vehicle they took all the way apart and put it together. He had gotten to the point where he could pull the engine and put it on a stand, weld things, paint, redo the wiring harness. I remember one day I went and looked at it and he sort of casually said, "I can do anything". Anyway, I think the diagram says something else to me. It says he understands what his setup does enough to show it/explain it to someone else.
- gtf21 6y ago
  I had this with my bicycle at some point -- learning to fix and tweak oneself without having to go to a mechanic was eye-opening. Reminds me of the core premises in Zen and the Art of Motorcycle Maintenance.
csomar 6y ago
The good thing (and one of the reasons I like Linux and dotfiles) is that you can start right away and keep sophisticating your setup as you go. You don't lose that configuration which is akin to knowledge. I bought a Qnap NAS a month ago. I thought I would get it setup right away for my Linux machines, Macbooks, and network. I was wrong. But I'm slowly learning every couple days and now I have a systemd service that loads two volumes using NFS to my Linux machine.
shirleyquirk 6y ago
I want what you want, yet how can society reward the work involved in creating a turnkey version of such, other than through the standard capitalist selfinterested paradigm?
oppositelock 6y ago
These complicated setups which we see are complex because they try to save costs by using some part of the cloud. Shared VM resources in the cloud, which is all you really need, are dirt cheap compared to the really simple alternative. Renting space in a rack at a colo facility and putting an nginx server on it is really simple, but it's also expensive compared to the complex solution in the original post.
poisonborz 6y ago
Depends on what you want - for file sync, there is definitely similar things there. As for generic homelab, I'm writing a similar article atm, and I can tell you, it will be maybe simpler to follow, but definitely not less complex. Everything depends on your needs. What's important here that ideally you only need to do this once, and then only do light maintenance.
MrDresden 6y ago
Like has been pointed out, this can (and probably should) be done over time. I have migrated from using cloud provided storage to Nextcloud (been running that for over 4 years now without issues), and have my calendar and contacts in there as well. My ongoing task is to fully migrate all my images, videos and calibre library from Dropbox to other self hosted entities. It is a process made over a long time.

gsreenivas 6y ago

check out Helm - thehelm.com

merb 6y ago

just use k3s + (restic + velero [backup]) it's soo much easier you can basically install everything with the same tooling and update everything with the same tooling. if something breaks, bam you can just restore the whole cluster with velero (including local volumes)

imwillofficial 6y ago

I used Cloudron.io for most of the above. Check them out. Most polished self hosting experience by far.

babelfish 6y ago
And they have a good 'eject' option.

SkipperCat 6y ago

We always have this debate at work. Do we build the system ourselves or do we purchase a product? On prem Prometheus or push everything to DataDog? I'm always a fan of building things myself because I like building things, but my company compares engineering time vs product cost.

clairity 6y ago

i've actually daydreamed about starting a computing appliance company that would make a variety of services plug and play for consumers and small businesses, from email to storage, to networking, to security, and to smart home. it's actually the direction apple is headed, but they're encumbered by the innovator's dilemma, which leaves an opportunity for an upstart. google and facebook are similarly too focused on adtech, while amazon on commerce, to lock up this market yet.

mikepurvis 6y ago
It would be fun stuff to build, but I feel like you'd struggle to make money. Google and Amazon can afford to give away the hardware, and they can smuggle their ecosystem into your house as a thermostat or a smart speaker or a phone app, or whatever. Like, how do you persuade the audience of enthusiasts (think: Unifi buyers) to pay for a subscription to managed software they run on their own computers, raspis, whatever? I would probably spend $10/mo on something like that, but much above that and you'd be fighting against the armchair commentary of users who won't appreciate the effort that goes into stability and will basically have a "no wireless, less space than a Nomad, lame" attitude.
- clairity 6y ago
  there are actually tons of companies in this space already making money (e.g., wyze), but it’s highly fragmented and none have a unified vision or product strategy yet. so yes, they’re vulnerable to the behemoths right now, but those dynamics aren’t locked in yet. it’s mostly tough because of the high upfront capital costs (manufacturing, r&d, and marketing). people still talk fondly about discontinued apple routers and what nest could have been as an independent venture, for example.
  - mikepurvis 6y ago
    Maybe I'm misunderstanding the pitch from the GP, but Wyze seems like it's pretty clearly a hardware + cloud services play, similar to most other IOT ecosystems except maybe Hue. The (optional) monthly cost paid there is for loosened restrictions on an already existing works-anywhere setup— it's an upsell for power users, not a cost of entry. This seems a lot easier to me than on-prem cloud services, either in BYOH form ("but it's just software") or as a packaged appliance ("another hub to install, really?"). I would say that the closest thing to this right now for paid is coming from the storage side— NAS providers like Synology using hardware sales to support a limited ecosystem of "one click" deployable apps. And for free, it's ecosystems like HomeAssistant, which a lot of people just deploy as a fire-and-forget RPi image, but as expected with a free ecosystem, as soon as you get off the ultra-common use cases, you're reading source code to figure out how it works, and wading through a tangle of unmaintained "community" plugins that only do half of what you want.

mikepurvis 6y ago

> The diagram alone is more than enough of an argument to dissuade me from giving this a shot right now - it's simply too complicated and too much to manage for the amount of time I can dedicate to it. Yeah. I have a basic home server and I feel like even with fairly modest needs/desires (Jellyfin, Deluge, Zoneminder, some kind of file syncing, I gave up on photos because my whole family uses Google for that), it's hard to find a reasonable workflow/setup that covers it all. It was basically down to partitioning by VM (proxmox) or partitioned by container (docker), and I went with Docker + Portainer, but I'm not really happy with it; even basic functionality like redeploying a Compose configuration has sat as a feature-ask for three years [1]. Maybe I'm wanting it to be something that it just isn't, and I'd be happier with microk8s and managing the apps as Helm charts. But is that just inviting additional complexity where none is needed? [1]: https://github.com/portainer/portainer/issues/1753 https://github.com/portainer/portainer/issues/1753

vorpalhex 6y ago
I used to have a portainer centric setup.. now I just use docker-compose directly. I have my compose split into different files with a makefile to keep things "make start" simple. Highly recommend.
- mikepurvis 6y ago
  That is helpful to know— I may try to move more in this direction the next time I'm hacking on it.
- simplecto 6y ago
  This is my setup. Compose and Makefiles Ben running like this for 3 years. No fuss.
heavyset_go 6y ago
You can migrate from docker-compose to Swarm very easily, and it'll handle redeployments automatically for you.

belval 6y ago

I think the diagram gives a skewed view of how hard this actually is. I run a very similar setup only my VPS is only a proxy for my home server and it requires very little maintenance. I run everything with docker-compose and I haven't had to work on my setup at all this year and only about 8 hours in 2020 to setup the Wireguard network to replace the ssh tunnels I was using previously for VPS -> server communications. At the end of the day YMMV and use what you are comfortable with, but it's not as crazy undertaking as it sounds.

divbzero 6y ago
Yes, and many popular applications are prepackaged as one-click apps by cloud providers like Vultr [1] or Digital Ocean [2]. [1]: https://www.vultr.com/features/one-click-apps/ https://www.vultr.com/features/one-click-apps/ [2]: https://marketplace.digitalocean.com https://marketplace.digitalocean.com You can also enable automatic backups for your servers.
- deleted 6y ago
  [deleted]
- Semaphor 6y ago
  Somewhat OT, but never realized how expensive those cloud instances are. For comparison, I pay $4.95/month (billed annually) for a KVM VPS with 2 Ghz, 2 GB RAM, 40gb SSD, 400 GB HDD in the Netherlands. That seems a lot better for selfhosting where you probably want more raw storage than more SSD space.
  - rkachowski 6y ago
    That's pretty awesome, what provider are you with?
    - Semaphor 6y ago
      https://www.prometeus.net/site/terakvm.php https://www.prometeus.net/site/terakvm.php But it looks like I got it on a sale, TERAKVM-400 comes down to $6.94 at normal prices.

turtlebits 6y ago

People like to bash on the complexity of Kubernetes, but this is exactly what it's for. Use a cloud provider or your own automation to create a cluster, then apply a set of configs to bring your services up.

ssivark 6y ago

From what I hear, it’s all pretty nicely containerized/turnkey already. There are even several “meta apps” (Eg: Homelab OS, YUNoHost, etc) which are like the base layer on which many of these services are available as “applications” which have been pre-configured and can be trivially instantiated. Those curious can check out /r/SelfHosted.

lucideer 6y ago

After reading through it all, I think this is more a condemnation of the author's diagram (or at least their decision to put that particular one up-front), than of their process in general, nor the challenge. Breakdown of (my) issues with the diagram: - author's interaction with each device is explicitly included, adding unnecessary noise - "partial" and "full" real-time sync are shown as separate processes, whereas there's no obvious need to differentiate them in such a high-level overview - devices with "partial" and "full" sync (see above) are colour-coded differently; again differentiation unnecessary - including onsite & off-site backups in the same diagram is cool but would probably be nicer living in a dedicated backup diagram for better focus Here's a simplified version of the same diagram: ┌───────────────────┐ ┌───────────┐ ┌───────────────► │ │ nextcloud │ │ │ phone │ │ music │ │ └───────────────────┤ │ videos ◄─────realtime sync├───────────────► │ │ photos │ │ │ laptop │ │ docs │ │ ├───────────────────┤ │ calendar │ │ │ │ ├───────────┤ │ │ │ │ │ ├───────────────► desktop │ │ crm ◄─────┐ │ │ │ │ │ │ │ │ │ ├───────────┤ │ │ ├───────────────────┤ │ │ │ │ │ │ │ analytics ◄─────┤ │ │ │ │ │ │ └───────────────► │ ├───────────┤ │ │ │ │ │ │ │ │ │ web │◄── daily sync────────────────────► │ │ │ │ synology │ ├───────────┤ │ │ │ │ │ │ │ │ │ git ◄─────┤ │ │ │ │ │ │ │ ├───────────┤ │ └───────────────────┘ │ │ │ │ devtools ◄─────┘ │ │ └───────────┘

alcover 6y ago

m463 6y ago

I think do a little at a time and keep at it. Over time it adds up. At sometime you will hit something interesting: Personal Sovereignty. I've seen other folks hit this in weird ways. My friend started working on cars with his buddy. They finally got to an old vehicle they took all the way apart and put it together. He had gotten to the point where he could pull the engine and put it on a stand, weld things, paint, redo the wiring harness. I remember one day I went and looked at it and he sort of casually said, "I can do anything". Anyway, I think the diagram says something else to me. It says he understands what his setup does enough to show it/explain it to someone else.

gtf21 6y ago
I had this with my bicycle at some point -- learning to fix and tweak oneself without having to go to a mechanic was eye-opening. Reminds me of the core premises in Zen and the Art of Motorcycle Maintenance.

csomar 6y ago

The good thing (and one of the reasons I like Linux and dotfiles) is that you can start right away and keep sophisticating your setup as you go. You don't lose that configuration which is akin to knowledge. I bought a Qnap NAS a month ago. I thought I would get it setup right away for my Linux machines, Macbooks, and network. I was wrong. But I'm slowly learning every couple days and now I have a systemd service that loads two volumes using NFS to my Linux machine.

shirleyquirk 6y ago

I want what you want, yet how can society reward the work involved in creating a turnkey version of such, other than through the standard capitalist selfinterested paradigm?

oppositelock 6y ago

These complicated setups which we see are complex because they try to save costs by using some part of the cloud. Shared VM resources in the cloud, which is all you really need, are dirt cheap compared to the really simple alternative. Renting space in a rack at a colo facility and putting an nginx server on it is really simple, but it's also expensive compared to the complex solution in the original post.

poisonborz 6y ago

Depends on what you want - for file sync, there is definitely similar things there. As for generic homelab, I'm writing a similar article atm, and I can tell you, it will be maybe simpler to follow, but definitely not less complex. Everything depends on your needs. What's important here that ideally you only need to do this once, and then only do light maintenance.

MrDresden 6y ago

Like has been pointed out, this can (and probably should) be done over time. I have migrated from using cloud provided storage to Nextcloud (been running that for over 4 years now without issues), and have my calendar and contacts in there as well. My ongoing task is to fully migrate all my images, videos and calibre library from Dropbox to other self hosted entities. It is a process made over a long time.