4 ms·

I can see the appeal. I've been working on containerizing apps at work. There's a lot of complication there. But I also really really want to be able to run upd

by jerrac 4y ago

I can see the appeal. I've been working on containerizing apps at work. There's a lot of complication there. But I also really really want to be able to run updates whenever I want. Not just during a maintenance window when it's ok for services to be offline. And containers are the most likely route to get me there. At home, well, I enjoy my work, so I containerize all my home stuff as well. And I have a plan to switch my main linux box to ProxMox and then host both a Nomad cluster and a Rancher cluster. If I weren't interested in the learning experience, I'd just stick with docker-compose based app deployment and Ansible for configuring the docker host vm.

j45 4y ago
Docker especially with Portainer is pretty serviceable for an at home setup. Once the line has been crossed with "I should be backing this up", "I should have a dev copy vs the one i'm using", the existing setups can port quite well/into something like ProxMox to keep everything in a homelab setup running relatively like an appliance with a minimum of manual system administration, maintenance and upgrading.
- jerrac 4y ago
  If Docker Swarm was a bit better, I'd have all my test instances at work running a Docker Swarm, Portainer, and Traefik stack. Unfortunately Swarm has some quirks that make running stateful apps a bit difficult. For home use, I've been experimenting with Portainer. It seems to work well for apps I'm not developing and am just running.
  - ancieque 4y ago
    Could you elaborate on the quirks regarding stateful apps?
    - jerrac 4y ago
      So, for context, my experience is limited to trying to get a MariaDB Galera cluster running. Specifically using the Bitnami image. So my issues might not apply to every single stateful app out there. I'm also running all of this on a vsphere in our own data center. Not in the cloud. Swarm does not support dependencies between services. See [0]. It also does not support deploying replicas one at a time. See [1] where I'm asking for that support. In the case of Galera, you need a master node to be up and running before you add any new nodes. I'm pretty sure that when you're initiating any kind of stateful clustered app, you'd want to start with one node at a time to be safe. You can't do that in Swarm using a replicated service. All replicas start at the same time. Using a service per instance might work, but you need to be sure you have storage figured out so that when you update your stack to add a new service to the stack, the initial service will get the data it was initiated with. (Since when you restart a stack to add the new service, the old service will also get restarted. If I'm remembering what I found correctly.) Then there's updating services/replicas. You cannot have Swarm kill a service/replica until after the replacement is actually up and running. Which means you'll need to create a new volume every time you need to upgrade, otherwise you'll end up with two instances of your app using the same data. To complicate things, as far as I can tell, Swarm doesn't yet support CSI plugins. So you're pretty much stuck with local or nfs storage. If you're using local storage when deploying new replicas/services, you better hope the first replica/service starts up on same node it was on before... All that combined means I haven't figured out how I can run a Galera cluster on Swarm. Even if I use a service per instance, updates are going to fail unless I do some deep customization on the Galera image I'm using to make it use unique data directories per startup. Even if I succeed in that, I'll still have to figure out how to clean out old data... I mean, I could manually add a new volume and service, then manually remove the old volume and service for each instance of Galera I'm running. But at the point, why bother with containers? Anyway, I'm pretty sure I've done my research and am correct on all of this, but I'd be happy to be proven wrong. Swarm/Portainer/Traefik is a really really nice stack... [0] https://github.com/moby/moby/issues/31333 https://github.com/moby/moby/issues/31333 [1] https://github.com/moby/moby/issues/43937 https://github.com/moby/moby/issues/43937
      - ancieque 4y ago
        If you are interested in making this work with any of the constraints, I am sure that there is a way to work around all these issues. About [0]/[1]: I guess you are right in this not working out of the box, but this could possibly be worked around with a custom entrypoint that behaves differently on which slot the task is running in. > (Since when you restart a stack to add the new service, the old service will also get restarted. If I'm remembering what I found correctly.) Are you sure the Docker Image digest did not change? Have you tried pinning an actual Docker Image digest? > Then there's updating services/replicas. You cannot have Swarm kill a service/replica until after the replacement is actually up and running. Which means you'll need to create a new volume every time you need to upgrade, otherwise you'll end up with two instances of your app using the same data. Is this true even with "oder: stop-first"? > To complicate things, as far as I can tell, Swarm doesn't yet support CSI plugins. So you're pretty much stuck with local or nfs storage. If you're using local storage when deploying new replicas/services, you better hope the first replica/service starts up on same node it was on before... True, but there are still some volume plugins that work around and local storage should work if you use labels to pin the replicas to nodes.
woopwoop24 4y ago
if you have more than one pod running at the same time, otherwise will be downtime if the container spawns somewhere else or am i missing something?
- jerrac 4y ago
  I assume you mean at work? Yes, we'd have more than one instance of the app running. Ideally 3 or 5 instances. And they'd be load balanced. So I could update the app one instance at a time with no downtime. At least that is the goal.

by jerrac 4y ago

j45 4y ago

Docker especially with Portainer is pretty serviceable for an at home setup. Once the line has been crossed with "I should be backing this up", "I should have a dev copy vs the one i'm using", the existing setups can port quite well/into something like ProxMox to keep everything in a homelab setup running relatively like an appliance with a minimum of manual system administration, maintenance and upgrading.

jerrac 4y ago
If Docker Swarm was a bit better, I'd have all my test instances at work running a Docker Swarm, Portainer, and Traefik stack. Unfortunately Swarm has some quirks that make running stateful apps a bit difficult. For home use, I've been experimenting with Portainer. It seems to work well for apps I'm not developing and am just running.
- ancieque 4y ago
  Could you elaborate on the quirks regarding stateful apps?
  - jerrac 4y ago
    So, for context, my experience is limited to trying to get a MariaDB Galera cluster running. Specifically using the Bitnami image. So my issues might not apply to every single stateful app out there. I'm also running all of this on a vsphere in our own data center. Not in the cloud. Swarm does not support dependencies between services. See [0]. It also does not support deploying replicas one at a time. See [1] where I'm asking for that support. In the case of Galera, you need a master node to be up and running before you add any new nodes. I'm pretty sure that when you're initiating any kind of stateful clustered app, you'd want to start with one node at a time to be safe. You can't do that in Swarm using a replicated service. All replicas start at the same time. Using a service per instance might work, but you need to be sure you have storage figured out so that when you update your stack to add a new service to the stack, the initial service will get the data it was initiated with. (Since when you restart a stack to add the new service, the old service will also get restarted. If I'm remembering what I found correctly.) Then there's updating services/replicas. You cannot have Swarm kill a service/replica until after the replacement is actually up and running. Which means you'll need to create a new volume every time you need to upgrade, otherwise you'll end up with two instances of your app using the same data. To complicate things, as far as I can tell, Swarm doesn't yet support CSI plugins. So you're pretty much stuck with local or nfs storage. If you're using local storage when deploying new replicas/services, you better hope the first replica/service starts up on same node it was on before... All that combined means I haven't figured out how I can run a Galera cluster on Swarm. Even if I use a service per instance, updates are going to fail unless I do some deep customization on the Galera image I'm using to make it use unique data directories per startup. Even if I succeed in that, I'll still have to figure out how to clean out old data... I mean, I could manually add a new volume and service, then manually remove the old volume and service for each instance of Galera I'm running. But at the point, why bother with containers? Anyway, I'm pretty sure I've done my research and am correct on all of this, but I'd be happy to be proven wrong. Swarm/Portainer/Traefik is a really really nice stack... [0] https://github.com/moby/moby/issues/31333 https://github.com/moby/moby/issues/31333 [1] https://github.com/moby/moby/issues/43937 https://github.com/moby/moby/issues/43937
    - ancieque 4y ago
      If you are interested in making this work with any of the constraints, I am sure that there is a way to work around all these issues. About [0]/[1]: I guess you are right in this not working out of the box, but this could possibly be worked around with a custom entrypoint that behaves differently on which slot the task is running in. > (Since when you restart a stack to add the new service, the old service will also get restarted. If I'm remembering what I found correctly.) Are you sure the Docker Image digest did not change? Have you tried pinning an actual Docker Image digest? > Then there's updating services/replicas. You cannot have Swarm kill a service/replica until after the replacement is actually up and running. Which means you'll need to create a new volume every time you need to upgrade, otherwise you'll end up with two instances of your app using the same data. Is this true even with "oder: stop-first"? > To complicate things, as far as I can tell, Swarm doesn't yet support CSI plugins. So you're pretty much stuck with local or nfs storage. If you're using local storage when deploying new replicas/services, you better hope the first replica/service starts up on same node it was on before... True, but there are still some volume plugins that work around and local storage should work if you use labels to pin the replicas to nodes.

woopwoop24 4y ago

if you have more than one pod running at the same time, otherwise will be downtime if the container spawns somewhere else or am i missing something?

jerrac 4y ago
I assume you mean at work? Yes, we'd have more than one instance of the app running. Ideally 3 or 5 instances. And they'd be load balanced. So I could update the app one instance at a time with no downtime. At least that is the goal.