6 ms·

how do you handle no downtime deployments? how do you backup data? I rather would use kompose+kustomize to convert it to k8s and than use k3s with ingress-ngin

by merb 5y ago

how do you handle no downtime deployments? how do you backup data? I rather would use kompose+kustomize to convert it to k8s and than use k3s with ingress-nginx with hostPath and backup local volumes with velero.

mamcx 5y ago
You don't always need "no downtime deployments", few minutes are tolerated in many environments (I work in enterprise software, where it could be days!). Backups? You only need backups or the Db, and that have zero impact in the use of docker or not* *ie: I know what very complex setups the micro-service crow use for serve their smalls-size disruptive app. I still think cron + pg_dump + rsync is more than enough...
- merb 5y ago
  well I'm asking because I have a docker-compose file and I use kustomize+kompose on it, because volume backup is a thing and the file comes from an upstream vendor and we can actually miss a few minutes/hours/days of uptime, but since the databases also use compose and sadly in a way that I can't remove it (migrate.sh script that makes use of compose commands), else I can use my kustomize+kompose setup which basically replaces the kustomize+kompose with a custom init container that I built, however everytime they change their shell scripts I need to fix my stuff... but they say that I should just backup my whole volume/vm via lvm snapshots or via vmware whatever. which I find a little bit akward and I do not want to backup my data with disk snapshots. another way would be to natively backup stuff via compose, but they use many databases and it's way harder than just use velero.
vbsteven 5y ago
I’m not the parent but I’ll pitch in on this as I have a few client projects using docker compose for production. No downtime deployments: I don’t bother for these projects. A minute of downtime in off-peak hours for a restart is fine. Data backup: just like you would backup any other local data: cron, rsync, pg_dump. And there isn’t much local data anyway as most services are stateless, using S3 or a database for long term storage. I choose docker compose for production when I want to keep complexity to a minimum: All I need is a cloud vm, an ansible playbook to install docker and configure backups. Typically in these scenarios I run the database as an OS service and only use docker for nginx/redis and application services.
- angrais 5y ago
  Why do you use the database as an OS service and not in docker? Using it in docker would allow easier local development, and shorter setup time.
  - vbsteven 5y ago
    For local development I run the full environment including the database in docker compose. For production some of these projects run alongside each other on the same host and share the database service. Sometimes the database is on another host. A typical host like this would be a 50/month Hetzner dedicated machine with only docker daemon and postgres installed natively. And then multiple projects deployed with docker compose. Sometimes nginx is native as well with vhosts pointing to the docker projects. Edit: s/ghosts/vhosts/
  - axlee 5y ago
    It is really not recommended to run (production) databases in Docker, unless you know what you are doing. https://vsupalov.com/database-in-docker/ https://vsupalov.com/database-in-docker/
lmeyerov 5y ago
We've been doing this because alternatives like k8s burned out time, $, and other team members, and our on-prem customers like it operationally vs k8s. SW teams ran high uptime stuff before k8s, and you can still do most of those tricks. We're revisiting, but selectively and only b/c we're now getting into trickier scenarios like multi-az and potentially elastic GPUs for our SaaS offering Ex: - blue/green stateless nodes - DNS/load balancer pointer flips - Do the DB as a managed saas in the cloud and only ^^^ for app servers - Backups: We've been simply syncing (rsync, pg_dump, ..) w/ basic backup testing as part of our launch flow (stressed as part of blue/green flow), and cloud services automate blob backups. We're moving to DBaaS for doing that better, which involves taking the DB out of our app for our SaaS users In practice, our service availability has been largely degraded by external infra factors, like Azure's DNS going down, than by not using k8s/swarms for internal infra. Likewise, our own logic bugs causing QoS issues that don't show up in our 9's reports.
- rualca 5y ago
  > We've been doing this because alternatives like k8s burned out time, $, and other team members, and our on-prem customers like it operationally vs k8s. It doesn't seem you bothered looking into alternatives. Take for example docker swarm. It already uses docker compose files (well, there are some differences) , they support blue-green deployments of stacks, and they even support multi-node deployments and horizontal scaling.
  - lmeyerov 5y ago
    Swarm was/is a bad bet due to looking like Docker gave up on it, esp. for deploying to our on-prem users where we can't just change our mind the next day, and wanting something we'd have some faith in for 2-3 years. Nomad was more appealing from the simplicity view, esp. as we'd need to do some rewriting either way.
zmmmmm 5y ago
> I rather would use ... you'd rather use six complex things instead of one simple one?
- merb 5y ago
  well I asked, because everybody answered either with the following: > we do not care about no downtime or stateless nodes and a lb in front and b: > backup with the standard tooling or stateless nodes so basically they just use it mostly to "package" the application and everything else is probably done with different tooling.
- rualca 5y ago
  > you'd rather use six complex things instead of one simple one? By that measuring stick, using Docker Swarm instead of docker compose is even simpler as you only need to use docker alone instead of having to install a separate script.
  - octopoc 5y ago
    docker-compose is now built into the docker command line. e.g., `docker compose` instead of `docker-compose`
aprdm 5y ago
How's it different than running things with pure docker or before docker? There were backups and no downtime deployments before docker became a thing... docker compose is already a layer above docker (but really just docker).
- merb 5y ago
  yeah but stuff like that is hard and not solved with compose, you can update your compose but that would result in downtime, so you can either do a/b (blue/green) deploys or you somehow reload the content inside the container?! everything that needs special tooling.
  - aprdm 5y ago
    You can have a compose with multiple backend services and an nginx load balancer in it proxing back to it each one, then down/up each each backend service individually when upgrades for example..
    - ta988 5y ago
      Which may force you to handle two versions of databases schemes instead of a single migration. This adds complexity and sometimes 1min of down time is more acceptable than that additional work.
      - aprdm 5y ago
        Yeah for sure, but I feel that's a software design problem and not directly related to using docker compose or not, you could have this same issue regardless of how you go about deploying it.
    - rualca 5y ago
      > You can have a compose with multiple backend services and an nginx load balancer in it proxing back to it each one, then down/up each each backend service individually when upgrades for example.. Things are way simpler with Docker Swarm. You just run a single ingress controller as a separate stack and route all traffic internally. If you use Traefik as your ingress controller, all you need to do is set labels in your docker compose files and everything just works. There is absolutely no reason or excuse to use docker compose in anything resembling production.
mhuffman 5y ago
Another one here! I use docker-compose, normal periodic DB backups to S3 and a twilio script that can use a lambda function to reload the entire system or parts of it by simple phone commands when necessary. Site does millions of views per day.
- mixmastamyk 5y ago
  Nice. Reminds me fondly of many years ago when I sent commands via pager to long-running jobs running at work from the comfort of a nearby stool at the pub. Was eating dinner of course. ;-)
nickjj 5y ago
> how do you handle no downtime deployments? I take the few second downtime hit on each deploy. I know, it's iNsAniTy but it's really not. A 5 second blip a few times a week is no biggie for most SAAS apps or services. To me that's well worth the trade offs of not having to deal with a load balancer, a container orchestrator tool, dealing with database migrations that need to be compatible for 2 versions of your app as it's being rolling restart and all of the other fun complexities of running a zero down time service. You can still architect your app in a flexible way (env vars, keeping your web servers stateless, uploading user content to S3 or another object store, etc.) so that moving to something like Kubernetes is less painful if / when the time comes. It's just for me, that time has never come. > how do you backup data? A background job / cron job runs every few hours and does a pg_dump to a directory in a block storage device. Could easily change that to be S3 too. Just comes down to preference. Personally I found it easier to drop it into block storage since it's just copying the file to a directory.
- goodoldneon 5y ago
  Your pragmatism is refreshing. This simple solution meets all your requirements so there isn’t a need to add complexity. Kudos!
- efrecon 5y ago
  Thank you for a sane comment! In many cases, downtime can even be contractually ok.
- adflux 5y ago
  Nice and simple, love it
- mr-karan 5y ago
  This is a great response and refreshing to see things can just be simple, if one tries to cover 99% of usecases and not go crazy overboard trying to solve just 1% of edge cases with "container orchestration" :)
- strzibny 5y ago
  I like this pragmatism and I teach it as well (https://deploymentfromscratch.com/ https://deploymentfromscratch.com/). Just want to say that you can employ systemd socket activation for systemd services, and you can have zero-downtime deployment without load balancers.
mahalol 5y ago
I know this wasn't intended as satire, but that's how I read it. Very modern stack but totally overengineered.
xorcist 5y ago
How do you do any deployments? It's common to have a load balancer, that routes traffic to nodes that are up. That part is mostly straightforward. It could be the application's responsibility or the load balancer's. The tricky bit is maintenance on the load balancing parts.
bcoughlan 5y ago
I've done zero-downtime deployments with the same Docker compose file running on multiple nodes. I just had a Python script (~100 lines) that would take instances off the load balancer in turn, update the docker-compose file, add them back, wait for a warm-up period and move on to the next ones.

by merb 5y ago

mamcx 5y ago

You don't always need "no downtime deployments", few minutes are tolerated in many environments (I work in enterprise software, where it could be days!). Backups? You only need backups or the Db, and that have zero impact in the use of docker or not* *ie: I know what very complex setups the micro-service crow use for serve their smalls-size disruptive app. I still think cron + pg_dump + rsync is more than enough...

merb 5y ago
well I'm asking because I have a docker-compose file and I use kustomize+kompose on it, because volume backup is a thing and the file comes from an upstream vendor and we can actually miss a few minutes/hours/days of uptime, but since the databases also use compose and sadly in a way that I can't remove it (migrate.sh script that makes use of compose commands), else I can use my kustomize+kompose setup which basically replaces the kustomize+kompose with a custom init container that I built, however everytime they change their shell scripts I need to fix my stuff... but they say that I should just backup my whole volume/vm via lvm snapshots or via vmware whatever. which I find a little bit akward and I do not want to backup my data with disk snapshots. another way would be to natively backup stuff via compose, but they use many databases and it's way harder than just use velero.

vbsteven 5y ago

I’m not the parent but I’ll pitch in on this as I have a few client projects using docker compose for production. No downtime deployments: I don’t bother for these projects. A minute of downtime in off-peak hours for a restart is fine. Data backup: just like you would backup any other local data: cron, rsync, pg_dump. And there isn’t much local data anyway as most services are stateless, using S3 or a database for long term storage. I choose docker compose for production when I want to keep complexity to a minimum: All I need is a cloud vm, an ansible playbook to install docker and configure backups. Typically in these scenarios I run the database as an OS service and only use docker for nginx/redis and application services.

angrais 5y ago
Why do you use the database as an OS service and not in docker? Using it in docker would allow easier local development, and shorter setup time.
- vbsteven 5y ago
  For local development I run the full environment including the database in docker compose. For production some of these projects run alongside each other on the same host and share the database service. Sometimes the database is on another host. A typical host like this would be a 50/month Hetzner dedicated machine with only docker daemon and postgres installed natively. And then multiple projects deployed with docker compose. Sometimes nginx is native as well with vhosts pointing to the docker projects. Edit: s/ghosts/vhosts/
- axlee 5y ago
  It is really not recommended to run (production) databases in Docker, unless you know what you are doing. https://vsupalov.com/database-in-docker/ https://vsupalov.com/database-in-docker/

lmeyerov 5y ago

We've been doing this because alternatives like k8s burned out time, $, and other team members, and our on-prem customers like it operationally vs k8s. SW teams ran high uptime stuff before k8s, and you can still do most of those tricks. We're revisiting, but selectively and only b/c we're now getting into trickier scenarios like multi-az and potentially elastic GPUs for our SaaS offering Ex: - blue/green stateless nodes - DNS/load balancer pointer flips - Do the DB as a managed saas in the cloud and only ^^^ for app servers - Backups: We've been simply syncing (rsync, pg_dump, ..) w/ basic backup testing as part of our launch flow (stressed as part of blue/green flow), and cloud services automate blob backups. We're moving to DBaaS for doing that better, which involves taking the DB out of our app for our SaaS users In practice, our service availability has been largely degraded by external infra factors, like Azure's DNS going down, than by not using k8s/swarms for internal infra. Likewise, our own logic bugs causing QoS issues that don't show up in our 9's reports.

rualca 5y ago
> We've been doing this because alternatives like k8s burned out time, $, and other team members, and our on-prem customers like it operationally vs k8s. It doesn't seem you bothered looking into alternatives. Take for example docker swarm. It already uses docker compose files (well, there are some differences) , they support blue-green deployments of stacks, and they even support multi-node deployments and horizontal scaling.
- lmeyerov 5y ago
  Swarm was/is a bad bet due to looking like Docker gave up on it, esp. for deploying to our on-prem users where we can't just change our mind the next day, and wanting something we'd have some faith in for 2-3 years. Nomad was more appealing from the simplicity view, esp. as we'd need to do some rewriting either way.

zmmmmm 5y ago

> I rather would use ... you'd rather use six complex things instead of one simple one?

merb 5y ago
well I asked, because everybody answered either with the following: > we do not care about no downtime or stateless nodes and a lb in front and b: > backup with the standard tooling or stateless nodes so basically they just use it mostly to "package" the application and everything else is probably done with different tooling.
rualca 5y ago
> you'd rather use six complex things instead of one simple one? By that measuring stick, using Docker Swarm instead of docker compose is even simpler as you only need to use docker alone instead of having to install a separate script.
- octopoc 5y ago
  docker-compose is now built into the docker command line. e.g., `docker compose` instead of `docker-compose`

aprdm 5y ago

How's it different than running things with pure docker or before docker? There were backups and no downtime deployments before docker became a thing... docker compose is already a layer above docker (but really just docker).

merb 5y ago
yeah but stuff like that is hard and not solved with compose, you can update your compose but that would result in downtime, so you can either do a/b (blue/green) deploys or you somehow reload the content inside the container?! everything that needs special tooling.
- aprdm 5y ago
  You can have a compose with multiple backend services and an nginx load balancer in it proxing back to it each one, then down/up each each backend service individually when upgrades for example..
  - ta988 5y ago
    Which may force you to handle two versions of databases schemes instead of a single migration. This adds complexity and sometimes 1min of down time is more acceptable than that additional work.
    - aprdm 5y ago
      Yeah for sure, but I feel that's a software design problem and not directly related to using docker compose or not, you could have this same issue regardless of how you go about deploying it.
  - rualca 5y ago
    > You can have a compose with multiple backend services and an nginx load balancer in it proxing back to it each one, then down/up each each backend service individually when upgrades for example.. Things are way simpler with Docker Swarm. You just run a single ingress controller as a separate stack and route all traffic internally. If you use Traefik as your ingress controller, all you need to do is set labels in your docker compose files and everything just works. There is absolutely no reason or excuse to use docker compose in anything resembling production.

mhuffman 5y ago

Another one here! I use docker-compose, normal periodic DB backups to S3 and a twilio script that can use a lambda function to reload the entire system or parts of it by simple phone commands when necessary. Site does millions of views per day.

mixmastamyk 5y ago
Nice. Reminds me fondly of many years ago when I sent commands via pager to long-running jobs running at work from the comfort of a nearby stool at the pub. Was eating dinner of course. ;-)

nickjj 5y ago

> how do you handle no downtime deployments? I take the few second downtime hit on each deploy. I know, it's iNsAniTy but it's really not. A 5 second blip a few times a week is no biggie for most SAAS apps or services. To me that's well worth the trade offs of not having to deal with a load balancer, a container orchestrator tool, dealing with database migrations that need to be compatible for 2 versions of your app as it's being rolling restart and all of the other fun complexities of running a zero down time service. You can still architect your app in a flexible way (env vars, keeping your web servers stateless, uploading user content to S3 or another object store, etc.) so that moving to something like Kubernetes is less painful if / when the time comes. It's just for me, that time has never come. > how do you backup data? A background job / cron job runs every few hours and does a pg_dump to a directory in a block storage device. Could easily change that to be S3 too. Just comes down to preference. Personally I found it easier to drop it into block storage since it's just copying the file to a directory.

goodoldneon 5y ago
Your pragmatism is refreshing. This simple solution meets all your requirements so there isn’t a need to add complexity. Kudos!
efrecon 5y ago
Thank you for a sane comment! In many cases, downtime can even be contractually ok.
adflux 5y ago
Nice and simple, love it
mr-karan 5y ago
This is a great response and refreshing to see things can just be simple, if one tries to cover 99% of usecases and not go crazy overboard trying to solve just 1% of edge cases with "container orchestration" :)
strzibny 5y ago
I like this pragmatism and I teach it as well (https://deploymentfromscratch.com/ https://deploymentfromscratch.com/). Just want to say that you can employ systemd socket activation for systemd services, and you can have zero-downtime deployment without load balancers.

mahalol 5y ago

I know this wasn't intended as satire, but that's how I read it. Very modern stack but totally overengineered.

xorcist 5y ago

How do you do any deployments? It's common to have a load balancer, that routes traffic to nodes that are up. That part is mostly straightforward. It could be the application's responsibility or the load balancer's. The tricky bit is maintenance on the load balancing parts.

bcoughlan 5y ago

I've done zero-downtime deployments with the same Docker compose file running on multiple nodes. I just had a Python script (~100 lines) that would take instances off the load balancer in turn, update the docker-compose file, add them back, wait for a warm-up period and move on to the next ones.