3 ms·

what is good to use if you need to do a rolling deploy midday without interruption for clients?

otterley 5y ago
Unfortunately not Kubernetes, because it decoupled pod lifecycles from load balancers by design. K8S has no idea how to coordinate with load balancers to ensure new requests are flowing to new pods and to allow in-flight requests on moribund pods to complete before terminating them. This leads to at least a small number of 5xx errors being returned to clients on almost every deployment unless you implement workarounds like delaying pod shutdowns with arbitrary sleep times.
- dharmab 5y ago
  You can do this in Kubernetes by combining Envoy's draining support and graceful shutdown in the backend Pod (via handling SIGTERM or adding a preStop hook to delay termination). The big users of Envoy+Kubernetes are mostly running their own custom ingress controllers to coordinate this across the ingress system (easier than it sounds). https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... Source: I helped build a system like this for a major tech company that handles many tens of billions request daily.
  - otterley 5y ago
    That sounds like a great deal of work that ought to have been needless. And Envoy is a second system, not part of K8S proper.
- magicalhippo 5y ago
  In our current non-web application we have a launcher, which allows users to select between the current and a few previous versions. This helps us iterate faster due to less show-stoppers. If a user experiences a blocking bug in current version, they log out and back into an older version. For our web version I was thinking of doing something similar by having a launcher page, and then having each version run in its own container on a subdomain. So far I tested this using Docker Compose with Traefik routing the requests to the appropriate app container, and it worked fine. My deployment script modifies the docker-compose.yml config file to add a new container, and then Compose brings the additional container online. Current users get a notification that a new version is available but can continue working in the current version, and new logins have the new version selected by default (but can be overridden as mentioned). Would something like this not work with Kubernetes? I'm not yet familiar with Kubernetes but I've been assuming we might end up using it.
  - dharmab 5y ago
    There are ingress controllers for kubernetes that support setting up routing rules that would fit your needs. For example, the user's browser could set a header with a desired version and the ingress controller could route the request to a set of containers running that version. There are many other ways to skin this cat (all the way up to complex systems like GraphQL). Pick one that matches the size and complexity of your organization.
- cassianoleal 5y ago
  I've been managing a high traffic website running as 100s of Kubernetes microservices. We have 0 issues with 500s due to deployments. As long as you have well set up probes and old pods have enough time to fulfil lingering requests before going away, there's no reason why you should have 500s at all.
  - otterley 5y ago
    There’s no question that it can be worked around by adding arbitrary delays in places, but finding the right delay then becomes a guessing game and a trade-off between safety and paying to run pods that will do no more work. Different applications and L4/L7 transports also require different behaviors. It also requires that the author of the podspec know to implement this workaround, instead of having this knowledge baked into the orchestrator where it rightly belongs. I suspect your early attempts at setting this up safely also failed.
    - krab 5y ago
      From my experience, it's enough if the application is able to finish existing requests when it receives the signal to terminate. Kubernetes removes the terminating pod from the service endpoints, so it won't receive new connections. Therefore, if the application can serve all remaining queries during the grace period (30s by default), it should be fine. Probes are useful for ensuring long term health and also for determining when to send traffic to a starting pod. They aren't strictly necessary in the termination process. > I suspect your early attempts at setting this up safely also failed. IMHO Kubernetes is too complicated for many use cases. The zero downtime deployment may require some testing, but it's definitely supported. It's not a rare requirement. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination https://kubernetes.io/docs/concepts/workloads/pods/pod-lifec...
      - otterley 5y ago
        First, although you are correct that the endpoint is removed from the proxies, the removal is asynchronous and so the proxy configurations may not be up to date before the shutdown is initiated. Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism. Second, the challenge is that app developers are rarely aware of the need, let alone how to write SIGTERM handlers that finish handling existing requests before exiting. For them it’s yet another cognitive and testing burden that is usually out of their business expertise domain, and to my knowledge few of the common web service frameworks implement a reliable shutdown handler out of the box. On the other hand, if the orchestrator simply waited for the LB to signal that the pod was removed from the backend list before initiating the shutdown process, then it would be more robust by design.
KronisLV 5y ago
Honestly, it depends on what's in your stack and how everything fits together and what's the architecture like. Almost any orchestrator (Nomad, Swarm and Kubernetes) can do a rolling deploy and draining of particular nodes if you ever want to do that either. Currently my homepage runs on a few containers (horizontal scaling) and a load balancer in front of it, and redeploys have very little impact on the overall experience: https://kronis-dev.betteruptime.com/ https://kronis-dev.betteruptime.com/ That said, for write heavy workloads you'll need to even manage your DB migrations as a multi step process to allow both old and new schema versions to work during redeploy, whereas in other circumstances you'll need to think more about how to roll back either the DB, the app containers/instances or even events in your system (depending on the architecture and how much can go wrong). In short: - horizontal scaling can help, as can having a load balancer with health checks - changes not being breaking between any 2 releases can help (e.g. multi-stage DB migrations) - complications may arise with stateful services and data stores mostly - apps and APIs being able to cope with others being offline, if only for a bit (e.g. request replay) can also help
- MuffinFlavored 5y ago
  I am currently using Terraform almost like Docker Compose to manage about ~5 services + a database + Redis in Docker containers. When I do a deploy, I swap version 0.0.1 for version 0.0.2 for example and it brings 0.0.1 down, then brings 0.0.2 up (aka, not very rolling deploy like). How would I migrate that to Swarm? I'm not using AWS/GCP/any of that. Just locally hosted Docker containers.
  - KronisLV 5y ago
    This is probably a good starting point: https://docs.docker.com/engine/swarm/swarm-tutorial/ https://docs.docker.com/engine/swarm/swarm-tutorial/ Essentially, if you already have a node or multiple nodes that have Docker, you can initialize a swarm with: docker swarm init --advertise-addr <MANAGER-IP> And, optionally, add nodes as workers to the cluster: docker swarm join --token <JOIN-TOKEN> <MANAGER-IP>:2377 And then you can deploy stacks: docker stack deploy --compose-file docker-compose.yml my_stack Here's a tutorial about stacks: https://docs.docker.com/engine/swarm/stack-deploy/ https://docs.docker.com/engine/swarm/stack-deploy/ You'll probably want to read up on the Compose specification and the values that Docker Swarm and Docker Compose support since last i checked there were some differences in v2 / v3 (which many people still use): https://docs.docker.com/compose/compose-file/ https://docs.docker.com/compose/compose-file/ Of course, if you want a less steep learning curve for actually running Swarm, depending on your personal preferences (GUI vs CLI), you might want to have a look at Portainer, which is a lovely and simple dashboard: https://www.portainer.io/ https://www.portainer.io/ From there, it should mostly be a matter of running multiple replicas of each of your services and doing rolling updates, while having a stable load balancer in front of everything: https://docs.docker.com/compose/compose-file/compose-file-v3/#update_config https://docs.docker.com/compose/compose-file/compose-file-v3... Also, you'll probably want health checks because Swarm can then route traffic to containers only after they have truly started: https://docs.docker.com/compose/compose-file/compose-file-v3/#healthcheck https://docs.docker.com/compose/compose-file/compose-file-v3...

Unfortunately not Kubernetes, because it decoupled pod lifecycles from load balancers by design. K8S has no idea how to coordinate with load balancers to ensure new requests are flowing to new pods and to allow in-flight requests on moribund pods to complete before terminating them. This leads to at least a small number of 5xx errors being returned to clients on almost every deployment unless you implement workarounds like delaying pod shutdowns with arbitrary sleep times.

dharmab 5y ago
You can do this in Kubernetes by combining Envoy's draining support and graceful shutdown in the backend Pod (via handling SIGTERM or adding a preStop hook to delay termination). The big users of Envoy+Kubernetes are mostly running their own custom ingress controllers to coordinate this across the ingress system (easier than it sounds). https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv... Source: I helped build a system like this for a major tech company that handles many tens of billions request daily.
- otterley 5y ago
  That sounds like a great deal of work that ought to have been needless. And Envoy is a second system, not part of K8S proper.
magicalhippo 5y ago
In our current non-web application we have a launcher, which allows users to select between the current and a few previous versions. This helps us iterate faster due to less show-stoppers. If a user experiences a blocking bug in current version, they log out and back into an older version. For our web version I was thinking of doing something similar by having a launcher page, and then having each version run in its own container on a subdomain. So far I tested this using Docker Compose with Traefik routing the requests to the appropriate app container, and it worked fine. My deployment script modifies the docker-compose.yml config file to add a new container, and then Compose brings the additional container online. Current users get a notification that a new version is available but can continue working in the current version, and new logins have the new version selected by default (but can be overridden as mentioned). Would something like this not work with Kubernetes? I'm not yet familiar with Kubernetes but I've been assuming we might end up using it.
- dharmab 5y ago
  There are ingress controllers for kubernetes that support setting up routing rules that would fit your needs. For example, the user's browser could set a header with a desired version and the ingress controller could route the request to a set of containers running that version. There are many other ways to skin this cat (all the way up to complex systems like GraphQL). Pick one that matches the size and complexity of your organization.
cassianoleal 5y ago
I've been managing a high traffic website running as 100s of Kubernetes microservices. We have 0 issues with 500s due to deployments. As long as you have well set up probes and old pods have enough time to fulfil lingering requests before going away, there's no reason why you should have 500s at all.
- otterley 5y ago
  There’s no question that it can be worked around by adding arbitrary delays in places, but finding the right delay then becomes a guessing game and a trade-off between safety and paying to run pods that will do no more work. Different applications and L4/L7 transports also require different behaviors. It also requires that the author of the podspec know to implement this workaround, instead of having this knowledge baked into the orchestrator where it rightly belongs. I suspect your early attempts at setting this up safely also failed.
  - krab 5y ago
    From my experience, it's enough if the application is able to finish existing requests when it receives the signal to terminate. Kubernetes removes the terminating pod from the service endpoints, so it won't receive new connections. Therefore, if the application can serve all remaining queries during the grace period (30s by default), it should be fine. Probes are useful for ensuring long term health and also for determining when to send traffic to a starting pod. They aren't strictly necessary in the termination process. > I suspect your early attempts at setting this up safely also failed. IMHO Kubernetes is too complicated for many use cases. The zero downtime deployment may require some testing, but it's definitely supported. It's not a rare requirement. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination https://kubernetes.io/docs/concepts/workloads/pods/pod-lifec...
    - otterley 5y ago
      First, although you are correct that the endpoint is removed from the proxies, the removal is asynchronous and so the proxy configurations may not be up to date before the shutdown is initiated. Also, more and more load balancers are bypassing node ports these days, preferring instead to route traffic directly to pods, and so it’s becoming a less effective mechanism. Second, the challenge is that app developers are rarely aware of the need, let alone how to write SIGTERM handlers that finish handling existing requests before exiting. For them it’s yet another cognitive and testing burden that is usually out of their business expertise domain, and to my knowledge few of the common web service frameworks implement a reliable shutdown handler out of the box. On the other hand, if the orchestrator simply waited for the LB to signal that the pod was removed from the backend list before initiating the shutdown process, then it would be more robust by design.

KronisLV 5y ago

Honestly, it depends on what's in your stack and how everything fits together and what's the architecture like. Almost any orchestrator (Nomad, Swarm and Kubernetes) can do a rolling deploy and draining of particular nodes if you ever want to do that either. Currently my homepage runs on a few containers (horizontal scaling) and a load balancer in front of it, and redeploys have very little impact on the overall experience: https://kronis-dev.betteruptime.com/ https://kronis-dev.betteruptime.com/ That said, for write heavy workloads you'll need to even manage your DB migrations as a multi step process to allow both old and new schema versions to work during redeploy, whereas in other circumstances you'll need to think more about how to roll back either the DB, the app containers/instances or even events in your system (depending on the architecture and how much can go wrong). In short: - horizontal scaling can help, as can having a load balancer with health checks - changes not being breaking between any 2 releases can help (e.g. multi-stage DB migrations) - complications may arise with stateful services and data stores mostly - apps and APIs being able to cope with others being offline, if only for a bit (e.g. request replay) can also help

MuffinFlavored 5y ago
I am currently using Terraform almost like Docker Compose to manage about ~5 services + a database + Redis in Docker containers. When I do a deploy, I swap version 0.0.1 for version 0.0.2 for example and it brings 0.0.1 down, then brings 0.0.2 up (aka, not very rolling deploy like). How would I migrate that to Swarm? I'm not using AWS/GCP/any of that. Just locally hosted Docker containers.
- KronisLV 5y ago
  This is probably a good starting point: https://docs.docker.com/engine/swarm/swarm-tutorial/ https://docs.docker.com/engine/swarm/swarm-tutorial/ Essentially, if you already have a node or multiple nodes that have Docker, you can initialize a swarm with: docker swarm init --advertise-addr <MANAGER-IP> And, optionally, add nodes as workers to the cluster: docker swarm join --token <JOIN-TOKEN> <MANAGER-IP>:2377 And then you can deploy stacks: docker stack deploy --compose-file docker-compose.yml my_stack Here's a tutorial about stacks: https://docs.docker.com/engine/swarm/stack-deploy/ https://docs.docker.com/engine/swarm/stack-deploy/ You'll probably want to read up on the Compose specification and the values that Docker Swarm and Docker Compose support since last i checked there were some differences in v2 / v3 (which many people still use): https://docs.docker.com/compose/compose-file/ https://docs.docker.com/compose/compose-file/ Of course, if you want a less steep learning curve for actually running Swarm, depending on your personal preferences (GUI vs CLI), you might want to have a look at Portainer, which is a lovely and simple dashboard: https://www.portainer.io/ https://www.portainer.io/ From there, it should mostly be a matter of running multiple replicas of each of your services and doing rolling updates, while having a stable load balancer in front of everything: https://docs.docker.com/compose/compose-file/compose-file-v3/#update_config https://docs.docker.com/compose/compose-file/compose-file-v3... Also, you'll probably want health checks because Swarm can then route traffic to containers only after they have truly started: https://docs.docker.com/compose/compose-file/compose-file-v3/#healthcheck https://docs.docker.com/compose/compose-file/compose-file-v3...