5 ms·

> Zero downtime deployments are very rare What do you mean? Rolling deployments are a very basic feature of container orchestrators.

by snypox 7y ago

nijave 7y ago
You don't even need containers--it's a feature of load balancers containers rely on. Start new container, wait for it to become healthy, add it to the load balancer, remove old container (with an optional, let connections to old container close first)
- PunchTornado 7y ago
  what if it involves db updates?
  - rblatz 7y ago
    Where I work we had that same thought. Until we stopped doing breaking database changes.
  - sethammons 7y ago
    We have run a lot of db changes. It requires multiple steps. Add a column, migrate data, update code to use new column and old column, update code to use only the new column, update table to drop old column. Similar for migrating to new dbs, tables, etc. It is rare for us to need to disable writes for a version bump or something.
z3t4 7y ago
If you aim for zero downtime there will be a point where the complexity work against you. for example: load balancer goes down, corrupted state due to two different versions running simultaneous, a bottleneck causing congestion, etc. My favorite "load balancing" for web-pages is using "DNS round robin" where you add several A records, and the browser automatically try another IP if the request fails.
- nicolaslem 7y ago
  > ...the browser automatically try another IP if the request fails. The browser tries another IP if it cannot connect to the first one. If a server throws 500s left and right DSN RR won't help, whereas a load balancer would remove the offending server from the pool.
- SteveNuts 7y ago
  DNS round robin provides redundancy, not load balancing (though it ipso facto does somewhat evenly distribute requests). You really need an application aware load balancer that can detect http status codes and route around failures at the very least.
- vimslayer 7y ago
  Zero downtime deployments is not the same as zero downtime generally ever. We deploy to production several times a day. We have an in-house product that our own emplyees use and they absolutely love that they can ask a feature in the morning and get it in a few hours (in best case scenario). That wouldn't be at all feasible if all those deployments caused errors to users. Luckily that's not a problem, because these days it's really easy to get zero downtime deployments with all this containers and "serverless" functions and whatnot.
- sneak 7y ago
  The browser does not try another IP if the request fails; it tries the first one only (the DNS server chooses a random order for the response), unless browsers have changed their behavior recently. DNS round robin is not failover. If one of your four A records points to a dead box, 25% of your customers are SOL.
  - z3t4 7y ago
    Its supported in IE, Chrome and Firefox, havent tried Safari and others. They will wait until timeout, then try the next IP. IE will give up if the second IP is down, while Firefox and Chrome will try many IPs
  - duncaen 7y ago
namelosw 7y ago
I don't know what he refers to. But there is a specific kind of zero downtime deployments which can restart without losing in-memory state, like Erlang. Erlang also provides "fix in minutes, in production" stories he refers. For most of languages, it's very hard to remotely get access to an object, let alone fix things directly. With Erlang, you can inspect every object in the cluster to understand what's going on for any user, and fix the plane in the mid-air instead of rebuilding a plane with the same issue, then go through the process to fix it.

> Zero downtime deployments are very rare What do you mean? Rolling deployments are a very basic feature of container orchestrators.

by snypox 7y ago

> Zero downtime deployments are very rare What do you mean? Rolling deployments are a very basic feature of container orchestrators.

nijave 7y ago

You don't even need containers--it's a feature of load balancers containers rely on. Start new container, wait for it to become healthy, add it to the load balancer, remove old container (with an optional, let connections to old container close first)

PunchTornado 7y ago
what if it involves db updates?
- rblatz 7y ago
  Where I work we had that same thought. Until we stopped doing breaking database changes.
- sethammons 7y ago
  We have run a lot of db changes. It requires multiple steps. Add a column, migrate data, update code to use new column and old column, update code to use only the new column, update table to drop old column. Similar for migrating to new dbs, tables, etc. It is rare for us to need to disable writes for a version bump or something.

z3t4 7y ago

If you aim for zero downtime there will be a point where the complexity work against you. for example: load balancer goes down, corrupted state due to two different versions running simultaneous, a bottleneck causing congestion, etc. My favorite "load balancing" for web-pages is using "DNS round robin" where you add several A records, and the browser automatically try another IP if the request fails.

nicolaslem 7y ago
> ...the browser automatically try another IP if the request fails. The browser tries another IP if it cannot connect to the first one. If a server throws 500s left and right DSN RR won't help, whereas a load balancer would remove the offending server from the pool.
SteveNuts 7y ago
DNS round robin provides redundancy, not load balancing (though it ipso facto does somewhat evenly distribute requests). You really need an application aware load balancer that can detect http status codes and route around failures at the very least.
vimslayer 7y ago
Zero downtime deployments is not the same as zero downtime generally ever. We deploy to production several times a day. We have an in-house product that our own emplyees use and they absolutely love that they can ask a feature in the morning and get it in a few hours (in best case scenario). That wouldn't be at all feasible if all those deployments caused errors to users. Luckily that's not a problem, because these days it's really easy to get zero downtime deployments with all this containers and "serverless" functions and whatnot.
sneak 7y ago
The browser does not try another IP if the request fails; it tries the first one only (the DNS server chooses a random order for the response), unless browsers have changed their behavior recently. DNS round robin is not failover. If one of your four A records points to a dead box, 25% of your customers are SOL.
- z3t4 7y ago
  Its supported in IE, Chrome and Firefox, havent tried Safari and others. They will wait until timeout, then try the next IP. IE will give up if the second IP is down, while Firefox and Chrome will try many IPs
- duncaen 7y ago

namelosw 7y ago

I don't know what he refers to. But there is a specific kind of zero downtime deployments which can restart without losing in-memory state, like Erlang. Erlang also provides "fix in minutes, in production" stories he refers. For most of languages, it's very hard to remotely get access to an object, let alone fix things directly. With Erlang, you can inspect every object in the cluster to understand what's going on for any user, and fix the plane in the mid-air instead of rebuilding a plane with the same issue, then go through the process to fix it.