3 ms·

You deploy gradually, initially to 1% or so of requests. If your metrics, KPIs, etc. don’t suffer, it continues to roll-out until 100% of requests are using tha

You deploy gradually, initially to 1% or so of requests. If your metrics, KPIs, etc. don’t suffer, it continues to roll-out until 100% of requests are using that code-path - within a few hours or so. Takes a bit of infra effort, but for a company like Intercom, that’s very easily achievable.

ben_jones 9y ago
what would be some good resources to learn how to build such infrastructure? I've seen it as a feature of various platforms but would imagine it could become complex otherwise.
- BillinghamJ 9y ago
  It wouldn’t be practical without some kind of underlying infrastructure orchestration system. Obviously some companies have gone the extra step to actually build these systems - e.g. Google made Kubernetes. For any not-huge companies, I think it pretty much just falls to choosing one of the many existing tools. Obviously it is possible to just throw together a bunch of scripts and vaguely make it all happen, I’m not sure I’d be comfortable to do that in production.
- toast0 9y ago
  The simple way to do this can apply if you're mostly doing things in the context of a user. Write the change to check if a hash of the user identifier is less than a target; increase the target over time.
AznHisoka 9y ago
this is easy and doable if your change is a front end change. good luck trying this when its a backend change involving distributed threads, locks, resources, etc where production load is 100x more than staging/test.
- greenleafjacob 9y ago
  If the cost of downtime is high enough you can mirror some or all traffic onto a shadow copy of the infrastructure.
- BillinghamJ 9y ago
  Obviously in those cases, you’d take more than a minute to review prior to deployment... There are plenty of backend changes where your list of complexities do not apply.

3 ms·

You deploy gradually, initially to 1% or so of requests. If your metrics, KPIs, etc. don’t suffer, it continues to roll-out until 100% of requests are using tha

by BillinghamJ 9y ago

ben_jones 9y ago
what would be some good resources to learn how to build such infrastructure? I've seen it as a feature of various platforms but would imagine it could become complex otherwise.
- BillinghamJ 9y ago
  It wouldn’t be practical without some kind of underlying infrastructure orchestration system. Obviously some companies have gone the extra step to actually build these systems - e.g. Google made Kubernetes. For any not-huge companies, I think it pretty much just falls to choosing one of the many existing tools. Obviously it is possible to just throw together a bunch of scripts and vaguely make it all happen, I’m not sure I’d be comfortable to do that in production.
- toast0 9y ago
  The simple way to do this can apply if you're mostly doing things in the context of a user. Write the change to check if a hash of the user identifier is less than a target; increase the target over time.
AznHisoka 9y ago
this is easy and doable if your change is a front end change. good luck trying this when its a backend change involving distributed threads, locks, resources, etc where production load is 100x more than staging/test.
- greenleafjacob 9y ago
  If the cost of downtime is high enough you can mirror some or all traffic onto a shadow copy of the infrastructure.
- BillinghamJ 9y ago
  Obviously in those cases, you’d take more than a minute to review prior to deployment... There are plenty of backend changes where your list of complexities do not apply.