5 ms·
You can't test everything in production. Be it an infra change or application. It should always be like: Staging -> Canary -> Production If you invest time i
by lomkju 6y ago
You can't test everything in production.
Be it an infra change or application. It should always be like:
Staging -> Canary -> Production
If you invest time in making the above pipeline automated it will cause less outages.
P.S we use feature flags but it won't solve many other challenges around infra and application changes.
Some examples:
- Changing from Classic ELBs to ALBs (We care about latencies so ALBs served 1% traffic in the start)
- Zero downtime single mysql master upgrade. (ProxySql)
- New k8s controllers.
- New metrics adapters.
Staging should be like a playground to test all edge cases before you go to canary and then to prod.
- trynewideas 6y ago> Be it an infra change or application. It should always be like: > Staging -> Canary -> Production Ironically, the author says exactly this when she presents on it. Why she wrote this article in this fashion is beyond me. https://www.youtube.com/watch?v=adPQCuotAr4 https://www.youtube.com/watch?v=adPQCuotAr4
- joshuamorton 6y agoThe article is a lot more recent than that presentation, its possible the author's opinion has shifted.
- loopz 6y agoSo to "solve" her staging problem, she pushes the complexity of multiple feature flags and branching unto developers? Sounds a bit short-sighted given how quickly those complexities is known to multiply. Who is going to pay down those growing technical debts?
- yjftsjthsd-h 6y agoHow's canary different?
- jeffbee 6y agoGenerally canaries get a fraction of real production traffic while staging only gets traffic from test fixtures.