4 ms·
Good monitoring, logs, metrics, feature flagging (allowing for opening a branch of code for a % of users), blue/green deployment (allowing a release to handle a
by midrus 5y ago
Good monitoring, logs, metrics, feature flagging (allowing for opening a branch of code for a % of users), blue/green deployment (allowing a release to handle a % of the user's traffic) and good tooling for quick builds/releases/rollback, in my experience, are far better tools than intermediate staging environments.
I've had great success in the past with a custom feature flags system + Google's App Engine % based traffic shifting, where you can send just a small % of traffic to a new service, and rollback to your previous version quickly without even needing to redeploy.
Now, not having those tools as a minimum, and not having either staging environment is just reckless. No unit/integration/whatever tests are going to make me feel safe about a deploy.
- midrus 5y agoAnd yes, you need blue/green deployments in addition to feature flags, as it is not easy to feature flag certain things, such as a language runtime version update or a third party library upgrade, among many other things.
- mtoddsmith 5y agoIf you're running in containers aren't those version upgrades isolated?
- midrus 5y agoThe trick is to be able to route users traffic to different deployments. You can run two versions of your application concurrently, and have a dial to progressively shift traffic to the new version, as soon as you notice anything wrong you shift it back to the previous version which wasn't stopped at all. After 100% of the traffic is in the new version, and no customer complaints for 1h then you can shut down the old version. Google App Engine had all of this at least 5 or 6 years ago.