5 ms·
Sometimes you have to quickly roll forward.
by tleasure 6y ago
Sometimes you have to quickly roll forward.
- Raidion 6y agoThose sometimes should be super rare, and you should build testing infrastructure to prevent that from needing to happen. When you release something, you move traffic from the alb over to the new instance, if you have an issue, just move it back. If you are deploying breaking changes and don't provide yourself a stable upgrade and downgrade path, yea, you're gonna have trouble.
- tantalor 6y agoPlease do not.
- gregorymfoster 6y agoIncidents _requiring_ rolling forward are extremely rare. In the cases you have to, just build the image and deploy to your cluster with a high max-surge configured. If you image has correct caching, rebuilding it shouldn't take much time. Most of your time is likely spent in CI and rolling deployments, both of which you can manually skip.
- PragmaticPulp 6y ago> If you image has correct caching This is the hangup for most CI/CD systems with containers. Typical configurations (e.g. Gitlab basic setup) don't leverage any caching, so every container is built 100% from scratch, every time. Adjusting the system to properly utilize caching and ordering your container builds in a way that the most volatile steps are as late as possible in the build will massively speed up container builds.
- tashoecraft 6y agoI’ve learned to take the time and go through the normal deploy steps for any hot fix. More often then not, rushing the steps leads to longer outages, missing the actual bug, creating a new bug, etc. Don’t cowboy it, deploy properly and you’ll be more relaxed in the long term.
- tleasure 6y agoYeah, I should have been more clear. I'm 100% for using normal deploy steps and I'm not recommending cowboy-ing updates in a container. He was asking about using non-containerized infr though. If you can commit a code hotfix and quickly deploy the code package, you can roll forward without the slow container build/deploy.