7 ms·
Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back." General
by robbya 7y ago
Specifically, I tell Jenkins to deploy the commit hash that was last known good. Jenkins just deploys, and doesn't really know that it's a "roll back."
Generally, going back to a known clean state should be easier, safer and relatively quick (DNS flip is fast, redeploy of old code is fast if your automation works well).
In some cases changes to your data may make rolling back cause even more problems. I've seen that happen and we were stuck doing a rapid hot fix for a bug, which was ugly. We did a lot more review to ensure we avoided breaking roll back. So I'd advise code review and developer education of that risk.
- mooreds 7y agoHow do you find the commit hash that is the last known good? Looking through jenkins release logs, asking someone, something else?
- zootam 7y agohaving the jenkins build history reflect release deployments generally works well enough. if build #20 is bad, build #19 deployed with hash XYZ was last known good, click 'rebuild' or 'replay' and it'll deploy #19 again.
- httpsterio 7y agoIn our case its quite simple. All commits should be tested and if we need to roll back, it means either that tests have failed and it was pushed nonetheless, tests were missing or tests didn't catch the issue. In the first two cases, we revert to the commit where untested or failed code was introduced to the master. This basically never happens. In the third case, you need to do some debugging and try to figure out why it's broken and either fix it or revert it. Basically just look at the git history. If the code has dependencies then you might need to do code triage and produce a hotfix. If it's a relatively isolated piece, then just revert and fix it at a better time. We use semver and gitlab's tags so we know just by the versioning if the code that is broken is important or not and if we can roll back.
- StreamBright 7y agoWhat about logical errors? math.pow(2, 4) vs math.pow(4, 2)
- ISL 7y agoA test can test for those: "does the code give mathematically correct results?"
- StreamBright 7y agoI meant pow as an example to point out that there are logical errors in code that you cannot catch with unit tests.
- weaksauce 7y agothose kinds of errors should be caught a little higher in the hierarchy... some kind of feature or integration test.
- Scarblac 7y agoEven the best tests only catch like 50% of the bugs though.
- WrtCdEvrydy 7y agoThat's not really correct. Properly input space partitioned tests have something like 90 to 95 accuracy if properly written. The issues always come from people not wanting to add sufficient tests.
- cortesoft 7y agoDoes that mean that 1 in 20 deploys breaks, then?
- 7y ago
- msbarnett 7y agoTag your released commits in git. Then just ask your deployment infrastructure to deploy whatever tag was in production immediately prior to the failed deployment.
- pedalpete 7y agoWe create a release branch for each release so we know we can always go to a previous branch which was safe.
- ccmcarey 7y agoHow do you deal with DB migrations?
- timdorr 7y agoMigrations should be separated out from other code changes. If you have a rolling deploy process, then you need to make sure your database changes are forwards and backwards compatible. Assuming you've got a CI in place, making the migrations a separate, testable commit will let you do this easily. We did this at my last company with a small GitHub bot and a CODEOWNERS file.
- kyriakos 7y agoExactly this. If your db changes are compatible in both directions then you are safe. It's not hard to achieve. Just make sure new columns have default values and your orm can deal with extra undefined columns and don't drop anything during migrations.
- seanwilson 7y agoWhen do you drop the old e.g. columns and tables that are no longer in use though? You schedule it for say a week later when you're sure you won't need to roll back that far?