2 ms·
Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes. I worked
by Johnny555 8y ago
Actually, what I learned is that being afraid to deploy on Friday means you're lacking in testing, verification, auto healing and rollback processes.
I worked in a shop like that -- they had such great testing policies that they did continuous deployment, code went from commit to production as soon as the tests passed.
Until the holiday weekend when two code changes had an unexpected interaction and ended up black-holing all new customer activity that weekend. (existing customers were fine, they only lost data for new customers).
They could have recovered the data from a log on the front-end servers, but one of the admins noticed an unusual amount of disk space used on the front ends monday morning and just replaced them all (since their auto-healing allowed this without any interruption of service)... and since those logs were only used for debugging problems, they weren't persisted anywhere.
It turns out that tests aren't perfect - they only test what you think you need to test.
If the service goes down and nobody loses money, do you really need your engineers working overtime to fix it?
Money is not the only way to value a service.... but if the service goes down and no one cares, why run the service at all?
- nojvek 8y agoWell ... there are different kinds of deploys with different kinds of risks. I usually don’t like a blanket don’t deploy on Friday rule. We can usually rollback with one command easily, have good monitoring and health checks so even though something makes it into prod, it’s super easy to go back. Unless you have changes like you mentioned, weird side effects, db scheme changes, config changes that affect machine configuration. Those are unknown unknown changes. Good practice to hold them back. As to web and assets changes to a css file or a self contained js change, that should only re-deploy the files that changed and generally low risk.