47 ms·
This. A large fraction of outages are an unintended result of a planned change, so it's wise to freeze those on weeks when no one is around and sometimes other
by scottlamb 6y ago
This. A large fraction of outages are an unintended result of a planned change, so it's wise to freeze those on weeks when no one is around and sometimes other critical times (like during Black Friday sales for a shopping site). If you do this, holiday weeks are usually super quiet.
My worst holiday oncall shift as a SRE was when a couple SWEs on a semi-related team didn't honor any of our best practices, instead rolling out an unannounced change during the production freeze with no premortem or rollback plan. (This was 12 years ago, and clearly the larger org had failed in communicating cultural expectations to everyone.)
- ericbarrett 6y agoI would go so far as to say that the vast majority of outages are caused directly by code changes. If your org has a lot of commit churn, you’ll find a few more bugs in the calm of a holiday change freeze, as hundreds of patches take their first journey past 72 hours of uptime.