3 ms·
Background: I work on products that are less than 3 years old. Deployments are the #1 cause of outages. We don't write up fancy postmortems for the vast majori
by trjordan 10y ago
Background: I work on products that are less than 3 years old.
Deployments are the #1 cause of outages. We don't write up fancy postmortems for the vast majority of these outages, because it's things like "We forgot to ship the config files before pushing new code, guess we should automate that now" or "We automatically pushed the config files before code and the code wasn't backward-compatible, guess we should code review for that now." They're easy fixes, the downtime is almost always small, and it's relatively easy to fix in production.
The main cause of multi-day, turns-you-hair-grey outages are cascading failures that start with a deployment. My worst one was a migration that happened at the same time as a bad IOPS day on EBS where we also ran out of disk space because we weren't rotating logs properly. That took some 18 straight hours to clean up, which sucked.
If you ask my customers, AWS issues are the main cause of outages. But really, deployments are the thing that cause the most loss. It's just that most people never catch that downtime because it's an extra 5, 10, 20 minutes of slowness or a maintenance page during off-peak hours.
- _Codemonkeyism 10y ago"maintenance page during off-peak hours." This is not towards you in any way, I always get angry when there is maintenance in off-peak hours, because usually that means day time in Europe.