3 ms·
I've been involved in putting out a lot of production fires but disapointingly don't have m/any big ones that I recall causing. There are a few that I allowed t
by karmakaze 5y ago
I've been involved in putting out a lot of production fires but disapointingly don't have m/any big ones that I recall causing. There are a few that I allowed to happen under shared team watch, e.g. rollover 31- or 32-bit ids.
Maybe some database indexing changes that performed a lot worse for lots of users that had to be reverted. Certainly deploying some protocol incompatibilities either inadvertently or out of sequence.
One surprising one was using composite primary keys for a misc table then realizing that some downstream Go service was getting { "id": [1, 2], ... } from the upstream Ruby one. We need to validate schema on write rather than waiting for them to fail to parse.
Disaster recovery stories are much more interesting like Hollywood blockbusters. One of my faves is un-f*ing an OS/2 HPFS partition on the west coast over the phone using DOS Norton Utilities 'nu'. Luckily the client was IBM and they had lots of identically configured machines, so just blast the central drive shape definitions (in specific sectors at the start and middle of the drive) from a neighbouring machine and run checkdisk with the recover anything that looks like a valid HPFS structure option.