3 ms·
tl;dr: Engineer fat-fingered a command and shut everything down. Booting it back up took a long time. Then the backlog was huge, so getting back to normal took
by rkuykendall-com 10y ago
tl;dr: Engineer fat-fingered a command and shut everything down. Booting it back up took a long time. Then the backlog was huge, so getting back to normal took even longer. We made the command safer, and are gonna make stuff boot faster. Finally, we couldn’t report any of this on the service status dashboard, because we’re idiots, and the dashboard runs on AWS.
- fixermark 10y agoEverything except the "We're idiots" part I'd agree with. Self-hosting your diagnostic tools is an easy mistake to make, and I've seen both startups and large, multi-decade-experienced companies make it.
- rkuykendall-com 10y agoOf course. Meant it as more of an "I just spent 10 minutes searching for my car keys while holding them, because I'm an idiot." No disrespect to the engineers.