4 ms·

I've been looking forward to reading this. It never fails to amaze me how these sort of incidents are caused by a cascade of small, unrelated problems, any one

I've been looking forward to reading this. It never fails to amaze me how these sort of incidents are caused by a cascade of small, unrelated problems, any one of which on its own would likely not have caused the end problem. The post-mortem has plenty of details but without being wishy washy in any way. Just the fact's ma'am, ending with a sincere apology and steps to prevent a recurrence. Well done!

RobSpectre 13y ago
Very true - in this business, it nearly always is a waterfall of small errors that produces large consequences. Part of why I love being a software person - the devil lies literally in the details.
- ricardobeat 13y ago
  "literally" http://blog.fittothefinish.com/wp-content/uploads/2013/03/devil-in-the-details.jpg http://blog.fittothefinish.com/wp-content/uploads/2013/03/de...
dangrossman 13y ago
The only thing a bit wishy washy is the "only affected 1.4%" part. If the billing system was reporting $0 balances for everyone, then only 1.4% of Twilio's accounts have auto-recharge cards on file and made at least one call/text in a 1.5 hour window. A large portion of the other 98.6% could be abandoned/inactive/toy accounts or they'd have been affected too. The percentage of active accounts affected was probably much higher, for some reasonable definition of active. That said, I'm being pedantic to mention it. I think the situation was very well handled.