7 ms·
Billing Incident Update
- jmandzik 13y agoThis is about as good as you can handle an incident like this, PR wise.
- ajtaylor 13y agoSimple, honest, open and straight to the point. Nice to see that they recognise it's not just about the money, but the trust people had in Twilio. I'm looking forward to reading the promised post-mortem, because I can't recall reading one from them before.
- RobSpectre 13y agoIt has been some time since we have had an incident this disruptive for our customers. Looking forward to delivering that investigation summary to you as soon as we feel we got it right.
- deleted 13y ago[deleted]
- aroman 13y agoVery well handled. Facts, gestures of good will, and straight-forwardness. Call me crazy, but I tend to think issues like this ultimately serve, if handled well, to improve a company's public standing. Though I suppose that depends on how one defines "well"...
- jkestner 13y agoYou're not crazy. I've found that a small customer service interaction early on (regardless of fault) helps solidify a good first impression. Not that I intentionally create problems...
- michaelrbock 13y agoThis isn't crazy at all, in fact it's been proven to be true. The Marriott hotel chain found that guests who had something bad happen during their stay but which the Marriott fixed were actually more likely to return than those who had nothing bad happen at all. Source: www.cbsnews.com/8301-505125_162-48540174/how-customer-complaints-can-improve-business/
- smackfu 13y agoReminds me a bit of people who say "Apple is the best! My phone died but they replaced it on the spot."
- mikeash 13y agoThinking about it, it makes sense to me. Everybody makes mistakes, and everybody experiences bad luck. If you only experience good times with a company, then you have no idea how they respond to bad times. They might be great, or they might be awful. You just have no idea. If you experience a problem with a company, then you know how they respond to problems. If they do poorly, you'll avoid them. But if they do well, you now know that they handle problems well. The time then comes to choose a company for something. Which do you choose: an unknown, or a company that you know handles problems well?
- zaidf 13y agoCall me crazy, but I tend to think issues like this ultimately serve, if handled well, to improve a company's public standing. I royally screwed up the embed script testing for our very first customer resulting in their entire site not functioning and a panicking customer on the other end. I ended up writing a 3 page post-martem and apology. We debated a bit internally to be as open as I'd been, essentially admitting basic sloppiness on my part and promising to learn from the mistake if given a chance. Fast forward 7 months, the same customer(originally acquired from a cold call) recently introduced us--without asking--to one of the top seed funds locally.
- agilebyte 13y agoHappened to me. I made an SQL update without specifying the conditions on a live database (project running, surprise surprise, late). Our team has sent some flowers and chocolates too. I am going to do the same thing next time I screw up, and I will. These days it is too much about the electronic and not enough of the "physical" in my mind.
- stinky613 13y agoIn 1982 some nutjob went around to supermarkets and pharmacies and put potassium cyanide in Tylenol bottles. Johnson & Johnson responded by issuing a recall for ALL Tylenol products nationwide. This incident is anecdotally credited with having directly led to those paper or foil seals on over-the-counter pill bottles. https://en.wikipedia.org/wiki/Chicago_Tylenol_murders https://en.wikipedia.org/wiki/Chicago_Tylenol_murders
- BHSPitMonkey 13y agoWow. That is utterly tragic and horrifying.
- HNLogInShit 13y agoHow is it tragic?
- erichurkman 13y agohttps://en.wikipedia.org/wiki/Chicago_Tylenol_murders https://en.wikipedia.org/wiki/Chicago_Tylenol_murders is the link in case anyone has not heard of the case.
- colkassad 13y agoCorrect me if I'm wrong, but I believe that also lead to the seals we see today on a wide variety of products.
- RobSpectre 13y agoThank you very much for that - our work is still not done. Through the night we have identified what we believe to be the root cause and have reconstructed the timeline that produced the incident. We still owe you a full technical post mortem of the incident and we are responding to all the servicing fee reimbursement requests at help@twilio.com. Deeply appreciate your support - more to come.
- benjamincburns 13y agoReplace the word "company" in "a company's public standing" with "well-established startup that's already known for good service" and I 100% agree. If Twilio's overall service wasn't excellent, people wouldn't be nearly as willing to view this from the positive angle. If Twilio weren't a startup - if it were, say, SalesForce.com - this also would hurt more than it would help. Fortunately, Twilio seems to know this. They're absolutely in no way resting on their existing positive reputation. Even though they've earned the trust and respect of so many customers, they're still treating each and every customer as if they must still continue to earn their trust and respect. Word of advice to Twilio execs: when this is all done and over with give your PR people a bonus and/or a raise. They did a fantastic job here in a nearly impossible situation, and I'm sure the recruiters are circling like sharks.
- RobSpectre 13y agoTruthfully? Our PR guy is on his honeymoon. Not kidding. Glad to hear you got the information you needed when you needed it - still a lot of work to do to resolve this to our customers' complete satisfaction. We'll get it done.
- benjamincburns 13y agoHa! Maybe I should've said "give the people who are handling PR for this..."
- Volscio 13y agoAfter reading some of the comments posted on the blog, just wanted to give you a shout-out for your having to put up with douchey comments and responding in a humble fashion to them. Can't be easy. :)
- throwaway1979 13y agoThat's how it is done. Classy.
- mikegirouard 13y agoIt was surprising to read they will be reimbursing any overdraft fees caused by the problem. Good move.
- jpiasetz 13y agoAt the end of the day aren't they likely liable?
- benjamincburns 13y agoI don't think they are, but it's the right thing to do. One would hope the banks would be understanding enough about the situation (haha, I know, right?) to reverse the fees. But even in that case Twilio is still causing you to, at a minimum, call your bank, wait on hold, spend 15 minutes getting transferred around to someone who has the ability to reverse the fees, etc. If they want minimal impact for their customers, they'll just reimburse the fees. I played this game with Dreamhost and SunTrust back with their whole billing debacle in Jan 2008. Dreamhost didn't respond to my support requests to reimburse "non-sufficient funds fees", The bank manager at SunTrust said that regardless of the reason funds weren't available in my account when the other transactions posted, so the NSF fees that I was incurring were legitimate. In the end I think I paid a little over $300 for their mistake.
- cosmie 13y agoI would hope they have a business liability insurance policy, in which cases it should cover the costs associated with this unfortunate mess, including any reimbursed fees.
- grimlck 13y agoWhy are you surprised they are reimbursing overdraft fees? That is the bare minimum that I would expect from any company which made this mistake
- maaaats 13y agoOf course it is to be expected, but unfortunately one cannot expect it happen. If you get it?
- ckdarby 13y agoSo many updates on their status page! One of the first companies I felt like I knew wtf was going on during the issue.
- dustingetz 13y agoAnd you can see status from two years ago! Google won't even recognize that gmail is down unless 5% of requests are failing (!!!! Try getting your work done all day if 1/20 actions fails.)
- RobSpectre 13y agoVery glad you found the stashboard useful during the incident. This is actually one of my favorite open source projects of ours. If you would like a similar service for your company, check it out here: http://www.stashboard.org http://www.stashboard.org
- brianstorms 13y agoTwilio should really consider a lighter gray color for their text on the white background. Right now the text is simply too legible still! Why, I can almost read it, if I crane my neck and look at the screen at just the right angle. What could they have been thinking? Hopefully they will take this feedback to heart so that future urgent messages like these will be completely obscured.
- maaaats 13y agoKinda ironic that this comment shows up as light grey on a grey background. :)
- RobSpectre 13y agoAppreciate the feedback. Will get it fixed.
- dredmorbius 13y agoI've been a fan of ContrastRebellion, though I've lately simply taken to modifying websites' CSS that I visit often enough that it annoys me. E.g., I've swapped the border and content background colors on Hacker News. What the heck, here it is: http://stylebot.me/styles/2945 http://stylebot.me/styles/2945 I use Stylebot, but Stylish or other CSS management plugins should work just fine, copy/paste the CSS source.
- contingencies 13y agoFor anyone looking for multiple (redundant) SMS provisioning, I have had not perfect but pretty good experience with http://fortytwotele.com/ http://fortytwotele.com/ over a good few years. I am not connected to them at all, just throwing the option out there. As a bonus, they're European, which means you are better positioned for compliance with EU privacy laws for non-US clients.
- bdcravens 13y agoInteresting. I just came back from an event here at Lone Star Ruby Conference they sponsored (movie - they used Twilio powered service to pick the movie we watched at a local brew theater). Would have made for some interesting questions had more known about this problem :-)
- josh2600 13y agoCongrats to Twilio on a textbook example on how to handle a billing outage. While I hope to never to through this I do admire a team that can execute like this. Cheers, from one telco nerd to your team of hackers.
- RobSpectre 13y agoAlways appreciate your support Josh. much work to do before this incident is resolved fully.
- happywolf 13y agoNice response, bad text color. The gray on bright white background really strains my eyes. Ouch.
- neurostimulant 13y agoI use iReader extension whenever the website's color scheme strain my eyes too much. It's essentially a clone of "Reader" functionality on safari. https://chrome.google.com/webstore/detail/ireader/ppelffpjgkifjfgnbaaldcehkpajlmbc?hl=en https://chrome.google.com/webstore/detail/ireader/ppelffpjgk...
- RobSpectre 13y agoGood feedback. Thank you.
- HNLogInShit 13y agoWhat "billing incident"? Nice title, DUHHH.
- brador 13y agoWhy not batch process billing every hour or 30 minutes? That way you can run simple checks to look for anomalies.
- RobSpectre 13y agoIt's a good question. As our service is delivered through a utility pricing model, most of our API requests are billing events - each call, text message or recording ends up having a debit to the account balance associated with it. As you can imagine then, our auto-recharge feature could be called any moment to make sure the balance doesn't run out. That said, we do have the monitoring you describe in place which is how we were alerted to the incident and limited the damage to 1.4% of the account base.