17 ms·
Twilio is erroneously over-billing and suspending accounts
- SCDaniel 13y agoYeap had this ourselves. Auto recharge actually took $500 rather than $20!
- RobSpectre 13y agoAbsolutely unacceptable. As I indicated above Daniel, if you haven't shot a note to help@twilio.com indicating you're affected by the incident, please do so we can communicate directly when this $480 mistake is corrected.
- dazbradbury 13y agoWe've been charged about 10k and going up... Looks like their auto-recharge balance test keeps returning 0 or less than 0, it charges your card, and loop. EDIT: Looks like they're doing something - the UI at least now displays $500, rather than $10k before. The only real confidence I'll get is when I can see todays account transactions from the bank.
- jerguismi 13y agoI had like $240 billed, when there was nothing used... fuck.
- RobSpectre 13y agoI think that reaction is entirely appropriate in this case. We just unsuspended the accounts affected by this incident and are currently working on the balances. If you haven't already, please send an email to help@twilio.com indicating you are affected by the incident so we can communicate directly as we correct these erroneous charges.
- SCDaniel 13y agoDear god I hope the $500 in shows in my UI isn't just a ruse to stop me worrying before knowing it's actually in the XXXX/X price range!
- RobSpectre 13y agoThe account balance in your Dashboard UI will have some changes as we continue our recovery work. If you haven't already, please send an email to help@twilio.com so we can let you know directly the resolution for this billing error.
- RobSpectre 13y agoWe are actively working on the incident and have identified the accounts affected. There will be account balance changes in the Dashboard UI as we address the erroneous billing. If you haven't already, please send us an email to help@twilio.com so we can keep you directly updated on our recovery work.
- andyhmltn 13y agoOh dear, not nice at all if you're on a budget
- RobSpectre 13y agoDefinitely not nice for anybody. We will make this right.
- btucker 13y agoI'm glad they (and most of us) are not in the business of making insulin pumps or other software where such bugs could have potentially lethal consequences.
- themckman 13y agoYou mean you don't want to see a medical supply startup that "moves fast and breaks stuff"? I'm surprised we haven't yet...
- tbrake 13y agoDisrupt Health!
- knodi 13y agoWith death.
- potatolicious 13y agoLike Soylent?
- thebooktocome 13y agoSoylent hasn't broken anything... yet.
- jiggy2011 13y agoDon't medical people get around that by ensuring the first customers are mice?
- ashmud 13y agoPerhaps with nano-tech we can eliminate this? https://en.wikipedia.org/wiki/List_of_withdrawn_drugs https://en.wikipedia.org/wiki/List_of_withdrawn_drugs
- wheaties 13y agoUnfortunately you use very different software development practices which are designed to move slower for things where lives are on the line. Things more akin to Spiral and Waterfall.
- robobrobro 13y agoWow. Seriously? Learn to code.
- 67726e 13y agoHow is that constructive at all? Shit happens. My code has bugs, your code has bugs. Just about everyone has something break at some point. At least Twilio has some reasonable disclosure of these issues on their dashboard to inform people of what is going on. I'm sure they will rectify the issue as soon as they have everything back up and running.
- pjmlp 13y agoThat is why companies should be made accountable for software errors, just like most industries.
- peterjancelis 13y agoHow about we just stick to what we have (freedom of contract) and solve disputes according to the terms and conditions we agreed on, maybe with some PR gestures on top. Imagine a world where Saas vendors would be liable for economic loss their software bugs generated. You'd have to fill in a huge questionnaire to sign up and everybody would be paying a tailored, very high price to cover all the risks. Every software vendor would be in the insurance business.
- pjmlp 13y agoAs someone once putted, "the software industry is like buying shoes that explode when you try them on". We only get away with things like outsourcing, low quality work full of bugs, side jumpers into the industry, weekend developers, because no one is held accountable for software bugs and their consequences. The only place people seem to care is when people life's are at risk, and even then we know how Therac-25 went out.
- nkozyra 13y agoYour code has bugs, my code has bugs, but when our organization fails to implement a system to effectively catch bugs - particularly catastrophic ones - it's a bit dismissive to throw a "shit happens" at it.
- drx 13y agoThis has gotta suck. Many cards have steep overdraft fees.
- jerguismi 13y agoGoodbye, twilio...
- RobSpectre 13y agoI'm very sorry to hear that, but understand your frustration. As I indicated above, please shoot us a note to help@twilio.com so we can make this right for you.
- k-mcgrady 13y agoThis is bad. It'll badly affect people with low bank balances and high overdraft charges - it'll be interesting to see how Twilio reimburses people for charges banks levy on customers.
- freehunter 13y agoOr if it's on a credit card and you hit your credit limit and they keep trying to charge. How do you reimburse for a drop in credit rating?
- sdoering 13y agoNormally, if these errors occur and are dealt intraday, there should not be a problem. So I would wait and see, how this is handled. I see updates about every half hour, so I suspect, that they will handle the rest as transparent as well... Disclaimer: Not a customer.
- nkozyra 13y agoThis is unlikely to happen given most cards report monthly with a 30-day delay. Assuming you're able to contest the charges there would be no effect on your rating. Debit/bank users are definitely the most susceptible.
- lftl 13y agoAdditionally as far as I know there is no credit rating penalty for repeated attempted charges above your limit. That sort of information isn't even on any report I've ever seen. The increased balance may impact your credit score, but as you've noted it's unlikely this would make it onto your credit report unless Twilio is rather slow to resolve the issue.
- freehunter 13y agoHowever, running close to your credit limit does impact your credit score, at least according to everything I've seen on the subject.
- lysol 13y agoHaha, the one time in my life having an outdated credit card on file paid off.
- uptown 13y agoBetter double-check. Sometimes credit card companies allow charges to continue to flow through old, expired, even cancelled / compromised credit cards. I learned first-hand that a card can still process charges after AMEX allowed about 7 months of Netflix charges to continue to be processed from a card number I'd reported compromised.
- aptwebapps 13y agoMy bank hit me with a fee when Dreamhost charged an expired card ... It was my fault for not keeping my account up-to-date but it does seem like the kind of thing their billing code could check.
- jusben1369 13y agoWhat may be happening there is that Netflix is connected to services provided by the card companies to stop interruptions to recurring billing for reasons like this. Long story short if Netflix receives an error code from the payment gateway that says something like "Card expired/Card changed" it'll make a call out to this service with the old card data. That'll do a look up and return the new data to Netflix who then updates their records and charges away happily.
- citricsquid 13y agoAs uptown said, providing valid credit card information creates an authorisation for a merchant to charge to you, if your card expires after that authorisation has already been created you can still be charged and that charge will still be approved by your credit card provider. This is a common problem for people who take out short term loans and then think they can cancel their card to escape being emptied out when due day comes. Cancelling a card (or letting it expire) prevents new authorisations, it does not cancel previous authorisations.
- mohamedahmed33 13y agoel.khloud@yahoo.com
- SCDaniel 13y agoI think the fact of the matter is as a developer I want to get angry at them and then remember this is my own worst nightmare - mistakes happen, the true test will be in how Twilio reimburse people.
- hobs 13y agoYeah, I see a lot of (completely fair) angry comments in this thread. But honestly the response has been pretty damn good for what happened, and we will see how it plays out. Any one of us could have made this mistake and Twilio is a great service, otherwise you wouldnt have seen many people here complaining at all!
- RobSpectre 13y agoResponse is not good until everyone affected has their money returned to them and we explain what happened. Very much appreciate your support through a difficult night and morning. More to come.
- benjamincburns 13y agoFor people commenting about overdraft charges, Dreamhost went through this a while back. I was in college at the time, and didn't realize they'd drafted a second charge to my account. A series of small transactions later, and I was $300 in the red with very little income to fix it. They offered to reimburse overdraft fees for those affected, but I never was reimbursed. This is also great as a "how not to tell customers that you just double billed them." At least Twilio is getting this right. http://dreamhost.com/dreamscape/2008/01/15/um-whoops/ http://dreamhost.com/dreamscape/2008/01/15/um-whoops/ Step 1: Don't complain to the customer how inconvenient your massive mistake was for you.
- mischanix 13y agoI'd say, as a general rule, don't put engineers into a PR role and expect it to go smoothly.
- dayjah 13y agoI think this very much depends on the empathy of the developers. We've a pretty strict rule that PR are not allowed to comment on technology; each time they have they've gotten it wrong and we end up looking "stupid". Now we have devs and PR work closely to get succinct and technically accurate updates out to our community.
- phaemon 13y agoWhat's wrong with the Dreamhost post? He specifically says: >"The end to this story is that of course, I’m very very sorry, we’re very very sorry, and I’m sure you’re very very sorry this happened. I really am. I understand the sort of problems that an unexpected large charge to your credit card (or worse yet, your debit card) can cause. If the tone of this blog post seemed a little light, I apologize I don’t mean to offend and I realize how serious an issue this is." I can see why you'd still be annoyed about not getting your charges refunded though. What happened there? Frankly, if your bank didn't refund them when you told them what happened, they seem pretty scummy. I hope you dumped them as soon as you could!
- 13y ago
- eksith 13y agoThis sort of thing has happened before to another company, but that was a case of human error compounded by the fact that there were no software safeguards in place. http://dreamhost.com/dreamscape/2008/01/15/um-whoops/ http://dreamhost.com/dreamscape/2008/01/15/um-whoops/ At the very least, it would be wise for any company to implement billing features that will under bill rather than overbill. Under billing at least gives you the option of notifying users later (or you just eat it). Overbilling can sour a professional relationship to the point of ending it.
- jakejake 13y agoUnfortunately when automation goes wrong, it usually tends to do too much - double billing, aggressively canceling things, etc. Having built several automation systems that bill millions of dollars, I can say that it's quite challenging to build things that have no visible output, run by themselves and often sit there for years without being touched, while everything changes around them! It's not sexy work, but when it goes wrong the shit really can hit the fan!
- aculver 13y agoHere's a choice piece of code we have in production along the lines of what you suggest: if prorated_amount > (cost_per_month * 2) raise "critical! we were about to prorate-charge someone #{prorated_amount}" end It's never been triggered, but this is probably the only place in our system where we're actually responsible for an exact amount that ends up on an invoice, so having a reasonable upper limit seems like a no-brainer.
- ronaldx 13y agoshouldn't it be >=, to catch double-charging?
- hartleybrody 13y agoWhat if this code is called multiple times in quick succession? Based on the fact that I got a dozen separate emails over the span of 10 minutes, I'm guessing Twilio's overcharging happened in a series of transactions, not all at once. This code -- while good for catching bad single calculations -- wouldn't catch that.
- conformal 13y agopainful reminder that you need to test payment processing code, by hand or however, after updating it. i've run some ecommerce sites and this is basic stuff - new payment modules, time to test card processing works properly.
- RobSpectre 13y agoRob from Twilio here. We are still actively working on this incident and aware that erroneous auto-recharge billing and account suspensions have occurred for a number of customers. If this has happened to your account, please send an email to help@twilio.com where we have our support team working on making this right for everyone affected. We are spinning up additional resources to make sure every customer issue gets resolved as quickly as we can. Link for this article goes to our status board which serves as the authoritative source of information on this incident. We will be delivering updates every half hour or as new information becomes available. We are very sorry. This is far short of our commitment to you. We'll have more on the incident and what we will do to make this right later today.
- WillieBKevin 13y agoRob, best of luck to you and the Twilio team while fixing and recovering from this incident. We've been with Twilio for years and have received excellent service. Billing issues are always serious, but its nothing compared to a call routing issue, and when it comes to call routing Twilio has been flawless. I'd much rather call my bank and discuss an overdraft fee then call a client and explain why their calls were not routed properly. In the years we've been with Twilio, we have not experienced any significant downtime. Compare that to one of their main competitors that experienced eight consecutive hours of downtime during business hours earlier this year, and I'm confident we're in good hands.
- RobSpectre 13y agoThank you very much for that. We came up way short for you this morning - committed to making it right.
- benjamincburns 13y agoSeconding Willie's comments. I'm not your customer today, but seeing how y'all are handling this would make me feel confident to use Twilio should the need arise. When you get some time (I'm sure you have none right now), read some of the Dreamhost discussion below. You folks appear to be doing an excellent job responding to this.
- martin-adams 13y agoThis is one of the reasons I like to use my credit card over a debit card. Knowing that errors like this don't affect my mortgage payments makes me feel safer.
- claudius 13y agoThe web portal of my bank has a link ‘cancel’ next to every direct withdrawal which lets me cancel the charge immediately (at the cost of the people charging me, of course).
- edanm 13y agoI see a lot negative comments here. Frankly, I understand people's frustration, but I think Twilio are dealing with this in the right way. I was one of the customers affected. Luckily for me, my credit card automatically blocked the transaction after the 10th time I was auto-recharged, meaning I had a cap on how much I paid. I can totally understand the frustration of people discovering that they suddenly have 100's or even 1000's of dollars worth of charges to their credit card. On the other hand, by the time I noticed that something strange was going on, Twilio's status page already said they were looking into it. I contacted their support right away, and they answered within 20 minutes, during what must be a customer support crisis. And now RobSpectre is on HN giving very helpful answers. So, no, it's not their finest work (as RobSpectre has said), but at least they're dealing with it quickly, and hopefully professionally as well. Let's let this play out before we jump up and down on a great company that we all love.
- RobSpectre 13y agoThank you very much for the support. Lot of work to do to make up for this shortcoming.
- screwt 13y agoApologies for nit-picking at punctuation, but occasionally it matters :-) "Twilio is over-billing ..." = Twilio is charging more than they should. "Twillio is over billing ..." = Twilio has given up on billing. It's so over. Just my personal bugbear - but sometimes it's worth getting this sort of thing correct. Apologies if this is just a US/UK difference, also (I'm in the UK).
- peterjmag 13y agoSince we're already nitpicking, "overbilling" (without the hyphen) is probably better. https://www.google.com/search?q="overbilling" https://www.google.com/search?q="overbilling" vs. https://www.google.com/search?q="over-billing" https://www.google.com/search?q="over-billing"
- 300bps 13y agoI'm from the US and I also originally thought the headline was saying something along the lines of "Trilio [has ended the practice of] billing and suspending accounts". When I clicked on the article I realized that they were "over-billing" instead of "over billing".
- sjtgraham 13y agoNot a single person was confused by this, including me, a British dude sitting in the middle of London.
- krisoft 13y agoNow you got me confused. You are saying that no one was confused, not even you. Or you say that it was confusing not only for a single person, but for you too?
- sjtgraham 13y agoI'm happy to take the karma hit I'll get for this, but IMO this is emblematic of a large problem with HN, i.e. asinine bike-shedding. I address an unhelpful nitpicking comment and get a nitpicking reply. It's turtles all the way down.
- mkoble11 13y agoMistakes like this really suck, but I have no doubt that Twilio will make it right.
- shakes 13y agoI've been using Twilio for over 3 years now and haven't experienced any significant downtime or service issues. Their support has always been amazing. Obviously this situation sucks but if there's any company I'd trust to make it right it's Twilio. From a developer point of view, we all have bugs. I remember the first time one of my bugs impacted customers and it was awful. If there's any community that should be understanding that things like this happen in the tech world, it should be us.
- RobSpectre 13y agoThank you very much for that shakes. Grateful we have you in our community. We fell short of your expectations today. More later today on how we'll get this right.
- TallboyOne 13y agoShould have used <= instead of < Best of luck
- hndl 13y agoHit -- but we only see an overcharge (3x). Service seems to be back up. Do you have a support number our ops team can contact (they're paranoid).
- RobSpectre 13y agoVery sorry for this mistake. Can you have your ops team reach out via help@twilio.com and we'll get them connected with a resource to keep them informed?
- imroot 13y agoMy account was impacted by this as well, however, it looks like my credit card company blocked the fifth recurring bill (and I woke up to a call from the Fraud Team at my credit card), so no real damage has been done. Support answered my email within 10 minutes, and promised to help me clean everything up.
- RobSpectre 13y agoVery sorry for the inconvenience here and appreciate you opening a ticket for it. More to come.
- josh2600 13y agoI work in telecom, in a slightly different vein from Twilio's and I deeply empathize with the pain of billing. Even if Twilio's somehow able to ignore the pain of telco engine rating and least-cost-routing, there are still so many places you can make an error and cause something like this. Kudos to Rob and team for owning the issue and I hope the Twilio folks get this sorted out (I'm confident they will). It's worth mentioning that I've never seen another billing issue of this size from Twilio so one in 3-4 years isn't too awful. Compared to AT&T it's probably generous! (Facetious but with a grain of truth). Good luck guys and keep rocking. Here's hoping everything works out.
- MichaelGG 13y agoI'd be really interested to know what ends up being the root cause. I know properly transactional databases ain't so popular in some parts of telecom, but it's pretty simple to insert a record of the charge transactionally. Then the only real failure path is the lack of distributed transactions to the credit card processor, but it's unlikely you're going to run into many repeated commit failures.
- josh2600 13y agoOur experience: We run BigCouch for Database and BrainTree for processing. We're using DTH Software for our rating engine at the moment. I don't believe any of these technologies have any failsafes built-in related to credit card processing (or overcharging), and so anyone who uses these systems has to perform their own sanity checks. It's extremely rare for billing to screw up in Telecom, because, essentially, that is the crux of all Telco business. Twilio is probably using a homegrown or Non-telecom billing system, but I don't have any insight into their operations. I would hazard a guess that it's homebrewed though as I can't imagine a processing company introducing an error like this (stranger things have happened). So yeah, I think this will be a one time incident, and frankly, their response was fantastic.
- mathrawka 13y agoThe worst part is that the calls aren't getting updated with prices right away... so if you built a system that relies on getting the price info at the end of the call, you probably aren't pretty happy now.
- RobSpectre 13y agoWe are working cautiously on restoring full billing service to all accounts. We are trying to err on minimizing further billing impact to our customers as we restore service and, as such, taking each step carefully. This will also affect the Usage API for some users. If you would like an immediate update when these are restored, use the OP link (http://status.twilio.com/services/account-portal http://status.twilio.com/services/account-portal) or send an email to help@twilio.com. We're moving as expeditiously as safety allows.
- mathrawka 13y agoThanks, I have been in touch with my account manager... just making a note that developers should consider this type of failure to try and build resilient systems using Twilio.
- RobSpectre 13y agoGood to know. Very sorry again.
- RobSpectre 13y agoRob from Twilio again. Our CEO Jeffiel just posted an update on the situation on our blog which can be found here: http://www.twilio.com/blog/2013/07/billing-incident-update.html http://www.twilio.com/blog/2013/07/billing-incident-update.h... Full text also here: At 3:28am PDT/11:28am GMT, our monitoring systems reported an anomaly in our billing systems, which resulted in erroneous credit card charges and in some cases account suspensions. This incident affected 1.1% of customer accounts. The on call team immediately began an incident response, using the Twilio status dashboard at status.twilio.com to update customers at regular intervals. By 6:24am PDT/2:24pm GMT all suspended accounts had had service restored. The same problem briefly re-occurred at approximately 12:30pm PDT/8:30pm GMT, affecting 0.3% of customer accounts, which were immediately remedied. At this time, the Twilio billing system is offline and account-balances are not being updated in real-time until we fully resolve the issue. We are actively processing credit card refunds, and you will see a transaction void or refund on your credit card statement shortly (most banks process these within 24-48 hours). While the billing issue is being resolved, all voice and messaging services continue to operate normally. In addition to refunding erroneous credit card transactions, Twilio will also be crediting affected accounts an additional 10% of their last 30 days’ spend. We recognize that it’s not about the money, but our responsiveness to the situation that matters to you. If your account was affected, please consider this credit an acknowledgement of the inconvenience we’ve caused you. If affected customers incurred overdraft or over-limit fees due to this incident, we will also make them whole. We are in the process of contacting all affected customers via email. Additionally, we will be releasing a full postmortem on the incident once all events and root causes are known, as well as detailing the corrective steps we’ll be taking. Our focus is on providing you the best quality service and experience, and we recognize that today’s disruption came up short of what you expect from Twilio. Please accept our apologies and know it is our mission to always be earning your trust and business. As many have you have pointed out on Twitter, this is not a fun day for our team, and especially our engineers. We appreciate your support and your patience. Sincerely, Jeff Lawson CEO & Co-founder
- pdknsk 13y agoI suppose this is because it was written in haste, but you're an hour off on GMT times.
- deleted 13y ago[deleted]
- donaldmorgan 13y agoAnother example of the amateur nature of Twilio. Good for $30 a month developers but bad for businesses that depend on their phone system.