15 ms·

Twilio blocked our account and with it hundreds of our customers [resolved]

312 points by _ikfm 5y ago

I'm the founder of a tech startup in the field of telephony and we use Twilio for parts of our service. A little over 3 hours ago our entire account was suspended without reason. We received an email stating that fact without further explanation. Since then, we are trying to get in contact with them through every channel we could find: email, Twitter, LinkedIn, and I even tried emailing Jeff Lawson, their CEO. They don't have a support number we can call. We are not a small customer and we have been with them for more than 6 years. It's deeply frustrating that despite that fact, we don't seem to matter to them at all. If you work at Twilio or if you can help us get in touch with them, I'd be indebted to you forever!

slater 5y ago
They don't have a support number we can call. I take it you've tried (844) 814-4627 ?
- _ikfm 5y ago
  Yes. When I select the option for support they just play a message saying that I should open a ticket via email instead.
  - slater 5y ago
    Yeah looks like you're SOL. They also have some scheduled maintenance going on right now in the US, but doubtful that's to do with your situation. Also, it's 8PM here in SF, so I'm guessing e-mail will be the only option tonight.
    - _ikfm 5y ago
      Yeah, I guess so. I'll keep trying everything I can since our customers can't wait and are losing business because of the downtime. Thank you!
jrowley 5y ago
Can you open a new account and open a support ticket on that? What a nightmare situation. Wishing you the best of luck.
- _ikfm 5y ago
  I can open tickets with my existing account but they are not replying. Thank you! We are still here with the team trying to find someone who can help us.
simfree 5y ago
Port your phone numbers out ASAP, you can jump to Teli or BulkVS and pay quite a bit less with much broader numbering inventory available.
- _ikfm 5y ago
  Even if we wanted to do this, this process can take days and sometimes even weeks.
  - simfree 5y ago
    A simple port request (1 phone number) has to be resolved within 24 hours, and complex port requests have similarly short SLAs. The FCC has defined these SLAs and if you ever have a losing provider that is close to breaking the SLA on a port request, search for the providers local number portability escalation list and call each contact until resolved.
- wusatiuk 5y ago
  thanks for those 2 providers. will also check them out. do you know (beside voxbone) some further international DID providers?
  - simfree 5y ago
    Prior to Voxbone joining Bandwidth.com they were a provider we had chosen to avoid. Voxbeam and Anveo Direct are solid choices for international DIDs. Bandwidth.com has a poor reputation FYI, they pissed off a few Public Utility Commissions and were blacklisted from getting new phone numbers in some states.
atat7024 5y ago
Terrifying to rely on another team's support staff for your business to exist.
- _ikfm 5y ago
  In this time and age I think it's inevitable. Think AWS instead of Twilio for example.
  - oarsinsync 5y ago
    AWS provides a service that you can use as a commodity, and with appropriate backups, can easily recover from. DNS is the magic here. Unfortunately (with the exception of SIP dialling, which is extremely niche) there is no DNS for telephony. If your provider cuts you off, you can’t transparently move away, and number porting is slow.
    - mdoms 5y ago
      Do you really think the typical aws customer can easily recover from being banned?
      - jackson1442 5y ago
        From what I've heard surrounding the Parler incident, AWS worked _with_ Parler staff to get their data off the platform. Sure, it's not a hot-swap to Azure or metal or whatever, but it was nice to see that they didn't hold Parler's data hostage or just plain delete it. I'd assume the same courtesy would be extended to any customer, provided their ToS breaches aren't significantly worse (read: illegal) and it's not a billing-related suspension.
        ksec 5y ago
        >I'd assume the same courtesy would be extended to any customer Unpopular opinion. You see, I dont want assumption. I want guarantees. I also dont want a platform owner to remove you without some days if not weeks notice. I mean your landlord and court give you time for eviction. At least I hope that is true, I am not sure if time to vacate is universal across the world. They also give little to no time to solve any dispute. I also dont like it when media spin that AWS is at least helping them move their Data off the platform as if it is something good. I dont know why it is perceived that way in the US, but that was absolutely the least they need to do and expected in EU or UK. It is increasingly a worry sign. I have been yearning for an iOS Time Capsule for more than a decade. And the recent event makes it even more important. I want to own my things.
redis_mlc 5y ago
You might want to just spend a day and switch to Plivo. I use both Twilio and Plivo, and they're both fine for sms. (Twilio is running amok because there was a recent court case that providers could be held liable for actions of their clients, so last month you would have received update ToS from several cloud providers in your email.)
rahimnathwani 5y ago
Could this be the result of one of your customers breaching Twilio's TOS? If so, does using Twilio's subaccounts feature do anything to prevent your whole account being suspended for the actions of one of your customers?
- tluyben2 5y ago
  Yeah, they changed their TOS 2 days ago... This seems likely.
- nurgasemetey 5y ago
  Do they send mail about breaching?
floatingatoll 5y ago
What is your startup?
- eps 5y ago
  Incredibly, there's a link in the OP's profile.
- max_hammer 5y ago
  https://toky.co https://toky.co
villgax 5y ago
Vendor lock-in despite 6 years in the biz?
Raed667 5y ago
I got this email from Twilio a few days back, it could be related: As a current Twilio customer, we are notifying you that we are updating our Messaging Policy, available at https://www.twilio.com/legal/messaging-policy. We are sending this message to all customers, regardless of whether action is required to ensure compliance. [...] There are a few key sections we’d like to call out: Applicability. We clarified that the Twilio Messaging Policy applies to all organizations using our services, and that Independent Software Vendors (ISVs) are responsible for ensuring their users are in compliance. Messaging Policy Evasion. We clarified that content or techniques designed to avoid mechanisms that detect policy violations are not permitted. Usage Limitations - Content We Do Not Allow. We clarified that you may not send content that is illegal in the jurisdiction where the message recipient lives. For example, in the United States, this includes cannabis and CBD messages.
holoduke 5y ago
Welcome to the new world where tech companies rule. Where moral standards are gone. Where local laws do not apply. This shit is happening a lot. Artists on Youtube, App builders on Playstore/iOS. Marketing companies on Google search, pretty women on Instagram, political men on Facebook or Twitter. I feel for you. Hope it gets alright.
- blowski 5y ago
  Remove the tech-specific words from your comment, and you would probably find people saying the same thing throughout all of history.
  - dx034 5y ago
    Not sure if large car companies or clothes manufacturers (just two examples of large companies) were ever able to ignore legal rights of their customers. This has only become possible with digital services where companies refuse to acknowledge local laws and regulation.
    - halfdan 5y ago
      Car companies selling cars exceeding allowed emission levels and clothes manufacturers exploiting child labour in third world countries? Sure there were "fines" but as usual that was pocket change them..
      - varispeed 5y ago
        Cost of doing business factored in the product price...
    - blowski 5y ago
      For most of history, customers didn't have the kind of legal rights they have now. Businesses could refuse to serve you, warranties were non-existent until late 1800s. And even when consumers did get rights, there have been constant stories in newspapers of "big companies trampling on the rights of consumers".
ApolloRising 5y ago
Try this link: https://investors.twilio.com/governance/person-details/default.aspx?ItemId=8a5fc44e-87c5-4f3d-88c8-31b7aef0ea7f https://investors.twilio.com/governance/person-details/defau... Phone number on the page and an email address that may work for you. Let me know if that helps.
work_for_x 5y ago
I work for Twilio and forwarded it internally. If you are 6 years with the company, you probably have an AE assigned to your company or have been in touch with any other human being along the way? UPDATE: Your account should be active now.
- _ikfm 5y ago
  Yes, it's active now after almost 7 hours. Thank you! I plan to update everyone on what happened as soon as I get some sleep first. I've been up since 7 am and it's 2 am now. The incident started at 7 pm.
  - gscott 5y ago
    and setup another system to back this one up immediately because whatever algorithm brought down your account will do so again! If you spread the load across two platforms it would be a lot of work but you would be half-alive instead all-dead.
  - jariel 5y ago
    Try to find an alternative service supplier at very minimum for back up and consider legal recourse.
  - e40 5y ago
    We're still interested in an update!
- syabro 5y ago
  Why was it deactivated?
- cosmotic 5y ago
  What is preventing this from happening to someone else?
- morrbo 5y ago
  It's a bit shit that they had to jump through so many hoops, with HN being the only one that actually worked (after 7 hours) in order to get in touch. Serious kudos to you for helping them but I know a lot of people who wouldn't touch your company with a bargepole for these kind of reasons. Having to kick up a fuss on social media in order to get service, how very 2021..
ksajadi 5y ago
Sorry to hear. Twilio has changed since its early days. Last month, our bill went from $20 per month to $16,000. The reason: fraudsters use Twilio to setup premium numbers, then use Twilio customers (like us) to send OTPs to those numbers using bots. The result is us out of pocket, fraudsters and Twilio making money. Twilio is also the only Pay as you Go SaaS vendor that doesn’t have spend limits on accounts and advises customers to implement their own any fraud measures themselves. While this was going on, we got an email from their sales rep, congratulating us for increasing our spend by more than 8000% and wanted to have a chat. So I guess they do have the systems to monitor unusual patterns, they just turn a blind eye. You’d never know, someone in Twilio might even be selling customer lists on the black market to the fraudsters so they know who to target.
- smhg 5y ago
  FWIW: you can turn off automatic recharge and ask for a different reminder limit. Not ideal, but at least it prevents crazy bills.
  - ksajadi 5y ago
    We didn't have automatic charge. We now have an invoice to pay.
    - gumby 5y ago
      A Twilio invoice or a lawyer’s invoice?
  - throwaway2048 5y ago
    They still bill you no matter what, and if the bill is big enough they will sic collections on you.
    - mleonhard 5y ago
      This surprises me. I thought they would disable the service when the balance goes negative. I need to add some more monitoring and a failsafe rate limit...
ajimix 5y ago
We had the same problem some months ago with Mailgun. They blocked our account after many years using them without apparent reason or prior notice and took them 2 days to unblock it, leaving our thousands of clients stranded without receiving any of our transactional emails. Lesson learnt, always keep a backup for crucial services and move away from Mailgun
- framerate 5y ago
  We just moved to Mailgun because SendGrid did something similar to us! Fortunately we only used SMTP so it was a straightforward change, but having a backup for something like Twilio Video would be quite an effort.
benjaminwootton 5y ago
I have the same on Sendgrid, owned by Twilo. One day they just cut us off with no explanation, kneecapping our product. I raised two tickets, neither of which were answered. The next day, it all started working again. I’ve always been a big advocate for using 3rd party SaaS, but with the level of customer service I’m seeing lately, I’m starting to recommend avoiding it where possible.
- luckylion 5y ago
  That's 99% of why I'm generally trying to avoid SaaS. I always run into some bug or nuisance or there's some account trouble and unless you're spending insane amounts of money, I've found support to be severely lacking, often answering support tickets with canned responses without reading the issue, or not answering at all. I'd rather spend 100 hours upfront to have something that I know and can maintain than using a service, running into trouble and spend 50 hours chasing the support around to see whether they'd fix it, or worse, rely on their system, have it fail or be banned, and then having to spend 100 hours building a replacement, often while things are exploding left and right. Relying on SaaS feels incredibly fragile to me.
- paul_f 5y ago
  It's an insane amount of engineering, but maybe one solution is to have two back end telephony SaaS. If one turns off, rely on the other until you get it fixed? Plivo, Mexmo?
- MattyMc 5y ago
  I recently moved to Postmark. It’s been an extremely positive experience. They’re a small, dedicated team with amazing support.
- varispeed 5y ago
  I think this should be regulated - companies often take massive amount of customers and while they are most of the time able to deliver the product, they have no capacity to service support queries. I think there should be an SLA guaranteed by law and for a company it should be illegal to take more customers if they are unable to provide SLA on support. This way we wouldn't have a situation where you contact a company and in reality you talk to bots or nobody at all. This would also deal with the problem of companies growing too big to fail and not having reasonable competition.
jnsaff2 5y ago
Maybe time for a new HN headline convention, like: Tell FAANG: You blocked me, again FAANG Support: Important please call back ASAP
- thinkingemote 5y ago
  Could this be productised somehow? Would an investor be willing to invest in my startup which effectively un-blocks small companies from larger ones? It would do the leg work to compile the necessary information before contacting a real human. I'll call it Support As A Service.
  - mcny 5y ago
    Makes sense. 1. If someone works at a company, they can sign up on our website as a support agent. 2. We will only list companies with which we have an authenticated contact, maybe require people to verify they got our authentication token on their employee email? 3. When someone contacts us about an issue with a company we support, we verify the request is genuine. Perhaps charge a small fee to block spam. (This is the problematic part. How do we know a problem is genuine without asking for potentially confidential information?) 4. Add all vetted requests to a private channel that all the people who volunteered on step one. (Can we pay our volunteer support agents? How? Does paying make things worse? Maybe gamify this step and send our volunteers gifts?) 5. Monitor performance of support agents to avoid giving false hope to our customers.
  - exikyut 5y ago
    I posited the interestingness of such an idea a couple months ago in a thread about app store blocks: https://news.ycombinator.com/item?id=25988410 https://news.ycombinator.com/item?id=25988410 IMHO, most emphatically *yes* - this is a completely unserved/unserviced area of the B2B and B2C markets. I noted in the linked thread that such a service would likely start by leveraging individual contacts, and building a reputation for unwaveringly high signal. It definitely does seem to me that the vacuum between corporate and customers represents a giant sleeper industry that fundamentally won't shift very quickly. Which means you're unlikely to reach the usual core-problem-is-solved entrenchment/stagnation point that tends to happen when you ramp up on a complex-but-solve-once problem, then find yourself stuck with a bunch of intricate sausage machine bits that aren't doing anything...
yenoham 5y ago
Somewhat related anecdote. Whilst we didn't get cut off, we found a huge drop-off in time to receive, and the quality of, support from our live chat/support messaging provider. Having been a customer since our launch [1], and very early after theirs, it has been really sad and frustrating to see that friendly distrupter energy zap out of the service. I get why businesses go through that changes like this during growth, but it feels like all the things to let slide, your persona is not one of them! [1] https://mailosaur.com https://mailosaur.com (somewhat related to the thread we let you test SMS messages)
daniellarusso 5y ago
Our company phone system has several VOIP providers (including Twilio) as well as multiple inbound analog lines. Two weeks ago, all of the analog lines stopped working, which are local numbers, but our main 800 number forwards to those numbers. Calling the local provider was futile. The provider’s office was also experiencing the issue and could not receive calls. Although we do have the 800 number on business cards, we quickly updated our website and Intercom docs to use a recently provisioned 800 number from one of the VOIP providers. It took 2 days for the other lines to begin working again. This is the first POTS outage we have experienced in several years.
newbie578 5y ago
Yikes, talk about bad PR. I am sorry that this happened to you, but thank you for sharing this, you are not the first one, nor the last one by the looks of it. I intend to avoid Twilio in a wide range, don't need that kind of a "business partner" in my life.
jFriedensreich 5y ago
a lot of us have this as nightmare scenario. - long term we really should move to an industry wide due process system with legislation for compensation and other parameters. once there is some more structure there can also be insurances or similar ways to prevent something like this to ruin a company. - there maybe is a market for more intermediaries that bundle twilio and messagebird or similar providers and have an official strong communication channel to the providers to resolve issues and provide failover
- yowlingcat 5y ago
  100%. Disappointment from seeing a once upstart (which I was rooting on from the sidelines) become a faceless big co aside, I think insurance is the right way to handle this problem and prevent it from being unfairly externalized.
rapnie 5y ago
> Since then, we are trying to get in contact with them through every channel we could find: email, Twitter, LinkedIn, [..] They don't have a support number we can call. Tons of platforms seem to operate like this, and this kinda thing happens all the time. HN is turning into a desperate last resort as a PleaForHelp-as-a-Service (hoping that bad PR or a company insider is the trigger). But by now business decision-makers should really wake up and acknowledge the risk of choosing such platforms. Of course they should demand better service, but this is also something to research upfront ("is there a support desk and at least a phone number.") and then decide if they want to pay the extra money, or go somewhere else that addresses their needs.
macspoofing 5y ago
Sorry to hear that. I hope everything works out. Take this a learning experience. Once you get your business back, sit down with your team and come up with a mitigation plan because this may happen again. It's possible you may need to switch to another vendor, even one more expensive or smaller, but one with a dedicated support line and/or an account manager and stronger SLAs. Your business heavily relies on Twilio and you should have someone you can call at Twilio outside of the general support line. Maybe you need to attend their conferences and create a personal relationship with their developers or other employees so in cases like this you can backchannel your issue. If you're going to outsource critical business infrastructure to third party SAAS vendors, make sure you have a plan for when things go tits up, because things will go tits up. It looks like Twilio screwed up here, but Twilio isn't going to suffer the consequences of this like your business will.
dooglius 5y ago
Disclaimer: IANAL As far as telephony is concerned (things like SMS are a different story), a phone provider cannot legally do this, even if you signed some ToS agreement. According to the FCC, phone providers, including voip providers, have legal requirements around when they are allowed to terminate service: https://www.fcc.gov/consumers/guides/when-your-telephone-company-discontinues-service https://www.fcc.gov/consumers/guides/when-your-telephone-com...
- meepmorp 5y ago
  The page you link to says that this doesn’t apply to mobile providers in general.
  - dooglius 5y ago
    I was under the impression that voip is not considered "wireless" but I could be mistaken there.
    - meepmorp 5y ago
      I get the impression those protections are for residential phone service, not the kind of stuff twilio does. Leaving that aside, though, SMS (at issue here) is definitely in the wireless realm, even if the messages come from an API call rather than a mobile device's radio.
rkagerer 5y ago
This is why you have multiple vendors for every external function. Just like hardware manufacturers try to maintain at least two suppliers for every component.
alberth 5y ago
Twilio is a wet dream for spammers (robocalling, emailing, SMS). They’ve turned spamming into a positive ROI business.
sbestetti 5y ago
Hey there, sorry to hear that. Are you able to login to your account at all? If so, can you open a ticket and share the number here so I can take a look?
sirinath 5y ago
Many of these system now a days use automation with little human oversight. Algorithms classify and block accounts. The little human oversight is inadequate to even verify what happened. I believe support is spread thin with cost saving and automation. P.S. I am having a similar problem, but with Viber. I posted the details here: https://www.reddit.com/r/viber/comments/mrcu6f/viber_block_for_sending_greetings/ https://www.reddit.com/r/viber/comments/mrcu6f/viber_block_f...
Kim_Bruning 5y ago
It seems like there would be a market for reputable E2E vendors that provide you with a decent SLA. On the other hand, that would be a bit more expensive, and might therefore be out-competed in a race to the bottom.
harel 5y ago
We had an incident where our we got charged thousands (later refunded) by abusing dodgy routing and number verification process. The problem was that Twilio will allow calls through when the call balance is below zero, if the calls were queued before it was zero. You then end up with a negative balance. On the plus side, it does force you to start making defensive plans to protect your application.
cphoover 5y ago
Hmmmm they definitely have a phone number, I remember when I worked at a startup a few years ago I had many phone conversations with them regarding a bug in their mobile chat SDK. Perhaps we had a support contract tho...
dangerboysteve 5y ago
While this does not resolve your current issue. Have a look at https://signalwire.com https://signalwire.com its from the same people who created FreeSWITCH.
rsync 5y ago
Long time twilio user and evangelist here ... as I have mentioned here and in interviews[1], etc., I built my own little mini-telco out of twiml bins and have attempted, over the past 4-5 years, to move all of my telephony into the (twilio) cloud. I was encouraged when I attended Signal 2018[2] by twilio product and technical managers who intimated that such personal "utility" uses of twilio were favored by the CEO (Jeff Lawson) himself. Everything you need to know about how this has turned out - and which direction the company is going - can be imparted with two points: 1) Twilio straight up refuses to put an email verb in Twiml. This sounds like a nerdy, nit-picky point from my own personal use-case (and it is) but it's the most dead-obvious thing in the world, is technically trivial, and would make so many use-cases a world easier than they are. Instead of adding an email verb to twiml (so you could, for instance, cc: an email address with emergency SMS alerts, or perhaps cc: yourself a log of all received SMS) they chose to buy an entire email company. Now you can create these use-cases provided that you sign up a sendgrid account, and tie the two together with code hosted at a third party and blah blah blah. The reason they did this is providing telco utility tools is not interesting. Customer engagement at scale (or whatever bullshit) is what is interesting. Managing high value interactions. Whatever. 2) Q1 of 2021 they crammed phone only 2FA down the throats of all account holders. You may say "you can also use authy" but, of course, the initial setup of authy requires a real mobile number[3] and you can't log into twilio to enable authy without providing a mobile number first. So, if you're planning on moving your telephony into the cloud, you can't - you'll need to tie real phones and SIM cards to these accounts. Have multiple twilio accounts ? Congratulations - you can either be super sloppy and share your mobile SIM across all of them or you can buy three phones. This is for your security. Total bullshit. In reality, twilio has a massive spam/scam problem that is almost unsolvable. I don't envy them in this position. However, instead of solving this problem, they have decided to just throw sand in the gears and slow down the scammers by demanding that everyone prove a SIM identity. This doesn't bother most users because they aren't, ironically, trying to shed SIM cards by moving their telco infrastructure to twilio ... but it fucks my shit up immensely. Which is what happens when providing telco utility services has no path-to-unicorn and you need to turn yourself into an "engagement platform" that "enables customer interaction" for "high value use-cases". [1] https://console.dev/qa/rsync-john-kozubik/ https://console.dev/qa/rsync-john-kozubik/ [2] "peak twilio" [3] Which is so ironic since, in almost all cases, twilio numbers are NOT mobile numbers and cannot be used for "proper" 2FA since they cannot receive SMS from short-codes. So twilio demands a telephony use-case that they, themselves, cannot satisfy.
taylodl 5y ago
Lesson learned: always have a number to call, always have the ability to get a hold of a flesh-and-blood human being. My company mandates this as a minimal requirement for all our SaaS providers. With regards to having a mitigation strategy we take another tactic: what's the SaaS provider's mitigation strategy? How do they cope with data center loss, communications loss - what processes and procedures do they have in place? Any business-worthy SaaS provider has this information readily available. Any hemming or hawing is a sign of trouble.
- deleted 5y ago
  [deleted]
pauldd7 5y ago
We asked for a phone number transfer to another Twilio account, and the support rep took the wrong action by mistake. He wiped our entire account, causing us to lose tens of thousands of recordings, logs etc. We were told sorry, no way to recover them - only the underlying phone numbers. It's cheap, but I'd never trust them again.

312 points by _ikfm 5y ago

slater 5y ago

They don't have a support number we can call. I take it you've tried (844) 814-4627 ?

_ikfm 5y ago
Yes. When I select the option for support they just play a message saying that I should open a ticket via email instead.
- slater 5y ago
  Yeah looks like you're SOL. They also have some scheduled maintenance going on right now in the US, but doubtful that's to do with your situation. Also, it's 8PM here in SF, so I'm guessing e-mail will be the only option tonight.
  - _ikfm 5y ago
    Yeah, I guess so. I'll keep trying everything I can since our customers can't wait and are losing business because of the downtime. Thank you!

jrowley 5y ago

Can you open a new account and open a support ticket on that? What a nightmare situation. Wishing you the best of luck.

_ikfm 5y ago
I can open tickets with my existing account but they are not replying. Thank you! We are still here with the team trying to find someone who can help us.

simfree 5y ago

Port your phone numbers out ASAP, you can jump to Teli or BulkVS and pay quite a bit less with much broader numbering inventory available.

_ikfm 5y ago
Even if we wanted to do this, this process can take days and sometimes even weeks.
- simfree 5y ago
  A simple port request (1 phone number) has to be resolved within 24 hours, and complex port requests have similarly short SLAs. The FCC has defined these SLAs and if you ever have a losing provider that is close to breaking the SLA on a port request, search for the providers local number portability escalation list and call each contact until resolved.
wusatiuk 5y ago
thanks for those 2 providers. will also check them out. do you know (beside voxbone) some further international DID providers?
- simfree 5y ago
  Prior to Voxbone joining Bandwidth.com they were a provider we had chosen to avoid. Voxbeam and Anveo Direct are solid choices for international DIDs. Bandwidth.com has a poor reputation FYI, they pissed off a few Public Utility Commissions and were blacklisted from getting new phone numbers in some states.

atat7024 5y ago

Terrifying to rely on another team's support staff for your business to exist.

_ikfm 5y ago
In this time and age I think it's inevitable. Think AWS instead of Twilio for example.
- oarsinsync 5y ago
  AWS provides a service that you can use as a commodity, and with appropriate backups, can easily recover from. DNS is the magic here. Unfortunately (with the exception of SIP dialling, which is extremely niche) there is no DNS for telephony. If your provider cuts you off, you can’t transparently move away, and number porting is slow.
  - mdoms 5y ago
    Do you really think the typical aws customer can easily recover from being banned?
    - jackson1442 5y ago
      From what I've heard surrounding the Parler incident, AWS worked _with_ Parler staff to get their data off the platform. Sure, it's not a hot-swap to Azure or metal or whatever, but it was nice to see that they didn't hold Parler's data hostage or just plain delete it. I'd assume the same courtesy would be extended to any customer, provided their ToS breaches aren't significantly worse (read: illegal) and it's not a billing-related suspension.
      - ksec 5y ago
        >I'd assume the same courtesy would be extended to any customer Unpopular opinion. You see, I dont want assumption. I want guarantees. I also dont want a platform owner to remove you without some days if not weeks notice. I mean your landlord and court give you time for eviction. At least I hope that is true, I am not sure if time to vacate is universal across the world. They also give little to no time to solve any dispute. I also dont like it when media spin that AWS is at least helping them move their Data off the platform as if it is something good. I dont know why it is perceived that way in the US, but that was absolutely the least they need to do and expected in EU or UK. It is increasingly a worry sign. I have been yearning for an iOS Time Capsule for more than a decade. And the recent event makes it even more important. I want to own my things.

redis_mlc 5y ago

You might want to just spend a day and switch to Plivo. I use both Twilio and Plivo, and they're both fine for sms. (Twilio is running amok because there was a recent court case that providers could be held liable for actions of their clients, so last month you would have received update ToS from several cloud providers in your email.)

rahimnathwani 5y ago

Could this be the result of one of your customers breaching Twilio's TOS? If so, does using Twilio's subaccounts feature do anything to prevent your whole account being suspended for the actions of one of your customers?

tluyben2 5y ago
Yeah, they changed their TOS 2 days ago... This seems likely.
nurgasemetey 5y ago
Do they send mail about breaching?

floatingatoll 5y ago

What is your startup?

eps 5y ago
Incredibly, there's a link in the OP's profile.
max_hammer 5y ago
https://toky.co https://toky.co

villgax 5y ago

Vendor lock-in despite 6 years in the biz?

Raed667 5y ago

I got this email from Twilio a few days back, it could be related: As a current Twilio customer, we are notifying you that we are updating our Messaging Policy, available at https://www.twilio.com/legal/messaging-policy. We are sending this message to all customers, regardless of whether action is required to ensure compliance. [...] There are a few key sections we’d like to call out: Applicability. We clarified that the Twilio Messaging Policy applies to all organizations using our services, and that Independent Software Vendors (ISVs) are responsible for ensuring their users are in compliance. Messaging Policy Evasion. We clarified that content or techniques designed to avoid mechanisms that detect policy violations are not permitted. Usage Limitations - Content We Do Not Allow. We clarified that you may not send content that is illegal in the jurisdiction where the message recipient lives. For example, in the United States, this includes cannabis and CBD messages.

holoduke 5y ago

Welcome to the new world where tech companies rule. Where moral standards are gone. Where local laws do not apply. This shit is happening a lot. Artists on Youtube, App builders on Playstore/iOS. Marketing companies on Google search, pretty women on Instagram, political men on Facebook or Twitter. I feel for you. Hope it gets alright.

blowski 5y ago
Remove the tech-specific words from your comment, and you would probably find people saying the same thing throughout all of history.
- dx034 5y ago
  Not sure if large car companies or clothes manufacturers (just two examples of large companies) were ever able to ignore legal rights of their customers. This has only become possible with digital services where companies refuse to acknowledge local laws and regulation.
  - halfdan 5y ago
    Car companies selling cars exceeding allowed emission levels and clothes manufacturers exploiting child labour in third world countries? Sure there were "fines" but as usual that was pocket change them..
    - varispeed 5y ago
      Cost of doing business factored in the product price...
  - blowski 5y ago
    For most of history, customers didn't have the kind of legal rights they have now. Businesses could refuse to serve you, warranties were non-existent until late 1800s. And even when consumers did get rights, there have been constant stories in newspapers of "big companies trampling on the rights of consumers".

ApolloRising 5y ago

Try this link: https://investors.twilio.com/governance/person-details/default.aspx?ItemId=8a5fc44e-87c5-4f3d-88c8-31b7aef0ea7f https://investors.twilio.com/governance/person-details/defau... Phone number on the page and an email address that may work for you. Let me know if that helps.

work_for_x 5y ago

I work for Twilio and forwarded it internally. If you are 6 years with the company, you probably have an AE assigned to your company or have been in touch with any other human being along the way? UPDATE: Your account should be active now.

_ikfm 5y ago
Yes, it's active now after almost 7 hours. Thank you! I plan to update everyone on what happened as soon as I get some sleep first. I've been up since 7 am and it's 2 am now. The incident started at 7 pm.
- gscott 5y ago
  and setup another system to back this one up immediately because whatever algorithm brought down your account will do so again! If you spread the load across two platforms it would be a lot of work but you would be half-alive instead all-dead.
- jariel 5y ago
  Try to find an alternative service supplier at very minimum for back up and consider legal recourse.
- e40 5y ago
  We're still interested in an update!
syabro 5y ago
Why was it deactivated?
cosmotic 5y ago
What is preventing this from happening to someone else?
morrbo 5y ago
It's a bit shit that they had to jump through so many hoops, with HN being the only one that actually worked (after 7 hours) in order to get in touch. Serious kudos to you for helping them but I know a lot of people who wouldn't touch your company with a bargepole for these kind of reasons. Having to kick up a fuss on social media in order to get service, how very 2021..

ksajadi 5y ago

Sorry to hear. Twilio has changed since its early days. Last month, our bill went from $20 per month to $16,000. The reason: fraudsters use Twilio to setup premium numbers, then use Twilio customers (like us) to send OTPs to those numbers using bots. The result is us out of pocket, fraudsters and Twilio making money. Twilio is also the only Pay as you Go SaaS vendor that doesn’t have spend limits on accounts and advises customers to implement their own any fraud measures themselves. While this was going on, we got an email from their sales rep, congratulating us for increasing our spend by more than 8000% and wanted to have a chat. So I guess they do have the systems to monitor unusual patterns, they just turn a blind eye. You’d never know, someone in Twilio might even be selling customer lists on the black market to the fraudsters so they know who to target.

smhg 5y ago
FWIW: you can turn off automatic recharge and ask for a different reminder limit. Not ideal, but at least it prevents crazy bills.
- ksajadi 5y ago
  We didn't have automatic charge. We now have an invoice to pay.
  - gumby 5y ago
    A Twilio invoice or a lawyer’s invoice?
- throwaway2048 5y ago
  They still bill you no matter what, and if the bill is big enough they will sic collections on you.
  - mleonhard 5y ago
    This surprises me. I thought they would disable the service when the balance goes negative. I need to add some more monitoring and a failsafe rate limit...

ajimix 5y ago

We had the same problem some months ago with Mailgun. They blocked our account after many years using them without apparent reason or prior notice and took them 2 days to unblock it, leaving our thousands of clients stranded without receiving any of our transactional emails. Lesson learnt, always keep a backup for crucial services and move away from Mailgun

framerate 5y ago
We just moved to Mailgun because SendGrid did something similar to us! Fortunately we only used SMTP so it was a straightforward change, but having a backup for something like Twilio Video would be quite an effort.

benjaminwootton 5y ago

I have the same on Sendgrid, owned by Twilo. One day they just cut us off with no explanation, kneecapping our product. I raised two tickets, neither of which were answered. The next day, it all started working again. I’ve always been a big advocate for using 3rd party SaaS, but with the level of customer service I’m seeing lately, I’m starting to recommend avoiding it where possible.

luckylion 5y ago
That's 99% of why I'm generally trying to avoid SaaS. I always run into some bug or nuisance or there's some account trouble and unless you're spending insane amounts of money, I've found support to be severely lacking, often answering support tickets with canned responses without reading the issue, or not answering at all. I'd rather spend 100 hours upfront to have something that I know and can maintain than using a service, running into trouble and spend 50 hours chasing the support around to see whether they'd fix it, or worse, rely on their system, have it fail or be banned, and then having to spend 100 hours building a replacement, often while things are exploding left and right. Relying on SaaS feels incredibly fragile to me.
paul_f 5y ago
It's an insane amount of engineering, but maybe one solution is to have two back end telephony SaaS. If one turns off, rely on the other until you get it fixed? Plivo, Mexmo?
MattyMc 5y ago
I recently moved to Postmark. It’s been an extremely positive experience. They’re a small, dedicated team with amazing support.
varispeed 5y ago
I think this should be regulated - companies often take massive amount of customers and while they are most of the time able to deliver the product, they have no capacity to service support queries. I think there should be an SLA guaranteed by law and for a company it should be illegal to take more customers if they are unable to provide SLA on support. This way we wouldn't have a situation where you contact a company and in reality you talk to bots or nobody at all. This would also deal with the problem of companies growing too big to fail and not having reasonable competition.

jnsaff2 5y ago

Maybe time for a new HN headline convention, like: Tell FAANG: You blocked me, again FAANG Support: Important please call back ASAP

thinkingemote 5y ago
Could this be productised somehow? Would an investor be willing to invest in my startup which effectively un-blocks small companies from larger ones? It would do the leg work to compile the necessary information before contacting a real human. I'll call it Support As A Service.
- mcny 5y ago
  Makes sense. 1. If someone works at a company, they can sign up on our website as a support agent. 2. We will only list companies with which we have an authenticated contact, maybe require people to verify they got our authentication token on their employee email? 3. When someone contacts us about an issue with a company we support, we verify the request is genuine. Perhaps charge a small fee to block spam. (This is the problematic part. How do we know a problem is genuine without asking for potentially confidential information?) 4. Add all vetted requests to a private channel that all the people who volunteered on step one. (Can we pay our volunteer support agents? How? Does paying make things worse? Maybe gamify this step and send our volunteers gifts?) 5. Monitor performance of support agents to avoid giving false hope to our customers.
- exikyut 5y ago
  I posited the interestingness of such an idea a couple months ago in a thread about app store blocks: https://news.ycombinator.com/item?id=25988410 https://news.ycombinator.com/item?id=25988410 IMHO, most emphatically *yes* - this is a completely unserved/unserviced area of the B2B and B2C markets. I noted in the linked thread that such a service would likely start by leveraging individual contacts, and building a reputation for unwaveringly high signal. It definitely does seem to me that the vacuum between corporate and customers represents a giant sleeper industry that fundamentally won't shift very quickly. Which means you're unlikely to reach the usual core-problem-is-solved entrenchment/stagnation point that tends to happen when you ramp up on a complex-but-solve-once problem, then find yourself stuck with a bunch of intricate sausage machine bits that aren't doing anything...

yenoham 5y ago

Somewhat related anecdote. Whilst we didn't get cut off, we found a huge drop-off in time to receive, and the quality of, support from our live chat/support messaging provider. Having been a customer since our launch [1], and very early after theirs, it has been really sad and frustrating to see that friendly distrupter energy zap out of the service. I get why businesses go through that changes like this during growth, but it feels like all the things to let slide, your persona is not one of them! [1] https://mailosaur.com https://mailosaur.com (somewhat related to the thread we let you test SMS messages)

daniellarusso 5y ago

Our company phone system has several VOIP providers (including Twilio) as well as multiple inbound analog lines. Two weeks ago, all of the analog lines stopped working, which are local numbers, but our main 800 number forwards to those numbers. Calling the local provider was futile. The provider’s office was also experiencing the issue and could not receive calls. Although we do have the 800 number on business cards, we quickly updated our website and Intercom docs to use a recently provisioned 800 number from one of the VOIP providers. It took 2 days for the other lines to begin working again. This is the first POTS outage we have experienced in several years.

newbie578 5y ago

Yikes, talk about bad PR. I am sorry that this happened to you, but thank you for sharing this, you are not the first one, nor the last one by the looks of it. I intend to avoid Twilio in a wide range, don't need that kind of a "business partner" in my life.

jFriedensreich 5y ago

a lot of us have this as nightmare scenario. - long term we really should move to an industry wide due process system with legislation for compensation and other parameters. once there is some more structure there can also be insurances or similar ways to prevent something like this to ruin a company. - there maybe is a market for more intermediaries that bundle twilio and messagebird or similar providers and have an official strong communication channel to the providers to resolve issues and provide failover

yowlingcat 5y ago
100%. Disappointment from seeing a once upstart (which I was rooting on from the sidelines) become a faceless big co aside, I think insurance is the right way to handle this problem and prevent it from being unfairly externalized.

rapnie 5y ago

> Since then, we are trying to get in contact with them through every channel we could find: email, Twitter, LinkedIn, [..] They don't have a support number we can call. Tons of platforms seem to operate like this, and this kinda thing happens all the time. HN is turning into a desperate last resort as a PleaForHelp-as-a-Service (hoping that bad PR or a company insider is the trigger). But by now business decision-makers should really wake up and acknowledge the risk of choosing such platforms. Of course they should demand better service, but this is also something to research upfront ("is there a support desk and at least a phone number.") and then decide if they want to pay the extra money, or go somewhere else that addresses their needs.

macspoofing 5y ago

Sorry to hear that. I hope everything works out. Take this a learning experience. Once you get your business back, sit down with your team and come up with a mitigation plan because this may happen again. It's possible you may need to switch to another vendor, even one more expensive or smaller, but one with a dedicated support line and/or an account manager and stronger SLAs. Your business heavily relies on Twilio and you should have someone you can call at Twilio outside of the general support line. Maybe you need to attend their conferences and create a personal relationship with their developers or other employees so in cases like this you can backchannel your issue. If you're going to outsource critical business infrastructure to third party SAAS vendors, make sure you have a plan for when things go tits up, because things will go tits up. It looks like Twilio screwed up here, but Twilio isn't going to suffer the consequences of this like your business will.

dooglius 5y ago

Disclaimer: IANAL As far as telephony is concerned (things like SMS are a different story), a phone provider cannot legally do this, even if you signed some ToS agreement. According to the FCC, phone providers, including voip providers, have legal requirements around when they are allowed to terminate service: https://www.fcc.gov/consumers/guides/when-your-telephone-company-discontinues-service https://www.fcc.gov/consumers/guides/when-your-telephone-com...

meepmorp 5y ago
The page you link to says that this doesn’t apply to mobile providers in general.
- dooglius 5y ago
  I was under the impression that voip is not considered "wireless" but I could be mistaken there.
  - meepmorp 5y ago
    I get the impression those protections are for residential phone service, not the kind of stuff twilio does. Leaving that aside, though, SMS (at issue here) is definitely in the wireless realm, even if the messages come from an API call rather than a mobile device's radio.

rkagerer 5y ago

This is why you have multiple vendors for every external function. Just like hardware manufacturers try to maintain at least two suppliers for every component.

alberth 5y ago

Twilio is a wet dream for spammers (robocalling, emailing, SMS). They’ve turned spamming into a positive ROI business.

sbestetti 5y ago

Hey there, sorry to hear that. Are you able to login to your account at all? If so, can you open a ticket and share the number here so I can take a look?

sirinath 5y ago

Many of these system now a days use automation with little human oversight. Algorithms classify and block accounts. The little human oversight is inadequate to even verify what happened. I believe support is spread thin with cost saving and automation. P.S. I am having a similar problem, but with Viber. I posted the details here: https://www.reddit.com/r/viber/comments/mrcu6f/viber_block_for_sending_greetings/ https://www.reddit.com/r/viber/comments/mrcu6f/viber_block_f...

Kim_Bruning 5y ago

It seems like there would be a market for reputable E2E vendors that provide you with a decent SLA. On the other hand, that would be a bit more expensive, and might therefore be out-competed in a race to the bottom.

harel 5y ago

We had an incident where our we got charged thousands (later refunded) by abusing dodgy routing and number verification process. The problem was that Twilio will allow calls through when the call balance is below zero, if the calls were queued before it was zero. You then end up with a negative balance. On the plus side, it does force you to start making defensive plans to protect your application.

cphoover 5y ago

Hmmmm they definitely have a phone number, I remember when I worked at a startup a few years ago I had many phone conversations with them regarding a bug in their mobile chat SDK. Perhaps we had a support contract tho...

dangerboysteve 5y ago

While this does not resolve your current issue. Have a look at https://signalwire.com https://signalwire.com its from the same people who created FreeSWITCH.

rsync 5y ago

Long time twilio user and evangelist here ... as I have mentioned here and in interviews[1], etc., I built my own little mini-telco out of twiml bins and have attempted, over the past 4-5 years, to move all of my telephony into the (twilio) cloud. I was encouraged when I attended Signal 2018[2] by twilio product and technical managers who intimated that such personal "utility" uses of twilio were favored by the CEO (Jeff Lawson) himself. Everything you need to know about how this has turned out - and which direction the company is going - can be imparted with two points: 1) Twilio straight up refuses to put an email verb in Twiml. This sounds like a nerdy, nit-picky point from my own personal use-case (and it is) but it's the most dead-obvious thing in the world, is technically trivial, and would make so many use-cases a world easier than they are. Instead of adding an email verb to twiml (so you could, for instance, cc: an email address with emergency SMS alerts, or perhaps cc: yourself a log of all received SMS) they chose to buy an entire email company. Now you can create these use-cases provided that you sign up a sendgrid account, and tie the two together with code hosted at a third party and blah blah blah. The reason they did this is providing telco utility tools is not interesting. Customer engagement at scale (or whatever bullshit) is what is interesting. Managing high value interactions. Whatever. 2) Q1 of 2021 they crammed phone only 2FA down the throats of all account holders. You may say "you can also use authy" but, of course, the initial setup of authy requires a real mobile number[3] and you can't log into twilio to enable authy without providing a mobile number first. So, if you're planning on moving your telephony into the cloud, you can't - you'll need to tie real phones and SIM cards to these accounts. Have multiple twilio accounts ? Congratulations - you can either be super sloppy and share your mobile SIM across all of them or you can buy three phones. This is for your security. Total bullshit. In reality, twilio has a massive spam/scam problem that is almost unsolvable. I don't envy them in this position. However, instead of solving this problem, they have decided to just throw sand in the gears and slow down the scammers by demanding that everyone prove a SIM identity. This doesn't bother most users because they aren't, ironically, trying to shed SIM cards by moving their telco infrastructure to twilio ... but it fucks my shit up immensely. Which is what happens when providing telco utility services has no path-to-unicorn and you need to turn yourself into an "engagement platform" that "enables customer interaction" for "high value use-cases". [1] https://console.dev/qa/rsync-john-kozubik/ https://console.dev/qa/rsync-john-kozubik/ [2] "peak twilio" [3] Which is so ironic since, in almost all cases, twilio numbers are NOT mobile numbers and cannot be used for "proper" 2FA since they cannot receive SMS from short-codes. So twilio demands a telephony use-case that they, themselves, cannot satisfy.

taylodl 5y ago

Lesson learned: always have a number to call, always have the ability to get a hold of a flesh-and-blood human being. My company mandates this as a minimal requirement for all our SaaS providers. With regards to having a mitigation strategy we take another tactic: what's the SaaS provider's mitigation strategy? How do they cope with data center loss, communications loss - what processes and procedures do they have in place? Any business-worthy SaaS provider has this information readily available. Any hemming or hawing is a sign of trouble.

deleted 5y ago
[deleted]

pauldd7 5y ago

We asked for a phone number transfer to another Twilio account, and the support rep took the wrong action by mistake. He wiped our entire account, causing us to lose tens of thousands of recordings, logs etc. We were told sorry, no way to recover them - only the underlying phone numbers. It's cheap, but I'd never trust them again.