9 ms·

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired,

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to SSH. This wasn’t totally their fault, but at the time it just all felt like a dodgy, shaky setup. This wasn’t *at all* their fault and there’s nothing dodgy about it. This was entirely your fault, and even mentioning this as a „partly“ negative for DO is imo very wrong. I imagine any other service provider, including Heroku, would have acted the same way. You can’t expect to mooch off a month of service without paying and without consequences. If you don’t pay your bills, service will be interrupted. It’s as simple as that.

tommica 3y ago
I read the "dodgy, shaky" parts as their own setup was that, not digital oceans
alberth 3y ago
But it kind of is. Merchant payment services exist that can automatically obtain the updated card-on-file credential when they are about to expire. Both Visa and Mastercard offer this directly, or you can get the same service from a PSP.
- TheNewsIsHere 3y ago
  It’s definitely not the merchant’s responsibility to catch when your card changes. We have this at work thanks to Stripe, and it’s wildly inconsistent. But beyond that, it won’t do anything for a closed/canceled card/cardholder account. That situation was fully on OP. This could have happened at GCP, AWS, Azure, Linode, OVH, Rackspace, Oracle Cloud, you name it.
  - TacticalCoder 3y ago
    > It’s definitely not the merchant’s responsibility to catch when your card changes. Well seen that the tech exists, they're not just leaving money on the table by not using it (to update card that just expired) but they're also burning money on pointless support, support which wouldn't need to happen in the first place had they updated the month/year of expiration. I mean: we're literally talking about credit cards often keeping the exact same owner name, the very same number and just having their MM/YY of expiration changed. It's not rocket science to update in a DB after an API call. And what's the catch on at least trying? If it works, you saved everybody time. If it doesn't work, it's not worse than your current "solution". I understand politicians and lawyer-minded people saying: "technically it's not our responsibility" but they're wasting everybody's time, leaving money on the table and wasting money on support. It's just poor judgment to react like that instead of thinking as to how life could be made better for everybody, starting with your paying customers (which, btw, are the reason you exist).
    - JohnFen 3y ago
      Would it be a better practice to use that tool? Yes. Are they at fault because their customer dropped the ball so badly about paying their bill? No.
      - relaxing 3y ago
        It doesn’t matter who’s at fault. They missed a chance to provide better service and lost a customer as a result.
        JohnFen 3y ago
        Sorry, I thought your comment was in support of alberth's upthread assertion that they bear some of the fault in this.
    - devmor 3y ago
      > Well seen that the tech exists, they're not just leaving money on the table by not using it (to update card that just expired) but they're also burning money on pointless support, support which wouldn't need to happen in the first place had they updated the month/year of expiration. To lend a bit of insight as someone who works in the payments industry: This functionality is generally not free. If most of your customers come and update their expired cards already - or you offer a service which is so essential to your customers that they generally would freely expend the effort to do so if notified, it doesn't make any sense from a financial standpoint to pay the fees to subscribe to card issuer updates.
- waboremo 3y ago
  These are regionally restricted on top of users being able to opt out of such service at both the visa/mastercard level and at their local banks. This means you cannot rely on the service entirely and have to create alternative flows even if the update system works perfectly. Both Visa and Mastercard detail for developers the manual methods they can use, which still require human monitoring/input. They also both warn people about monitoring this at the merchant level so you minimize disruptions, late fees, cancellations, etc. It's not as seamless as you're making it sound.
dangerboysteve 3y ago
Totally agree! Super dick move for dumping on DO when this was all your fault.
kodah 3y ago
Digital Ocean has an invoices API: https://docs.digitalocean.com/reference/api/api-reference/#operation/invoices_list https://docs.digitalocean.com/reference/api/api-reference/#o... No need to use email at all.
troyvit 3y ago
Yeah, Tanda's own terms of services[1] are more restrictive than DO's. They "restrict or suspend" all services (or else terminate the agreement) if the client is 14 days late. This vs ignoring billing alerts for a month and being surprised when the service is shut off? [1] https://www.tanda.co/terms-conditions/ https://www.tanda.co/terms-conditions/
voakbasda 3y ago
Whatever happened to companies actually reaching out with a phone call to an actual human in order to resolve these issues before they become a problem? A provider that simply throws a switch because a credit card expired is not one that I would trust. So, yeah, DO (and every other provider that operates that way) deserves a little bit of shade for actions that clearly demonstrate a lack of care for their customers.
- yifanl 3y ago
  They were paying all of $40 a month, I'd imagine that doesn't buy them very much customer support from any decently sized cloud provider.
withinboredom 3y ago
Not exactly. Some countries have laws in place that prevent a provider from removing service due to non-payment to a business. Instead they must follow a process, including physical paper notices, before removing service. DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.
- onion2k 3y ago
  DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty. DO should abide by the law of the country they operate in. If that means they can shut off services for non-payment then that's what they're entitled to do. By the sounds of it they gave plenty of warning and 30 days service they weren't getting paid for. Email is the expected form of business communication in 2023. I don't think any of the hosts I use even have my physical address. Maybe they could SMS me. I doubt they have my phone number either though. I would expect a billing warning notice when I SSH into the service though. Maybe that was in place for the 30 days before they shut the customer out. I certainly hope so.
  - ghiculescu 3y ago
    I SSH'd plenty. There was no billing notice. I think that + allowing customers to add more than one credit card to an account are reasonable suggestions. Maybe they do that now - this was 7 years ago. I'm not trying to shirk responsibility though. We didn't, for example, pass the buck by telling our customers our site was offline because of DO.
    - greenpeas 3y ago
      How do you deal with customers that don't reply to email notifications and don't have a phone number on file? I can appreciate how terrible must have been to suddenly find all your servers shut down, and I understand how easy it is to set up billing and forget, especially with the day to day stress of running a startup, but what would you have done if you were DO? > our site was offline because of DO. Or maybe it's because of the bank that expired your credit card?
chillfox 3y ago
I don’t think it’s unreasonable to expect that multiple channels of communication will be tried before cutting off service, especially not for services provided to a business.
- AndroTux 3y ago
  For a business that makes checks notes 50 bucks a month in revenue with DO? If this were the case, prices would be astronomically high because there would be tons of scammers and bad actors abusing this system, leaving their servers running for months, because apparently DO has to send them a letter via snail mail to Antarctica and wait for their response. All of this would then have to be paid for by legitimate customers as for DO to break even, which means a simple droplet would probably cost around $50 alone. So no, thank you. For larger businesses with thousands of dollars of revenue per month? Sure. For small $50 charges? No way.
  - pchangr 3y ago
    That’s exactly how it works for providers in Germany… as a foreigner I find it hilarious. You do get back-charged on the extra costs, tho.
    - robertlagrant 3y ago
      Why's it hilarious? If you stopped paying for something, how long should you get that thing for? What if you don't pay the back-charges?
      - pchangr 3y ago
        I find it hilarious because I find it extremely dysfunctional .. for everyone involved and yet… it’s true. I can tell you from personal experience… my partner somehow signed up for a stupid accounting online tool that we never used .. and we just find out 2 years later because we received a letter from a debt collection agency asking for the 2 years we “owed” for a service we never actually used .. but they kept “providing” us. And somehow even tho we never paid .. they never stopped. We very likely had to pay a lot more than what we would have to if they had stopped providing the service at some point in the first place. So yeah.. it just baffles me to think this is how things are supposed to work in this country and neither the provider nor we could do anything about it.
MattGaiser 3y ago
> The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. The using of an unmonitored email is the one that really gets to me.
- ghiculescu 3y ago
  Yeah. I'm not proud of it.
  - deleted 3y ago
    [deleted]
  - VoodooJuJu 3y ago
    Would you explain why you're trying to implicate DO as being in any way "dodgy" or at fault for your own failure to pay them? You got a month of service for free, and they attempted numerous times to contact you about your bill, you just didn't read their emails.
    - withinboredom 3y ago
      I don’t consider email to be a legal form of communication. Cutting services to a business effectively disabled their business (especially servers). You’d think they would send at least one letter.
      - greenpeas 3y ago
        > I don’t consider email to be a legal form of communication. The flipside of requiring a provider to submit paperwork to terminate your service, is the situation where in order to setup a new server with DO you'd have to file paperwork yourself. Can you imagine sending a paper letter every time you need to spin up a VM? Which one do you prefer? paperwork in both directions or no paperwork?
        withinboredom 3y ago
rocho 3y ago
And then there's Twitter who doesn't pay GCS and AWS for months and keeps operating normally.
- AndroTux 3y ago
  Yeah, Twitter is totally comparable in that case, $50 a month is exactly the same thing as however many Millions of Dollars Twitter is paying to GCS/AWS with presumably custom contracts. But just to humor you: https://techcrunch.com/2023/06/14/twitter-is-being-evicted-from-its-boulder-office-over-unpaid-rent/ https://techcrunch.com/2023/06/14/twitter-is-being-evicted-f...
  - robertlagrant 3y ago
    Apropos of nothing, $27k/mo for an office for 300 people!?
  - rocho 3y ago
    > many Millions of Dollars Twitter is paying to GCS/AWS But they aren't paying anymore!
- scott_w 3y ago
  I can guarantee you they’re not ignoring emails from their providers, if this is the case. Account management involves a lot of discussion and your account manager will keep the service running if they think the bill will eventually get paid.
Dowwie 3y ago
It's not as simple as that. Do you know how quickly a business loses customers without being flexible? It's also reputation damaging for the market to know that DO draws a hard line.
- karamanolev 3y ago
  What is being flexible? They gave 1) a month of unpaid bills 2) emailed the customer N times 3) gave the option to enter a backup credit card. The customer made it so neither of these worked. I expect that DO deals with a lot of non-paying customers. I wouldn't put it on them to what - figure out where they live and drive up to their house?
- pontiacbandit8 3y ago
  They literally send an email every single day for a month. And they are flexible enough that if you do write them they will happily extend that grace period. It’s like saying car is a shitty transportation service because it stops working once the gas runs out. Not paying attention the the reserve light, instrument msg and pings is solely your problem. More so because you are running a business…
  - jeremyjh 3y ago
    The car should arrange for a singing telegram to be delivered to your home, office AND your child’s daycare to notify you before shutting off its engine due to lack of fuel. In my country this is required by law.
    - Dowwie 3y ago
      Hilarious comment. Thanks for brightening my day.
nine_k 3y ago
I'd say that your service is dead unless proven otherwise. Your bills are unpaid unless proven otherwise. I'd be alerted by absence of regular emails telling me that my card was billed for the hosting service.
- dsr_ 3y ago
  It turns out that humans are very bad at noting the absence of a signal. Computers, on the other hand, are perfectly happy to use an expired timer to generate mail, a dashboard icon, or any other shout for attention that you please -- as long as you actually set it up. Paying attention to it is still a human problem.
- everybodyknows 3y ago
  > I'd say that your service is dead unless proven otherwise. I'd say you're dead right -- this is the only safe assumption. And any email client smart enough to check incoming email against a regexp could be extended to raise a not-easily-ignored alert if no pattern match (indicating success in processing a payment) had arrived by some chosen date of the month. Yet I've never come across such a feature in Gmail, Fastmail, nor Thunderbird. Maybe someone sometime has hacked such a thing into Gnus ...

tommica 3y ago

I read the "dodgy, shaky" parts as their own setup was that, not digital oceans

alberth 3y ago

But it kind of is. Merchant payment services exist that can automatically obtain the updated card-on-file credential when they are about to expire. Both Visa and Mastercard offer this directly, or you can get the same service from a PSP.

TheNewsIsHere 3y ago
It’s definitely not the merchant’s responsibility to catch when your card changes. We have this at work thanks to Stripe, and it’s wildly inconsistent. But beyond that, it won’t do anything for a closed/canceled card/cardholder account. That situation was fully on OP. This could have happened at GCP, AWS, Azure, Linode, OVH, Rackspace, Oracle Cloud, you name it.
- TacticalCoder 3y ago
  > It’s definitely not the merchant’s responsibility to catch when your card changes. Well seen that the tech exists, they're not just leaving money on the table by not using it (to update card that just expired) but they're also burning money on pointless support, support which wouldn't need to happen in the first place had they updated the month/year of expiration. I mean: we're literally talking about credit cards often keeping the exact same owner name, the very same number and just having their MM/YY of expiration changed. It's not rocket science to update in a DB after an API call. And what's the catch on at least trying? If it works, you saved everybody time. If it doesn't work, it's not worse than your current "solution". I understand politicians and lawyer-minded people saying: "technically it's not our responsibility" but they're wasting everybody's time, leaving money on the table and wasting money on support. It's just poor judgment to react like that instead of thinking as to how life could be made better for everybody, starting with your paying customers (which, btw, are the reason you exist).
  - JohnFen 3y ago
    Would it be a better practice to use that tool? Yes. Are they at fault because their customer dropped the ball so badly about paying their bill? No.
    - relaxing 3y ago
      It doesn’t matter who’s at fault. They missed a chance to provide better service and lost a customer as a result.
      - JohnFen 3y ago
        Sorry, I thought your comment was in support of alberth's upthread assertion that they bear some of the fault in this.
  - devmor 3y ago
    > Well seen that the tech exists, they're not just leaving money on the table by not using it (to update card that just expired) but they're also burning money on pointless support, support which wouldn't need to happen in the first place had they updated the month/year of expiration. To lend a bit of insight as someone who works in the payments industry: This functionality is generally not free. If most of your customers come and update their expired cards already - or you offer a service which is so essential to your customers that they generally would freely expend the effort to do so if notified, it doesn't make any sense from a financial standpoint to pay the fees to subscribe to card issuer updates.
waboremo 3y ago
These are regionally restricted on top of users being able to opt out of such service at both the visa/mastercard level and at their local banks. This means you cannot rely on the service entirely and have to create alternative flows even if the update system works perfectly. Both Visa and Mastercard detail for developers the manual methods they can use, which still require human monitoring/input. They also both warn people about monitoring this at the merchant level so you minimize disruptions, late fees, cancellations, etc. It's not as seamless as you're making it sound.

dangerboysteve 3y ago

Totally agree! Super dick move for dumping on DO when this was all your fault.

kodah 3y ago

Digital Ocean has an invoices API: https://docs.digitalocean.com/reference/api/api-reference/#operation/invoices_list https://docs.digitalocean.com/reference/api/api-reference/#o... No need to use email at all.

troyvit 3y ago

Yeah, Tanda's own terms of services[1] are more restrictive than DO's. They "restrict or suspend" all services (or else terminate the agreement) if the client is 14 days late. This vs ignoring billing alerts for a month and being surprised when the service is shut off? [1] https://www.tanda.co/terms-conditions/ https://www.tanda.co/terms-conditions/

voakbasda 3y ago

Whatever happened to companies actually reaching out with a phone call to an actual human in order to resolve these issues before they become a problem? A provider that simply throws a switch because a credit card expired is not one that I would trust. So, yeah, DO (and every other provider that operates that way) deserves a little bit of shade for actions that clearly demonstrate a lack of care for their customers.

yifanl 3y ago
They were paying all of $40 a month, I'd imagine that doesn't buy them very much customer support from any decently sized cloud provider.

withinboredom 3y ago

Not exactly. Some countries have laws in place that prevent a provider from removing service due to non-payment to a business. Instead they must follow a process, including physical paper notices, before removing service. DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty.

onion2k 3y ago
DO should have done something like that as well, just cutting off service because you sent some emails is kinda shitty. DO should abide by the law of the country they operate in. If that means they can shut off services for non-payment then that's what they're entitled to do. By the sounds of it they gave plenty of warning and 30 days service they weren't getting paid for. Email is the expected form of business communication in 2023. I don't think any of the hosts I use even have my physical address. Maybe they could SMS me. I doubt they have my phone number either though. I would expect a billing warning notice when I SSH into the service though. Maybe that was in place for the 30 days before they shut the customer out. I certainly hope so.
- ghiculescu 3y ago
  I SSH'd plenty. There was no billing notice. I think that + allowing customers to add more than one credit card to an account are reasonable suggestions. Maybe they do that now - this was 7 years ago. I'm not trying to shirk responsibility though. We didn't, for example, pass the buck by telling our customers our site was offline because of DO.
  - greenpeas 3y ago
    How do you deal with customers that don't reply to email notifications and don't have a phone number on file? I can appreciate how terrible must have been to suddenly find all your servers shut down, and I understand how easy it is to set up billing and forget, especially with the day to day stress of running a startup, but what would you have done if you were DO? > our site was offline because of DO. Or maybe it's because of the bank that expired your credit card?

chillfox 3y ago

I don’t think it’s unreasonable to expect that multiple channels of communication will be tried before cutting off service, especially not for services provided to a business.

AndroTux 3y ago
For a business that makes checks notes 50 bucks a month in revenue with DO? If this were the case, prices would be astronomically high because there would be tons of scammers and bad actors abusing this system, leaving their servers running for months, because apparently DO has to send them a letter via snail mail to Antarctica and wait for their response. All of this would then have to be paid for by legitimate customers as for DO to break even, which means a simple droplet would probably cost around $50 alone. So no, thank you. For larger businesses with thousands of dollars of revenue per month? Sure. For small $50 charges? No way.
- pchangr 3y ago
  That’s exactly how it works for providers in Germany… as a foreigner I find it hilarious. You do get back-charged on the extra costs, tho.
  - robertlagrant 3y ago
    Why's it hilarious? If you stopped paying for something, how long should you get that thing for? What if you don't pay the back-charges?
    - pchangr 3y ago
      I find it hilarious because I find it extremely dysfunctional .. for everyone involved and yet… it’s true. I can tell you from personal experience… my partner somehow signed up for a stupid accounting online tool that we never used .. and we just find out 2 years later because we received a letter from a debt collection agency asking for the 2 years we “owed” for a service we never actually used .. but they kept “providing” us. And somehow even tho we never paid .. they never stopped. We very likely had to pay a lot more than what we would have to if they had stopped providing the service at some point in the first place. So yeah.. it just baffles me to think this is how things are supposed to work in this country and neither the provider nor we could do anything about it.

MattGaiser 3y ago

> The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. The using of an unmonitored email is the one that really gets to me.

ghiculescu 3y ago
Yeah. I'm not proud of it.
- deleted 3y ago
  [deleted]
- VoodooJuJu 3y ago
  Would you explain why you're trying to implicate DO as being in any way "dodgy" or at fault for your own failure to pay them? You got a month of service for free, and they attempted numerous times to contact you about your bill, you just didn't read their emails.
  - withinboredom 3y ago
    I don’t consider email to be a legal form of communication. Cutting services to a business effectively disabled their business (especially servers). You’d think they would send at least one letter.
    - greenpeas 3y ago
      > I don’t consider email to be a legal form of communication. The flipside of requiring a provider to submit paperwork to terminate your service, is the situation where in order to setup a new server with DO you'd have to file paperwork yourself. Can you imagine sending a paper letter every time you need to spin up a VM? Which one do you prefer? paperwork in both directions or no paperwork?
      - withinboredom 3y ago

rocho 3y ago

And then there's Twitter who doesn't pay GCS and AWS for months and keeps operating normally.

AndroTux 3y ago
Yeah, Twitter is totally comparable in that case, $50 a month is exactly the same thing as however many Millions of Dollars Twitter is paying to GCS/AWS with presumably custom contracts. But just to humor you: https://techcrunch.com/2023/06/14/twitter-is-being-evicted-from-its-boulder-office-over-unpaid-rent/ https://techcrunch.com/2023/06/14/twitter-is-being-evicted-f...
- robertlagrant 3y ago
  Apropos of nothing, $27k/mo for an office for 300 people!?
- rocho 3y ago
  > many Millions of Dollars Twitter is paying to GCS/AWS But they aren't paying anymore!
scott_w 3y ago
I can guarantee you they’re not ignoring emails from their providers, if this is the case. Account management involves a lot of discussion and your account manager will keep the service running if they think the bill will eventually get paid.

Dowwie 3y ago

It's not as simple as that. Do you know how quickly a business loses customers without being flexible? It's also reputation damaging for the market to know that DO draws a hard line.

karamanolev 3y ago
What is being flexible? They gave 1) a month of unpaid bills 2) emailed the customer N times 3) gave the option to enter a backup credit card. The customer made it so neither of these worked. I expect that DO deals with a lot of non-paying customers. I wouldn't put it on them to what - figure out where they live and drive up to their house?
pontiacbandit8 3y ago
They literally send an email every single day for a month. And they are flexible enough that if you do write them they will happily extend that grace period. It’s like saying car is a shitty transportation service because it stops working once the gas runs out. Not paying attention the the reserve light, instrument msg and pings is solely your problem. More so because you are running a business…
- jeremyjh 3y ago
  The car should arrange for a singing telegram to be delivered to your home, office AND your child’s daycare to notify you before shutting off its engine due to lack of fuel. In my country this is required by law.
  - Dowwie 3y ago
    Hilarious comment. Thanks for brightening my day.

nine_k 3y ago

I'd say that your service is dead unless proven otherwise. Your bills are unpaid unless proven otherwise. I'd be alerted by absence of regular emails telling me that my card was billed for the hosting service.

dsr_ 3y ago
It turns out that humans are very bad at noting the absence of a signal. Computers, on the other hand, are perfectly happy to use an expired timer to generate mail, a dashboard icon, or any other shout for attention that you please -- as long as you actually set it up. Paying attention to it is still a human problem.
everybodyknows 3y ago
> I'd say that your service is dead unless proven otherwise. I'd say you're dead right -- this is the only safe assumption. And any email client smart enough to check incoming email against a regexp could be extended to raise a not-easily-ignored alert if no pattern match (indicating success in processing a payment) had arrived by some chosen date of the month. Yet I've never come across such a feature in Gmail, Fastmail, nor Thunderbird. Maybe someone sometime has hacked such a thing into Gnus ...