8 ms·

The same is true of DigitalOcean, fwiw. Shutting down machines used in production with no notice, outside working hours. I don’t know why these companies don’t

The same is true of DigitalOcean, fwiw. Shutting down machines used in production with no notice, outside working hours. I don’t know why these companies don’t care - they’re massively shooting themselves in their feet. Incidents like this turn customers off for life.

mro_name 4y ago
they are big and you are small. It's like walking over ants.
- arvindamirtaa 4y ago
  Ants pay the bill!
  - tinus_hn 4y ago
    Ants that don’t cause trouble pay the bill. Ants that cause trouble are a net loss. Customer service representation is one of the biggest expenses these companies have. That’s why they go to great lengths to make sure you can’t contact them and why they just close your account with no recourse as soon as you start causing trouble.
    - pid-1 4y ago
      Poor AWS must be close to bankruptcy then. When my account started spending 300-400 USD monthly, I was contacted by an account manager, who was at my office for a meeting two weeks later.
    - arvindamirtaa 4y ago
      Every time they do this, a few hundred ants decide, they may very well be next and move out. Supporting them through whatever they need may not be quite the net loss you claim it is.
Croftengea 4y ago
Care to share more info please? Was it because card didn't go through? I've got some important stuff there.
- michelb 4y ago
  I'd like some more info as well. Only case I know was from someone who didn't see the emails about issues with their payment.
- jiggawatts 4y ago
  It has been covered here on HN several times, but to summarize: Budget cloud hosting on a credit card allows scammers to use stolen CC numbers to spin up VMs to mine crypto. Essentially the hosting company is offering a cash equivalent for unreliable credit. This kind of abuse occurs at enormous scale, because it can be worth it for an attacker to use botnets to throw tens of thousands of credit cards at hosting companies. If you never actually expect to pay any money (especially your own!), then even thirty minutes of free compute is worthwhile to chase. There have been similar attacks against GitHub Actions, because they're also free and general-purpose. This is also why the free-tier, dev-only, education, or similar MSDN or VS Subscriber type cloud subscriptions never permit the use of GPU instances. Too tempting a target! The large providers like Azure and AWS are basically printing money, so they don't care about the small-fry scammers. They're too busy trying to hold on to the firehose of cash. Smaller, budget providers like Digital Ocean are running too lean to tolerate scammers, but also can't afford to have humans in the loop from the sheer scale of the attacks. So they use heuristics that are 99.99% accurate (or whatever), which means one in ten thousand legit customers gets axed along with the scammers. Oops. Essentially, as a customer of these small providers you are taking on some of their risk in exchange for the discount over the big-name vendors. This is especially true if you pay them with a credit card, irrespective of past payment history. Like I said, they simply can't afford to keep a human in the loop.[1] An example that came up here was a startup that had an account "ticking along" with a handful of dev stuff, went "live" with a big launch, and so almost all of their servers was loaded to nearly 100% capacity thanks to efficient containerisation and auto-scale. Good job devs... except that looks identical to a hacked account that has suddenly started to mine crypto on every machine. Axed. [1] It's even more complex than you think. Employees can and have been bribed to let the scammers through! The employees themselves might be mining crypto. The scammers can trick them, lie, beg, or just figure out the system from repeat conversations. The only recourse is to take the human out of the loop entirely, nothing else works for them. If you get to be the 0.01%, you generally have no recourse.
jabiko 4y ago
I actually managed to get my account suspended one day due to nonpayment. It was just a personal toy project and it was completely my fault: I was automatically filtering the mails into a folder that I didn't read until it was too late. If you look at the sheer amount of emails they sent me before suspending the account, Digital Ocean did everything right: https://i.imgur.com/ah13bEA.png https://i.imgur.com/ah13bEA.png They gave lots of warnings and also sent an email with the date the account was going to be closed a week in advance.
- terom 4y ago
  Just like you setup multiple instances across different availability zones for redundancy, you also need multiple billing contacts in case one of them fails and causes an outage. Not sure how to mitigate these risks for providers that only support one "project owner" with a single billing address :/
  - Xylakant 4y ago
    Use a “synthetic” project owner with an email address that forwards to the group.
    - CaptainZapp 4y ago
      That could backfire, since nobody really feels in charge. https://en.wikipedia.org/wiki/Bystander_effect https://en.wikipedia.org/wiki/Bystander_effect
      - Xylakant 4y ago
        But that's the same if you have multiple billing contacts. What is required is that the responsibility in the group of people that receive those notifications is clear. Having multiple people that receive billing notifications is strictly better than having a single person - what happens if that person is on vacation, sick, overloaded? No one even has a chance of catching this. If at least one other person receives those increasingly important emails, they at least have a chance to catch this before it breaks.
magicalhippo 4y ago
> outside working hours Working hours are relative. Our support routinely gets awoken at like 3am because someone in Singapore, where one of our larger customers has outsourced their IT to, just started their day and decided to follow up on a case. Though I'll agree that weekends should be considered off-limits regardless.
- qabqabaca 4y ago
  > Working hours are relative Sorry but for me this doesn't cut it when you have staff and availability zones in every continent on the planet. It isn't some mom and pop shop in one town. It shouldn't be much to ask a company of Google's size and breadth to consider your local timezone for urgent communications.
  - magicalhippo 4y ago
    It was more a comment on how it's difficult to pin down working hours in this international day and age. I mean we're based in Norway, but what if we outsourced our IT to Chile? How would Google or similar know that they need to contact our IT guys during Chilean working hours, and not Norwegian working hours? If it's an invoice that's not paid, well that's done here in Norway though, so definitely Norwegian working hours there...
    - kadoban 4y ago
      Worst case they can ask, somewhere in the account settings. Defaulting to the billing region should handle 99% of cases though.
kdumont 4y ago
We went through a similar event last week. Droplets shut down and account locked without warning and because of “violation of the DigitalOcean user agreement”. Luckily, it was on a sandbox environment we use for testing before taking anything to production. All we had done was set up a simple keycloak server. The real kicker is that they refused to tell us what terms we had allegedly violated because it’s “critical to maintaining our security systems.” And they had “just conducted a manual review” and confirmed the suspension. Then, after escalating further and a few choice words about how much I’ve spent on their platform, sure enough they conducted another “manual review” and found no issues. I’ve really tried to avoid moving to AWS over the years, but I’m really reconsidering.
rendaw 4y ago
At a previous employer we were hosting with DO via credit balance. Our finance department couldn't get things together and kept forgetting (almost every month) to add new credits to the account. Despite this DO never stopped or terminated our servers, and we were often days to weeks behind on the payments.
BrainVirus 4y ago
they’re massively shooting themselves in their feet It doesn't matter. Google operates at scale. As long as they save more money by not bothering to handle such incidents better, they will see it as a perfectly good business policy. Incidents like this turn customers off for life. Small customers are completely interchangeable for larger service providers. You can easily get new customers by throwing around some free credits to college kids and posting a few tech ads masqueraded as tutorials.
Sohcahtoa82 4y ago
Years ago, I used a cheap droplet on DO as an IRC bouncer to hide my home IP address. Someone came into a channel I was an op in and started spamming racial slurs, so I did what a normal op does and banned them. They proceeded to DDoS my droplet. Only a couple minutes later, DO responded by disconnecting my droplet from the Internet for 3 hours, supposedly to protect other customers. Thanks, DO, you let the DDoSer win. My droplet was handling the attack just fine, it just made my IRC connection a little laggy. As a $5/month customer, I was just small potatoes, for sure. But I had ideas for products in the future, and I decided that I would not use DO for them, knowing that some skiddie could just take it offline for 3 hours by just packet flooding it for a couple minutes.

mro_name 4y ago

they are big and you are small. It's like walking over ants.

arvindamirtaa 4y ago
Ants pay the bill!
- tinus_hn 4y ago
  Ants that don’t cause trouble pay the bill. Ants that cause trouble are a net loss. Customer service representation is one of the biggest expenses these companies have. That’s why they go to great lengths to make sure you can’t contact them and why they just close your account with no recourse as soon as you start causing trouble.
  - pid-1 4y ago
    Poor AWS must be close to bankruptcy then. When my account started spending 300-400 USD monthly, I was contacted by an account manager, who was at my office for a meeting two weeks later.
  - arvindamirtaa 4y ago
    Every time they do this, a few hundred ants decide, they may very well be next and move out. Supporting them through whatever they need may not be quite the net loss you claim it is.

Croftengea 4y ago

Care to share more info please? Was it because card didn't go through? I've got some important stuff there.

michelb 4y ago
I'd like some more info as well. Only case I know was from someone who didn't see the emails about issues with their payment.
jiggawatts 4y ago
It has been covered here on HN several times, but to summarize: Budget cloud hosting on a credit card allows scammers to use stolen CC numbers to spin up VMs to mine crypto. Essentially the hosting company is offering a cash equivalent for unreliable credit. This kind of abuse occurs at enormous scale, because it can be worth it for an attacker to use botnets to throw tens of thousands of credit cards at hosting companies. If you never actually expect to pay any money (especially your own!), then even thirty minutes of free compute is worthwhile to chase. There have been similar attacks against GitHub Actions, because they're also free and general-purpose. This is also why the free-tier, dev-only, education, or similar MSDN or VS Subscriber type cloud subscriptions never permit the use of GPU instances. Too tempting a target! The large providers like Azure and AWS are basically printing money, so they don't care about the small-fry scammers. They're too busy trying to hold on to the firehose of cash. Smaller, budget providers like Digital Ocean are running too lean to tolerate scammers, but also can't afford to have humans in the loop from the sheer scale of the attacks. So they use heuristics that are 99.99% accurate (or whatever), which means one in ten thousand legit customers gets axed along with the scammers. Oops. Essentially, as a customer of these small providers you are taking on some of their risk in exchange for the discount over the big-name vendors. This is especially true if you pay them with a credit card, irrespective of past payment history. Like I said, they simply can't afford to keep a human in the loop.[1] An example that came up here was a startup that had an account "ticking along" with a handful of dev stuff, went "live" with a big launch, and so almost all of their servers was loaded to nearly 100% capacity thanks to efficient containerisation and auto-scale. Good job devs... except that looks identical to a hacked account that has suddenly started to mine crypto on every machine. Axed. [1] It's even more complex than you think. Employees can and have been bribed to let the scammers through! The employees themselves might be mining crypto. The scammers can trick them, lie, beg, or just figure out the system from repeat conversations. The only recourse is to take the human out of the loop entirely, nothing else works for them. If you get to be the 0.01%, you generally have no recourse.

jabiko 4y ago

I actually managed to get my account suspended one day due to nonpayment. It was just a personal toy project and it was completely my fault: I was automatically filtering the mails into a folder that I didn't read until it was too late. If you look at the sheer amount of emails they sent me before suspending the account, Digital Ocean did everything right: https://i.imgur.com/ah13bEA.png https://i.imgur.com/ah13bEA.png They gave lots of warnings and also sent an email with the date the account was going to be closed a week in advance.

terom 4y ago
Just like you setup multiple instances across different availability zones for redundancy, you also need multiple billing contacts in case one of them fails and causes an outage. Not sure how to mitigate these risks for providers that only support one "project owner" with a single billing address :/
- Xylakant 4y ago
  Use a “synthetic” project owner with an email address that forwards to the group.
  - CaptainZapp 4y ago
    That could backfire, since nobody really feels in charge. https://en.wikipedia.org/wiki/Bystander_effect https://en.wikipedia.org/wiki/Bystander_effect
    - Xylakant 4y ago
      But that's the same if you have multiple billing contacts. What is required is that the responsibility in the group of people that receive those notifications is clear. Having multiple people that receive billing notifications is strictly better than having a single person - what happens if that person is on vacation, sick, overloaded? No one even has a chance of catching this. If at least one other person receives those increasingly important emails, they at least have a chance to catch this before it breaks.

magicalhippo 4y ago

> outside working hours Working hours are relative. Our support routinely gets awoken at like 3am because someone in Singapore, where one of our larger customers has outsourced their IT to, just started their day and decided to follow up on a case. Though I'll agree that weekends should be considered off-limits regardless.

qabqabaca 4y ago
> Working hours are relative Sorry but for me this doesn't cut it when you have staff and availability zones in every continent on the planet. It isn't some mom and pop shop in one town. It shouldn't be much to ask a company of Google's size and breadth to consider your local timezone for urgent communications.
- magicalhippo 4y ago
  It was more a comment on how it's difficult to pin down working hours in this international day and age. I mean we're based in Norway, but what if we outsourced our IT to Chile? How would Google or similar know that they need to contact our IT guys during Chilean working hours, and not Norwegian working hours? If it's an invoice that's not paid, well that's done here in Norway though, so definitely Norwegian working hours there...
  - kadoban 4y ago
    Worst case they can ask, somewhere in the account settings. Defaulting to the billing region should handle 99% of cases though.

kdumont 4y ago

We went through a similar event last week. Droplets shut down and account locked without warning and because of “violation of the DigitalOcean user agreement”. Luckily, it was on a sandbox environment we use for testing before taking anything to production. All we had done was set up a simple keycloak server. The real kicker is that they refused to tell us what terms we had allegedly violated because it’s “critical to maintaining our security systems.” And they had “just conducted a manual review” and confirmed the suspension. Then, after escalating further and a few choice words about how much I’ve spent on their platform, sure enough they conducted another “manual review” and found no issues. I’ve really tried to avoid moving to AWS over the years, but I’m really reconsidering.

rendaw 4y ago

At a previous employer we were hosting with DO via credit balance. Our finance department couldn't get things together and kept forgetting (almost every month) to add new credits to the account. Despite this DO never stopped or terminated our servers, and we were often days to weeks behind on the payments.

BrainVirus 4y ago

they’re massively shooting themselves in their feet It doesn't matter. Google operates at scale. As long as they save more money by not bothering to handle such incidents better, they will see it as a perfectly good business policy. Incidents like this turn customers off for life. Small customers are completely interchangeable for larger service providers. You can easily get new customers by throwing around some free credits to college kids and posting a few tech ads masqueraded as tutorials.

Sohcahtoa82 4y ago

Years ago, I used a cheap droplet on DO as an IRC bouncer to hide my home IP address. Someone came into a channel I was an op in and started spamming racial slurs, so I did what a normal op does and banned them. They proceeded to DDoS my droplet. Only a couple minutes later, DO responded by disconnecting my droplet from the Internet for 3 hours, supposedly to protect other customers. Thanks, DO, you let the DDoSer win. My droplet was handling the attack just fine, it just made my IRC connection a little laggy. As a $5/month customer, I was just small potatoes, for sure. But I had ideas for products in the future, and I decided that I would not use DO for them, knowing that some skiddie could just take it offline for 3 hours by just packet flooding it for a couple minutes.