5 ms·

“DigitalOcean Killed Our Company” (2019)

55 points by ddtaylor 4y ago

m-p-3 4y ago
Digital Ocean's post-mortem of the event https://web.archive.org/web/20220417154429/https://www.digitalocean.com/blog/an-update-on-last-weeks-customer-shutdown-incident https://web.archive.org/web/20220417154429/https://www.digit...
- oncewas 4y ago
- Espressosaurus 4y ago
  Non-archived version since the web.archive.org version seems to be slammed: https://www.digitalocean.com/blog/an-update-on-last-weeks-customer-shutdown-incident https://www.digitalocean.com/blog/an-update-on-last-weeks-cu...
  - m-p-3 4y ago
    Ironically I posted the Archived version because the DO blog was under maintenance.
josephcsible 4y ago
What are "fraudulent high-cpu-loads"? Imagine if your water company told you that taking long showers was "fraudulent" even though you were paying for the water.
- lmz 4y ago
  cryptocoin miners using stolen cards
  - josephcsible 4y ago
    But the stolen cards are all that's fraudulent, not the high CPU loads.
    - quesomaster9000 4y ago
      No, technically the use is fraudulent, which would be an incurred cost damage to the hosting provider (potentially including third-party breach of contract due to knock-on effects, SLAs etc), with another count of deception for paying with a stolen card... Anyway, I digress...
      - josephcsible 4y ago
        In any case, though, there were just high CPU loads and the credit card wasn't stolen, so there was no actual fraud at all.
      - shrubble 4y ago
        "Process for adding the Allow High CPU Utilization safety flag was not followed." so high CPU utilization is allowed under certain cases
- nhinck2 4y ago
  If you were using commercial amounts of water in a residential property you can bet your ass your service would be shut off. (Not a judgment on what happened here)
WalterGR 4y ago
If anyone's curious, the discussion from 2019 got 1,343 points and 601 comments: https://news.ycombinator.com/item?id=20064169 https://news.ycombinator.com/item?id=20064169
Mo3 4y ago
Something’s not quite right with this. The email states that droplets and services were kept running, and they did keep on running when the exact same thing happened to me after a script went haywire and spun up huge amounts of droplets. Second, if you have Fortune 500 companies using your service, why would you not keep offsite backups… I hope they learned from this…
car_analogy 4y ago
> Cryptocurrency mining mitigation detects suspicious behavior, including very high CPU utilization on an account with no payment history While it was only a few hundred dollars, actually using the compute I paid for in advance still resulted in a permanent ban, and Digital Ocean kept the remaining balance. It was only around ten droplets, created manually. What exactly should one do if one actually needs the compute?
- 10000truths 4y ago
  I don’t understand why “cryptocurrency mining mitigation” even exists. Why care at all what someone does with an instance they paid for, as long as it’s not illegal? Even if you do care for whatever reason, isn’t this something that’s better addressed via pricing adjustment? You won’t have any miners if they spend more money renting the instance than they get by mining.
  - duxup 4y ago
    I believe because it isn’t cost effective in the cloud there just isn’t much if any legitimate mining being done. Most cases are fraudulent in some way. Fake credit cards, stolen credentials, misusing company resources, and other types of efforts to mine, but not actually pay. Cloud providers would rather just not provide that service, detect and stop it before it gets too out of hand. The handful of times I’ve personally encountered it, it was always some sort of fraud. Notices from the cloud provider were how a given company found out about it… before the surprise alerts or bills.
  - toast0 4y ago
    Pricing adjustment helps if there's a arbitrage on cost to run vs amount charged. There very well could be if mining uses more power and the increased power cost is material. Pricing adjustment doesn't help if cryptocurrency mining is being used to generate income from VMs leased with stolen credit cards. Might be more useful to have a CPU limiter that eases up as the VM (or payment) ages; although if that's not public knowledge, it will lead to support requests.
    - 4y ago
ilrwbwrkhv 4y ago
Ya since this incident I have moved away from Digital Ocean. Quite happy at render.com and linode for smaller tasks.
nicbvs 4y ago
Hello Hacker News, I'm the guy who made the Twitter post and wanted to clarify a few things because this Tweet keep coming back every year with the same comments. 1. About the thread: First, I would like to say that this thread was purposely dramatic with the only goal to get a quick answer from Digital Ocean. We did not have a communication channel with them after their last ban email, and it was a way for us to promptly establish a communication through social pressure. And with hindsight, probably a bit of immature social-media revenge of creating a shitstorm on social media. DO did not kill our company, they put us in an uncomfortable position. We did have two Fortune 500 clients at that time, but we had no critical data, just a few configuration variables that we were aware of and could probably recreate from our memory. 2. About the backups: We had two database backup, DO automatic disk backup and a backup script that uploaded to DO object storage. YES, IT WAS A MISTAKE to have all our data with a single provider. I've learned from this, and it was a valuable lesson for all the people that followed this story. At the time, I was the only person with tech skills in a two-person company. My tasks ranged from UI/UX to NLP to DevOps, I had professional experience with frontend / backend, but it was my first role where I had to handle system administration. I think I did a fantastic job with it for a junior, outside not setting up multi-provider backups. 3. About what we did to get ban: It really was a python script that had to process ~500k records and fuzzy match them against another DB with 1M records. It was pure business logic and nothing illegal / shady / or against DO TOS. As stated in the postmortem, we did not have a payment history with DO, as we were still on free credit at this time. We had no malicious intent to avoid payment and had the mean to pay for what we used once the credit ran off. 4. About Digital Ocean: I do not hold any grief against DO, it was a human a process error, it happens to everyone, big and small companies. What matters is that they fixed quickly, did a postmortem on it, and acted to prevent this situation in the future. Nobody got killed, no data got lost, and we all learned from this event. Hope this clarifies the story a bit. And please stop harassing me about multi-cloud backups :s
- znpy 4y ago
  Hi Nic, As a person that has been watching this kind of horror stories from outside, I just wonder why people choose to make business with companies that have no human in the loop when stuff gets serious. AFAIK, for AWS for example, when things get serious you are prompted to upgrade to a the big boi support plan (1.5k$/month) and are assigned a TAM (Technical Account Manager). In a situation like that, if you were running on AWS, a TAM or any human in general would have probably fixed the whole thing in like 30 minutes (if anything, because they would know you and that your usage is legitimate).
jesterson 4y ago
Another reminder for not having provider as single point of failure. Being in the industry for years I've heard similar stories for XXXX (insert your favourite, extremely "friendly" and reliable provider). Never build your service on single provider. At least 2, preferably 3+ for redundancy.

55 points by ddtaylor 4y ago

Digital Ocean's post-mortem of the event https://web.archive.org/web/20220417154429/https://www.digitalocean.com/blog/an-update-on-last-weeks-customer-shutdown-incident https://web.archive.org/web/20220417154429/https://www.digit...

oncewas 4y ago
Espressosaurus 4y ago
Non-archived version since the web.archive.org version seems to be slammed: https://www.digitalocean.com/blog/an-update-on-last-weeks-customer-shutdown-incident https://www.digitalocean.com/blog/an-update-on-last-weeks-cu...
- m-p-3 4y ago
  Ironically I posted the Archived version because the DO blog was under maintenance.

josephcsible 4y ago

What are "fraudulent high-cpu-loads"? Imagine if your water company told you that taking long showers was "fraudulent" even though you were paying for the water.

lmz 4y ago
cryptocoin miners using stolen cards
- josephcsible 4y ago
  But the stolen cards are all that's fraudulent, not the high CPU loads.
  - quesomaster9000 4y ago
    No, technically the use is fraudulent, which would be an incurred cost damage to the hosting provider (potentially including third-party breach of contract due to knock-on effects, SLAs etc), with another count of deception for paying with a stolen card... Anyway, I digress...
    - josephcsible 4y ago
      In any case, though, there were just high CPU loads and the credit card wasn't stolen, so there was no actual fraud at all.
    - shrubble 4y ago
      "Process for adding the Allow High CPU Utilization safety flag was not followed." so high CPU utilization is allowed under certain cases
nhinck2 4y ago
If you were using commercial amounts of water in a residential property you can bet your ass your service would be shut off. (Not a judgment on what happened here)

WalterGR 4y ago

If anyone's curious, the discussion from 2019 got 1,343 points and 601 comments: https://news.ycombinator.com/item?id=20064169 https://news.ycombinator.com/item?id=20064169

Mo3 4y ago

Something’s not quite right with this. The email states that droplets and services were kept running, and they did keep on running when the exact same thing happened to me after a script went haywire and spun up huge amounts of droplets. Second, if you have Fortune 500 companies using your service, why would you not keep offsite backups… I hope they learned from this…

car_analogy 4y ago

> Cryptocurrency mining mitigation detects suspicious behavior, including very high CPU utilization on an account with no payment history While it was only a few hundred dollars, actually using the compute I paid for in advance still resulted in a permanent ban, and Digital Ocean kept the remaining balance. It was only around ten droplets, created manually. What exactly should one do if one actually needs the compute?

10000truths 4y ago
I don’t understand why “cryptocurrency mining mitigation” even exists. Why care at all what someone does with an instance they paid for, as long as it’s not illegal? Even if you do care for whatever reason, isn’t this something that’s better addressed via pricing adjustment? You won’t have any miners if they spend more money renting the instance than they get by mining.
- duxup 4y ago
  I believe because it isn’t cost effective in the cloud there just isn’t much if any legitimate mining being done. Most cases are fraudulent in some way. Fake credit cards, stolen credentials, misusing company resources, and other types of efforts to mine, but not actually pay. Cloud providers would rather just not provide that service, detect and stop it before it gets too out of hand. The handful of times I’ve personally encountered it, it was always some sort of fraud. Notices from the cloud provider were how a given company found out about it… before the surprise alerts or bills.
- toast0 4y ago
  Pricing adjustment helps if there's a arbitrage on cost to run vs amount charged. There very well could be if mining uses more power and the increased power cost is material. Pricing adjustment doesn't help if cryptocurrency mining is being used to generate income from VMs leased with stolen credit cards. Might be more useful to have a CPU limiter that eases up as the VM (or payment) ages; although if that's not public knowledge, it will lead to support requests.
  - 4y ago

ilrwbwrkhv 4y ago

Ya since this incident I have moved away from Digital Ocean. Quite happy at render.com and linode for smaller tasks.

nicbvs 4y ago

Hello Hacker News, I'm the guy who made the Twitter post and wanted to clarify a few things because this Tweet keep coming back every year with the same comments. 1. About the thread: First, I would like to say that this thread was purposely dramatic with the only goal to get a quick answer from Digital Ocean. We did not have a communication channel with them after their last ban email, and it was a way for us to promptly establish a communication through social pressure. And with hindsight, probably a bit of immature social-media revenge of creating a shitstorm on social media. DO did not kill our company, they put us in an uncomfortable position. We did have two Fortune 500 clients at that time, but we had no critical data, just a few configuration variables that we were aware of and could probably recreate from our memory. 2. About the backups: We had two database backup, DO automatic disk backup and a backup script that uploaded to DO object storage. YES, IT WAS A MISTAKE to have all our data with a single provider. I've learned from this, and it was a valuable lesson for all the people that followed this story. At the time, I was the only person with tech skills in a two-person company. My tasks ranged from UI/UX to NLP to DevOps, I had professional experience with frontend / backend, but it was my first role where I had to handle system administration. I think I did a fantastic job with it for a junior, outside not setting up multi-provider backups. 3. About what we did to get ban: It really was a python script that had to process ~500k records and fuzzy match them against another DB with 1M records. It was pure business logic and nothing illegal / shady / or against DO TOS. As stated in the postmortem, we did not have a payment history with DO, as we were still on free credit at this time. We had no malicious intent to avoid payment and had the mean to pay for what we used once the credit ran off. 4. About Digital Ocean: I do not hold any grief against DO, it was a human a process error, it happens to everyone, big and small companies. What matters is that they fixed quickly, did a postmortem on it, and acted to prevent this situation in the future. Nobody got killed, no data got lost, and we all learned from this event. Hope this clarifies the story a bit. And please stop harassing me about multi-cloud backups :s

znpy 4y ago
Hi Nic, As a person that has been watching this kind of horror stories from outside, I just wonder why people choose to make business with companies that have no human in the loop when stuff gets serious. AFAIK, for AWS for example, when things get serious you are prompted to upgrade to a the big boi support plan (1.5k$/month) and are assigned a TAM (Technical Account Manager). In a situation like that, if you were running on AWS, a TAM or any human in general would have probably fixed the whole thing in like 30 minutes (if anything, because they would know you and that your usage is legitimate).

jesterson 4y ago

Another reminder for not having provider as single point of failure. Being in the industry for years I've heard similar stories for XXXX (insert your favourite, extremely "friendly" and reliable provider). Never build your service on single provider. At least 2, preferably 3+ for redundancy.