4 ms·

How-to take down any DigitalOcean-hosted website.

31 points by pjbrunet 13y ago

Simply send an abuse complaint. Digital Ocean doesn't even read or verify the complaint. They automatically power down the server (within about 30 mins) whether or not the complaint refers to actual URLs on the server. This just happened to one of my clients today (hour ago) and Digital Ocean is not responding to support tickets and not responding on Twitter either.

pjbrunet 13y ago
This is the email you get 30 mins before they shut you down: "Oh no! We've found an issue with your account and issued you a new ticket that needs to be addressed as soon as possible. Please login to view the ticket: https://www.digitalocean.com/support https://www.digitalocean.com/support Thanks so much, DigitalOcean"
- pjbrunet 13y ago
  After 30 mins you get this: "Account Locked by DigitalOcean Staff We are sorry to inform you that your account has been locked by DigitalOcean Staff. None of your virtual servers have been destroyed. We have opened a support ticket with you regarding the cause of this lock. Please navigate to the Support Page to resolve the issue that has caused your account to be locked. Thank you, DigitalOcean Staff"
pjbrunet 13y ago
The account is unlocked now. Just downloaded a backup in case this happens again. Still no response to why they didn't bother reading the abuse complaint. (Edit) DigitalOcean wrote back a nice response. I'll follow up with them before commenting futher here.
zacharybk 13y ago
Hi PJ, This is Zach from DigitalOcean. I'm very sorry that this was your experience with us. We've replied to your ticket (and Tweet) and fully unlocked your account. We take abuse (DCMA complaints), account security and platform security seriously. Sometimes it's a tough balance, and when accounts are incorrectly locked we take quick action to resolve. Please feel free to let me know if you have any further thoughts: zach at digitalocean.com. Thank you, Zach DigitalOcean Director of Support
- pjbrunet 13y ago
  I feel bad posting this to Hacker News because I like Digital Ocean. But I take downtime seriously. It's not my website that was shut down and I'm saying this on my own, not on behalf of the client/owner. So I can't really comment on the details. I responded to your support ticket already, thank you. This is just my opinion. I already posted some of these suggestions on Twitter. - 30 mins notice before you power down a server sounds extreme. Why not a 24-hour notice? An emergency SMS, tweet or phone call would have been awesome. - Shutting down a server is a big deal. That particular server has several background processes running all day. Maybe I'm dreaming, but if you could have just shut down http (what the complaint was about) and not the entire server, that would have been great. That way I'd still have ssh access to delete files, etc. - I don't understand your policy of (paraphrasing) "Your account will continue to lock and re-lock as long as you have open abuse tickets." Even if the problem is resolved and the account is unlocked by support and everyone is happy, your system continues to re-lock the account until all abuse tickets are closed. The company complaining sends rapid-fire complaints (can't be that uncommon) and clicking "delete ticket" 100 times (waiting for refresh) is annoying. Seems like you could easily fix this by merging all the complaints into one ticket or with some kind of "checkbox" option. - I don't get the impression you have someone at DigitalOcean familiar with abuse complaints. Maybe you do, I don't know. I get the impression you don't read or filter the complaints. Seems like you just press the "off switch" willy nilly.
jMyles 13y ago
annnnd this made me close the tab about using ansible to create a droplet. Back to the pyrax docs!
raiyu 13y ago
Hi, Thanks for inquiring, but this hackernews title is very misleading. We take customer satisfaction very seriously at DigitalOcean. It wasn't one complaint, but 76, and it wasn't immediate, but 4 days with no responses that led to action being taken. Of those, about 50 of them were done on the 25th, for which we received no response. If we do not receive a response for 50+ abuse complaints we unfortunately have to take action, which was done in this case. Hopefully that clears this issue up. Thanks, Moisey Cofounder DigitalOcean
- deleted 13y ago
  [deleted]
- pjbrunet 13y ago
  Moisey, I suggest you review what actually happened. It was not 4 days of no response. According to your own ticket system, the first abuse complaint came in "04/25/14 at 16:10" and the 50 complaints (I did not count) came in rapid-fire succession, several per minute. Do you want a screenshot? I did not see the auto-generated emails you sent till after the client's server was shut down. I became aware of the problem Fri, Apr 25, 2014 at 12:12 PM thanks to Hyperspin, which is when you shut down the server. Not sure of the timezone difference but where are you getting 4 days? Did you even read the tickets? You can see my tweet on the 25th here: "@digitalocean turns out, nothing to do w/ DNS. Account locked due to abuse." 12:43 PM - 25 Apr 2014 https://twitter.com/php_austin/status/459749518464937984 https://twitter.com/php_austin/status/459749518464937984 I checked your status website which said you were having DNS issues. This client recently dropped your DNS servers due to errors so that wasn't the problem. Within a few minutes I found out the account was locked due to some images automatically pulled from "open graph" and Twitter meta tags. (In other words, images that were intended to be shared, but whatever that's another tangent.) Fri, Apr 25, 2014 at 12:44 PM told the client you allowed me to power on the server to delete the images. Then I had to upgrade MySQL because you didn't shut down the server properly. Fri, Apr 25, 2014 at 1:47 PM spoke with the client and he wants to delete the entire category of images so this would never happen again. All the images were in the same directory "rm -rf" was all I needed. So within about one hour of discovering there was a problem it was taken care of. Posted on 04/25/14 at 18:44 (support ticket time) "Thank you for responding to this manner quickly. Please let us know if there is anything else we can do to help." So everything was fine, until (To be continued...)
theshank 13y ago
And this is why I keep recommending Linode to all my clients !
deleted 12y ago
[deleted]
throwa 12y ago
It is troubling to hear that digital-ocean takes arbitrary action to shut down servers without giving atleast 24 hours notice. It is worrisome for me because I am a linode user and only recently created a digital-ocean droplet and after playing with it, I liked their service and was planning to host a new app with them instead of Linode while the existing one's remain with linode. But now I am not sure? Can any digital-ocean staff please provide a link that shows your policy for shutting down servers. I need to be sure this high handedness is not actually in your policy.

31 points by pjbrunet 13y ago

pjbrunet 13y ago

This is the email you get 30 mins before they shut you down: "Oh no! We've found an issue with your account and issued you a new ticket that needs to be addressed as soon as possible. Please login to view the ticket: https://www.digitalocean.com/support https://www.digitalocean.com/support Thanks so much, DigitalOcean"

pjbrunet 13y ago
After 30 mins you get this: "Account Locked by DigitalOcean Staff We are sorry to inform you that your account has been locked by DigitalOcean Staff. None of your virtual servers have been destroyed. We have opened a support ticket with you regarding the cause of this lock. Please navigate to the Support Page to resolve the issue that has caused your account to be locked. Thank you, DigitalOcean Staff"

pjbrunet 13y ago

The account is unlocked now. Just downloaded a backup in case this happens again. Still no response to why they didn't bother reading the abuse complaint. (Edit) DigitalOcean wrote back a nice response. I'll follow up with them before commenting futher here.

zacharybk 13y ago

Hi PJ, This is Zach from DigitalOcean. I'm very sorry that this was your experience with us. We've replied to your ticket (and Tweet) and fully unlocked your account. We take abuse (DCMA complaints), account security and platform security seriously. Sometimes it's a tough balance, and when accounts are incorrectly locked we take quick action to resolve. Please feel free to let me know if you have any further thoughts: zach at digitalocean.com. Thank you, Zach DigitalOcean Director of Support

pjbrunet 13y ago
I feel bad posting this to Hacker News because I like Digital Ocean. But I take downtime seriously. It's not my website that was shut down and I'm saying this on my own, not on behalf of the client/owner. So I can't really comment on the details. I responded to your support ticket already, thank you. This is just my opinion. I already posted some of these suggestions on Twitter. - 30 mins notice before you power down a server sounds extreme. Why not a 24-hour notice? An emergency SMS, tweet or phone call would have been awesome. - Shutting down a server is a big deal. That particular server has several background processes running all day. Maybe I'm dreaming, but if you could have just shut down http (what the complaint was about) and not the entire server, that would have been great. That way I'd still have ssh access to delete files, etc. - I don't understand your policy of (paraphrasing) "Your account will continue to lock and re-lock as long as you have open abuse tickets." Even if the problem is resolved and the account is unlocked by support and everyone is happy, your system continues to re-lock the account until all abuse tickets are closed. The company complaining sends rapid-fire complaints (can't be that uncommon) and clicking "delete ticket" 100 times (waiting for refresh) is annoying. Seems like you could easily fix this by merging all the complaints into one ticket or with some kind of "checkbox" option. - I don't get the impression you have someone at DigitalOcean familiar with abuse complaints. Maybe you do, I don't know. I get the impression you don't read or filter the complaints. Seems like you just press the "off switch" willy nilly.

jMyles 13y ago

annnnd this made me close the tab about using ansible to create a droplet. Back to the pyrax docs!

raiyu 13y ago

Hi, Thanks for inquiring, but this hackernews title is very misleading. We take customer satisfaction very seriously at DigitalOcean. It wasn't one complaint, but 76, and it wasn't immediate, but 4 days with no responses that led to action being taken. Of those, about 50 of them were done on the 25th, for which we received no response. If we do not receive a response for 50+ abuse complaints we unfortunately have to take action, which was done in this case. Hopefully that clears this issue up. Thanks, Moisey Cofounder DigitalOcean

deleted 13y ago
[deleted]
pjbrunet 13y ago
Moisey, I suggest you review what actually happened. It was not 4 days of no response. According to your own ticket system, the first abuse complaint came in "04/25/14 at 16:10" and the 50 complaints (I did not count) came in rapid-fire succession, several per minute. Do you want a screenshot? I did not see the auto-generated emails you sent till after the client's server was shut down. I became aware of the problem Fri, Apr 25, 2014 at 12:12 PM thanks to Hyperspin, which is when you shut down the server. Not sure of the timezone difference but where are you getting 4 days? Did you even read the tickets? You can see my tweet on the 25th here: "@digitalocean turns out, nothing to do w/ DNS. Account locked due to abuse." 12:43 PM - 25 Apr 2014 https://twitter.com/php_austin/status/459749518464937984 https://twitter.com/php_austin/status/459749518464937984 I checked your status website which said you were having DNS issues. This client recently dropped your DNS servers due to errors so that wasn't the problem. Within a few minutes I found out the account was locked due to some images automatically pulled from "open graph" and Twitter meta tags. (In other words, images that were intended to be shared, but whatever that's another tangent.) Fri, Apr 25, 2014 at 12:44 PM told the client you allowed me to power on the server to delete the images. Then I had to upgrade MySQL because you didn't shut down the server properly. Fri, Apr 25, 2014 at 1:47 PM spoke with the client and he wants to delete the entire category of images so this would never happen again. All the images were in the same directory "rm -rf" was all I needed. So within about one hour of discovering there was a problem it was taken care of. Posted on 04/25/14 at 18:44 (support ticket time) "Thank you for responding to this manner quickly. Please let us know if there is anything else we can do to help." So everything was fine, until (To be continued...)

theshank 13y ago

And this is why I keep recommending Linode to all my clients !

deleted 12y ago

[deleted]

throwa 12y ago

It is troubling to hear that digital-ocean takes arbitrary action to shut down servers without giving atleast 24 hours notice. It is worrisome for me because I am a linode user and only recently created a digital-ocean droplet and after playing with it, I liked their service and was planning to host a new app with them instead of Linode while the existing one's remain with linode. But now I am not sure? Can any digital-ocean staff please provide a link that shows your policy for shutting down servers. I need to be sure this high handedness is not actually in your policy.