11 ms·

Don't use Digital Ocean (DO) for production

120 points by vfulco2 5y ago

TLDR; account locked out suddenly without any follow-up to my inquiries from customer service over 48+ hours. Sadly, I used to love DO for their balanced pricing, range of services, docs, online written material. I thought they were among the best of the best. Long story short, I've had an account with them for roughly 5+ years as I recall, good credit card on file, monthly bill around $20-50, US address. I use a few droplets to run dokku/docker containers for dev and 1 private Google Outline VPN endpoint, only for me. I only ssh into these from non-root and root should be turned off completely (but I'm only a hacker and not a F/T systems person). Not sure if it matters but I don't do anything nefarious on these droplets-- no webscraping, no botnet creation, no crypto mining. I don't know what else would be considered verboten!?! Try to log in with github, won't work and asks for email login. Try to login from there, and it asks for credit card info for "authorization" although the card is the one on file and being charged for so many years. No recourse, and what a crappy way to treat clients even if they are small potatoes in the grand scheme of things. Thankfully I am building a new MVP all on AWS and have my company prod there too running for 5 years. So I don't have to put up with unprofessional dealings and major risks of service interruption. Assume through no fault of your own that you could be shut out of everything. You have been warned.

NicoJuicy 5y ago
This isn't really a complaint and i was lucky that it was my personal stuff. But somehow a droplet was part of a DDOS and network was booted off ( 1 network spike for 10 minutes). Let's just say it was stressful and very unwelcome at the time. I also went to mention that i did receive feedback and there was definitely a human from their end involved. But, I wouldn't want to imagine what would have happened if it were websites of clients. I took more measures by now, but i think the actual culprit was a contractor that did some work for me. I'm wondering how other providers handle that ( a VPS for example)? Anyone has some insights on how it can be handled better?
- kbenson 5y ago
  > I'm wondering how other providers handle that? Depending on what you mean by "booted off", any and every reputable provider will do the same. They all have AUPs that allow them to stop routing for your, at a minimum, if you're found to be sourcing traffic that's against their AUP. That's all normal. You can't expect a company to keep you online when you're causing problems for them and the rest of the internet, and their contracts/AUPs are all worded so they can take action if your systems are found to be a problem. What can you do to prevent this? Don't let your systems be part of a botnet or get hacked in general. More feasibly, stay up to date on all security issues and take multiple system backups so if you have to restore from a good prior backup, if that's possible. > I took more measures by now, but i think the actual culprit was a contractor that did some work for me. Could be, but my guess is not. Botnets survive on having massive numbers of servers in them, and are often automated exploits. More likely, you were running some exploitable software, and it was exploited. Updates need to be run, daemons often need to be restarted after updates because of older code still running in memory, etc. And if you were running some CMS like Wordpress or Drupal, well if you aren't doing it yourself and keeping on top of updates and new versions, you either better have someone on retainer to do it for you or give up and do something else, because you're a perfect target for a botnet at that point because those generally have problems on a regular basis (as well as the problems introduced by different modules they each might support).
  - NicoJuicy 5y ago
    > booted off The context is missing without my original sentence. "Network was booted off." Obviously not my account but "network access" and i could only connect through the console through their web dashboard. > Could be, but my guess is not. According to the files i found and the zipfiles he left behind. There would be a huge confidence of similar timing and folders involved. He even named the zip similar to the work he did. I also know how botnets work, backups and security updates. I already took measures as mentioned before. I never said and/or blamed DO for it. I'm not running sites on it, but some personal stuff as i mentioned ( again). I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better... Tbh. I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved. Which is really frustrating to respond too... As you're not talking to me, but a reader of the comments.
    - kbenson 5y ago
      > The context is missing without my original sentence. "Network was booted off." I didn't take it as written because it was too ambiguous to know. Often your actual IP or network range might be null routed from upstream providers. Or perhaps DO just turns off routing for the systems in question. Or it could be a poor description of something else. I work at an ISP. We will null route colocated customer IP addresses if they are causing problems until they resolve them and contact us, or if they are the target of a DDOS. The point of null routing is that other providers won't even send you the traffic anymore, so it neatly solves the problem. Sometimes we have to do it for IPs for our own services if they are the target of a DDOS. > I never said and/or blamed DO for it. I'm just letting you know how network providers work. If you cause hard to the network, and all their other customers, whether on purpose or just because you're the person responsible for the resources in question, they will shut you down. That's expected. It's a good thing they do, otherwise none of these providers could even offer useful service because every day there's be an event destroying their network, and your experience every day prior to the problem would have been crappy service. > I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better... The correct way to handle your service being part of a botnet is to immediately remove your server from the network, whether they do it or you do. You server is causing active harm to other people on the internet while it's left online in that state. It sucks for you that it goes down, but really, that's how the world works. When something you control is causing harm to the public, it will be dealt with whether you like how it happens or not. The way to deal with that problem is to do everything in your power to prevent if from getting to that state. Because of that, iit's impossible to be absolutely secure from a network provider shutting down your service. If it's that important, you need to have backup plans, le other services you can can bring up at a moment's notice and swing DNS to. People make good money designing and implementing disaster recovery and business continuity plans like this. That's likely not feasible for a personal site. But usually people understand and can deal with a few hours or even days of downtime for personal sites, while a business might not recover as well from that. > I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved. I think perhaps you didn't understand what I was trying to express. What you experienced is normal and the correct and expected way network providers will deal with a problem like that. I tried to express that. As for whether your consultant installed a botnet on your system, it's possible I thought it unlikely because the amount they could gain from one extra hour of repeat business from you likely dwarfs the gain from a single extra botnet computer, of which botnet networks have thousands. There's just not a lot of incentive for a consultant to put a system they're working on on a botnet. I would rate it as more likely that something the consultant did left the system in a vulnerable state, but it doesn't really matter does it? I wasn't trying to attack you or indicate you screwed up, just indicate that if your system was on a botnet it probably got there like 99.999% of other systems on them, which is some automated work that infects it, adds a control mechanism, and then scans to add other hosts. Or some script kiddie ran some scanner/infector tool and added the control mechanism to join it to a botnet. Because the time to do it manually isn't really worth the one extra system in a lot of cases, I would imagine.
- marcosdumay 5y ago
  I have made a mistake in a Linode VPS once, and it was coopted into a botnet. When they detected the problem, they took it offline and I received an email from the support. As soon as it was fixed, I emailed them and they put it online again.
JimDabell 5y ago
Something like this has happened before: DigitalOcean Killed Our Company: https://news.ycombinator.com/item?id=20064169 https://news.ycombinator.com/item?id=20064169
- stjohnswarts 5y ago
  This is why you should use FOSS as much as possible. So what if digital ocean shuts you down? You have a backed up copy at least once a day of your data at least once a day on off premises backup, right? You just move to another provider and make sure to tweet and HN about how awful DO (or whoever is) so they lose customers and your company keeps moving alone with a 1 day outage.
- rapind 5y ago
  This is concerning. We were considering a move to DO from Huroku to save quite a few $, but I’m thinking AWS now. I dislike the convoluted mess of a UI that AWS presents, but this happening would be way worse.
  - tailspin2019 5y ago
    I’m in a similar position at the moment and I’d suggest if you were considering DO, to take a look at Vultr and UpCloud. I’ve found support response time from both so far to be excellent (UpCloud in particular though their feature set is just slightly behind Vultr and DO currently). I had been testing the water with DO with a view to hosting some production stuff with them, but I’ve seen a few too many similar stories like this one and decided to get out of DO and rule them out for production stuff for now. Many of their direct competitors seem to provide much better support at lower levels of spend too.
  - usr1106 5y ago
    I use AWS for a hobby project. It just costs a couple of Euros a month. I am in EU and according to PSD2 every payment would need strong authentication. AWS warns about this but my bank has never required it. I always fear some day they will require the authentication and just these days I will be late with checking mails/accounts, wherever this will turn up and I will notice only when AWS has shut down my server and deleted my data. I would happily make a 20 or 50 EUR/USD payment in advance to avoid the risk. But AWS is not flexible enough to allow me to do so. I understand in some countries it's possible, but in EU there are no options.
maxk42 5y ago
I've been having this problem a lot recently. First Rackspace, later CloudFlare. Most recently Twitter accused me of being a robot and asked me to verify that I'm a human by posting a government-issued ID even though I rarely use it and I've never posted a comment there and could not have possibly broken any policies. Sorry that this is happening to you. I think tech companies at large are really failing individuals. Even if it's only 1 - 2% of users annually, it adds up across the dozens of services we use day-in, day-out for years and it really isn't good.
- adventured 5y ago
  It's not 1-2% of users annually. It's a small fraction of that. When it happens people tend to get loud about it though, so it shows up on social media or on HN and similar. Publicly traded companies like DigitalOcean aren't about to wipe out 2% of their paying customers by mistake every year; the problem is small enough relatively speaking that they don't much care (unless the customer can get enough attention to their plight), despite how it impacts the customer that is suffering from their negligence.
garbagecoder 5y ago
Reminder that the cloud is just someone else's computer.
- hpoe 5y ago
  In the same way that Uber is just someone else's car.
  - helaoban 5y ago
    Would you be willing to to sell all of your vehicles and sign a time-charter contract with Uber to manage all of your ground transportation for the foreseeable future?
    - GianFabien 5y ago
      Short answer: No $%^&* way !
    - hpoe 5y ago
      No but I am a single individual. If I was a company that produced widgets I would use FedEx or UPS rather than maintaining my own fleet of trucks, drivers, maintenance staff, etc.
- gary_0 5y ago
  Unless you have the up-front cash, time, and know-how to get your own box in a datacenter, isn't it the most logical option to rent something? The problem seems to be the market for servers is so large now, and incumbents have so many customers, they can afford to kick random people to the curb with no repercussions.
  - garbagecoder 5y ago
    It would be if cloud providers didn't keep changing the deal all the time. Ownership is control and control comes with a premium.
- usr1106 5y ago
pram 5y ago
I had something similar but it was DMCA complaints for a forum I ran. I complied with the deletions but after a handful they just deleted my stuff. I moved to OVH. What a joke.
BabyJohn 5y ago
Don't worry vfulco2, you've reached the front page of HN, the official support portal of Digital Ocean. Your account will be restored within two hours.
- ronsor 5y ago
  Woah, I thought it was just the support portal for Google
  - newsbinator 5y ago
    Also Facebook / Instagram accounts. And in my case, Stripe too.
- vfulco2 5y ago
  Thank you. I posted an update, let me know if there is anything else I should do.
- lovelearning 5y ago
  Even the estimated time was right. I'm impressed! HN does seem to be the best customer service hub for tech cos.
- joshxyz 5y ago
  HAHAHAHAHA. What a classic.
starburst 5y ago
That's very scary, I am building an MVP currently on DO and although I have daily backups of all the data pushed to another cloud provider as a safety, seeing how they closed your account suddenly out of nowhere with no recourse I feel like I should move out of there. This is insane, if I wouldn't have those backups and the same thing happened to me, I would be screwed. How come they don't contact you beforehand, or at least let you move out your data (keep the account open but cut the droplets from public internet if for whatever reason their algorithm detected nefarious activity on it, or a bogus DMCA, exploits happens, etc). I wanted to try an alternative to AWS but look like I'll be back to it soon. Does AWS have a similar track record of closing account out of the blue? Is there any other alternative that have better support and would at the very least try to contact the owner before taking drastic action like this?
tsian2 5y ago
I tried them after reading here about how their support is excellent and had an interesting one-day experience. Not long after logging in for the first time their site started having some issues, such as the GUI being jumbled because the asset weren't loading correctly. It went down officially a minute later and I didn't have time to hang around until it came back up. I got charged for the things that had enabled themselves or failed to really disable during that mess, so I asked for the money back. The first response gave me a guide to closing my account, so I tried again. The second response was to refund me some loose change (some dollars and cents).
yosito 5y ago
This is par for the course with free services, but this can't be legal if you're a paying customer and have signed agreement with them to provide you services in exchange for payment.
hgs3 5y ago
What does HN recommend these days for app hosting? I'm a solo dev without deep pockets. I was looking at Linode and DO, but posts like this have me worried about the later.
- i_am_proteus 5y ago
  I have never had any problems with Hetzner, but I suspect someone who has will reply to this comment.
- justrudd 5y ago
  YMMV - but I’ve had 2 wordpress blogs on Linode for 14 years now. I’ve spun up (and spun down) 3 different companies on Linode (all monoliths using Postgres, Memcached and Redis). I’ll be with them as long as I can. During this time, I can really only recall 2 times they emailed me out of the blue and said something was wrong with one my hosts. Both times were a couple of hours of downtime to which they credited me the time. I’m sure I’ve had more downtime, but it must not have been too bad because it isn’t readily available from memory.
- electroly 5y ago
  AWS Lightsail. Linode and DO are not cheaper. Seriously, compare the prices for yourself [0][1][2]. Linode and DO seem to have a reputation of being cheaper options, but they're not, and AWS isn't going to jerk you around with small-hoster issues like Linode and DO have been known to. [0] https://aws.amazon.com/lightsail/pricing/ https://aws.amazon.com/lightsail/pricing/ [1] https://www.digitalocean.com/pricing https://www.digitalocean.com/pricing [2] https://www.linode.com/pricing/ https://www.linode.com/pricing/
  - jbboehr 5y ago
    I've inadvertently done something against Linode's ToS twice and both times they contacted me to get the issue resolved, my account was never locked. So take this as a +1 for them, at least on this front.
  - m0lecules 5y ago
    What is a "small-hoster issue"?
Zababa 5y ago
I've heard stuff like that for pretty much any hosting company. Are there any companies that don't do that? How do they guarantee it?
bogota 5y ago
This could be written about any cloud provider. I for instance have had a much better time using DO than other cloud services but it’s all anecdotal. If your trying to optimize for customer support buy and run your own datacenter.
larrysalibra 5y ago
This post made me think about the personal DO droplets I have and how I’m backing them up. I’m currently using the built in back up feature…but that doesn’t protect against DO preventing me from accessing my backups. What does everyone do to easily and automatically back up their side project droplets offsite?
- vorpalhex 5y ago
  Rclone
- starburst 5y ago
  I have a daily CRON job, a simple script that backup the database with `pg_dump` and send the file to an S3 bucket (keeping the last X files) and also use `rclone` to sync my files on "Spaces" (DO clone of S3) to an S3 bucket. It's supposed to be a last resort kind of backup as I also use DO backup solution (managed database) but seeing how they can close an account out of nowhere, it ain't worth nothing.
  - larrysalibra 5y ago
    Thanks! That's super helpful!
  - toomanydoubts 5y ago
    I hope you periodically restore the off-premises backup to check they're working too. Not really fun to discover your backup is broken when you need it the most.
    - starburst 5y ago
      I admit I don't check them often enough, but I did makes sure to verify after this post!
- neurostimulant 5y ago
  Daily database dumps to a folder, followed by a rsync (database and users' files) to a 3rd party backup services (rsync.net). I used backblaze b2 in the past, but found that accessing historical backup data is easier on rsync.net. Just a simple sftp into the snapshot directory.
implying 5y ago
Digitalocean will automatically lock any account I create, because of an (admitted!) false positive in their fraud software. Literally any IP, any credit card, any email address I use- seconds after taking out a test deposit on the card, the account is locked. Every time I've messaged support for an account lock, 24 hours later I'm informed it was a false positive and the account is unlocked. Needless to say, this does not inspire confidence that my resources are going to be kept up and running past Kafkaesque fraud monitoring systems
- ffhhj 5y ago
  Sorry for that Mr. '); DROP TABLE...
  - cheschire 5y ago
    Little Bobby Tables, we call him.
- usr1106 5y ago
  > Every time I've messaged support for an account lock, 24 hours later I'm informed it was a false positive and the account is unlocked. Sounds like good customer service to me ;) I tried to open an account with Oracle cloud and it rejected my credit card. They never even answered to my messages. Well, I was not overly convinced I want to do business with Oracle anyway, so now AWS charges the same card every month. Oracle seemed somewhat cheaper for the use case, but nothing that kills me.
- nitn 5y ago
  Can confirm. I created an account for a client but I was using a VPN to avoid conflicting with my own personal account. It was blocked but DO unlocked it once I explained it in an email to them.
danaris 5y ago
Does anyone know of a similar VPS provider that a) is responsive in situations like the OP's, and b) is not on the UCEPROTECTL3 email blocklist? I've been happy with DigitalOcean for a while now, but reports like this concern me, and I've also discovered after investigating that the reason MS-hosted email accounts (@hotmail.com, @live.com, etc) reject all emails from my VPS is because the entire provider is on the UCEPROTECT blocklist, and apparently has no interest in changing this.
vfulco2 5y ago
*Update* to their credit, they've followed up with me. The github login was routing to the wrong account (a rarely used account) and was caught in their automatic fraud process. They've verified the main account (5 year old) is still intact and given me instructions to get back in. I am immensely relieved. They've done right by me.
- tailspin2019 5y ago
  Good to hear! Did they explain why it took over 48 hours to reply to you?
- AnonHP 5y ago
  So the HN post did work and you’re going back to a platform that treated you badly? If I were you, I’d copy what needs to be copied from there and close all accounts. Encouraging “customer support only through social media outrage” is not a very good choice. If you continue, you wouldn’t be doing right by the other customers of this provider.
  - ingvul 5y ago
    Which platform is exempt from this? I have heard similar stories from GCP, AWS, Linode, etc. I would even say: the bigger the platform, the worse the customer support (i.e., Google)
phendrenad2 5y ago
I used to like Digital Ocean. But it seems as though ever since they went IPO all the engineers became rich and absconded to Hawaii. Or they're at least not working. It's Noam Bardin's Law in effect. I'm looking for the next Digital Ocean. Something still bespoke, with higher cost but better customer support. So far oldschool VPS hosts are looking good.
AnonHP 5y ago
Don’t use Digital Ocean for anything, not just production. Your account may be locked up and customer service will keep sending boilerplate responses. There is no way I know of to contact a human and find out why the account is locked and what options exist. DO has been heavily understaffed for years, and the mass layoffs in early 2020 likely made things far worse. I see value in supporting smaller companies (relative to large ones like Amazon, Microsoft or Google), but smaller companies usually tend to have better customer support when things go wrong, and are generally a lot more helpful. With DO, it has the kind of offerings by a smaller company with the kind of support by a large company.
vfulco2 5y ago
*Update* to their credit, they've followed up with me. The github login was routing to the wrong account (a rarely used account) and was caught in their automatic fraud process. They've verified the main account (5 year old) is still intact and given me instructions to get back in. I am immensely relieved. They've done right by me. Kudos for a delayed but excellent response.
eaplmx 5y ago
It happened similar to us. A payment arrived late and everything was deleted. I promised to never use DO again, but the low price makes sense for a small project being made by an indie games company, so here we are again. Luckily for us we added redundancy on payments and alerts, so I hope we won't be in the same case again.

120 points by vfulco2 5y ago

NicoJuicy 5y ago

This isn't really a complaint and i was lucky that it was my personal stuff. But somehow a droplet was part of a DDOS and network was booted off ( 1 network spike for 10 minutes). Let's just say it was stressful and very unwelcome at the time. I also went to mention that i did receive feedback and there was definitely a human from their end involved. But, I wouldn't want to imagine what would have happened if it were websites of clients. I took more measures by now, but i think the actual culprit was a contractor that did some work for me. I'm wondering how other providers handle that ( a VPS for example)? Anyone has some insights on how it can be handled better?

kbenson 5y ago
> I'm wondering how other providers handle that? Depending on what you mean by "booted off", any and every reputable provider will do the same. They all have AUPs that allow them to stop routing for your, at a minimum, if you're found to be sourcing traffic that's against their AUP. That's all normal. You can't expect a company to keep you online when you're causing problems for them and the rest of the internet, and their contracts/AUPs are all worded so they can take action if your systems are found to be a problem. What can you do to prevent this? Don't let your systems be part of a botnet or get hacked in general. More feasibly, stay up to date on all security issues and take multiple system backups so if you have to restore from a good prior backup, if that's possible. > I took more measures by now, but i think the actual culprit was a contractor that did some work for me. Could be, but my guess is not. Botnets survive on having massive numbers of servers in them, and are often automated exploits. More likely, you were running some exploitable software, and it was exploited. Updates need to be run, daemons often need to be restarted after updates because of older code still running in memory, etc. And if you were running some CMS like Wordpress or Drupal, well if you aren't doing it yourself and keeping on top of updates and new versions, you either better have someone on retainer to do it for you or give up and do something else, because you're a perfect target for a botnet at that point because those generally have problems on a regular basis (as well as the problems introduced by different modules they each might support).
- NicoJuicy 5y ago
  > booted off The context is missing without my original sentence. "Network was booted off." Obviously not my account but "network access" and i could only connect through the console through their web dashboard. > Could be, but my guess is not. According to the files i found and the zipfiles he left behind. There would be a huge confidence of similar timing and folders involved. He even named the zip similar to the work he did. I also know how botnets work, backups and security updates. I already took measures as mentioned before. I never said and/or blamed DO for it. I'm not running sites on it, but some personal stuff as i mentioned ( again). I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better... Tbh. I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved. Which is really frustrating to respond too... As you're not talking to me, but a reader of the comments.
  - kbenson 5y ago
    > The context is missing without my original sentence. "Network was booted off." I didn't take it as written because it was too ambiguous to know. Often your actual IP or network range might be null routed from upstream providers. Or perhaps DO just turns off routing for the systems in question. Or it could be a poor description of something else. I work at an ISP. We will null route colocated customer IP addresses if they are causing problems until they resolve them and contact us, or if they are the target of a DDOS. The point of null routing is that other providers won't even send you the traffic anymore, so it neatly solves the problem. Sometimes we have to do it for IPs for our own services if they are the target of a DDOS. > I never said and/or blamed DO for it. I'm just letting you know how network providers work. If you cause hard to the network, and all their other customers, whether on purpose or just because you're the person responsible for the resources in question, they will shut you down. That's expected. It's a good thing they do, otherwise none of these providers could even offer useful service because every day there's be an event destroying their network, and your experience every day prior to the problem would have been crappy service. > I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better... The correct way to handle your service being part of a botnet is to immediately remove your server from the network, whether they do it or you do. You server is causing active harm to other people on the internet while it's left online in that state. It sucks for you that it goes down, but really, that's how the world works. When something you control is causing harm to the public, it will be dealt with whether you like how it happens or not. The way to deal with that problem is to do everything in your power to prevent if from getting to that state. Because of that, iit's impossible to be absolutely secure from a network provider shutting down your service. If it's that important, you need to have backup plans, le other services you can can bring up at a moment's notice and swing DNS to. People make good money designing and implementing disaster recovery and business continuity plans like this. That's likely not feasible for a personal site. But usually people understand and can deal with a few hours or even days of downtime for personal sites, while a business might not recover as well from that. > I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved. I think perhaps you didn't understand what I was trying to express. What you experienced is normal and the correct and expected way network providers will deal with a problem like that. I tried to express that. As for whether your consultant installed a botnet on your system, it's possible I thought it unlikely because the amount they could gain from one extra hour of repeat business from you likely dwarfs the gain from a single extra botnet computer, of which botnet networks have thousands. There's just not a lot of incentive for a consultant to put a system they're working on on a botnet. I would rate it as more likely that something the consultant did left the system in a vulnerable state, but it doesn't really matter does it? I wasn't trying to attack you or indicate you screwed up, just indicate that if your system was on a botnet it probably got there like 99.999% of other systems on them, which is some automated work that infects it, adds a control mechanism, and then scans to add other hosts. Or some script kiddie ran some scanner/infector tool and added the control mechanism to join it to a botnet. Because the time to do it manually isn't really worth the one extra system in a lot of cases, I would imagine.
marcosdumay 5y ago
I have made a mistake in a Linode VPS once, and it was coopted into a botnet. When they detected the problem, they took it offline and I received an email from the support. As soon as it was fixed, I emailed them and they put it online again.

JimDabell 5y ago

Something like this has happened before: DigitalOcean Killed Our Company: https://news.ycombinator.com/item?id=20064169 https://news.ycombinator.com/item?id=20064169

stjohnswarts 5y ago
This is why you should use FOSS as much as possible. So what if digital ocean shuts you down? You have a backed up copy at least once a day of your data at least once a day on off premises backup, right? You just move to another provider and make sure to tweet and HN about how awful DO (or whoever is) so they lose customers and your company keeps moving alone with a 1 day outage.
rapind 5y ago
This is concerning. We were considering a move to DO from Huroku to save quite a few $, but I’m thinking AWS now. I dislike the convoluted mess of a UI that AWS presents, but this happening would be way worse.
- tailspin2019 5y ago
  I’m in a similar position at the moment and I’d suggest if you were considering DO, to take a look at Vultr and UpCloud. I’ve found support response time from both so far to be excellent (UpCloud in particular though their feature set is just slightly behind Vultr and DO currently). I had been testing the water with DO with a view to hosting some production stuff with them, but I’ve seen a few too many similar stories like this one and decided to get out of DO and rule them out for production stuff for now. Many of their direct competitors seem to provide much better support at lower levels of spend too.
- usr1106 5y ago
  I use AWS for a hobby project. It just costs a couple of Euros a month. I am in EU and according to PSD2 every payment would need strong authentication. AWS warns about this but my bank has never required it. I always fear some day they will require the authentication and just these days I will be late with checking mails/accounts, wherever this will turn up and I will notice only when AWS has shut down my server and deleted my data. I would happily make a 20 or 50 EUR/USD payment in advance to avoid the risk. But AWS is not flexible enough to allow me to do so. I understand in some countries it's possible, but in EU there are no options.

maxk42 5y ago

I've been having this problem a lot recently. First Rackspace, later CloudFlare. Most recently Twitter accused me of being a robot and asked me to verify that I'm a human by posting a government-issued ID even though I rarely use it and I've never posted a comment there and could not have possibly broken any policies. Sorry that this is happening to you. I think tech companies at large are really failing individuals. Even if it's only 1 - 2% of users annually, it adds up across the dozens of services we use day-in, day-out for years and it really isn't good.

adventured 5y ago
It's not 1-2% of users annually. It's a small fraction of that. When it happens people tend to get loud about it though, so it shows up on social media or on HN and similar. Publicly traded companies like DigitalOcean aren't about to wipe out 2% of their paying customers by mistake every year; the problem is small enough relatively speaking that they don't much care (unless the customer can get enough attention to their plight), despite how it impacts the customer that is suffering from their negligence.

garbagecoder 5y ago

Reminder that the cloud is just someone else's computer.

hpoe 5y ago
In the same way that Uber is just someone else's car.
- helaoban 5y ago
  Would you be willing to to sell all of your vehicles and sign a time-charter contract with Uber to manage all of your ground transportation for the foreseeable future?
  - GianFabien 5y ago
    Short answer: No $%^&* way !
  - hpoe 5y ago
    No but I am a single individual. If I was a company that produced widgets I would use FedEx or UPS rather than maintaining my own fleet of trucks, drivers, maintenance staff, etc.
gary_0 5y ago
Unless you have the up-front cash, time, and know-how to get your own box in a datacenter, isn't it the most logical option to rent something? The problem seems to be the market for servers is so large now, and incumbents have so many customers, they can afford to kick random people to the curb with no repercussions.
- garbagecoder 5y ago
  It would be if cloud providers didn't keep changing the deal all the time. Ownership is control and control comes with a premium.
usr1106 5y ago

pram 5y ago

I had something similar but it was DMCA complaints for a forum I ran. I complied with the deletions but after a handful they just deleted my stuff. I moved to OVH. What a joke.

BabyJohn 5y ago

Don't worry vfulco2, you've reached the front page of HN, the official support portal of Digital Ocean. Your account will be restored within two hours.

ronsor 5y ago
Woah, I thought it was just the support portal for Google
- newsbinator 5y ago
  Also Facebook / Instagram accounts. And in my case, Stripe too.
vfulco2 5y ago
Thank you. I posted an update, let me know if there is anything else I should do.
lovelearning 5y ago
Even the estimated time was right. I'm impressed! HN does seem to be the best customer service hub for tech cos.
joshxyz 5y ago
HAHAHAHAHA. What a classic.

starburst 5y ago

That's very scary, I am building an MVP currently on DO and although I have daily backups of all the data pushed to another cloud provider as a safety, seeing how they closed your account suddenly out of nowhere with no recourse I feel like I should move out of there. This is insane, if I wouldn't have those backups and the same thing happened to me, I would be screwed. How come they don't contact you beforehand, or at least let you move out your data (keep the account open but cut the droplets from public internet if for whatever reason their algorithm detected nefarious activity on it, or a bogus DMCA, exploits happens, etc). I wanted to try an alternative to AWS but look like I'll be back to it soon. Does AWS have a similar track record of closing account out of the blue? Is there any other alternative that have better support and would at the very least try to contact the owner before taking drastic action like this?

tsian2 5y ago

I tried them after reading here about how their support is excellent and had an interesting one-day experience. Not long after logging in for the first time their site started having some issues, such as the GUI being jumbled because the asset weren't loading correctly. It went down officially a minute later and I didn't have time to hang around until it came back up. I got charged for the things that had enabled themselves or failed to really disable during that mess, so I asked for the money back. The first response gave me a guide to closing my account, so I tried again. The second response was to refund me some loose change (some dollars and cents).

yosito 5y ago

This is par for the course with free services, but this can't be legal if you're a paying customer and have signed agreement with them to provide you services in exchange for payment.

hgs3 5y ago

What does HN recommend these days for app hosting? I'm a solo dev without deep pockets. I was looking at Linode and DO, but posts like this have me worried about the later.

i_am_proteus 5y ago
I have never had any problems with Hetzner, but I suspect someone who has will reply to this comment.
justrudd 5y ago
YMMV - but I’ve had 2 wordpress blogs on Linode for 14 years now. I’ve spun up (and spun down) 3 different companies on Linode (all monoliths using Postgres, Memcached and Redis). I’ll be with them as long as I can. During this time, I can really only recall 2 times they emailed me out of the blue and said something was wrong with one my hosts. Both times were a couple of hours of downtime to which they credited me the time. I’m sure I’ve had more downtime, but it must not have been too bad because it isn’t readily available from memory.
electroly 5y ago
AWS Lightsail. Linode and DO are not cheaper. Seriously, compare the prices for yourself [0][1][2]. Linode and DO seem to have a reputation of being cheaper options, but they're not, and AWS isn't going to jerk you around with small-hoster issues like Linode and DO have been known to. [0] https://aws.amazon.com/lightsail/pricing/ https://aws.amazon.com/lightsail/pricing/ [1] https://www.digitalocean.com/pricing https://www.digitalocean.com/pricing [2] https://www.linode.com/pricing/ https://www.linode.com/pricing/
- jbboehr 5y ago
  I've inadvertently done something against Linode's ToS twice and both times they contacted me to get the issue resolved, my account was never locked. So take this as a +1 for them, at least on this front.
- m0lecules 5y ago
  What is a "small-hoster issue"?

Zababa 5y ago

I've heard stuff like that for pretty much any hosting company. Are there any companies that don't do that? How do they guarantee it?

bogota 5y ago

This could be written about any cloud provider. I for instance have had a much better time using DO than other cloud services but it’s all anecdotal. If your trying to optimize for customer support buy and run your own datacenter.

larrysalibra 5y ago

This post made me think about the personal DO droplets I have and how I’m backing them up. I’m currently using the built in back up feature…but that doesn’t protect against DO preventing me from accessing my backups. What does everyone do to easily and automatically back up their side project droplets offsite?

vorpalhex 5y ago
Rclone
starburst 5y ago
I have a daily CRON job, a simple script that backup the database with `pg_dump` and send the file to an S3 bucket (keeping the last X files) and also use `rclone` to sync my files on "Spaces" (DO clone of S3) to an S3 bucket. It's supposed to be a last resort kind of backup as I also use DO backup solution (managed database) but seeing how they can close an account out of nowhere, it ain't worth nothing.
- larrysalibra 5y ago
  Thanks! That's super helpful!
- toomanydoubts 5y ago
  I hope you periodically restore the off-premises backup to check they're working too. Not really fun to discover your backup is broken when you need it the most.
  - starburst 5y ago
    I admit I don't check them often enough, but I did makes sure to verify after this post!
neurostimulant 5y ago
Daily database dumps to a folder, followed by a rsync (database and users' files) to a 3rd party backup services (rsync.net). I used backblaze b2 in the past, but found that accessing historical backup data is easier on rsync.net. Just a simple sftp into the snapshot directory.

implying 5y ago

Digitalocean will automatically lock any account I create, because of an (admitted!) false positive in their fraud software. Literally any IP, any credit card, any email address I use- seconds after taking out a test deposit on the card, the account is locked. Every time I've messaged support for an account lock, 24 hours later I'm informed it was a false positive and the account is unlocked. Needless to say, this does not inspire confidence that my resources are going to be kept up and running past Kafkaesque fraud monitoring systems

ffhhj 5y ago
Sorry for that Mr. '); DROP TABLE...
- cheschire 5y ago
  Little Bobby Tables, we call him.
usr1106 5y ago
> Every time I've messaged support for an account lock, 24 hours later I'm informed it was a false positive and the account is unlocked. Sounds like good customer service to me ;) I tried to open an account with Oracle cloud and it rejected my credit card. They never even answered to my messages. Well, I was not overly convinced I want to do business with Oracle anyway, so now AWS charges the same card every month. Oracle seemed somewhat cheaper for the use case, but nothing that kills me.
nitn 5y ago
Can confirm. I created an account for a client but I was using a VPN to avoid conflicting with my own personal account. It was blocked but DO unlocked it once I explained it in an email to them.

danaris 5y ago

Does anyone know of a similar VPS provider that a) is responsive in situations like the OP's, and b) is not on the UCEPROTECTL3 email blocklist? I've been happy with DigitalOcean for a while now, but reports like this concern me, and I've also discovered after investigating that the reason MS-hosted email accounts (@hotmail.com, @live.com, etc) reject all emails from my VPS is because the entire provider is on the UCEPROTECT blocklist, and apparently has no interest in changing this.

vfulco2 5y ago

*Update* to their credit, they've followed up with me. The github login was routing to the wrong account (a rarely used account) and was caught in their automatic fraud process. They've verified the main account (5 year old) is still intact and given me instructions to get back in. I am immensely relieved. They've done right by me.

tailspin2019 5y ago
Good to hear! Did they explain why it took over 48 hours to reply to you?
AnonHP 5y ago
So the HN post did work and you’re going back to a platform that treated you badly? If I were you, I’d copy what needs to be copied from there and close all accounts. Encouraging “customer support only through social media outrage” is not a very good choice. If you continue, you wouldn’t be doing right by the other customers of this provider.
- ingvul 5y ago
  Which platform is exempt from this? I have heard similar stories from GCP, AWS, Linode, etc. I would even say: the bigger the platform, the worse the customer support (i.e., Google)

phendrenad2 5y ago

I used to like Digital Ocean. But it seems as though ever since they went IPO all the engineers became rich and absconded to Hawaii. Or they're at least not working. It's Noam Bardin's Law in effect. I'm looking for the next Digital Ocean. Something still bespoke, with higher cost but better customer support. So far oldschool VPS hosts are looking good.

AnonHP 5y ago

Don’t use Digital Ocean for anything, not just production. Your account may be locked up and customer service will keep sending boilerplate responses. There is no way I know of to contact a human and find out why the account is locked and what options exist. DO has been heavily understaffed for years, and the mass layoffs in early 2020 likely made things far worse. I see value in supporting smaller companies (relative to large ones like Amazon, Microsoft or Google), but smaller companies usually tend to have better customer support when things go wrong, and are generally a lot more helpful. With DO, it has the kind of offerings by a smaller company with the kind of support by a large company.

vfulco2 5y ago

eaplmx 5y ago

It happened similar to us. A payment arrived late and everything was deleted. I promised to never use DO again, but the low price makes sense for a small project being made by an indie games company, so here we are again. Luckily for us we added redundancy on payments and alerts, so I hope we won't be in the same case again.