6 ms·

Do not use Digital Ocean for production

55 points by MoSattler 5y ago

I just wanted to share my short cautionary tale about DO. I am running an application on managed Kubernetes. From a tech perspective, I was quite happy until this happened. I broke my production system by introducing a bug in the k8s infrastructure. So I went over to DO to login and fix it. It sends a 6 digit code to my email for me to enter in order to login. The only problem is, that the email never arrives in my Gmail inbox. I made double and triple sure it's not in spam, and checked that emails arrive normally in my inbox. But it's just not there. I guess there is a bug on DO's side. This is not their 2FA, but something they do when logging in from new browsers/locations, so no backup codes. So I open a ticket, as this is already quite urgent (production is down). But I get no answer. I waited a few hours and opened another ticket. With no success. By now it's be 24h, and I cannot get a hold of anybody at DO. I guess I got what I paid for. Digital Ocean is nice for playing around, but for serious projects one should look elsewhere. EDIT: I found a blog post describing a similar problem. Looks like sometimes DO is just silently blacklisting emails. https://qertoip.medium.com/you-will-randomly-lose-access-to-your-digitalocean-account-2fb7b4d799b5 https://qertoip.medium.com/you-will-randomly-lose-access-to-...

thesausageking 5y ago
If your bill is >$500 / mo, you get their "Business" support level which lets you use their customer success managers. If you're not spending this much, you won't get good support from them or any cloud vendor. AWS, Azure, and GCP all push you to the forums or submitted a ticket which may or may not get read.
- smorgusofborg 5y ago
  True though with a lot of the European ones you get good support and/or can pay explicitly for a level of support. The email disappearing with no trace sounds like Gmail hating DO address blocks though.. Gmail sends only a fraction of things it thinks are spam to the spam folder.
  - MoSattler 5y ago
    Just found a blog post with a similar issue - seems like DO silently blacklists your email in certain cases. https://qertoip.medium.com/you-will-randomly-lose-access-to-your-digitalocean-account-2fb7b4d799b5 https://qertoip.medium.com/you-will-randomly-lose-access-to-...
  - boring_twenties 5y ago
    And on the other hand, only a fraction of things it sends to the spam folder are actually spam! The universe is in balance.
puppycodes 5y ago
This reason exactly is why i'm terrified of using email based two factor. I find the OTP from 1Password more reliable because nothing has to be sent or received.
MattGaiser 5y ago
Cloud is too large and usually too cheap for most people to really get meaningful customer support. I have heard of the same challenges from Azure and AWS too.
hitpointdrew 5y ago
>I guess I got what I paid for. News flash, you won't get any better customer service from AWS or Azure, etc. (unless your dropping lots of cash). With AWS you have web forms to fill out to create a support ticket, good luck ever getting a real person on the phone. This is all of course subject to the size of your account. Sure you can get a dedicated AWS account manager if you throw enough money around. But running one small managed k8s cluster won't cut it.
- saurik 5y ago
  You can just pay for premium AWS support, and it isn't even that expensive.
- allynjalford 5y ago
  I had a real person on the phone from AWS today. They actually called me after I opened a ticket yesterday and hadn't responded to their latest ticket update.
- rozenmd 5y ago
  I have had an AWS support person call me a few minutes after I got locked out of my account (<$10/mo spend) when I broke my 2FA device.
- electroly 5y ago
  You can sign up for a paid AWS Business support plan, immediately use it to file a priority ticket, and cancel it after 30 days. The business support plan includes 24/7 phone support. For small peanuts users, this is $100.
  - Volundr 5y ago
    How do you sign up when you can't log in like the OP?
    - electroly 5y ago
      AWS doesn't send you codes via email like DigitalOcean does. The situation as stated doesn't exist at all. If you forgot your password AND lost access to your email so you can't use the reset link, you can contact them at https://aws.amazon.com/contact-us/ https://aws.amazon.com/contact-us/ without having to log in. They have a special system for lost login support requests.
trutannus 5y ago
The grass often looks greener. I've never had good results with any cloud company's customer support with the only exception of Google, but that was because the individual I was assigned actually put in the effort themselves. Azure/Microsoft for example didn't even call me back when I requested a windows VM quote for a client I was working with.
- dnautics 5y ago
  yeah I company I worked for had terrible customer service from google and great customer service from amazon, so I think it's just a roll of the roulette wheel as to who you get as your CSM (once you pay for it).
geocrasher 5y ago
I don't know how DO's 2FA works, but when you signed up were you not given a number of backup codes for a contingency? I wish more vendors did this. Also I think this isn't so much a caution about DO as it is poor implementations of 2FA without good customer service to back it up.
- anuragsoni 5y ago
  Digitalocean offers TOTP for 2fa, with backup codes! No email needed. If memory serves well they only offer a TOTP and SMS based 2fa so i'm not sure why an email would be sent for this. Edit: Their docs mention the 2fa modes that are supported: https://docs.digitalocean.com/products/accounts/security/2fa/ https://docs.digitalocean.com/products/accounts/security/2fa...
  - MoSattler 5y ago
    I never really activated 2FA - the email verification code routine is something they do when one logs in from a new browser, or when one deleted cookies it seems. Unfortunately there are no backup codes for that.
    - meepmorp 5y ago
      I've never gotten an email like that from them, but I also have 2FA set up.
      - MoSattler 5y ago
        I guess once you have 2FA setup it's not necessary anymore. I just set up a new account for testing purposes, and I get their 6 digit code email out of the box.
- bks 5y ago
  https://docs.digitalocean.com/products/accounts/security/2fa/ https://docs.digitalocean.com/products/accounts/security/2fa... Looks like they provide backup options.
api 5y ago
The exact same thing will happen with AWS, GCP, and Azure unless you are spending a minimum of about $1000/month or purchase elevated support.
- bks 5y ago
  AWS does not care about your spend - you can always add support and this is a % of your spend with a minimum. You can add it and remove support at will - https://aws.amazon.com/premiumsupport/plans/business/ https://aws.amazon.com/premiumsupport/plans/business/
punkspider 5y ago
I was thinking the same thing 1-2 days ago when I wasn't getting the 6 digit code. It wasn't anything urgent, and it arrived after ~5-10 minutes. But if it were something urgent then it would have been a huge problem.
acejam 5y ago
I unfortunately had a similar issue about 1-2 years ago also on DOKS. We upgraded some k8s infrastructure on our side, but something about the "managed-ness" of their k8s solution caused our entire cluster to break. I believe it was due to some sort of admission webhook controller that had a failurePolicy of Fail, causing dependent pods to break, preventing everything else from starting.
danpalmer 5y ago
We ran our backup server on Digital Ocean. Needed to restore our database backups but were bound by very low I/O read rates from the Digital Ocean disks. Another issue we had is that they decided we were being DDoS'd, despite a server running no externally accessible ports apart from SSH (which was not being hit). Their solution was to disable the networking, therefore preventing our services on the machine from running. They couldn't provide any further information on the DDoS. Their suggested solution was to "put Cloudflare in front of it", but of course with no services to put Cloudflare in front of, this was useless advice. They did not understand this point. They're easy, and not bad for a first pass/MVP, but not ideal for production.
veesahni 5y ago
2FA is typically implemented by using an authenticator app that generates time based codes. Looking at DO's docs, they do this: https://docs.digitalocean.com/products/accounts/security/2fa/ https://docs.digitalocean.com/products/accounts/security/2fa... 2FA by sending an email is kindof strange. I wonder if that's the method of security they use if you don't actually have 2FA setup? p.s. For anybody reading this, DO docs talk of 2FA via SMS codes. Don't do that - it's generally considered insecure.
- MoSattler 5y ago
  DO's email thing is not really 2FA - they do that when one logs in from a new browser or after deleting cookies it seems.
PaulRobinson 5y ago
It took GitHub a month to not fix my 2FA problems. I think we rely on a lot of systems that for solo devs (where a colleague can’t fix your credential issues), it’s a terrible experience.
exabrial 5y ago
Welcome to the new normal. Simple Bank attempted the "no customer service" stuff with banking products and fortunately it failed. Robinhood is attempting it as well, and, well idiots keep pouring money into it. Your email provider went that way a long time ago. It's profitable not to provide customer service, and companies are in a place where it's hard to sue them.
- bener 5y ago
  If they are using Google Workspace they will get pretty decent super from Google. You can also check the reports to see if the email even made it as far as the mail server - some items are blocked before they even hit the spam folder.
dave404500 5y ago
they provide backup codes for this very reason
- MoSattler 5y ago
  They do that for proper 2FA - but the email flow is not 2FA. They do that when you use a new browser or deleted cookies.
diveandfight 5y ago
Anyone got data points from Linode?
- MrWiffles 5y ago
  I was going to post a separate top-level comment asking about Linode as well, but you beat me to it! I've never used Linode for my own projects before, or for anything production, but a company I've worked for in the past has used it's k8s offering for non-production/lab usage and I found it very slick and easy to use from the UX point of view. It was stable, performed well, and generally looked like a great product. We did run into one issue, though, where some internal and undisclosed account limit on I think the number of containers you could run had been hit because we had some automation starting a bunch of containers in there and then killing them off real quick (we were testing something if memory serves). All of a sudden we start seeing these unusual failures going on, coming from Linode's API, that made zero sense. So a manager contacted their support folks, who resolved it the next day with the first response to the support ticket. It wasn't production and we didn't file it as an emergency priority thing (even though it caused us to miss a deadline by like, a day I think), so I was perfectly OK with "next business day" turn-around. That's my very limited experience with Linode, and based on that alone, I'd certainly be willing to give them a shot for a small-scale production environment, at least. And if that didn't go south very quickly, I'd probably be willing to do more. A co-worker of mine is ambivalent to Linode, but swears by vultr. Says he loves them to death. I have no experience there, just wanted to throw out the name for anyone looking to see what options exist, so you know what to evaluate.
mads 5y ago
One of our suppliers went bankrupt and left us hanging without proper access to their Digital Ocean account where they were hosting some of the services they had provided for us. I think our account was already configured to have access but only as billing contact or something like that, so we couldn’t do anything besides paying the bills. I wrote DO and explained them the situation. It took a few days and a bunch of documents back and forth before they managed to let our account have an admin role. All in all they provided us with pretty good support and we were not a big customer - a few hundred euros a month, I believe.
crazypython 5y ago
Linode has always had great customer support for me. Responds in a few hours, friendly, helpful.
gregjor 5y ago
You broke your production system, and didn’t have 2FA set up, or a recovery plan. None of that is Digital Ocean’s fault. Slow response to tickets is unfortunately a problem with many services, a problem of scale. I’ve hosted sites on DO for several years, never had any problems with them. On the few occasions I had to contact their support I got a reply right away. Use OTP 2FA. Log in to your hosting provider console before doing critical deployments or upgrades. Use a staging server and have a recovery plan you’ve tested. No hosting provider is perfect, some are more reliable and responsive than others. But instead of blaming this on DO maybe title your post “Don’t use my service in production.”
- MoSattler 5y ago
  > You broke your production system, and didn’t have 2FA set up, or a recovery plan. None of that is Digital Ocean’s fault. That their emails and logins don't work, is their fault though. Those things are so basic and essential that they go way beyond "No hosting provider is perfect". And then they offer no assistance in rectifying their buggy software. I find it hard to understand that you blame the user when a product is so broken that one cannot login. That I didn't have a recovery plan is an untrue assumption of yours. Luckily I was able to mitigate this "outage" of digital ocean and host my app elsewhere. But this costs me work and energy I would have liked to invest elsewhere, and hence I would appreciate a more stable provider, where the most basic functionality reliably works.
  - gregjor 5y ago
    The product may be broken for you. I have used DO for years and never had this problem. They offer OTP 2FA specifically so you don’t have to rely on emails. Emails can fail in many places, you shouldn’t rely on that. Next time make sure you can log in to your hosting provider’s console before you make possibly breaking changes. I get that you had a bad experience but that doesn’t imply that DO is broken or not worthy of trust.
oliwarner 5y ago
Some email providers block email from networks they consider spammy. Just drop them on connect, never accept the delivery. I can't get email from Linode to Outlook 365 for love nor money. I wouldn't be entirely surprised if DO had the same problem from time to time, if they deliver direct from their own network.
midrus 5y ago
That's what happens when you run your infrastructure on $5 servers, even if paying more, the company is optimized for people paying $5 and running mostly side projects. If you're doing anything serious, go with gcloud or AWS. Their support might suck as well, but 99,9% of the time the problem is your fault and not their service, because it is so widely used and so many businesses depending on it that they'll notice even the slightest issue on their platform.
xtanx 5y ago
Why was this flagged? It seems like a valid reason to post it on HN.

55 points by MoSattler 5y ago

thesausageking 5y ago

If your bill is >$500 / mo, you get their "Business" support level which lets you use their customer success managers. If you're not spending this much, you won't get good support from them or any cloud vendor. AWS, Azure, and GCP all push you to the forums or submitted a ticket which may or may not get read.

smorgusofborg 5y ago
True though with a lot of the European ones you get good support and/or can pay explicitly for a level of support. The email disappearing with no trace sounds like Gmail hating DO address blocks though.. Gmail sends only a fraction of things it thinks are spam to the spam folder.
- MoSattler 5y ago
  Just found a blog post with a similar issue - seems like DO silently blacklists your email in certain cases. https://qertoip.medium.com/you-will-randomly-lose-access-to-your-digitalocean-account-2fb7b4d799b5 https://qertoip.medium.com/you-will-randomly-lose-access-to-...
- boring_twenties 5y ago
  And on the other hand, only a fraction of things it sends to the spam folder are actually spam! The universe is in balance.

puppycodes 5y ago

This reason exactly is why i'm terrified of using email based two factor. I find the OTP from 1Password more reliable because nothing has to be sent or received.

MattGaiser 5y ago

Cloud is too large and usually too cheap for most people to really get meaningful customer support. I have heard of the same challenges from Azure and AWS too.

hitpointdrew 5y ago

>I guess I got what I paid for. News flash, you won't get any better customer service from AWS or Azure, etc. (unless your dropping lots of cash). With AWS you have web forms to fill out to create a support ticket, good luck ever getting a real person on the phone. This is all of course subject to the size of your account. Sure you can get a dedicated AWS account manager if you throw enough money around. But running one small managed k8s cluster won't cut it.

saurik 5y ago
You can just pay for premium AWS support, and it isn't even that expensive.
allynjalford 5y ago
I had a real person on the phone from AWS today. They actually called me after I opened a ticket yesterday and hadn't responded to their latest ticket update.
rozenmd 5y ago
I have had an AWS support person call me a few minutes after I got locked out of my account (<$10/mo spend) when I broke my 2FA device.
electroly 5y ago
You can sign up for a paid AWS Business support plan, immediately use it to file a priority ticket, and cancel it after 30 days. The business support plan includes 24/7 phone support. For small peanuts users, this is $100.
- Volundr 5y ago
  How do you sign up when you can't log in like the OP?
  - electroly 5y ago
    AWS doesn't send you codes via email like DigitalOcean does. The situation as stated doesn't exist at all. If you forgot your password AND lost access to your email so you can't use the reset link, you can contact them at https://aws.amazon.com/contact-us/ https://aws.amazon.com/contact-us/ without having to log in. They have a special system for lost login support requests.

trutannus 5y ago

The grass often looks greener. I've never had good results with any cloud company's customer support with the only exception of Google, but that was because the individual I was assigned actually put in the effort themselves. Azure/Microsoft for example didn't even call me back when I requested a windows VM quote for a client I was working with.

dnautics 5y ago
yeah I company I worked for had terrible customer service from google and great customer service from amazon, so I think it's just a roll of the roulette wheel as to who you get as your CSM (once you pay for it).

geocrasher 5y ago

I don't know how DO's 2FA works, but when you signed up were you not given a number of backup codes for a contingency? I wish more vendors did this. Also I think this isn't so much a caution about DO as it is poor implementations of 2FA without good customer service to back it up.

anuragsoni 5y ago
Digitalocean offers TOTP for 2fa, with backup codes! No email needed. If memory serves well they only offer a TOTP and SMS based 2fa so i'm not sure why an email would be sent for this. Edit: Their docs mention the 2fa modes that are supported: https://docs.digitalocean.com/products/accounts/security/2fa/ https://docs.digitalocean.com/products/accounts/security/2fa...
- MoSattler 5y ago
  I never really activated 2FA - the email verification code routine is something they do when one logs in from a new browser, or when one deleted cookies it seems. Unfortunately there are no backup codes for that.
  - meepmorp 5y ago
    I've never gotten an email like that from them, but I also have 2FA set up.
    - MoSattler 5y ago
      I guess once you have 2FA setup it's not necessary anymore. I just set up a new account for testing purposes, and I get their 6 digit code email out of the box.
bks 5y ago
https://docs.digitalocean.com/products/accounts/security/2fa/ https://docs.digitalocean.com/products/accounts/security/2fa... Looks like they provide backup options.

api 5y ago

The exact same thing will happen with AWS, GCP, and Azure unless you are spending a minimum of about $1000/month or purchase elevated support.

bks 5y ago
AWS does not care about your spend - you can always add support and this is a % of your spend with a minimum. You can add it and remove support at will - https://aws.amazon.com/premiumsupport/plans/business/ https://aws.amazon.com/premiumsupport/plans/business/

punkspider 5y ago

I was thinking the same thing 1-2 days ago when I wasn't getting the 6 digit code. It wasn't anything urgent, and it arrived after ~5-10 minutes. But if it were something urgent then it would have been a huge problem.

acejam 5y ago

I unfortunately had a similar issue about 1-2 years ago also on DOKS. We upgraded some k8s infrastructure on our side, but something about the "managed-ness" of their k8s solution caused our entire cluster to break. I believe it was due to some sort of admission webhook controller that had a failurePolicy of Fail, causing dependent pods to break, preventing everything else from starting.

danpalmer 5y ago

We ran our backup server on Digital Ocean. Needed to restore our database backups but were bound by very low I/O read rates from the Digital Ocean disks. Another issue we had is that they decided we were being DDoS'd, despite a server running no externally accessible ports apart from SSH (which was not being hit). Their solution was to disable the networking, therefore preventing our services on the machine from running. They couldn't provide any further information on the DDoS. Their suggested solution was to "put Cloudflare in front of it", but of course with no services to put Cloudflare in front of, this was useless advice. They did not understand this point. They're easy, and not bad for a first pass/MVP, but not ideal for production.

veesahni 5y ago

2FA is typically implemented by using an authenticator app that generates time based codes. Looking at DO's docs, they do this: https://docs.digitalocean.com/products/accounts/security/2fa/ https://docs.digitalocean.com/products/accounts/security/2fa... 2FA by sending an email is kindof strange. I wonder if that's the method of security they use if you don't actually have 2FA setup? p.s. For anybody reading this, DO docs talk of 2FA via SMS codes. Don't do that - it's generally considered insecure.

MoSattler 5y ago
DO's email thing is not really 2FA - they do that when one logs in from a new browser or after deleting cookies it seems.

PaulRobinson 5y ago

It took GitHub a month to not fix my 2FA problems. I think we rely on a lot of systems that for solo devs (where a colleague can’t fix your credential issues), it’s a terrible experience.

exabrial 5y ago

Welcome to the new normal. Simple Bank attempted the "no customer service" stuff with banking products and fortunately it failed. Robinhood is attempting it as well, and, well idiots keep pouring money into it. Your email provider went that way a long time ago. It's profitable not to provide customer service, and companies are in a place where it's hard to sue them.

bener 5y ago
If they are using Google Workspace they will get pretty decent super from Google. You can also check the reports to see if the email even made it as far as the mail server - some items are blocked before they even hit the spam folder.

dave404500 5y ago

they provide backup codes for this very reason

MoSattler 5y ago
They do that for proper 2FA - but the email flow is not 2FA. They do that when you use a new browser or deleted cookies.

diveandfight 5y ago

Anyone got data points from Linode?

MrWiffles 5y ago
I was going to post a separate top-level comment asking about Linode as well, but you beat me to it! I've never used Linode for my own projects before, or for anything production, but a company I've worked for in the past has used it's k8s offering for non-production/lab usage and I found it very slick and easy to use from the UX point of view. It was stable, performed well, and generally looked like a great product. We did run into one issue, though, where some internal and undisclosed account limit on I think the number of containers you could run had been hit because we had some automation starting a bunch of containers in there and then killing them off real quick (we were testing something if memory serves). All of a sudden we start seeing these unusual failures going on, coming from Linode's API, that made zero sense. So a manager contacted their support folks, who resolved it the next day with the first response to the support ticket. It wasn't production and we didn't file it as an emergency priority thing (even though it caused us to miss a deadline by like, a day I think), so I was perfectly OK with "next business day" turn-around. That's my very limited experience with Linode, and based on that alone, I'd certainly be willing to give them a shot for a small-scale production environment, at least. And if that didn't go south very quickly, I'd probably be willing to do more. A co-worker of mine is ambivalent to Linode, but swears by vultr. Says he loves them to death. I have no experience there, just wanted to throw out the name for anyone looking to see what options exist, so you know what to evaluate.

mads 5y ago

One of our suppliers went bankrupt and left us hanging without proper access to their Digital Ocean account where they were hosting some of the services they had provided for us. I think our account was already configured to have access but only as billing contact or something like that, so we couldn’t do anything besides paying the bills. I wrote DO and explained them the situation. It took a few days and a bunch of documents back and forth before they managed to let our account have an admin role. All in all they provided us with pretty good support and we were not a big customer - a few hundred euros a month, I believe.

crazypython 5y ago

Linode has always had great customer support for me. Responds in a few hours, friendly, helpful.

gregjor 5y ago

You broke your production system, and didn’t have 2FA set up, or a recovery plan. None of that is Digital Ocean’s fault. Slow response to tickets is unfortunately a problem with many services, a problem of scale. I’ve hosted sites on DO for several years, never had any problems with them. On the few occasions I had to contact their support I got a reply right away. Use OTP 2FA. Log in to your hosting provider console before doing critical deployments or upgrades. Use a staging server and have a recovery plan you’ve tested. No hosting provider is perfect, some are more reliable and responsive than others. But instead of blaming this on DO maybe title your post “Don’t use my service in production.”

MoSattler 5y ago
> You broke your production system, and didn’t have 2FA set up, or a recovery plan. None of that is Digital Ocean’s fault. That their emails and logins don't work, is their fault though. Those things are so basic and essential that they go way beyond "No hosting provider is perfect". And then they offer no assistance in rectifying their buggy software. I find it hard to understand that you blame the user when a product is so broken that one cannot login. That I didn't have a recovery plan is an untrue assumption of yours. Luckily I was able to mitigate this "outage" of digital ocean and host my app elsewhere. But this costs me work and energy I would have liked to invest elsewhere, and hence I would appreciate a more stable provider, where the most basic functionality reliably works.
- gregjor 5y ago
  The product may be broken for you. I have used DO for years and never had this problem. They offer OTP 2FA specifically so you don’t have to rely on emails. Emails can fail in many places, you shouldn’t rely on that. Next time make sure you can log in to your hosting provider’s console before you make possibly breaking changes. I get that you had a bad experience but that doesn’t imply that DO is broken or not worthy of trust.

oliwarner 5y ago

Some email providers block email from networks they consider spammy. Just drop them on connect, never accept the delivery. I can't get email from Linode to Outlook 365 for love nor money. I wouldn't be entirely surprised if DO had the same problem from time to time, if they deliver direct from their own network.

midrus 5y ago

That's what happens when you run your infrastructure on $5 servers, even if paying more, the company is optimized for people paying $5 and running mostly side projects. If you're doing anything serious, go with gcloud or AWS. Their support might suck as well, but 99,9% of the time the problem is your fault and not their service, because it is so widely used and so many businesses depending on it that they'll notice even the slightest issue on their platform.

xtanx 5y ago

Why was this flagged? It seems like a valid reason to post it on HN.