6 ms·

I'm more worried about the following post in that thread: <begin quote> -------------------------------------------------------- We were loyal customers of Di

by mizzao 11y ago

I'm more worried about the following post in that thread: <begin quote> -------------------------------------------------------- We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No biggie, we'll just update it, turn the server back on and be on our way. Nope. We had to contact customer service to get back in to our account. After updating the card info we realize that all our droplets are gone. We reach out to customer service again in which case they let us know that when a CC expires, an automated process kicks off and deletes the droplets. We've been a customer for 2 years, surely they could pick up the phone and call us. We've spent thousands of dollars with them. Plan B, let's restore them from the backups we've been paying for. Nope! When they delete your droplets, they also delete your backups. This is where we ask to speak to them on the phone and are denied. I then ask if they are insane and why they would delete someone's servers and backups via an automated process without a human at least checking to see if it is a loyal multi year customer who has a simple lapse with their CC expiry. We had to find a backup on a developers machine from over a month prior and rebuild data using various megtods that took close to a week. In the end, DI gave us a $500 credit. You get what you pay for. Anyone who uses DI for production or anything more than a hobby app is playing with fire. They do not care, they are apathetic, they will screw you over and the throw you a credit for their terrible service as a half assed apology.

mizzao 11y ago
One time my CC expired on DO because there was a fraudulent purchase on it and I had to get a new one. I got a brief e-mail from DO noting there was a problem charging my card, and updated it the same day. What if I was on vacation, and didn't see this e-mail for a few days? shudder In what world is automating the deletion of customer data a good idea?
- dualboot 11y ago
  > In what world is automating the deletion of customer data a good idea? The data destruction world :)
- xyzzy4 11y ago
  The world where employees care more about management satisfaction than customer satisfaction? And it's difficult to prove that bothering some customers here and there is impacting corporate revenue. Policies like this slip through the cracks.
  - fleitz 11y ago
    Yes, it's digital oceans fault that people want to pay bargain basement prices and then wonder why they are getting bargain basement service.
    - mixmastamyk 11y ago
      Sorry, that doesn't excuse the purely bad idea of scripting deletion of data at the first sign of trouble.
      - Tassels 11y ago
        It's not at the first sign of trouble, though. It's after ~15 email notifications over the course of over a month, and no contact back.
        mixmastamyk 11y ago
        Both anecdotes mentioned timeframes of about 15 days after longer than a year of service.
        Tassels 11y ago
        The timeframe is 3 weeks until suspension, then another 2 weeks until deletion. At the time this user was affected, it was 3 weeks until suspension, then 3 days until deletion, but that period was raised to 2 weeks due to feedback that 3 days wasn't enough. I'm not saying that excuses things, but it's far from "at the first sign of trouble".
- deleted 11y ago
  [deleted]
the_common_man 11y ago
I had the exact same experience! Except I got no credit. My CC expired but I was on vacation and didn't have access to email. When I was back, the droplet and all backups and snapshots were completely removed. For the record, I was only 15 days late. Very sad. I tried my best to get the backups from the support engineer but no luck.
vidyesh 11y ago
I really don't know to admire or criticise DO for this. On one hand horrible situation to be in, at least could have called/mailed/SMSes/notified the customer before nuking the account. On the other hand, it is a relief to know your data is secure and gone one you stop paying, no way they have a copy or keep it in archive/deep backup.
- happywolf 11y ago
  Nope, the only assurance you can get is you won't be access your data, I am not sure how one can come down with conclusion of "no way they have a copy or keep it in archive/deep backup" Having a backup and letting you to access it are two entirely different issues, and most of the time customer service staff don't know how the data management is being done by the engineering/operations team.
  - vidyesh 11y ago
    If you have used shared hosting you know as soon as you pay for your expired account, everything is back as it was. Never thought this was a problem until you try to test how long you can go until you pay and still restore you data. If DO's CC can't handle data management problems they probably will connect you to their engg/ops team. So I would assume they would prefer having it resolved ( by giving the data if they had ) instead of keeping it and let user rant/post about it( which they know will probably happen ).
    - happywolf 11y ago
      I agree with the first paragraph. The second paragraph hinges on the rationale that if customer data were available, then CS (Customer Service) agents would have made every effort to contact the ops/engineering team to retrieve those to avoid a social disaster. Unfortunately, this isn't true in the companies that I worked for, which are big customer-facing MNCs. First of all, for US consumers, the CS teams most likely are contractors in Philippines, Malaysia, or India, they are trained with scripted responses. For any issue that is beyond the scripts, their standard response will be "Sorry we can't help you because we can't do x", one reason is they have no incentive to escalate the issue and get oneself more work to follow up, other times, it is the management explicitly decided this is to prevent too many cases hit the ops and eventually, the engineering teams. It is not that the management is stupid or nasty, just that the 20:80 rule dictates 80% of the issues usually are raised by 20% of particularly picky customers. For the 20% customers who generate 80% of the revenue, each of them will be assigned an AM (Account Manager) who can get things done much faster and more efficient than the CS team. This group of high-value customers are often given access to VPs to make sure all their issues are heard and handled properly Second reason is most teams in a big company will not concern anything that doesn't hit them directly. For example the CS team in my ex-company upon getting repeated complaint on an erratic component that didn't get bug fix for months, eventually the CS told the upset customer to fuck off and complain to CEO directly since it was engineering department's fault. Unbelievable? Maybe, and that company remains one of the largest high-tech companies in the world.
joenathan 11y ago
I'm looking for a good host, was considering DO, but now I'm not. Any suggestions?
- joshmn 11y ago
  Needs vary, but here's a brief list of competitors: * OVH.com — huge, great price point and performance * Linode.com — also huge, favorite of many developers * Vultr.com — owned by Choopa (bigger player) * RamNode.com — doesn't scream "super modern", but reviews are great There are many others, but those are off the top of my head that correlate with DO.
  - jacquesm 11y ago
    - leaseweb - 1andone Vote against linode here, yes, they're large but they are also not all that good.
- toadi 11y ago
  According to me DO is still a good hosting company. Works good for the price you pay. Taking offsite backups is always a good idea.
rorykoehler 11y ago
For what it's worth this happened to me through a client's account (client wasn't checking emails and didn't see warning mails re CC expiring). I reached out to customer support once the site went down and even though we were beyond the cut-off deadline they re-activated all the droplets. Perhaps they have updated their process in this regard. I also learned my lesson and made sure I got forwarding on all mails related to DO.
mrweasel 11y ago
>We've been a customer for 2 years, surely they could pick up the phone and call us. I don't understand that reasoning from customers to be honest. That right there, throws away the $5 and $10 plans. Having some one trying to contact a customer, just because their credit card expired, cost more than the actual hosting. Top tier service does comes from companies with $5 a month plans. I'm sorry it just doesn't.
- josephagoss 11y ago
  > We've spent thousands of dollars with them. That's a decent amount of money right? It sounds like they were not using the $5 plans.
  - mrweasel 11y ago
    Over two years, that's what, eight $10 droplets or sixteen $5 droplet. I would guess that if you want a hosting company to be proactive you need $100+ per month.
- aagha 11y ago
  Because $5/$10/month is not the same thing as Customer Lifetime Value.
techstrategist 11y ago
This is a big danger of "move fast and break things". I imagine the process they put in place seemed reasonable internally before catching this edge case, but what a crappy result.
josephagoss 11y ago
Thank you for this, I have a large client where we were considering DigitalOcean and the yearly spend could easily be in the thousands. Now I'll strike them off of the list. Automated deletion of VM's and backups, absolutely unacceptable. The VM hosting market isn't looking good at the moment, seems like each provider has some dark history at this point, it's really frustrating.
toadi 11y ago
Offsite backups? We use AWS and DO to host applications for our clients. We still take offsite backups. Isn't that what your clients paying you for?
josephmx 11y ago
I had the opposite experience, my CC expired with DO, but I didn't have a new one for about 2 weeks. I didn't have any issues with just waiting to the next billing cycle and paying double then to cover the negative.
fweespeech 11y ago
This is why you use two providers with two credit cards. I'm quite serious. Our production environment operates on two cards with different issue & expiration dates from two different merchants and uses 2 different Cloud providers.
andrewthetechie 11y ago
Check out the followup: <begin quote> Hey Matthew, I'm Zach, Director of Support here at DigitalOcean. Thanks for raising this topic. I was able to locate your account, and I see that I was the one who granted the credit and followed up via email. I hadn't heard back from you until now, and I'm happy that we have an open line of communication. I'm hopeful that this thread starts a conversation, that it clarifies what steps we take, and we might even uncover a different solution that works well for everyone. To start with, I want to be really clear, this is the absolute worst case scenario. I absolutely don't want it to happen to any customer, let alone someone who has been with us for such a long time. There's really a delicate balance that we must strike between customers who forget to pay and customers who do not want to pay. Currently, our notifications for overdue balances are sent via email. If a customer account is unpaid on the first of the month, we send ~15 emails total notifying the customer of the situation, and subsequently power off servers 21 days after the account is on hold as a further way to gain your attention. At this time, droplets are removed from your account 14 days following power off, which is an increased amount of time from what you experienced. It was increased from 3 days based on past user feedback. Why do we do it this way? In the past, we had no hold or suspension process for non-payment, which enabled bad actors to run for months and months without paying. From a business perspective, we made a decision to put a scalable process in place that limited how long an account could go unpaid. As a support team and business, we are always willing to work with our customers who are unable to pay. We are always available via ticket, our contact form, and have made wide-ranging attempts to help users who aren't able to pay due to banking regulations: https://insidedigitalocean.com.. https://insidedigitalocean.com... I'd like some input, so I'll publicly ask for feedback on questions that I've asked privately before. I'd also like any other feedback that we can consider on how to make this a more positive experience for everyone -Knowing that we do not do phone support, what's the best way to notify you of a past-due balance? -Is SMS effective at times like this? I would love to hear your thoughts. Thank you, Zach zach@digitalocean.com
- mizzao 11y ago
  3 days seemed like an accident waiting to happen. This updated policy seems a lot better; it would probably be great to ask people for secondary contact information such as SMS for emergencies.

by mizzao 11y ago

mizzao 11y ago

One time my CC expired on DO because there was a fraudulent purchase on it and I had to get a new one. I got a brief e-mail from DO noting there was a problem charging my card, and updated it the same day. What if I was on vacation, and didn't see this e-mail for a few days? shudder In what world is automating the deletion of customer data a good idea?

dualboot 11y ago
> In what world is automating the deletion of customer data a good idea? The data destruction world :)
xyzzy4 11y ago
The world where employees care more about management satisfaction than customer satisfaction? And it's difficult to prove that bothering some customers here and there is impacting corporate revenue. Policies like this slip through the cracks.
- fleitz 11y ago
  Yes, it's digital oceans fault that people want to pay bargain basement prices and then wonder why they are getting bargain basement service.
  - mixmastamyk 11y ago
    Sorry, that doesn't excuse the purely bad idea of scripting deletion of data at the first sign of trouble.
    - Tassels 11y ago
      It's not at the first sign of trouble, though. It's after ~15 email notifications over the course of over a month, and no contact back.
      - mixmastamyk 11y ago
        Both anecdotes mentioned timeframes of about 15 days after longer than a year of service.
        Tassels 11y ago
        The timeframe is 3 weeks until suspension, then another 2 weeks until deletion. At the time this user was affected, it was 3 weeks until suspension, then 3 days until deletion, but that period was raised to 2 weeks due to feedback that 3 days wasn't enough. I'm not saying that excuses things, but it's far from "at the first sign of trouble".
deleted 11y ago
[deleted]

the_common_man 11y ago

I had the exact same experience! Except I got no credit. My CC expired but I was on vacation and didn't have access to email. When I was back, the droplet and all backups and snapshots were completely removed. For the record, I was only 15 days late. Very sad. I tried my best to get the backups from the support engineer but no luck.

vidyesh 11y ago

I really don't know to admire or criticise DO for this. On one hand horrible situation to be in, at least could have called/mailed/SMSes/notified the customer before nuking the account. On the other hand, it is a relief to know your data is secure and gone one you stop paying, no way they have a copy or keep it in archive/deep backup.

happywolf 11y ago
Nope, the only assurance you can get is you won't be access your data, I am not sure how one can come down with conclusion of "no way they have a copy or keep it in archive/deep backup" Having a backup and letting you to access it are two entirely different issues, and most of the time customer service staff don't know how the data management is being done by the engineering/operations team.
- vidyesh 11y ago
  If you have used shared hosting you know as soon as you pay for your expired account, everything is back as it was. Never thought this was a problem until you try to test how long you can go until you pay and still restore you data. If DO's CC can't handle data management problems they probably will connect you to their engg/ops team. So I would assume they would prefer having it resolved ( by giving the data if they had ) instead of keeping it and let user rant/post about it( which they know will probably happen ).
  - happywolf 11y ago
    I agree with the first paragraph. The second paragraph hinges on the rationale that if customer data were available, then CS (Customer Service) agents would have made every effort to contact the ops/engineering team to retrieve those to avoid a social disaster. Unfortunately, this isn't true in the companies that I worked for, which are big customer-facing MNCs. First of all, for US consumers, the CS teams most likely are contractors in Philippines, Malaysia, or India, they are trained with scripted responses. For any issue that is beyond the scripts, their standard response will be "Sorry we can't help you because we can't do x", one reason is they have no incentive to escalate the issue and get oneself more work to follow up, other times, it is the management explicitly decided this is to prevent too many cases hit the ops and eventually, the engineering teams. It is not that the management is stupid or nasty, just that the 20:80 rule dictates 80% of the issues usually are raised by 20% of particularly picky customers. For the 20% customers who generate 80% of the revenue, each of them will be assigned an AM (Account Manager) who can get things done much faster and more efficient than the CS team. This group of high-value customers are often given access to VPs to make sure all their issues are heard and handled properly Second reason is most teams in a big company will not concern anything that doesn't hit them directly. For example the CS team in my ex-company upon getting repeated complaint on an erratic component that didn't get bug fix for months, eventually the CS told the upset customer to fuck off and complain to CEO directly since it was engineering department's fault. Unbelievable? Maybe, and that company remains one of the largest high-tech companies in the world.

joenathan 11y ago

I'm looking for a good host, was considering DO, but now I'm not. Any suggestions?

joshmn 11y ago
Needs vary, but here's a brief list of competitors: * OVH.com — huge, great price point and performance * Linode.com — also huge, favorite of many developers * Vultr.com — owned by Choopa (bigger player) * RamNode.com — doesn't scream "super modern", but reviews are great There are many others, but those are off the top of my head that correlate with DO.
- jacquesm 11y ago
  - leaseweb - 1andone Vote against linode here, yes, they're large but they are also not all that good.
toadi 11y ago
According to me DO is still a good hosting company. Works good for the price you pay. Taking offsite backups is always a good idea.

rorykoehler 11y ago

For what it's worth this happened to me through a client's account (client wasn't checking emails and didn't see warning mails re CC expiring). I reached out to customer support once the site went down and even though we were beyond the cut-off deadline they re-activated all the droplets. Perhaps they have updated their process in this regard. I also learned my lesson and made sure I got forwarding on all mails related to DO.

mrweasel 11y ago

>We've been a customer for 2 years, surely they could pick up the phone and call us. I don't understand that reasoning from customers to be honest. That right there, throws away the $5 and $10 plans. Having some one trying to contact a customer, just because their credit card expired, cost more than the actual hosting. Top tier service does comes from companies with $5 a month plans. I'm sorry it just doesn't.

josephagoss 11y ago
> We've spent thousands of dollars with them. That's a decent amount of money right? It sounds like they were not using the $5 plans.
- mrweasel 11y ago
  Over two years, that's what, eight $10 droplets or sixteen $5 droplet. I would guess that if you want a hosting company to be proactive you need $100+ per month.
aagha 11y ago
Because $5/$10/month is not the same thing as Customer Lifetime Value.

techstrategist 11y ago

This is a big danger of "move fast and break things". I imagine the process they put in place seemed reasonable internally before catching this edge case, but what a crappy result.

josephagoss 11y ago

Thank you for this, I have a large client where we were considering DigitalOcean and the yearly spend could easily be in the thousands. Now I'll strike them off of the list. Automated deletion of VM's and backups, absolutely unacceptable. The VM hosting market isn't looking good at the moment, seems like each provider has some dark history at this point, it's really frustrating.

toadi 11y ago

Offsite backups? We use AWS and DO to host applications for our clients. We still take offsite backups. Isn't that what your clients paying you for?

josephmx 11y ago

I had the opposite experience, my CC expired with DO, but I didn't have a new one for about 2 weeks. I didn't have any issues with just waiting to the next billing cycle and paying double then to cover the negative.

fweespeech 11y ago

This is why you use two providers with two credit cards. I'm quite serious. Our production environment operates on two cards with different issue & expiration dates from two different merchants and uses 2 different Cloud providers.

andrewthetechie 11y ago

Check out the followup: <begin quote> Hey Matthew, I'm Zach, Director of Support here at DigitalOcean. Thanks for raising this topic. I was able to locate your account, and I see that I was the one who granted the credit and followed up via email. I hadn't heard back from you until now, and I'm happy that we have an open line of communication. I'm hopeful that this thread starts a conversation, that it clarifies what steps we take, and we might even uncover a different solution that works well for everyone. To start with, I want to be really clear, this is the absolute worst case scenario. I absolutely don't want it to happen to any customer, let alone someone who has been with us for such a long time. There's really a delicate balance that we must strike between customers who forget to pay and customers who do not want to pay. Currently, our notifications for overdue balances are sent via email. If a customer account is unpaid on the first of the month, we send ~15 emails total notifying the customer of the situation, and subsequently power off servers 21 days after the account is on hold as a further way to gain your attention. At this time, droplets are removed from your account 14 days following power off, which is an increased amount of time from what you experienced. It was increased from 3 days based on past user feedback. Why do we do it this way? In the past, we had no hold or suspension process for non-payment, which enabled bad actors to run for months and months without paying. From a business perspective, we made a decision to put a scalable process in place that limited how long an account could go unpaid. As a support team and business, we are always willing to work with our customers who are unable to pay. We are always available via ticket, our contact form, and have made wide-ranging attempts to help users who aren't able to pay due to banking regulations: https://insidedigitalocean.com.. https://insidedigitalocean.com... I'd like some input, so I'll publicly ask for feedback on questions that I've asked privately before. I'd also like any other feedback that we can consider on how to make this a more positive experience for everyone -Knowing that we do not do phone support, what's the best way to notify you of a past-due balance? -Is SMS effective at times like this? I would love to hear your thoughts. Thank you, Zach zach@digitalocean.com

mizzao 11y ago
3 days seemed like an accident waiting to happen. This updated policy seems a lot better; it would probably be great to ask people for secondary contact information such as SMS for emergencies.