7 ms·

Just started using the new Kubernetes offering on Digital Ocean. Still in Beta but works pretty good. If this is a sign for how they will do databases, I'm all

by tnolet 8y ago

Just started using the new Kubernetes offering on Digital Ocean. Still in Beta but works pretty good. If this is a sign for how they will do databases, I'm all ears. Add CI/CD and you have a Heroku competitor.

joering2 8y ago
Glad you are having fun working on your after hours side project with DO. But please DO NOT host with DO if you plan to run a real production systems, build a company, hire people etc. I have never found a better company to spun off a server and play with some settings but when their algorithm decides there is something fishy, bye bye your account, servers, backups, you never going to see any of this ever again. The last startup I heavily pushed with switching to OVH or even Rackspace was exact sample of what happens when DO algorithm decides you are not genuine. That's it. No explanation, no phone number to call, nothing. These people blindly decided to believe their algorithm and never wanted even discuss resuming the account or even get us backup of data. There was nothing shady going on I assure you. Funny part is as of today DigitalOcean is still in violation of GPPR, as we have requested to hopefully find out what was wrong with our account by filing request for info. Nope, zit, nada, totally ignored. We have filed complain with proper authority and also notified Attorney General in NY and continue await result. Build all you want on DigitalOcean, but please understand not people but their weird algorithm is in charge of the future of your startup, future of your company and future salary or lack therefore when you are forced to fire team of people because you fall behind with payroll. In other words: be warned and build at your own risk.
- whitepoplar 8y ago
  This is frightening. Can anyone from DO chime in?
  - raiyu 8y ago
    Just making sure to reply to each person that raised a concern. There is a lot of fraud that comes into every single cloud provider as root access to a virtual server can be used for a lot of malicious activity. As a result every cloud provider has automatic and manual processes that they run to find these accounts and flag them. In the case of DigitalOcean when an account is flagged it is locked, which simply prevents a user from creating more resources and an email notification goes out to the user to establish a line of communication. There is no service interruption, and certainly the account, it's droplets, and other resources are not deleted, and never deleted automatically when the account is initially flagged. There are numerous communications that go out even if a user is unresponsive.
- tnolet 8y ago
  This sounds terrible. I never had any issues with DO. They actually refunded me once when I actually made a mistake. Would love to hear the DO side on this.
- esistgut 8y ago
  Our team is currently migrating our whole production system from Linode and GCP to Digital Ocean. Your comment is raising warnings, I'd like to hear more from DO itself.
  - joering2 8y ago
    Out for curiosity but asking honestly: what made you chose DO from all providers out there? It sounds like you have a serious setup that most likely is production and probably makes money. You do know that DO does not provide over the phone help, right? So if your server goes down you open the ticket and you wait... Yes they do offer SLA because I was in heavy email chain back and forth when moving one of my clients from Rackspace and they were sold on DO being the right choice, but again I was explained no phone support - only that your emails to customer support are prioritized.
  - jarland 8y ago
    I'd love to chat with you. If you have some time, send an email over to jdonnell@digitalocean.com and let's talk. I promise nothing but honesty, transparency, ideas, and maybe a few laughs :)
    - zifnab06 8y ago
      Any chance at an invite to the k8s beta for a large-ish open source project?
      - jarland 8y ago
        Toss me an email, I'll see what I can do :)
- Scarbutt 8y ago
  This is bad, but your data should have also resided somewhere outside DO.
  - joering2 8y ago
    I was only technical adviser for a while, but I know they did offsite backups but its hard to have these up to a minute. So they tried to recover the latest version of their DB so that truly no records are missing. Edit: my original post got downvoted severely in few minutes from posting. Hello DigitalOcean staff and/or owners!
    - dewey 8y ago
      I think more background information would be more appropriate than conspiracy theories of what happened to you. Usually there are two sides to a story.
    - greglindahl 8y ago
      How did you figure out that DigitalOcean staff and/or owners were the downvoters?
      - thecus 8y ago
        Don't participate on HN a lot, but I couldn't resist responding to the GDPR violation in the same line as engaging with the NY Attorney General.
    - gamblor956 8y ago
      I downvoted your original comment because it was a bunch of unsupported conspiracy BS and at least one of your claims was directly refuted by the company itself. If you have a real complaint against DO, please provide specific allegations and support for those allegations.
- sgc 8y ago
  Chiming in with everyone else. I would like more info from OP and DO since I host all my sites there.
- plainOldText 8y ago
  These are some serious allegations. If this is true then DogitalOcean needs to explain themselves. I guess, one should also have backups off site. Relying on just one provider constitutes a single point of failure.
  - thecus 8y ago
    I suspect that any cloud provider has enormous amounts of fraud on it and no tooling is 100% error free. It's also important to note that GDPR has specific exemptions for companies responding suspected security/fraud related issues. I love DO because the performance for the price is great. Also have no issue supporting emerging tech companies with cultures I connect with.
    - plainOldText 8y ago
      No question fraud exists on all these platforms, however, if anyone is wrongly flagged by an algorithm, reaching out to the company must be followed by a prompt and timely response, so any misfortune can be remediated. I would assume fraudsters wouldn't typically reach out to have their accounts reinstated. In the digital age, startups and businesses rely on cloud providers for their livelihood. These providers must be reliable and trustworthy, otherwise they shouldn't get a penny. Also, I'm not intimately familiar with OP's situation, my comments are just common sense generalizations; I think!
      - raiyu 8y ago
        We have an entire fraud and safety team whose sole purpose is to deal with these situations. Every account that is flagged is notified. Every account is communicated with and there are always replies sent. Unless a droplet is actively being malicious, such as sending out a DDoS attack, or performing some other sort of determined malicious activity, there is absolutely no interruption in service. The account is locked so that the account can not create more resources, but there is no disruption to the underlying running resources such as droplets and otherwise. The intent here is to establish a dialogue with the user and determine if the activity is fraudulent or otherwise.
- jarland 8y ago
  Hey friends! My name is Jarland and I'm on the support team at DigitalOcean. We do have a number of fraud and abuse algorithms, and when we are alerted to potentially fraudulent activity, we take appropriate action, which includes notifying and communicating with individual users. I also want to confirm that we are fully compliant with GDPR.
  - drchiu 8y ago
    Thanks @jarland for chiming into this thread. I love how DO has evolved over the past several years and want to comment about that. I use DO for production and have gradually migrated my infrastructure away from AWS and Linode to Digital Ocean as the platform improved. Just a quick question: If the algorithm is triggered (regardless if it is a false positive or not) and the user is notified, what happens with the droplets in the meantime? Is there a grace period for the user to act before DO takes action? And is the whole account frozen or just the offending droplets? It seems the major concern amongst commenters here is the sudden lost of service. Thanks for the great service, and I look forward to your insight on this.
    - donmcronald 8y ago
      Yeah, I want to know if the execution is before or after the trial. Part of DO's appeal to me is the simplicity and predictable (low) cost. It would be really great if they published well defined account termination procedures. Do I get a phone call? An email? Do I get to respond before being disconnected? Is there an appeals process?
      - raiyu 8y ago
        As anyone who runs a service that provides full root access to servers understands there is a tremendous amount of opportunity for potential abuse. It becomes a game of cat and mouse to catch the abusers and prevent them from creating numerous accounts which ultimately impact system performance and can lead to potential problems for real legitimate customers. Those guidelines aren't published specifically because if they were, then the abusers would immediately begin to route around them, so it's meant to be opaque for a reason, but that is against fraudulent use, not legitimate use.
- tracker1 8y ago
  It may be worthwhile to have a blog post with a (LOT) more details (maybe any correspondence with DO included) and include a link to that.
- js4ever 8y ago
  This is exactly why i'm ok with paying 10x more for AWS. You can speak with them in few minutes, you have a grace period before service shutdown unlike ovh, DO and most others
- thirstybanach 8y ago
  I've had an experience like this but with less consequence. I was running a server years ago that scraped information from various cryptocurrency sites and it was flagged as fraudulent. Support refused to say what the issue was or offer temporary access to the server for data retrieval. This wasn't some cowboy unbounded crawler either, it was making very specific requests, around 1000 a day for around two weeks, probably less than an average day of web browsing for the majority of people. I'll never use DO for anything, even testing and mucking around. At the very least you would have to provide excessive evidence of identity if you get flagged by their magic algorithm, and if they restore your service it could be after days of wrangling with support (it took support over a day to answer my support tickets at the time).
  - greglindahl 8y ago
    I glanced at the DO terms of service, and crawling is prohibited.
- ng3 8y ago
  I just had my account locked with no warning or explanation. All 8 droplets were turned off. Account was unlocked 40 minutes later (also with no notification), and I could go in and turn droplets back on. Over 2 hours later and still no response to my support ticket asking why and how it happened. I'll be interested in the response I get. Unless there's a good reason why, or a plan to prevent the how from happening again, I'll be shifting anything critical away from DO, and go back to just using it for spinning things up to play with or test on.
  - joering2 8y ago
    Ufortunately nobody here is going to listen to you. A flock of DO fanboys downvoted my comments into obl livion despite I was just merely posting my experience. Some ambulance chaser even turned out to be a psychic reader because he knows better than me myself whether we been provided with GDPR related response or not, because - well, some DO CS worker told so on this forum. I guess getting your droplets cutoff in the middle of business day and waiting 4 days for customer support copy and paste template answer has to happen to everyone before they themselves realize how crucial over the phone support is when it comes to hosting a production website. Its all good anyways.
tracker1 8y ago
You can do custom agents with Azure DevOps (formerly VSTS) or GitLab pretty easily. Haven't played with the new Github stuff. How are you liking the K8s on DO? I just got access a couple days ago but haven't touched it yet, I seem so out of the loop.
- dsumenkovic 8y ago
  Thanks for mentioning GitLab. You can find more information about Auto Devops and creating Kubernetes clusters from withing GitLab at this doc https://docs.gitlab.com/ee/topics/autodevops/quick_start_guide.html#creating-a-kubernetes-cluster-from-within-gitlab https://docs.gitlab.com/ee/topics/autodevops/quick_start_gui...

by tnolet 8y ago

joering2 8y ago

Glad you are having fun working on your after hours side project with DO. But please DO NOT host with DO if you plan to run a real production systems, build a company, hire people etc. I have never found a better company to spun off a server and play with some settings but when their algorithm decides there is something fishy, bye bye your account, servers, backups, you never going to see any of this ever again. The last startup I heavily pushed with switching to OVH or even Rackspace was exact sample of what happens when DO algorithm decides you are not genuine. That's it. No explanation, no phone number to call, nothing. These people blindly decided to believe their algorithm and never wanted even discuss resuming the account or even get us backup of data. There was nothing shady going on I assure you. Funny part is as of today DigitalOcean is still in violation of GPPR, as we have requested to hopefully find out what was wrong with our account by filing request for info. Nope, zit, nada, totally ignored. We have filed complain with proper authority and also notified Attorney General in NY and continue await result. Build all you want on DigitalOcean, but please understand not people but their weird algorithm is in charge of the future of your startup, future of your company and future salary or lack therefore when you are forced to fire team of people because you fall behind with payroll. In other words: be warned and build at your own risk.

whitepoplar 8y ago
This is frightening. Can anyone from DO chime in?
- raiyu 8y ago
  Just making sure to reply to each person that raised a concern. There is a lot of fraud that comes into every single cloud provider as root access to a virtual server can be used for a lot of malicious activity. As a result every cloud provider has automatic and manual processes that they run to find these accounts and flag them. In the case of DigitalOcean when an account is flagged it is locked, which simply prevents a user from creating more resources and an email notification goes out to the user to establish a line of communication. There is no service interruption, and certainly the account, it's droplets, and other resources are not deleted, and never deleted automatically when the account is initially flagged. There are numerous communications that go out even if a user is unresponsive.
tnolet 8y ago
This sounds terrible. I never had any issues with DO. They actually refunded me once when I actually made a mistake. Would love to hear the DO side on this.
esistgut 8y ago
Our team is currently migrating our whole production system from Linode and GCP to Digital Ocean. Your comment is raising warnings, I'd like to hear more from DO itself.
- joering2 8y ago
  Out for curiosity but asking honestly: what made you chose DO from all providers out there? It sounds like you have a serious setup that most likely is production and probably makes money. You do know that DO does not provide over the phone help, right? So if your server goes down you open the ticket and you wait... Yes they do offer SLA because I was in heavy email chain back and forth when moving one of my clients from Rackspace and they were sold on DO being the right choice, but again I was explained no phone support - only that your emails to customer support are prioritized.
- jarland 8y ago
  I'd love to chat with you. If you have some time, send an email over to jdonnell@digitalocean.com and let's talk. I promise nothing but honesty, transparency, ideas, and maybe a few laughs :)
  - zifnab06 8y ago
    Any chance at an invite to the k8s beta for a large-ish open source project?
    - jarland 8y ago
      Toss me an email, I'll see what I can do :)
Scarbutt 8y ago
This is bad, but your data should have also resided somewhere outside DO.
- joering2 8y ago
  I was only technical adviser for a while, but I know they did offsite backups but its hard to have these up to a minute. So they tried to recover the latest version of their DB so that truly no records are missing. Edit: my original post got downvoted severely in few minutes from posting. Hello DigitalOcean staff and/or owners!
  - dewey 8y ago
    I think more background information would be more appropriate than conspiracy theories of what happened to you. Usually there are two sides to a story.
  - greglindahl 8y ago
    How did you figure out that DigitalOcean staff and/or owners were the downvoters?
    - thecus 8y ago
      Don't participate on HN a lot, but I couldn't resist responding to the GDPR violation in the same line as engaging with the NY Attorney General.
  - gamblor956 8y ago
    I downvoted your original comment because it was a bunch of unsupported conspiracy BS and at least one of your claims was directly refuted by the company itself. If you have a real complaint against DO, please provide specific allegations and support for those allegations.
sgc 8y ago
Chiming in with everyone else. I would like more info from OP and DO since I host all my sites there.
plainOldText 8y ago
These are some serious allegations. If this is true then DogitalOcean needs to explain themselves. I guess, one should also have backups off site. Relying on just one provider constitutes a single point of failure.
- thecus 8y ago
  I suspect that any cloud provider has enormous amounts of fraud on it and no tooling is 100% error free. It's also important to note that GDPR has specific exemptions for companies responding suspected security/fraud related issues. I love DO because the performance for the price is great. Also have no issue supporting emerging tech companies with cultures I connect with.
  - plainOldText 8y ago
    No question fraud exists on all these platforms, however, if anyone is wrongly flagged by an algorithm, reaching out to the company must be followed by a prompt and timely response, so any misfortune can be remediated. I would assume fraudsters wouldn't typically reach out to have their accounts reinstated. In the digital age, startups and businesses rely on cloud providers for their livelihood. These providers must be reliable and trustworthy, otherwise they shouldn't get a penny. Also, I'm not intimately familiar with OP's situation, my comments are just common sense generalizations; I think!
    - raiyu 8y ago
      We have an entire fraud and safety team whose sole purpose is to deal with these situations. Every account that is flagged is notified. Every account is communicated with and there are always replies sent. Unless a droplet is actively being malicious, such as sending out a DDoS attack, or performing some other sort of determined malicious activity, there is absolutely no interruption in service. The account is locked so that the account can not create more resources, but there is no disruption to the underlying running resources such as droplets and otherwise. The intent here is to establish a dialogue with the user and determine if the activity is fraudulent or otherwise.
jarland 8y ago
Hey friends! My name is Jarland and I'm on the support team at DigitalOcean. We do have a number of fraud and abuse algorithms, and when we are alerted to potentially fraudulent activity, we take appropriate action, which includes notifying and communicating with individual users. I also want to confirm that we are fully compliant with GDPR.
- drchiu 8y ago
  Thanks @jarland for chiming into this thread. I love how DO has evolved over the past several years and want to comment about that. I use DO for production and have gradually migrated my infrastructure away from AWS and Linode to Digital Ocean as the platform improved. Just a quick question: If the algorithm is triggered (regardless if it is a false positive or not) and the user is notified, what happens with the droplets in the meantime? Is there a grace period for the user to act before DO takes action? And is the whole account frozen or just the offending droplets? It seems the major concern amongst commenters here is the sudden lost of service. Thanks for the great service, and I look forward to your insight on this.
  - donmcronald 8y ago
    Yeah, I want to know if the execution is before or after the trial. Part of DO's appeal to me is the simplicity and predictable (low) cost. It would be really great if they published well defined account termination procedures. Do I get a phone call? An email? Do I get to respond before being disconnected? Is there an appeals process?
    - raiyu 8y ago
      As anyone who runs a service that provides full root access to servers understands there is a tremendous amount of opportunity for potential abuse. It becomes a game of cat and mouse to catch the abusers and prevent them from creating numerous accounts which ultimately impact system performance and can lead to potential problems for real legitimate customers. Those guidelines aren't published specifically because if they were, then the abusers would immediately begin to route around them, so it's meant to be opaque for a reason, but that is against fraudulent use, not legitimate use.
tracker1 8y ago
It may be worthwhile to have a blog post with a (LOT) more details (maybe any correspondence with DO included) and include a link to that.
js4ever 8y ago
This is exactly why i'm ok with paying 10x more for AWS. You can speak with them in few minutes, you have a grace period before service shutdown unlike ovh, DO and most others
thirstybanach 8y ago
I've had an experience like this but with less consequence. I was running a server years ago that scraped information from various cryptocurrency sites and it was flagged as fraudulent. Support refused to say what the issue was or offer temporary access to the server for data retrieval. This wasn't some cowboy unbounded crawler either, it was making very specific requests, around 1000 a day for around two weeks, probably less than an average day of web browsing for the majority of people. I'll never use DO for anything, even testing and mucking around. At the very least you would have to provide excessive evidence of identity if you get flagged by their magic algorithm, and if they restore your service it could be after days of wrangling with support (it took support over a day to answer my support tickets at the time).
- greglindahl 8y ago
  I glanced at the DO terms of service, and crawling is prohibited.
ng3 8y ago
I just had my account locked with no warning or explanation. All 8 droplets were turned off. Account was unlocked 40 minutes later (also with no notification), and I could go in and turn droplets back on. Over 2 hours later and still no response to my support ticket asking why and how it happened. I'll be interested in the response I get. Unless there's a good reason why, or a plan to prevent the how from happening again, I'll be shifting anything critical away from DO, and go back to just using it for spinning things up to play with or test on.
- joering2 8y ago
  Ufortunately nobody here is going to listen to you. A flock of DO fanboys downvoted my comments into obl livion despite I was just merely posting my experience. Some ambulance chaser even turned out to be a psychic reader because he knows better than me myself whether we been provided with GDPR related response or not, because - well, some DO CS worker told so on this forum. I guess getting your droplets cutoff in the middle of business day and waiting 4 days for customer support copy and paste template answer has to happen to everyone before they themselves realize how crucial over the phone support is when it comes to hosting a production website. Its all good anyways.

tracker1 8y ago

You can do custom agents with Azure DevOps (formerly VSTS) or GitLab pretty easily. Haven't played with the new Github stuff. How are you liking the K8s on DO? I just got access a couple days ago but haven't touched it yet, I seem so out of the loop.

dsumenkovic 8y ago
Thanks for mentioning GitLab. You can find more information about Auto Devops and creating Kubernetes clusters from withing GitLab at this doc https://docs.gitlab.com/ee/topics/autodevops/quick_start_guide.html#creating-a-kubernetes-cluster-from-within-gitlab https://docs.gitlab.com/ee/topics/autodevops/quick_start_gui...