24 ms·
"DigitalOcean Killed Our Company"
- garlandcrow 7y agoThis same experience happened to me 6mos ago as well, actually I tend to read this online every month or so. No response from them for weeks, they without warning locked and deleted all instances, data, backups, everything. It wasn't until I posted on HN that I got a response. They said it was a mistake and apologized but...yea...whoops we deleted everything and killed your start up, want a free month of service? So no matter how slick the UI, I will go out of my way to never use this trigger happy company that is killing peoples startups on a regular basis.
- duxup 7y agoIt's interesting how many companies simply shut down service rather than say give a warning and wait for a response (or at least start a clock). Granted that would require people to communicate and use some form of reason. Even DMCA for all its warts fires up a warning and has a response mechanism (granted other issues there).
- jerf 7y ago"It's interesting how many companies simply shut down service rather than say give a warning and wait for a response (or at least start a clock)." I'm sure many people have started their companies firmly convinced that they'll give plenty of warnings and never automatically shut anything down. The problem is, you rapidly discover that doesn't scale, not even on a human level. You send your notice. 48 hours later, you've gotten no response. If you act now, it isn't materially different from your point of view as if you simply acted right away. Also, in a cloud environment, even Digital Ocean, as many people have learned the hard way with leaked credentials, you can rack up charges faster than the relevant humans can even conceivably be notified. As the hosting company, you can't just let abusive or accidental usage go. You can refund their money, but that's still resources of yours that went to something that failed to produce revenue rather than something that did; you can't absorb that indefinitely. I'm pretty sure you'll inevitably discover that you have no choice but to put automation in.
- kelp 7y agoThis is exactly why AWS has relatively low default account limits, and you have to open a support ticket to raise them. It's largely to prevent run-away costs from surprising the customer.
- busterarm 7y agoI accidentally left a 24xlarge instance running for a month without realizing it and they looked at the activity and were totally cool about zeroing the bill for that instance for the month. Basically gave me us a $2000 credit. It does probably help that I said I would be careful not to do that again and had already put in a CloudWatch Alarm to automatically power-off the instance after a set period of idleness before filing the ticket.
- scarface74 7y agoThere have been so many stories of AWS accounts being “hacked” (actually they weren’t. someone posted their keys to a public github repo), the person panicking, then sending a ticket to AWS and then getting a refund. AWS support is excellent - especially on the business tier and above. I will gladly pay the extra money for AWS than to even think about DO or even GCP for a money making project. But more on topic: with Aurora/MySQL you can have an on-site hosted read replica from an AWS hosted database. That would be a cheap, easy real time backup solution if I were really worried about AWS screwing me over.
- duxup 7y agoThe actual cost to Amazon is so low it probably isn't worth insisting on charging the mistakes that contact support.
- dfrage 7y agoThe good will generated by the stream of customer testimonials of this process we hear about is priceless. The proposition seems to go something like this: it's a new thing, mistakes are statistically expected, you make an honest one and plead "oops!" and we refund you, no doubt pointing you to resources on best practices and account throttling. As long as the customer takes the lesson to heart, everyone wins.
- deogeo 7y agoThey locked my account, without refunding the ~$200 balance, with no reason given except "We reviewed the account and found it matches unusual patterns associated with violations of our Terms of Service and Acceptable Use Policy." When asked, they would not reveal which terms were violated. No warning was given, and no way to retrieve any data. Fortunately nothing essential was lost.
- LinuxBender 7y agoI know this doesn't help you now, but you may want to consider distributing your site or setting up DR sites across multiple VPS providers. If your application supports it, you may even want to consider using a DNS provider that can do health checks and fail over the site for you.
- blacksmith_tb 7y agoI have happily used DO for small things, like hosting a Ghost blog, or running an Algo VPN, but I am a little surprised to see people doing bigger infrastructure with them - not that they deserve to lose everything for making that choice, but it seems like it would have been clearly riskier than AWS/GC/Azure?
- tracker1 7y agoI'm surprised that people put significantly more faith into GCP, AWS, Azure etc... similar things have happened in lots of scenarios. Not to mention when registrars have taken down sites. This is a shame, and imho it sucks a lot. One of my biggest points of paranoia are with backups and scripting a return of a site on another provider should the worst happen. Getting ready to launch something barely more than a hobby and was planning on DO because the hosted Postgres and a small K8s cluster is significantly less for what's there than the alternatives. Frankly, I don't want to go from ~$100/month to over $200 for another provider for something that likely won't lead anywhere. Goosfrabah... goosfrabah...
- deleted 7y ago[deleted]
- machbio 7y agoFrom the twitter thread - seems like the problem was resolved.
- deleted 7y ago[deleted]
- nwsm 7y agoHow's that? The last tweet from him I see is asking them for their data still. https://twitter.com/w3Nicolas/status/1134529379701334017 https://twitter.com/w3Nicolas/status/1134529379701334017
- magissima 7y agoDO's cofounder seems to have intervened. https://twitter.com/moiseyuretsky/status/1134547532149854208 https://twitter.com/moiseyuretsky/status/1134547532149854208
- ookblah 7y agothat last response from them is pretty damning. it's like what happens when your customer success turns off their brain and just applies blanket rules to everything. yikes, just when i thought their kubernetes and managed db's were looking attractive...
- zbobet2012 7y agoThey are, just don't rely one _one_ service. You need to be cloud agnostic and this is why.
- scarejunba 7y agoStrong anti-recommendation for this advice. The cost in dev time plus keeping things running is not worth it. This is a rare event. Just use a more well-established cloud service.
- 7y ago
- atonse 7y agoI’m certainly getting downvoted for this. Not at all trying to blame the original posters and victims: While they seem great for hobbyist and small business sites, there’s no way I’d trust Fortune 500 client business to something like DigitalOcean. I just don’t see the benefits over a more established operation like AWS, Azure or GCP. Saving $50 here and there isn’t worth it.
- jressey 7y agoThis is obvious. Start your app on Digital Ocean or Heroku, and when it becomes a viable business get that shit on a grown up platform.
- weberc2 7y agoAccording to the thread the company is 2 people.
- pgm8705 7y agoI can't speak on Digital Ocean and I guess it depends how you define "viable" but I've been running my upper six-figure revenue business on Heroku with tremendous success over the past 6 years. Customer support is super responsive and very helpful. The minimal downtime I've faced had has largely been the fault of AWS.
- mjcuva 7y agoCouldn't agree more! We've been using Heroku at ReadMe for ~5 years and it is easily my favorite piece of technology we use. Off the top of my head I can't think of a single issue that entire time that has been their fault.
- efficax 7y agoHeroku is expensive but fantastic. Unless you have complicated needs and the price point doesn’t shock you, it’s a fantastic way to use AWS while sparing yourself a lot of headaches.
- weberc2 7y ago
- ziddoap 7y agoAn expensive lesson learned in redundancy. It's a shame to see things play out this way, but sometimes a lesson is taught in a brutal fashion. I imagine the author will be more careful in the future regarding off-site backups, additional technical partners, contingency plans, etc.
- Hamuko 7y agoThere's a pretty hard cap on the level of redundancy you can do with a two-man company, as I assume a two-man company does not bring in a lot of money.
- ziddoap 7y agoThe number of employees shouldn't be the deciding factor when you are a tech company that apparently has fortune 500 companies as customers. I'm not talking 5 9's redundancy. I'm talking grab a backup once a week or something, anything, to help mitigate a scenario like this. According to the thread, they lost ~1 year of data. That should be unfeasible to a company serving customers, let alone Fortune 500 customers. Disaster recovery planning is key for a technical company to succeed. It is clear they never considered a scenario where there DO account would be closed/compromised/down.
- Hamuko 7y ago>It is clear they never considered a scenario where there DO account would be closed/compromised/down. I don't think the chance of DigitalOcean automatically freezing your account to a point where only a co-founder can do something about it has been well publicised.
- ziddoap 7y agoIn all practicality, DO freezing your account has the same effect of DO being down (or closing, etc.), or your account being compromised and you being locked out of it. A contingency plan should ideally have been in place for a scenario where, regardless of root cause, you have lost access to your DO account.
- tus87 7y agoCool watching this in real time. Co-founder just intervened and got it back up. Still....
- Hamuko 7y agoRather unfortunate customer service experience that you can't get help from the actual support and you can't get help from the official Twitter account. You just need to pray that your Twitter thread gets noticed hard enough for the actual co-founder to notice it so they can make the call that saves your company. Also, even the co-founder doesn't seem to know exactly why the service was suspended, even though he clearly managed to arrange things.
- rc_kas 7y agoRight? I hate that justice in these cases relies on the person tweeting and then that tweet hooking on and getting popular on Hacker News. So depressing.
- Hamuko 7y agoThis reinforces my gut feeling that companies like DigitalOcean and Scaleway are fine for hobby usage, but for any serious business operations, I'd go with AWS.
- ApolloFortyNine 7y agoWell if anything else this case just made that abundantly clear. Locking a customer resources without warning, and not allowing access to their data, is unacceptable, I'd argue even for hobby services. There's a lot of VPS providers cheaper than DigitalOcean, I don't believe they have room to act like this.
- ApolloFortyNine 7y agoNot even giving the account a warning is honestly enough for me not to go with DigitalOcean for even small projects in the future. Their prices are not that good for fly by night VPSs if that's how they're going to act.
- kwcts 7y agoIf all your backups are in the same account of the same cloud provider then you have no backups.
- bsg75 7y agoThe saying "2 is one, and 1 is none" applies to hosting providers as well.
- kangnkodos 7y agoThe article should be called "I killed my company by having all my backups in one spot."
- amelius 7y agoWhat other business shuts their customers out if they use too much of their product?
- Hamuko 7y agoBars.
- jacobsenscott 7y agoI've never seen that happen.
- detaro 7y agoAny that suspects fraud and doesn't want to be on the hook for the inevitable chargeback.
- taffer 7y agoCaps or quotas are a better way to achieve this IMO. Random shutdowns do not make it sound as if DO is ready for production.
- harryh 7y agoIt's not just the volume of usage that can indicates fraud, but the pattern. In addition, relying on quotas creates a system that is easily games by perpetrators of fraud. Caps or quotas are not sufficient to deal with this problem.
- taffer 7y agoIf a cloud service is supposed to be ready for production, then customers should be safe to assume that they will not simply be shut down, especially not without warning. Otherwise, the provider must make clear that the service is only for hobby use and not for commercial use.
- StudentStuff 7y agoDigitalOcean is operating worse than a fly by night host (like AlphaRacks, GreenValueHost, etc). The reasonable course of action would've been to email the customer and throttle their API access to prevent load spikes, but DO instead locked their entire account (not just the service that DO felt was being abused). A fly by night will often only suspend the VM or database that is in question, not other services on the account (having been in that position before).
- busterarm 7y agoReally? Big European VPS hosts like OVH just turn off your stuff until the problem goes away. Hardly fly-by-night.
- StudentStuff 7y agoHmm, I've done lots of stuff on OVH that other hosts consider abusive, with nary a word of complaint from OVH. OVH seems to have this whole automation thing down cold, monitoring boxes for dying components and letting me run hog wild on their VMs without limit... The one time I used their phone support, the guy I got was fairly helpful. They seem like a hands off company overall.
- busterarm 7y agoIf you get DDOS'd they will stick your VMs in a black hole until the traffic stops to protect their network, no matter how much you plead with them. Good practice, actually, but it's something people usually get frustrated with and they're literally famous for doing it.
- StudentStuff 7y agoThis doesn't seem unreasonable, OVH is not advertising DDOS protection (nor do they seem to be structured to offer it). Some hosts will can your VM and throw out its data, which is a much worse outcome. Edit: I stand corrected ¯\_(ツ)_/¯
- reaperducer 7y agoIs it odd that they didn't keep backups elsewhere? I don't ever back up to the same service that I use for production. Or is that just me being paranoid?
- detaro 7y ago> Or is that just me being paranoid? Clearly not.
- beardedman 7y agoSad sad day for DO. Will be moving my infrastructure ASAP that is with them. Poorly POORLY handled.
- jlv2 7y agoBest Twitter comment (grammar errors and all): "And if your full business relies on one tech partner (no offsite backups) your not doing your tech job right."
- spamizbad 7y agoI mean, sure, but it sounds like it’s a 1-man technical operation.
- Kovah 7y agoBeing just one person is no excuse. I am a 1-man technical operation too, and I know that backups have to be stored on two different locations.
- spamizbad 7y agoYou're correct. But there are also unicorns that don't follow this rule. It's just that their compute activity would never trigger a false-positive, so everyone (except their ops team) is blissfully unaware of their fragility.
- BinaryIdiot 7y agoMaybe. But don't forget this is a small company of just two people. Yes the backups should have been off site but relying on just one digital partner at that scale isn't the worst (they're unlikely to have the money or time to federate out to other services). Yes, ideally they would have already tested their back-up solution, the back-ups would be offsite and, if something like this happened, they could stand-up on another provider. But that just ignores the reality of them being a super tiny business. Almost no one at that size is going to do that.
- tssva 7y agoI disagree. Many 1 and 2 person shops implement proper off-site backups because they understand that loosing their data is a death sentence for their business. Proper off-site backups are neither expensive nor time consuming to establish these days. There is no good excuse for even the tiniest of companies to not implement them.
- nhooyr 7y agoA few years ago I used to run a VPS on DO with a mail server, VPN and some code I was writing. Once I was done with everything, I used their snapshots feature to backup my VPS and shut it down. Two years later I wanted to restore the VPS but turns out my snapshot become "outdated" and they stopped supporting the format for restoration... Support was completely useless, wouldn't even let me download the snapshot, said at most they could mount it into a new VPS and I could recover the data myself. Very unprofessional.
- mountainofdeath 7y agoYou know, there is usually an easy way to mitigate this. When you create an account, be sure to supply business information e.g EIN in the USA and see if you can do invoice billing. Once you get set-up as a business, most providers assume you know what you are doing. These sorts of things tend to slip through the cracks. If you are running is a business capacity, make sure you treat everything like any other business would. Even as a two-man company, you can't afford the cost and potential liability of not operating as a registered business. It also shows that this is a serious business and not a hobby.
- jaden 7y ago> Once you get set-up as a business, most providers assume you know what you are doing. Is there any evidence for this? For all we know his DO account was set up as a business.
- JohnClark1337 7y agoUsing the same cloud service for your backup as your production environment seems like a bad idea
- Kpourdeilami 7y agoNot even surprised by the way Digital Ocean have handled this. They pulled something similar on me back in 2014 at a previous company I used to work in. They essentially shut down my account and did not even let me get my backups out.
- wvenable 7y agoIt seems like there should be a middle-ground between all-on and all-off. If I'm paying customer, I should be able to access my account in some capacity even if some abuse related issue closes off server access.
- mugwort13 7y agoDude has no disaster recovery plan, so he throws a public hissy fit like a spoiled child. Public shaming as a customer support proxy is shameful. It’s a form of manipulation.
- deleted 7y ago[deleted]
- morpheuskafka 7y agoWho is good about not locking accounts or taking similar actions? Apple and Google are both notorious for blocking things for no reason, is AWS or Azure any better?
- keypusher 7y agoThey are all going to have some form of automated protection against malicious activity, but I suspect AWS and Google’s algorithms are better than the others. My experience with AWS and Google in general is that your treatment varies with your support plan. With business or enterprise level, you have dedicated resources within the company that are going to be aware of such issues or can escalate and sort it out quickly. I understand not wanting to shell out the base cost for enterprise if you are a small company on a budget, but paying at least for business support is a good idea if you are actually running a business. I have never actually heard of this happening to such a customer though, so perhaps they have extra processes in place?
- brlewis 7y agoI've only had one instance where Linode contacted me about suspicious activity. I responded promptly, it was obvious they understood my answer, and nothing came of it. Another comment thread on this story has people saying Linode's support is better.
- _bxg1 7y agoLooks like Moisey Uretsky personally intervened fairly quickly: https://twitter.com/moiseyuretsky/status/1134547532149854208 https://twitter.com/moiseyuretsky/status/1134547532149854208 That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.
- vbezhenar 7y agoProbably because of publicity. How many of those companies went bankrupt silently, because their case did not cause much attention in news?
- _bxg1 7y agoSure. Hopefully it results in a change of policy, or at least a public statement of some kind. Everyone can't depend on the cofounder to come in and save them from bad automation.
- ummonk 7y agoFrom the looks of it they will be posting a public postmortem to their status page.
- hn_throwaway_99 7y agoAgreed, but it's not like the original poster had a huge platform, he just posted about it on Twitter. I may despise Twitter for a bunch of different reasons, but I can't deny it's a great tool for raising issues to companies.
- jeremyjh 7y agoHe has 4300 followers on twitter. What if he had 50?
- scotchio 7y agoAccidents can happen. Don't really blame Digital Ocean for the accidental locking but this response is insane: https://pbs.twimg.com/media/D76ocofXoAY_xB5.png https://pbs.twimg.com/media/D76ocofXoAY_xB5.png
- bww 7y agoI can understand why a cloud platform would proactively disable workloads that appear to be acting dangerously. At the same time it seems quite unreasonable that there's no mechanism for a customer to get their data back when their account is unilaterally shut down.
- vbezhenar 7y agoThey are reckless to trust a single company with their backups worth of money. I don't trust Google and Amazon with backups of my wedding video nobody cares about, so I'm using my home NAS with raid as a third additional storage (along with Glacier and Coldline). And they just trust DigitalOcean with all their data, assets? Crazy. Don't put all eggs in one basket. Register domain from one company, host DNS in another company, host your servers in third company, keep your backups in multiple places owned by multiple companies, test your backups periodically, have migration plan if one company decides to ban you. That's another reason that I'm very skeptical of Amazon and Google offerings of their proprietary API. Rent virtual servers, no problems, you can rent those servers from a thousands of hosters. But if you're using their proprietary APIs, you will have to rewrite your software to migrate and you will have very tight time frame.
- wpietri 7y agoDoes anybody know what they actually do? I dug around to find a Wayback version that actually had text. I understand the words on their own, but when put together I get nothing: https://web.archive.org/web/20181030015237/https://raisup.com/en https://web.archive.org/web/20181030015237/https://raisup.co...
- foota 7y agoSounds like they give companies a feed of startups they might be interested in acquiring?
- the_common_man 7y agoI think it helps large companies track up and coming startups.
- ndiscussion 7y agoTypo on the front page, how you do one thing is how you do everything. >powerfull
- jjeaff 7y agoYa ya, and sometimes a small typo just slips through.
- cpeterso 7y agoAutomated off-site backups for startups
- shuzchen 7y agoIf you're reading this and concerned for your own backup story, fret not! In 2019 secure off-premise backups are super easy to implement, even for a 1 person shop. Get something like Restic or Borg or any one of the enumerated options here: https://github.com/restic/others https://github.com/restic/others I've recently implemented backups with Restic, the static binary and plethora of supported storage backends was extremely appealing. The easiest seems to be to just point it at a S3 bucket, but given most people have infrastructure on AWS (off-premise means off-premise) having other options supported out of the box is pretty handy.
- type0 7y ago> The easiest seems to be to just point it at a S3 bucket, but given most people have infrastructure on AWS (off-premise means off-premise) having other options supported out of the box is pretty handy. Sure, having other options is good: https://min.io/ https://min.io/
- shuzchen 7y agoMinio is great! I use it! But spinning up and managing yet another service when you're already a small shop adds more barriers to entry. Maybe find an already s3-compatible store (like Wasabi) or find something cheap and easy to spin up that's supported by the tool (like https://www.hetzner.com/storage/storage-box https://www.hetzner.com/storage/storage-box)
- hartator 7y agoHow do you restore DO Spaces files? Real backups are freaking hard.
- jammygit 7y agowhat considerations go into choosing between one option and another? Is there a blog post I could read on the topic?
- copperx 7y agoHow do you verify your off-site backups? Do you periodically download the entire backup set and bit compare to the originals? That sounds like a lot of bandwidth usage, not to mention costs.
- chrshawkes 7y agoShould have went with Linode
- pkrefta 7y agoThe quality of DigitalOcean communications in this case - especially email responses look like worst examples from some pretty terrible corporations. I thought DO was (is?) trying to differ from them.
- bitL 7y agoHeh, that reminds me when a director of one unit of a "world's top 10 brand" I was working at on big data architecture told me that they always knew when I ran anything on the cloud as I brought it down within a few seconds of my heavily multi-threaded processing scripts. I guess DigitalOcean decided that instead of fixing their infrastructure they just kill their own smart clients. I find this enforced mediocrity pretty appalling. With barely functional "anomaly detection" Deep Learning models with dubious decision making (I did some so I am familiar with the "landscape") it's gonna be a lot of fun for anything slightly deviating from whatever vague norm that can't be explained nor tested against.
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- drugme 7y agoWe lost everything, our servers, and more importantly 1 year of database backups. We now have to explain to our clients, Fortune 500 companies why we can’t restore their account. And yet, the explanation is very simple: Because you neglected basic principles and elected to put all of your backup eggs in one basket.
- zupreme 7y agoMy reply (pasted from original tweet) This is why you need, at minimum, a nightly mirror. Ideally you stack on top of that a load-balancing device or service to redirect traffic in the event of an outage from your primary farm. Never go on holiday again until you have backups and failover. #idothisforaliving
- unixhero 7y agoOkay. Brb. Deleting my DO account and instances. Migrating to elsewhere.
- charlus 7y agoThis isn't assuring to read at all. I'm trying to build and still relying heavily on DO, for what is becoming thousands in profit quite quickly. Frankly, I'm very uninterested in resolving a problem in hosting - mostly because it's boring, but because DO seems nice to use in every other way. On the other hand - I'm becoming increasingly aware of the inevitable Twitter-social-media-pile-on for any company ever.
- sergiotapia 7y agoDigitalOcean better respond, expeditiously (lol TI) or I'll never use them for anything ever again. Extremely dangerous behavior.
- thaumaturgy 7y agoSome people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it from getting set up. I sent out a support request, and a while later, over an hour I think, I got an equally unhelpful support response back. Things got cleared up by taking it to Twitter, where their social media support folks have got a great ground game going, but I really don't want to have to rely on Twitter support for critical stuff. DO seems to have gone with the "hire cheap overseas support that almost but doesn't quite understand English" strategy, whereas the tier 1 guys at Linode have on occasion demonstrated more Linux systems administration expertise than I've got.
- slig 7y agoI've been with Linode since 2009, and only got bitten once when their entire Atlanta DC went down near Christmas. I was eager to try the new DO managed Postgres service, but I guess I won't after this blunder.
- millettjon 7y ago+1 I have used both as well. Linode support has always been good and DO support has been a blocking barrier.
- znpy 7y agoI have interviewed with DO and they tried diverting me towards a support position. They told me that on a single day a support engineer was supposed to help/advice customers on pretty much whatever the customer was having issue with and also handle something between 80-120 tickets per day. It's nice to see that DO is willing to help on pretty much anything they (read: their team) has knowledge about, but with 80-120 tickets per day I cannot expect to give meaningful help. Needed EDIT: it seems to me that this comments is receiving more attention than it probably deserves, and I feel it's worth clarifying some things: 1. I decided not to move forward with the interview as I was not interested in that support position, so I have not verified that's the volume of tickets. 2. From their description of tickets, such tickets can be anything from "I cannot get apache2 to run" to "how can I get this linucs thing to run Outlook?" (/s) to "my whole company that runs on DO is stuck because you locked my account".
- MrStonedOne 7y agoI see no need for DO to shut down accounts and active long established nodes for a fraud check. Disable the offending node, disable making new ones, and limit editing existing ones to off/on controls. Regardless of this example of a false positive, locking whole accounts over that is unwise.
- coldtea 7y ago>We lost everything, our servers, and more importantly 1 year of database backups. We now have to explain to our clients, Fortune 500 companies why we can’t restore their account. I think what you have to explain is why there wasn't a contingency plan, with your own servers, colocation, another cloud offering, etc...
- PostPost 7y agoWhen AWS has gone down in the past, it's severely impacted massive tech companies like Netflix and Spotify. Why would there be an expectation that a 2-man shop have "another cloud offering" as a contingency plan when some of the biggest and best tech companies do not? People use services like AWS or DO because they are the contingency plan - they have the size and scale that smaller companies cannot afford or implement.
- prophesi 7y agoThe difference is that when AWS goes down, Netflix/Spotify still have backups and could adapt infrastructure if the outage involved permanent data-loss. You're talking about the people who built https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey I'd argue that it should be _easier_ for a 2-man company to adapt to cloud service outages, as they likely don't have to keep up with nearly as many backups or moving parts.
- nikanj 7y agoAnother practical difference: When AWS goes down, it makes the news outside the tech bubble too. Customers are much more likely to forgive you, if you send a link to NY Times that says "Amazon is suffering a global outage, affecting tens of thousands of companies"
- Dylan16807 7y agoSo pretend they had offsite backups. That's a separate issue from an entire contingency plan. The ability to adapt is not the same thing. This company could certainly adapt to a new host if they had an extra backup.
- hyperpape 7y agoTo everyone justifying the lack of a backup strategy by saying they’re a two man show: As far as sympathy goes, you’re not wrong. But you’re also justifying every pain in the ass procurement process you’ve ever dealt with. Your attitude is why so many companies won’t go near a two man shop.
- tarsinge 7y agoI would agree if they were reckless, but here they had a backup strategy, just the risk of a complete sudden shutdown from their provider was not taken into it. This is a new risk that must be taken into account, and I’m not sure procurement would have scanned this. Also the pain in procurement is usually because it’s in fact on irrelevant arbitrary administrative things and not on real risks like this.
- tomcam 7y agoBeautifully put.
- NoblePublius 7y agoSounds like you killed your company by not keeping a local physical backup.
- throwaway9870 7y agoFortune 500 customers with no off-site DR backup? Hm, not really a good plan.
- unethical_ban 7y agoI see a lot of people saying "They should have had more than one provider" and "They should have had better backups". What I'm saying is: If you ARE going to put your business at the mercy of one company from top to bottom, would it not be wise to try to get some kind of account rep? Or have some sort of communication with the company as to the nature of your operation? And if that isn't an option, should you do business with them?
- grayed-down 7y agoWe do a fair amount of business with DO to the tune of about 40+ droplets used together and separately for various tasks. While we could certainly survive the loss of these assets, the recovery would be long and costly. So I would certainly say that this story gives me a great deal of pause and will take up some mental space this weekend as I think about future dealings with DO.
- docker_up 7y agoI thought Fortune 500 companies (at least the enterprise companies I dealt with) that had checklists for their SAAS vendors that had to check off things like Disaster Recovery Readiness, etc. At the very least this company learned a hard lesson about Disaster Recovery best practices. Hopefully all the up and coming companies reading this story learns as well. Also please remember that a backup that isn't tested IS NOT A BACKUP! I've been in so many situations where backups were corrupted, so part of the disaster recovery is to test the backups and make sure you can really recover. There are a million different scenarios where their data could be lost and it not be Digital Ocean's fault. It's the company's responsibility to have protected their customers from this. I worked at a company with no real Disaster Recovery plan. I was told that "we can get the servers up and running within 18 hrs if we had an outage", which not only was absurdly slow but probably an underestimate. Only by the grace of God did we not suffer a real outage but if we did, it was totally the VP's fault for not addressing my concerns.
- the8472 7y agoSuch checklists exist, but not all services are of equal importance to the customer, so they are not vetted in equal detail. Often the supplier has to fill out the checklist themselves. > do you have backups? Manager: [X] > are they offsite? Dev: cloud provider docs say so Manager: [X] Customer: Great, you won the bid.
- korethr 7y agoMy personal experience such questionnaires is that the questions can be vague, redundant, or not even really ask anything at all. But, even when the questions are concise and precise, it's all to easy to "word smith" an answer that seems to give the desired answer without every actually saying anything. And the person administering said checklist may not even know enough about what the checklist is checking to vet the answers.
- m-p-3 7y agoThey literally put all their eggs in the same basket by having the backup and the production environment at the same provider. Not saying this shouldn't have happened, and hopefully they have learned from this experience.
- ehutch79 7y agoSoooo... an automated script flagged them. they get themselves unflagged, and proceed to do the same thing immediately. without any real confirmation that they wouldn't be flagged again. They've basically been flagged as abusing the system multiple times, and they're surprised they had to kick up a storm to get themselves reactivated again? Not to mention, that process they need to run every couple of months, that takes 1s, but they still need to parallelize over a bunch of vms, that's weird and sounds like something that needs to be rearchitected at the very least.
- harryh 7y agoMy interpretation of that was that the batch job takes 1s per row or something like that rather than 1s total. It obviously wouldn't make sense to spin up 10 nodes to turn a 1s job into 10 0.1s jobs. Just a guess.
- keypusher 7y agoGiven that the author was quite vague about the nature of this “pipeline” and that their product is an “AI-powered Startup Selection engine”, I have a suspicion they were probably crawling and scraping a whole bunch of pages for new startups. It’s possible that this was totally legit and it just looked like a ddos attack, or that it was something else entirely, but everyone here seems to have taken him at his word that what they were doing was actually above board.
- voldemort1968 7y agoI agree that we're not getting the full story. On the other hand, completely shutting down all their services without a quick conversation...
- deleted 7y ago[deleted]
- dsl 7y agoHaving been on the receiving end of terribly broken "pipelines" at startups wanting to hammer away at my resources, the right response is terminate first discuss later. I know of a company that explicitly had a "call us to discuss first" clause in their contract with a smaller cloud provider. Everyone was on holiday and not answering the phones while their hacked account was being used to spin up dozens of boxes launching a DoS attack against a crypto scam site. Guess who had to eat the bill on that one?
- chris_wot 7y agoSurely there was a support contract option where there were on-call people?
- timvisee 7y agoIn case something looking like a DDoS, they tend to disable network access to droplets until you manually 'resolve' it. Instead of destroying the whole account.
- 7y ago
- csomar 7y agoI think, going forward, a data backup is not going to be enough. You’ll need a full devops plan b/c where you just redeploy to another hosting provider. Even better: run your infrastructure across multiple hosting providers with something like consul. DO might cause slower service but not a death sentence.
- jacobsenscott 7y agoA corollary is keep your infrastructure simple, and know how it works, so you can stand it up anywhere. Web server/db/memcached can run anywhere easily and is enough for almost every company out there. Are you sure you need {cool new service}? Are you really really sure? Your database can probably do it.
- BinaryArcher 7y agoThis exact same thing happened to me last year. I accessed my account abroad and they perma banned me. Support was useless and even with evidence did not believe who I was. I then somehow convinced them to give me temp access, which in my opinion is even worse. They didn't believe me about who I was and then gave me temporary access to an account. DO can't be trusted when their support team could so easily be socially engineered.
- holdenc 7y agoObviously concerning. Care you provide more details on this? I respect your desire to stay anon, but any details to add some color would be great. For example, did you completely leave Digital Ocean, and if so, where did you go?
- fendy3002 7y ago> They didn't believe me about who I was and then gave me temporary access to an account. What???
- system2 7y agoOkay, if this is real, I am concerned. We have 40+ droplets with many clients. If anything happens like this we will lose our entire operation, as well as all of our client's confidential ecommerce data.
- dahfizz 7y agoI think a real takeaway here is to avoid going all in on one hosting provider. You should always have "off-site" backups for mission critical data. Yes, it sucks that DO did this. But this is hardly the first time someone got screwed over by some poor AI automated security. Backing up your data to backblaze or AWS would be the cheapest insurance policy you could buy.
- deleted 7y ago[deleted]
- EGreg 7y agoThis is very sad, and why we all need to make the Web decentralized and own our own data. Here, several Fortune 500 companies were relying on a two person team, which itself was relying on a hosting provider with full discretion to shut it all down. What could possibly go wrong? https://qbix.com/platform https://qbix.com/platform is one of many projects working to tackle this. Tim Berners-Lee’s SOLID project and others are, also.
- luhn 7y agoIt's easy to think your data is safe when your cloud provider advertises 11 9's of durability. But three replicas of your data doesn't protect against things like this. Even in the cloud age, offsite backups are important: Different provider, different region, different payment method. Unfortunately it doesn't help that cloud egress bandwidth is criminally expensive.
- mettamage 7y agoOut of curiosity, does anyone have some resources to have a homegrown DO/AWS server space? Say I have a friend at all the continents (including Antartica for hypthetical fun-ness), they are all willing to allocate some square meters to plunk down some servers for whatever is needed. How would I go about it and build this small(ish) infrastructure myself?
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- rocky1138 7y agoThe old saying still stands: There is no such thing as the cloud, there is only someone else's server. Hopefully everything gets righted and these folks can start making solid, multi-site backups.
- cptskippy 7y agoLet me see if I get this straight... Developer has a Python script that takes 1 second per record to execute and he has 500,00 records to process, so he spins up 10 distinct VMs each running the same Python script to parallelize the task. The provider shuts him down and cites a section of the EULA that says "You shall not take any action that imposes an unreasonable... load on our infrastructure." Basically saying "Hey whatever you did, don't do that." Developer gets his account restored and then proceeds to do exactly what he did to get it locked out the first time around. Also Developer has all his eggs in one basket. Shitty customer/product service aside, someone explain to me why DigitalOcean is at fault here?
- truncate 7y ago`unreasonable load` sounds pretty vague. What counts as unreasonable? 10 VMs doesn't sound like much, and I believe if I'm renting a VM with XYZ specs, I should be allowed to use up-to max capacity it says so in specs. What am I missing here?
- unilynx 7y agoAlso, digitalocean has a hard droplet (VMs) limit that requires a ticket to raise. You would expect them to be okay with you staying in that limit.. (And you cant just start 10 hugely expensive VMs either, the larger sizes are initially locked too)
- aftbit 7y agoWhy would DO be upset about spinning up 10 VMs then spinning them down again? Isn't this exactly the point of cloud providers? This is what they bill me for, right?
- cptskippy 7y agoI guess it depends on what each of those VMs is doing. I don't trust the Dev's explanation tbh. > to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done. Tildes preceding numbers means "approximately". Why doesn't he know exactly how many VMs he spun up? Was he actually spinning up 1 VM per record and only allowing 10 VMs to be running concurrently? I'm not a Pythong dev but why can't you execute 10 instances of Python on a single VM? If you need to dedicate an entire VM to processing 1 row, what the hell is it doing?
- andrewstuart 7y agoWhy is killing accounts part of the way they do business in any way, EVER? That's what destroys company reputation. I may be wrong but my understanding is that the gold standard - Amazon Web Services - will only ever suspend your account until an issue is sorted out. Whoever runs Digital Ocean needs to stand up and say very loud and clear to this community that they will never, ever delete accounts - if he doesn't then he can live with the business destroying reputation of Digital Ocean being "one of those account killing companies". What company would ever host on "an account killer" - the risk is way too high.
- duskwuff 7y ago> Why is killing accounts part of the way they do business in any way, EVER? Because fraud and abuse exist. Sometimes customers really are doing malicious things which need to be stopped immediately, and the only thing that will make that happen is disabling their account. Trying to make accommodations for those users is a fast path to getting sued, getting blacklisted by mail providers, and/or losing your upstream connection entirely.
- dcbadacd 7y agoDisabling != deleting.
- emdowling 7y agoWhile DO is not without fault in this story, any Fortune 500 company would have to be pretty stupid to work with this company after they admit to not storing backups in other places. I feel for you, but a backup is not a backup if it’s subject to the same weaknesses as the master copy.
- unixhero 7y agoTrue. Any sane Fortune500 should do it's supplier Due Diligence before signing over business critical functionality to a third party.
- joeldg 7y agoWell, there goes me ever using digital ocean for anything
- oedmarap 7y agoI agree with the other comments about not having backups in the same place, and ensuring that you distribute your assets (domain, DNS, compute, backups, etc.) across as many providers as possible. One thing I will add is that, especially for a small shop or project, assume from the get go that by renting infrastructure from DO (or any provider) that user-hostile actions can and will be taken when it comes to any issue regarding TOS violations that you are unaware of. This assumption helps to build redundancy in your mindset. Have a production website or app in DO for example? Droplet backup, periodic snapshots, B2 server backup, S3 tar backups, containerize apps if possible, have equally provisioned (smaller, idle VMs) infra on another DO account or another provider if possible, and so on. I know this is overkill but paranoid sysadmins/devops are always rewarded. Just to add some context for DO specifically, they're a great provider in my anecdotal experience and they are constantly rolling out services aimed at medium to large scale workloads, such as managed databases and k8s. That being said, it's entirely possible to transfer snapshots [0] to another DigitalOcean user account or teams account. So at the very least, create an entirely new DO account just for holding snapshots, outside of the native droplet backups and third-party backups you're doing on an application level. [0] https://www.digitalocean.com/docs/images/snapshots/how-to/change-owners/#transfer-a-snapshot https://www.digitalocean.com/docs/images/snapshots/how-to/ch...
- INTPenis 7y agoAs someone who has spent the last 8 years in big enterprise it amuses me slightly that small businesses are discovering this now. Because where I work our customers often have contracts that make us accountable for every single hour of downtime. Requiring us to pay customers penalty fees if their service level is affected.
- deleted 7y ago[deleted]
- bcooks 7y agoAs DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community. This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happen, and we take every effort to get customers back online as quickly as possible. In this particular scenario, we were slow to respond and had missteps in handling the false positive. This led the user to be locked out for an extended period of time. We apologize for our mistake and will share more details in our public postmortem.
- harshalizee 7y agoSure, but the email he received basically said "your account is locked. No other info. Thank You". That to me is a much scarier thing than anything else in the thread. How can anyone trust in your infrastructure if your standard protocol is literally just shutting down their entire operation without any form of review or communication?
- throwaway34241 7y agoYou can't, obviously. Even though I've used them before I really doubt I'll ever use DigitalOcean again. I can almost understand terminating customers (with notice) via automated heuristics for suspicious behavior, especially on the low end of the hosting market, but locking out a legitimate paying customer from backups with no notice or recourse is terrifying.
- jasonjayr 7y agoWe have a relatively large spend ($5k+) @ DO, for a unique client (most of our other clients can be served by our colocated facility), and I'm going to second this. Or with any other provider. They should always explain exactly which rule was broken. If the customer is legit + genuine, they will promptly fix the issue and won't be a further problem. Being vague makes it super troublesome to rely on any service that takes that tactic. (Like Google, for example) If they continue to re-offend, and find other ways to skirt the rules, that's when you move on to account termination.
- deleted 7y ago[deleted]
- warp_factor 7y agoI would love to hear what would all the advocates of going "100% in the cloud" think about this.
- tgsovlerkhgsel 7y agoThe timeline is interesting in itself: This story was posted 4 hours ago on Twitter, 3 hours ago on HN. The cofounder picked it up 3h ago. DO responded and apologized from the official account 2 hours ago, claiming it was fixed, and is actively responding to people tweeting at them, doing damage control, since about 1 hour, promising a public postmortem. While it's sad that a social media escalation was needed (and it confirms that getting attention on social media is the only effective way to resolve hard issues like this), the response after that was quite fast. Let's see how well and how quickly they deliver on the postmortem.
- rhacker 7y agoI am a software developer and yes software is eating the world. One of the side effects of software eating the world is out of control software. * Autobans in facebook * Cheated instacart drivers * $10000 stolen from thousands of bank accounts (and returned hopefully) on Etsy * Tesla cars literally killing people (it now feels like it's once a month right?) Now the software that runs software is running amok. The interesting thing about software is that it runs very quickly and it acts as a giant lever that affects the entire world. You can think of it like a giant airport suddenly being installed in your back yard and just start having planes take off, changing your $300k investment into a $120k valued house overnight. That's how quickly software is changing the real world. I know there is at least one HN browser writing a book on it. But I would love to see more books on how the internet, and software, is messing our world up.
- karlmcguire 7y agoThis reminds me of recent talk by Jonathan Blow [1], where he talks about how we've made very little progress in the field of software and anything that appears to be progress is just software leveraging better hardware. It's quite scary how low our standards have gotten. [1]: https://www.youtube.com/watch?v=pW-SOdj4Kkk https://www.youtube.com/watch?v=pW-SOdj4Kkk
- fogetti 7y agoI agree with you, and I was coming here to find similar thinking folks in the comment section and the most worrying thing is that I haven't found any except you. Most people seem to take it as a normal thing that software simply ruins lives and businesses. Todays developers' simply have no ethics. Also most developers seem to think that they can do whatever and they can get away with it, because y'know "Reasons". Sigh. :( I really don't understand this sheeple thinking, for example most people simply don't understand that an automated fraud detection system is not a technologically important thing, it's an economically important cost management system effectively. Just like companies externalize the cost of helpdesk personnel by operating an automated call center (and by proxy making the customer bear the cost), the goal is the same with fraud systems. But we cannot just simply throw our arms up in the air or shrug our shoulders when the companies leverage our lives this way. Take Facebook for example, they acted like they had no responsibility or any power to review and take down or prevent toxic and/or hateful comments by employing human reviewers until they were forced to do so in some countries. And guess what, they had no trouble doing so, their profit might have reduced somewhat, but not that much. So all in all, anyone who thinks he has integrity as a developer should take a look into himself when he justifies systems like fraud systems (or any other unnecessary cost reducing actions) as necessary. They are economically beneficial, yes; necessary, no.
- elcritch 7y ago> After sending multiple emails and DM on Twitter they unlocked our account, we got 12h of downtime and got a nice Wait their entire business was effectively shutdown and all they did was send an email? Granted the DO handling of a possible abuse situation is shocking, but to allow your business to go down for 12 hours and not be trying to call any and every actual human being at DO seems to be negligent on their part. While us technical types love interfacing via digital means, some situations benefit from actually talking to a live human.
- megaremote 7y agoThis is a 2 person startup, one of them was enroute to portugal on holiday.
- perrygeo 7y ago> Every 2-3 months we had to execute a python script that takes 1s on all our data (500k rows), to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done. Wait, a whole distributed computing sub-system to make a 1s process faster? > I got their final message right after arriving in Portugal. Did he initiate the script from a IP address outside France? Though that might explain why DO's fraud detection was triggered, it doesn't excuse their actions. Send an email first, jeez.
- RootKitBeerCat 7y agoThe way Digital Ocean handled this, I am willing to bet the OP never has trouble with his DO relationship again.
- simonebrunozzi 7y agoIf you run mission-critical apps for F500 companies, you HAVE to have a backup/DR policy in place, using a different IT provider than your main one (and/or a different infrastructure site). DO in my experience is not the greatest company to work with; but at the same time, your "incompetence" killed your company, not DO.
- dmh2000 7y agoQuestion : so what do you do besides host it yourself? Do you set up a backup on a different cloud service so that at least you can failover without too much downtime? you then have to at least pay for the some of the resources even if there is no traffic. or backup locally but have a process set up with the different service so you can provision and get back up quickly? or is there a better solution?
- ramraj07 7y agoAt least have a plan (which you test out once) of how you will migrate to another provider in case your current one screws you like this? All you will need to do continuously then is dump a backup copy of that data into they providers storage..
- superasn 7y agoI think an important thing you can learn from this story is that you should keep your backup on a different host(s) or better even have replication enabled. In these days, most apps generally can be migrated to a new host in seconds as long as you have the data source alive. If they had access to thier data they probably should have been able to spin up a similar ec2 instance in minutes and say goodbye to DO forever.
- ericax 7y agoWe had a similar incident with DO 8 days ago. It didn't kill our company, but we got hit hard. Our business is Dynalist, an online outliner app. Many of our users store all their notes on Dynalist, so uptime is really important. Starting 7 PM last Tuesday, we saw a slowdown in request handling. We filed a ticket with DO 2 hours after that (we also posted our initial tweet to keep our users informed: https://twitter.com/DynalistHQ/status/1131087411797270529 https://twitter.com/DynalistHQ/status/1131087411797270529). A few hours later, we started to experience full downtime. Still no reply from DO. We filed another ticket with the prefix "[URGENT]". Still no reply. We waited for 24 hours for their reply. We took turns taking naps because we're only a 2-person team. After 24 hours, we tweeted @ DO (https://twitter.com/DynalistHQ/status/1131397013306847232 https://twitter.com/DynalistHQ/status/1131397013306847232). 2 hours later we finally got a support person working on our ticket. We didn't want to take it to the social media, but there doesn't seem to be any other way at that point. DO doesn't have phone support, and us "bumping" our support ticket didn't work either. After 2 hours going back and forth on the support ticket and providing logs, DO's support person identified the issue and offered to move us to a less crowded server. They asked us what's a good time to do a manual migration if a live migration fails, and we replied immediately saying whenever is fine (we're experiencing downtime anyway). We thought it's over, but we were so wrong. They didn't reply in another 4 hours. That was 4 hours of more downtime. Sometimes, CPU steal is down a bit and our server could catch up some requests, although it would still take 10 seconds for our users to open Dynalist. But most of the time, our web app was totally inaccessible. Watching the charts on our dashboard go up and down felt like some of the hardest hours of my life... mainly because there's nothing we could do. 4 hours in, I realized we had to post another angry tweet to get a solution. There's nothing else to do other than trying to stay awake anyway. So I posted another tweet: https://twitter.com/DynalistHQ/status/1131497962184564737 https://twitter.com/DynalistHQ/status/1131497962184564737 This tweet didn't seem to work. Nothing happened in the next 3.5 hours and things started to feel surreal. I didn't know how much longer this downtime is going to last, and I didn't know what we were going to do about it. At that time, it was 9:30 AM EDT and people were starting their day. We were getting more and more emails and tweets asking what is going on and where are their notes. A few customers were angry, but most were understanding and supportive. At 9:55 AM EDT, DO finally did the live migration a few minutes before the time limit we gave them, which was 10 AM. That was the end of the incident; CPU steal was down to < 1% and Dynalist was finally up again. However, we couldn't trust DO any more. This weekend we're migrating to a dedicated server provider which has phone and live chat support. DO is pretty good for spinning up a $5 box quickly to test something, but we learned the hard way we shouldn't rely on it. Our postmortem post: https://talk.dynalist.io/t/2019-05-22-dynalist-outage-post-mortem/4872 https://talk.dynalist.io/t/2019-05-22-dynalist-outage-post-m...
- a2tech 7y agoYou can not rely on one company. Let me repeat that. You. Can. Not. Rely. On. One. Company. If you are you’re negligent and you deserve what you get. You need your backups to be good and to be with a second provider. This is NOT rocket science. Treat your company seriously if you expect to sell services.
- 55555 7y agoI dont have a Twitter but maybe I'll get one. It seems like a cheat code to getting real support in 2019.
- elagost 7y agoIt's always sad to see these stories because they always seem like a preventable tragedy. DigitalOcean, Azure, and AWS seem like the go-to for start-ups these days, instead of self-hosting your stuff at home or even in a colocation space. Even though it's a "dirty word" these days, on-prem does have its huge benefits. Professional stuff is one thing, but that's not to mention anything personal - anything I care about I won't put exclusively on someone else's computer. I want to have absolute control over as much of my stack as I can. Seems really scary that some company has control of your entire infrastructure and can ban-hammer you without notice, permanently, at any time of day or night (or while you're on vacation).
- jasonlotito 7y agoThis is not the first time DO has done things like this. I would not trust DO with anything critical, and have advised people in the past to not use DO. Nothing here surprises me.
- adem666 7y agoI now think twice about digitalocean, which was looking to me very nice company before.
- brucemoose 7y ago"DigitalOcean Caused an Outage for Our Company" seems like a more accurate title.
- nine_k 7y agoSomething that made me scratch my head: * The company quickly creates and starts 10 VMs and triggers an automated lock-down, with a message mentioning a sudden spike of activity. * The company gets the account unlocked. * The company again quickly creates and starts 10 VMs, and triggers the auto-lockdown again. Note to self: when something damaging happens as a result of a seemingly normal action, avoid doing that same seemingly normal action immediately again, lest the damaging consequences hit again.
- ginkgotree 7y agoWow. well, I will avoid DO at all costs for the rest of my career. There is a reason Enterprises trust AWS, GCP, Azure, and pay for premium support (and startups should, to the best of their financial abilities). He needs to sue or negotiate to recover material damages caused by this.
- dkersten 7y agoWhile this incident certainly puts me off DO too, the GCP horror stories are equally bad.
- sidlls 7y agoDO didn't handle this we'll, but a company that wants to serve "Fortune 500" customers ought to have a more mature process for handling an outage like this. The fact that they didn't makes it hard to view them as a serious, credible business.
- robertAngst 7y agoHoly crap. Well that is a story worth sharing. So... Never trust DigitalOcean for anything important.
- cavisne 7y agoBookmarking this for next time there's a HN comment "why is Netflix using AWS, they should just get some cheap VPS's from DO" The rise of the public cloud is pretty fortunate for companies like DO, they can make the same assumptions that legacy VPS companies did (most vm's sit idle so oversell them massively, most customers will create a single vm and nothing else) while branding themselves as having the same strong infrastructure as AWS.
- freehunter 7y agoHas anyone ever actually said Netflix should switch from AWS to "some cheap VPS's from DO"? AWS offers a lot that DO doesn't. They're hardly even comparable.
- xvector 7y agoRan a Minecraft server in college via DigitalOcean. Was my first experience with a VPS. They locked it down after looking at the console logs and deciding it wasn’t “educational” enough for them.
- corobo 7y agoWere you using credit from something education related?
- cybdestroyer 7y ago1st rule of businesses...don’t keep all your eggs in one basket. Is this amateur hour?
- kristopolous 7y agoThis is why I always use many providers. I'm talking about $10/month or so packages, this is pretty cheap. They're also not all on the same card in case one gets locked I use 4 for my current company and have redundancy spread over them so that if, say, AWS goes down or I get an account locked or whatever, nothing is lost, things continue to be operational, slight degradation happens and that's it. This really isn't that hard to set up. Under a day or so and then just do stuff in ssh config and the shell rc to act as helpers so you remember how to do things. It's super cheap, pretty easy, and robust. It's awful what happened to this guy but it's kinda like the person who backs up to the same harddrive as the originals. Awful to lose stuff, it shouldn't happen, but also don't do that.
- Epopeehief54 7y agoThis is why I never rely on just one provider. Brutal to find this happen to your company. I can’t imagine the heart sink feeling you had with this.
- neop1x 7y agoMaybe they should have their own physical server in a datacenter. In addition to more flexibility, it would also be cheaper. Cloud providers are trying to convince people it is easier by using them, but in the end, if your cloud grows a lot, you are still going to need a team dedicated for management, etc. They are trying to convince it is cheaper to use them, but I can easily have 32GB of RAM or far far more on a single node at the fraction of the cost of the similar virtual offering (if they even offer such big VMs).
- aembleton 7y agoOr just copy backups away from your cloud provider, to local storage, another cloud such as AWS or dropbox. No point making backups if you can't access them.
- stupidcar 7y agoIt won't be easier, and it won't be cheaper. When you're a small team trying to build a business, there are a lot of business functions where you won't personally have the expertise or the time to do it yourself efficiently, and your requirements won't be large enough to justify hiring somebody full-time. In the case of these business functions, the standard and correct approach is to outsource them to a 3rd-party service provider. You do this with accountancy, legal representation, facilities, office management, recruitment, etc. If you try to bring all these things in-house from the get-go you'll never get around to building a product, and it's financially and logistically sensible to do it with IT as well. This calculation may change over time as your business grows, but if you can't comprehend that the correct strategy for a fledgling business may not be the same as an established one, then you're simply not suited to run a business in the first place. Of course, in every case, you're taking a calculated risk by relying on a service provider: They may go bust, they may be incompetent or malicious, they might ramp up their prices. Your job as the manager of a company is to accept and manage these risks as best you can. Risks cannot be eliminated, only managed, and attempting to do is a fool's errand. If things do go wrong, you'll always have people lining up to tell you how you could have avoided this problem, usually it's by making a decision that can be justified with the benefit of hindsight. You should ignore these people. The only question is: did you make the correct decision at the time, based on the facts to hand?
- yardstick 7y agoAre there any best practices I should follow when using AWS, GCP, Azure, DO, and others to avoid these sorts of situations? I have heard although can’t confirm that using on account billing rather than using a credit card makes them less likely to just disable your account for billing issues. Things like that would be helpful to know. Should I be letting them know more contact info about us, asking for an account manager, etc? Does it make a difference in how they react to you if you are spending low thousands a month vs tens of thousands a month vs hundreds of thousands etc? Or is everything always automated to death?
- aembleton 7y agoMake backups to a different cloud provider, or onto your own infrastructure.
- beagle3 7y agoAmazon recently paid $200M or so for CloudEndure - I don’t know if they support DO, but they do let you easily move between and among cloud providers, which is something every cloud based company must plan for.
- SQL2219 7y agoHow about a support emergency button. Clicking it will cost you $1,000, but would immediately put you in contact with a human in escalated support mode.
- jcauldron 7y agoHow disappointing of DO. Clearly shows their hypocrisy and lack of care for customers with private tickets versus a public forum. Will not be recommending DO.
- ksec 7y agoWhen will the wheel turn again, when we learn it is best to have at least one copy of your Data in your hand? Not your Cloud VM, Server Rented somewhere, but actually in a HDD within your own premises.
- mark_l_watson 7y agoI am sorry they had this problem. Stories like this about edge cases with cloud deployments make my conviction stronger to always: 1) have a data backup plan that uses a different cloud provider unless you have so much data that you can’t afford the extra bandwidth and storage costs. 2) if possible don’t rely on unique cloud platform services and be ready to bring a system up quickly on another service provider.
- kissgyorgy 7y agoI really liked https://hyper.sh https://hyper.sh, they provided hosting for running Docker containers. They handled this same case by having a limit on every account by default. When I requested to increase that limit, they asked about my workload, I explained them briefly, they allowed the limit increase and all was good.
- chj 7y agoOver the years, I have moved most of my VPS to vultr, and now only 1 droplet left in DO. This story again proves my mistrust towards them. I don't like AWS because it's too complicated for my use, but now I think we better have some backups.
- pushpop 7y agoNot excusing DO here but there’s also a lesson to be learned about backups: make sure they’re kept off site in case the whole site itself because compromised. Usually when we say this we mean in terms of a physical disaster (eg fire) but in this case it’s also in case of a logical disaster (account getting shit down). So the lesson is don’t leave your backups with the same cloud provider that host your database. You should also pull local copies as well
- paulcarroty 7y agoUsed DO ~2 years ago, the storage was very slow and feels like cheap Seegates. So, I just switched to AWS. Also, had troubles with trial too - activate $5-10 coupon was quite impossible without PayPal payment.
- janjanson 7y agoThis is probably going to get buried in the replies, but I had a similar experience with DigitalOcean about a year ago with my account getting permanently locked with very little explanation and no way of getting it back. It's still locked to this day. I was just a student using my Github student package credit, but I was pretty appalled by the service from DO and vowed never to buy from them. Unfortunately, my ticket no longer appears under closed tickets. I was still able to dig up my original ticket message and all the responses their support made to me through my email though. Here they are: https://imgur.com/a/NJMRnyY https://imgur.com/a/NJMRnyY Between the replies I asked about what I could do to verify my account. As you can see, they didn't even give me a single chance to do so. They told me to hang on twice then just permanently closed it up. I'm not sure how I even got flagged. All I did was turn on a droplet and delete it. I checked the audit logs, and there was nothing suspicious there either. It was just me logging in and out. I thought about making a deal about it on Twitter, but I didn't bother because I don't have any followers and it wasn't a huge loss to me either. Maybe it's the only way?
- babyloneleven 7y agoI hope DO's CTO review your case too.
- sbarre 7y agoThe emails from their "Trust and Safety" team are extremely tone-deaf... "We've locked you out, no explanation" "Sorry for any inconvenience" Seriously? That last line is like a slap in the face. No one should talk to a customer like that in this situation, if only because (a) if this is real abuse, you don't need to be "fake nice" and (b) if it's a false-positive, you've just come across as extremely smug when you're in the wrong.
- crikli 7y ago“Warmest regards”. Wow. F*ck you too.
- apple4ever 7y agoWow that is just terrible. "We've tried nothing and we are all out of ideas! LOCKED" is basically their response. I'd complain on Twitter just to see what happens.
- scarecrow112 7y agoJeez this is scary! This very much sounds like the domain blocking incident [1] that Zoho faced sometime back. [1] - https://news.ycombinator.com/item?id=18059792 https://news.ycombinator.com/item?id=18059792
- proyb 7y agoI’m fortunate to use UpCloud provider with regional/dedicated technical support being responsive within minutes, and resolve issues is quite speedy.
- juskrey 7y ago/writes down the note/ "never forget to replicate elementary backup infrastructure on the second provider" PS: Warheads have THREE copies of navigation systems
- corobo 7y agoCalling my upcoming linode-digitalocean-vultr setup Project Warhead, thank you
- vsl 7y agoI was evaluating DO the past two months, but reading this, I'm staying at Linode - their poor security incident handling in the past is a theoretical concern, this is a more immediate one. It's not this anecdote in itself, but that it corroborates my experience during trial that I ignored and dismissed as support incompetence (which should have been a warning sign in itself). After setting up the account, adding a payment card, I wasn't able to enter our VAT ID as part of billing details, with some nondescript error. So I asked support. Two days(!) later, they responded by asking for incorporation documents, which was frankly bizarre (and a first in ~10 years of running business): they're not exactly a bank with KYC requirements. When I responded, basically, WFT?, and told them to check the billing data in VIES, they eventually fixed it. But what I got from it was a distinct impression that their default assumption is that the customer is trying to defraud them, even when it makes no sense. To this day, I have no idea what kind of fraud could they possibly be anticipating there (they allowed the card). This story is on the same general subject, and so are others surfaced here and on twitter in reaction: the customer is presumed scumbag.
- delzennejc 7y agoThis is really scary. Many developers are using Digital Ocean.
- creeble 7y agoStill waiting for response 22 hours in. The response (and its timing) will determine whether we continue with DO as a host for the (admittedly tiny) bit of infrastructure we host there. And there had better be a reasonable response on HN if they value their HN-reading customers; it is where we got the first recommendations for them years ago.
- detaro 7y ago? They have already responded, including directly in this thread.
- creeble 7y ago>We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short. I'm referring to _this_ response, i.e., the one where they explain why they did what they did.
- girmad 7y agoTheir terms disclaim it, but I'd sue for consequential damages anyway. IANAL
- vincedotca 7y agoCloud Contingency via restic. https://revenni.com/cloud-contingency-when-the-ban-hammer-drops/ https://revenni.com/cloud-contingency-when-the-ban-hammer-dr...
- vincedotca 7y agoCloud Contingency When The Ban Hammer Drops, or, restic for the masses... don't be caught without an offsite backup! https://revenni.com/cloud-contingency-when-the-ban-hammer-drops/ https://revenni.com/cloud-contingency-when-the-ban-hammer-dr...
- citizenpaul 7y ago"Welcome to Costco. I love you"