14 ms·
DigitalOcean lost our server
- rakem 11y agoI see this as a win for your processes (having backups) and DO for being honest. Stuff breaks, it happens. Good for you for having backups.
- therealmarv 11y agoNo surprise for me. It also happened on DO for me one time. It seemed I've had my machine on a wrong rack/server there. At the end you get what you pay for I would say... you can have luck on your DO server but there is a reason why Rackspace and Amazon still exist nowadays! If you can afford one of the bigger ones go for it.
- xytop 11y agoRackspace!!1 ha ha ha ha ha.. (neurotic laugh..) They took our whole infrastructure down for a day just because their highly paid Linux support team changed all root passwords and screwed firewalls and added unneed balancers so that servers weren't able to communicate with each other.. We didn't even request anything (as I remember).. Just bought their support and this shit happened. That was a nightmare
- vidyesh 11y agoOh.ho. When did this happen? I seem to have no recollection of reading about this.
- vellis 11y agoTypical PHP user, thinking that cloud is like their shared HostGator box. Pathetic.
- halis 11y agoLucy!? You got some splainin' to do...
- ljoshua 11y agoI hadn't heard of the backup providers mentioned in the article. Does anyone have experience with those or with other recommended solutions? I can re-implement the infrastructure of my VMs fairly easily (thank you Ansible), but backing up content outside of the provider's built-in options is something I haven't played with yet, and would obviously be the crucial piece.
- lobster_johnson 11y agoWe use Backupninja to drive backups, mostly Duplicity against S3. Some things, like Postgres dumps, are custom commands that just upload individual files to S3. The reason you want to wrap things in Backupninja is to centralize logging, scheduling and monitoring. Note that we only back up specific things like databases, logs (central syslog server) and data directories. We use Puppet to configure our boxes, and consider everything to be expendable that isn't data.
- csomar 11y agoTwo things to learn from this: 1. Never have a single point of failure. Relying on DO for Server+Backups is putting your eggs on one basket. 2. Your server state should be programmable. This is not quite easy for complex configurations. But today, DO has an API, we have Docker, and quite modern deployment tools. Here is setup: 1. Github for the server state. Basically, a repository to configure and deploy my infrastructure. 2. Enable DO backups in case I mess up something and want a quick come-back. 3. File backups through Tarsnap. Since I use Docker, I have a volume container. Backup the volume container with Tarsnap.
- peterwwillis 11y agoThere's always a single point of failure. Most commonly it's a human.
- tehbmar 11y agoWelp, I'm verifying back ups on my server now.
- ryanmarr 11y agoThe comment thread on this post makes me very happy. Having recently switched all my production instances to individual docker containers, I'm very happy to hear that the consensus seems to be that server instances should be killed and spun back up like bacteria. There's hope for humanity yet.
- krinchan 11y agoI use DO as a test bed for a lot of stuff. I ended up destroying/recreating so many droplets I just started using Chef. Also, I come from a major background in AWS, ChaosMonkey in prod, and a very strict push-button + nuke & pave deployment strategy. So, all this strikes me as "lol you didn't know this?"
- 20centuryboys 11y agoHow exactly do you "ssh" into your DO droplet?
- waxjar 11y agoA droplet is just a VPS.
- deleted 11y ago[deleted]
- eccstartup 11y ago"One of the main reasons is data privacy; for this reason, it’s expected that each customer will maintain the backup solution that works for their needs and specific situation."
- bikamonki 11y agoThis is yet another reason to push forward the idea of static websites/webapps. There is no single argument to support the idea of using a database to power a 5-6 page website. Not even a blog with daily posts. How many sites out there have less than 10 pages, are updated maybe once per month, sport a contact form or newsletter subscription box? 90%? And how many of them use an insecure behemonth like WP or Joomla? My new model: client wants a pretty responsive theme, I get the HTML version of it. Turn into a template I can use to spit out a static site (from my homebrewed CMS). I tell clients your monthly support charges are $0. If you ever need updates I charge hourly (most never call in months). Some clients want "a list of something" say products. If small and won't grow: a static csv file that also gets munched by the homebrewed CMS. Large list? A BAAS service with API. I really don't care if the servers are gone/hacked/ransomware. Git clone, go.
- vidyesh 11y agoThose 'behemoth like' CMSes have spur out a new range of amateur web-developing agencies who spit out a design, theme and finish the complete project with less cost, less maintenance and fast delivery. The client (who doesn't want to contact the agency again)is given instructions how to update which makes them happy. And then such projects spread like wildfire. /rant over.
- the_watcher 11y agoHoly hell, obviously where critical data is concerned, redundancy is something you should insist on, but this definitely sucks. Seems ridiculous that all you got from DigitalOcean is a $15 credit though. I'd have expected something like 6 months of paid backups comped.
- andrenotgiant 11y ago$15 credit gets you 15 months of paid backups comped on a $5/month droplet, or 7.5 months comped on a $10 droplet.
- vidyesh 11y agoWell if the droplet lost was $5 then as promised 3 months worth of it sounds fine.
- phamilton 11y agoI'm glad the author didn't turn this into a negative review of DO. "What will you do when your server is lost?" is exactly the right question to learn from a scenario like this.
- sarciszewski 11y agoThat's what I half-expected when I read the title, as well.
- freekmurze 11y agoI'm well aware that this could happen with any hosting provider. Sooner or later everyone will encounter this kind of problems. Forewarned is fore armed!
- cortesoft 11y agoI think it was an ad for some of their own services instead.
- pdabbadabba 11y agoAgreed. There may also be a customer service lesson in that refund email. Typically, your customer is not happy with you even after they receive a refund, so exclamation points and language like "Booyah!" is really not a good idea for that sort of message. It's always going to sound a little bit tone deaf -- and very tone deaf when the refund was for a major incident like the loss of an entire server. Keep it professional.
- dubcanada 11y agoIt's the standard DO credit email, which typically is from referring friends or doing something else to acquire credit. Not getting a refund because DO accidentally deleted your droplet. Though their should be another avenue for such things.
- deleted 11y ago[deleted]
- gdgtfiend 11y agoThis seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"
- Fizzadar 11y agoI have to disagree - DO's "cloud servers" are equivalent to virtual private servers, which you would never expect to loose in this manner from other providers. The lack of explanation is what worries me most - it leads me to think this might have been a case of "we forgot to replace a bad drive, then the second in the pair failed".
- bitshepherd 11y agoWelcome to "The Cloud". VPS can be effectively rebranded as "cloud", except when a server has problems you shoot first and ask later.
- RubyPinch 11y agoI'd expect to lose them. Fire happens, floods happen, electrical faults happen, mistakes happen. I can't really expect any human or humans to be perfect.
- e12e 11y agoI remember a ~36 hour down time with a serious uk provider (in my case just a shell account, but they also housed managed and unmanaged servers). They had redundant fiber links to the NOC, from competing providers. But turns out they both ran through the same box under a high way. And then someone blew that box up in order to knock out alarms while they committed a robbery...
- speeder 11y agoMy family business website got killed once by a flood. And many years ago it suffered a truck crash ( the truck crashed on the lamp post right outside the server company building and destroyed the telecom-related wires)
- halfdan 11y ago..and the second due to a flood of people from HN. Server seems down :-/
- d0ugie 11y agoWhere's that internet permanence when you need it?
- captn3m0 11y agoCache Link: https://webcache.googleusercontent.com/search?q=cache%3Ahttps%3A%2F%2Fmurze.be%2F2016%2F02%2Ftoday-digitalocean-lost-our-entire-server%2F&oq=cache%3Ahttps%3A%2F%2Fmurze.be%2F2016%2F02%2Ftoday-digitalocean-lost-our-entire-server https://webcache.googleusercontent.com/search?q=cache%3Ahttp...
- freekmurze 11y agoMy blog is up again.
- autotune 11y ago>Error establishing a database connection. I award OP the "Most Accurate Title Of The Day" Award.
- freekmurze 11y agoHaha :-) It's up again.
- bsg75 11y agoTL;DR: Backup servers no matter where hosted, because two is one and one is none.
- brayton 11y agoThey had to make space for the YC portfolio companies getting $250k of credits ;)
- fweespeech 11y agoI'd still not host production on DO unless your setup allows you to failover to a backup DC with a different provider.
- andrewthetechie 11y agoWhy?
- fweespeech 11y agoThey regularly are unable to create new droplets [or other control issues where you can't perform normal functions with normal latency] and/or have full DC outages. Pretty much any use case where I'd use something like DO being unable to create new VMs, etc. is the same as an outage.
- andrewthetechie 11y agoHuh, that's pretty odd. I asked why as I work for DO's operations team and I wanted to know what your concerns were. Do you have a specific instance of a problem? What you're describing would be considered a MAJOR outage for us and we have not had one in quite some time.
- fweespeech 11y agohttps://status.digitalocean.com/ https://status.digitalocean.com/ * Jan 6th * Dec 11th * Dec 4th * Nov 27th * Oct 10th * Oct 2nd That is annoying enough I'd want to be able to failover if it exceeded 15 minutes. I get it might not be an issue for anyone else but failing over is less complex than mitigating latency issues with droplet creation or whatever. I get "regularly" to me might mean something different to you but if the 2 DO DCs I was using are hit literally every month with a problem of some kind...it seems regular to me.
- r1ch 11y agoI definitely wouldn't recommend DO's built in backup service in production based on my experience. It brings my site down every week since all I/O freezes during the backup. Sometimes the droplet doesn't recover and needs a hard reset. Apparently it's a known issue but hard to fix.
- mrnismo92 11y agoYou should always build for failure, especially if you are running on IaaS. Good post & a great reminder for many!
- justinhj 11y agoThis has happened to me with Amazon servers too, it's just the nature of the cloud. I think Digital Ocean handled this as well as can be expected.
- Glyptodon 11y agoBetween the lines takeaway: hardware raid is dangerous.
- protomyth 11y agoI'd be curious what their RAID is setup. I agree and run ZFS for our data storage.
- andrewthetechie 11y agoCheck out https://www.digitalocean.com/community/questions/how-did-do-setup-their-ssd?answer=3675 https://www.digitalocean.com/community/questions/how-did-do-... for your answer. It's a bit old, but still true.
- protomyth 11y agoIs it single RAID controller or is there a redundant setup?
- ekimekim 11y agoAnecdotally: Every company I've worked for that uses hardware raid controllers, it feels like the controllers break about as often as one of the disks. Sure, still an improvement over no RAID, but still ridiculous compared to any decent software raid (md or ZFS).
- scurvy 11y agoYou lose a RAID card as often as spinning metal disks? Sure that's not a wild exaggeration? Sure sounds like one. In 15+ years of doing this, I've only been a part of a RAID card swap three times. Once was just to be safe with an overly paranoid customer. That said, I'm a fan of moving away from RAID. It's just more complexity in the face of a movement towards simpler architectures. The one benefit of a good RAID card: stupid fast, safe writes with a BBWC.
- 11y ago
- seanwilson 11y agoWhat if you need to scale up to more servers? What if your server gets hacked and you have to recreate it? What if you accidentally delete the server or perform an upgrade that completely breaks it? None of these would be an issue if you've scripted server recreation from scratch and permanent data is stored in high availability shared databases and services like S3. If you're relying on weekly backups to save your server state and customer data you're just asking for trouble. I like that Heroku recreates your server once a day to force you to do this properly.
- z3t4 11y agoI don't know how to write this without upsetting quite a lot of people, but RAID!???! I run Redundant disk controllers and hard drives on ZFS. And it's cheaper than a decent RAID system. First thing I do with a new server is disabling RAID.
- helper 11y agoHow many servers? 1, 100, 1000,10,000? Where are you hosting them, in your home or in a datacenter? How much are you paying for bandwidth and power? What SLAs do you have in place for network and power? How much do you pay for your remote hands? I hope you see what I'm getting at. While I love ZFS and think self-hosting can make a lot of economic sense, without knowing all the variables you can't really make a fair comparison.
- z3t4 11y agoI understand there are huge benefits of cloud hosting. But why are they running RAID? Why can't my setup be used at scale!?
- cannonball 11y agoNot surprising. DO is an epic scam. Their systems were designed by incompetent engineers and the CEO and his brother are both Coke heads. I can't even begin to tell you how many single points of failure there are in their platform.
- limeyy 11y agoAnd crappy documentation, too.
- fidget 11y agoAs if this can't happen on physical machines also.
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- chrismartin 11y agoProbably due to optimization of the conversion funnel, it's VERY easy to build things on DigitalOcean without understanding that - unlike many other *-as-a-service - droplets do not include backups and that is your responsibility. As a sysadmin, this is fine since I don't trust a single provider for both prod and backups anyhow, but I fear for those who are just getting started and don't realize this. I agree that DO handled this situation appropriately, but they should warn people a little more deliberately when they sign up and create a droplet. A checkbox along the lines of "I, user, understand that I'm responsible for backing up my server", with a few links to some how-to articles of the excellent quality that DO is known for, would suffice and empower without reducing the conversion rate.
- mikeash 11y agoI would really hope that the presence of a "Backups" checkbox on the droplet creation page is enough to tell the (presumably tech-savvy) user that backups don't happen unless you check that box.
- aesthetics1 11y agoThat's fine, but they didn't introduce this functionality until recently. One day, I logged into my DO console (I don't have a reason to do this very often), and saw a new backup tab. Older customers might not even be aware. I don't host anything I couldn't live without though, so I'm not really worried.
- mikeash 11y agoThe functionality has been there for a long time. It's possible the checkbox on droplet creation is new, and it used to require activating backups afterwards, I don't recall.
- bowlich 11y agoI still don't trust them to back it up. I checked the box, still make my own local backup.
- deleted 11y ago[deleted]
- goatforce5 11y ago> the same reliability level of a physical machine ie, it could suffer a catastrophic failure at any point in time?
- rascul 11y ago"Virtual private server" would be more accurate. How is it wrong, though?
- Johnny555 11y agoSo what is the correct interpretation? At a high level, it seems like a reasonable simplification.
- mdellabitta 11y agoHey, anybody remember LeafyHost? http://arstechnica.com/civis/viewtopic.php?f=25&t=238085 http://arstechnica.com/civis/viewtopic.php?f=25&t=238085
- trustfundbaby 11y agoThe weekly backup deal is something I really do not like with Digital Ocean, I think Linode does it better, I've been planning to move for a while now, I've just been super lazy, but this might be the push I need. 7 days of lost data is unacceptable to me, especially when I'm paying for backups. > Three backup slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old backup. A fourth backup slot is available for on-demand backups.
- chrismartin 11y agoI see DO's weekly backup as a convenience feature for if you need to restore your server, not as your primary backup method. Use the DO snapshot to restore most of the way, then run an up-to-date differential restore from your other (real) backup solution (that is hosted somewhere else).
- gedrap 11y agoBut wouldn't it be much more convenient (largely because of the complexity) to have the complete backup solution at the same place (i.e. DO)? Obviously, you want some offsite backups rather than completely relying on DO but that should be for disaster recovery (massive failures in DO's DCs, etc) rather than a more routine one.
- ryanlol 11y agoIf you're going to switch to linode you might just as well use any public torrent tracker to store your backups.
- sauere 11y agoI had two DO machines with a unrepairable file system after a crash and reboot. Shit happens. Never rely on a _any_ VPS. I treat them as throw-away boxes that can fail anytime.
- codazoda 11y agoAlthough I have backups of my critical files, I don't want to try to rebuild from those files if I can help it. After learning that Digital Ocean does no backups of their own (even for critical hardware failure on their own side) I've enabled weekly backups for an additional $1 per month. Glad you posted this so that I know that option is available and really necessary. And who wouldn't pay $1? Edit: It costs 20% but I only pay $5 for my personal server.
- toomuchtodo 11y agoIf you're running in "the cloud", you should _always_ be able to destroy your instances and have them auto-rebuild from configuration management and a persistent object storage system (git repo + S3 or Google Nearline).
- notwhereyouare 11y agodo you have any guides or tutorials I could possibly follow to have a setup like that? I'm working on configuring a site and would like the ability to 1) scale as quickly as possibly or 2) rebuild in event of total failure. I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs down? Any suggestions?
- brianwawok 11y agoIts a semi-religious topic, so you will get varied and loud opinions. My opinion is ansible. Basically python + yml, you list out steps to set up your server. Takes a few minutes longer round 1, but you can remake the server in 60 seconds in 6 months. Lots of ansible tutorials around, try it out..
- moondev 11y agoSource code lives in a repo user data lives in redundant databases / storage service server setup lives in a repo via bash scripts or ansible etc (and bake that into an ami or container) setup your launch-configuration for your servers to setup all of the above and you can destroy and spin up servers on a whim
- twunde 11y agoDO just gave a talk (~2 weeks ago) where they mentioned that they were moving to software RAID.
- vardump 11y agoSounds great, for example ZFS has proven to be more reliable than PCI-e card based hardware RAID solutions. EMC and Netapp are of course another matter, they're great. If you can afford them.
- brobinson 11y ago>ZFS has proven to be more reliable than PCI-e card based hardware RAID solutions I'm a fan of ZFS, but this is the first time I've heard this claim. Do you have any links to articles/stories talking about it?
- vinceguidry 11y agoIf you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it. For database servers you need to have procedures in place to quickly switch the production application over to a database you just spun up and migrated the data from a recent backup to in order to test your data backups. You want this to happen as smoothly as possible in the case of a failure. Keep data backups in three different places and test your procedure on all of them. On a production system, do not keep the database on the same server as the application.
- snuxoll 11y agoIt's so insanely easy to get started with puppet or ansible I wouldn't start rolling out a product without using them from the get-go at this point. At the very least properly package your software (read: DEB or RPM, not docker containers) so it's easy to reinstall quickly if need be.
- vinceguidry 11y agoPersonally, I like platform libraries rather than OS packages. I have several Ruby projects set up as gems. What's neat about that is it allows code reuse across projects.
- snuxoll 11y agoSo go ahead and vendor them in, outside of inclusion in distribution software packages (with exceptions) there's nothing stopping you from including your own dependencies. Packaging your application properly makes deployment a hell of a lot easier and less error-prone.
- lqdc13 11y agoWhat? Why is it a bad thing? I guess it depends on someone's definition of a database.
- outworlder 11y agoOh well, It's not like DigitalOcean doesn't tell you to enable backups. And you should not expect that hardware failures are now a thing of the past just because it's in "the cloud".
- kentonv 11y agoI think Google Compute Engine does the right thing here: by default it uses "persistent disks" (network-attached redundant/highly-available block devices) for all disks. The only case I've heard of where persistent disk data was lost was a few acknowledged writes occurring just before an unusual lightning-induced power outage: https://status.cloud.google.com/incident/compute/15056 https://status.cloud.google.com/incident/compute/15056 For added protection, you can take regular snapshots. You only pay to store the diff from the last snapshot (so go ahead and snapshot often), and snapshot storage is geographically distributed. (Note: I have no idea what EC2 does, maybe it's similar.)
- tshtf 11y agoElastic storage on AWS has a fairly low AFR (https://aws.amazon.com/ebs/details/ https://aws.amazon.com/ebs/details/): Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size and performance of the volume. This makes EBS volumes 20 times more reliable than typical commodity disk drives, which fail with an AFR of around 4%. EBS also supports a snapshot feature, which is a good way to take point-in-time backups of your data.
- mwcampbell 11y agoI've been suspicious of network-attached block storage ever since the Amazon EBS cascading failure of April 2011 and these three blog posts that Joyent did in response: https://www.joyent.com/blog/on-cascading-failures-and-amazons-elastic-block-store https://www.joyent.com/blog/on-cascading-failures-and-amazon... https://www.joyent.com/blog/magical-block-store-when-abstractions-fail-us/ https://www.joyent.com/blog/magical-block-store-when-abstrac... https://www.joyent.com/blog/network-storage-in-the-cloud-delicious-but-deadly https://www.joyent.com/blog/network-storage-in-the-cloud-del... Besides Joyent, I think DigitalOcean and Vultr do it right, with RAID-backed local storage and a backup option.
- kentonv 11y agoI'm having trouble unpacking useful information out of these rambly blog posts. It sounds like the basic complaint is that abstractions add complexity, which is more trouble when it goes wrong. I agree that if you have a large-scale sophisticated operation you will probably want to choose local disks and handle availability your own way (and GCE provides those options). But for small-scale operations that can't justify as much engineering, abstractions like persistent disks and automatic migration will save a ton of time and avoid data loss. Ironically, Digital Ocean seems to target this smaller scale, yet they've set the wrong defaults for their target market.
- apocalyptic0n3 11y agoDoes anyone have any good backup solutions to mitigate this? My agency uses a script we wrote in-house to more-or-less rsync our data to an AWS instance, but it's always seemed a poor way to handle it. I'm using DO's backup service for my personal site which was always supposed to be a temporary solution (the timing of the backups is inconvenient as mentioned by the article). A better solution would be wonderful.
- lobster_johnson 11y agoWe use Backupninja to drive backups, mostly Duplicity against S3. Some things, like Postgres dumps, are custom commands that just upload individual files to S3. The reason you want to wrap things in Backupninja is to centralize logging, scheduling and monitoring. Note that we only back up specific things like databases, logs (central syslog server) and data directories. We use Puppet to configure our boxes, and consider everything to be expendable that isn't data.
- chrismartin 11y agoThis isn't exactly turnkey but for my personal servers, I use rsync.net (ask for the HN discount) and attic [0]. It's easy to script and call with cron. (Of course, this is still somewhat DIY, and any backup solution is only as good as your monitoring + restore testing.) [0] https://attic-backup.org/ https://attic-backup.org/
- Tassels 11y agorsync.net is the shit. I can vouch for them 110%. Also backblaze's b2 is pretty damn good/cheap. I use it as a secondary backup.
- justizin 11y ago"Luckily, we made the decision at Spatie to host every site on it’s own droplet, so only one site was affected." I think that's a poor lesson learned here. Were this me, I would have said: "Luckily, all of our sites run on several servers, access data in a shared, replicated cluster, and a small shell script I wrote kept me from writing this entire blog post." IaaS has only surfaced what has always been true: your data lives on little physical things that are screwed into a thing and goes through a controller that could fuck up due to cosmic rays. Do better for your customers.
- hinkley 11y agoMy sentiments exactly. If they lost their db replica that would get the blood pumping but the customers would never see an actual outage.
- nikdow 11y agoTell me if I'm doing something wrong: my two EC2 servers both use EBS including for the root device, so I'm not using any device-attached storage. If Amazon "lost" my instance, it would appear to me to be just a reboot. And yes, I do a nightly backup of databases, webroot, /etc and some other directories.
- ju-st 11y agoDo you have two separate AWS accounts? Is your backup outside of amazons influence?
- tomphoolery 11y agoThat really stinks, but this should be a warning to you that in the future, you should have some kind of redundancy plan so data does not get lost. Even if Amazon decided to terminate all of my app instances right now, I would just need to rebuild them. No data is actually lost, and furthermore backups are made frequently so that if a server decides to explode one day, we're still covered. DigitalOcean droplets are really not meant to be holding your database and application server...you shouldn't have experienced "data loss" by losing a droplet because your database should not have been hosted on a droplet.
- tzakrajs 11y agoWhich is the irony of the entire post because the tldr was: "our DO instance was lost, fortunately we keep multiple backups of the data so we were able to restore without any data loss". It seems more like a passive aggressive note against Digital Ocean.
- ricardobeat 11y agoWhy not? How is a droplet different than any other server, including your own hardware? All of them will fail eventually, the important bit is having a sound replication/backup strategy as you said yourself.
- beachstartup 11y agoyou should hear what people say when we suggest that they make their own backups in addition to the ones we provide for them. adults acting like children. nothing more, nothing less. take responsibility for your own data. back it up yourself.
- jvoorhis 11y agoSounds like a Tuesday at AWS.
- mizzao 11y agoI'm more worried about the following post in that thread: <begin quote> -------------------------------------------------------- We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No biggie, we'll just update it, turn the server back on and be on our way. Nope. We had to contact customer service to get back in to our account. After updating the card info we realize that all our droplets are gone. We reach out to customer service again in which case they let us know that when a CC expires, an automated process kicks off and deletes the droplets. We've been a customer for 2 years, surely they could pick up the phone and call us. We've spent thousands of dollars with them. Plan B, let's restore them from the backups we've been paying for. Nope! When they delete your droplets, they also delete your backups. This is where we ask to speak to them on the phone and are denied. I then ask if they are insane and why they would delete someone's servers and backups via an automated process without a human at least checking to see if it is a loyal multi year customer who has a simple lapse with their CC expiry. We had to find a backup on a developers machine from over a month prior and rebuild data using various megtods that took close to a week. In the end, DI gave us a $500 credit. You get what you pay for. Anyone who uses DI for production or anything more than a hobby app is playing with fire. They do not care, they are apathetic, they will screw you over and the throw you a credit for their terrible service as a half assed apology.
- mizzao 11y agoOne time my CC expired on DO because there was a fraudulent purchase on it and I had to get a new one. I got a brief e-mail from DO noting there was a problem charging my card, and updated it the same day. What if I was on vacation, and didn't see this e-mail for a few days? shudder In what world is automating the deletion of customer data a good idea?
- dualboot 11y ago> In what world is automating the deletion of customer data a good idea? The data destruction world :)
- rubberstamp 11y agoWhats the point of raid if raid card failure resulted in complete loss of data? I came across an article a week or two ago that discussed various alternatives, I think ZFS with checksums, which unlike raid will not replicate corrupt data from drive nearing failure to the healthy drive.
- scurvy 11y agoThey should have been able to swap out the RAID card and import the configs from the disks. This is pretty common practice with LSI/Avago-based cards. Curious as to why they didn't do that. Maybe the card went really bad and started writing garbage all over the place?
- arihant 11y agoI do understand the part where they terminated and killed of the VMs. I do not, at all, understand why they deleted the backups. How can the CEO sit in his chair and make that decision? If he didn't and he doesn't know, why is he still in that chair? Even damn Netflix preserves data for 10 months. Deleting backups is making 100% sure that the customer doesn't come back to you. It's amazingly stupid. Unless you're as smart and as big as Google, use the damn phone. Don't use the word automated at all. It's not automated if one fool made a decision and another one wrote the script.
- charliedevolve 11y agoIf you're paying for backups and they get deleted the moment there's an issue with payment, ur doin' it rong. I don't care whatever the TOS says, they're being completely shitty to customers. Sure there should be regular off-site backups. No one is arguing against that. But the fastest restore is usually from the closest source, and that's partly why you'd pay them for it.
- Lanari 11y agoI recommend having a backup using Git on some reliable Git hosting(s). I can't think on anything safer and more practical.
- vidyesh 11y agoTo my surprise, this HN thread has no link to any external backup solution guide or little-to-no suggestion of best way to backup your server to an external service or another VPS/backup server.
- aprdm 11y agoI use Tarsnap and really enjoy it!
- tvvocold 11y agoLucky to u, they just delete my server just for i set up a VPN server...
- scurvy 11y agoAnyone have any experiences with running VM's with ceph storage? I've used it for other things, but I know that VM hosting is quite popular. Care to share any stories?
- kull 11y agoWe are also experiencing a huge problem with performance of DO servers as we scale. Especially databases. With stories like that , a decision of moving to AWS seems pretty obvious.