10 ms·

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it wasn't, because every night I run a PG backup and copy it to AWS S3. I just had to download the backup from the other cloud vendor and restore it on my DO server. Did DO fuck up? Yes. Did it cost me downtime? Yes. Was I mad? Yes, and I still am. But I still do business with DO because it costs ~half the price of a comparable EC2 instance, and writing a 10 line bash script to move my database backups to another cloud vendor isn't that hard. Storing that backup on S3 costs pennies per month. I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse. When it comes down to it, you are solely responsible to your customers. They're not going to care whose fault it was, because it was your fault. Don't trust any of your vendors.

dmix 7y ago
The article addresses this in the "Why not just do x?" section. > - Why were you only hosting one JUST ONE cloud provider? > - Why didn't you have backups outside of JUST ONE cloud provider? Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not to have extended downtime. This is much bigger issue than having external backups (which every startup should still do).
- zamu 7y ago
  Then save it locally as well? There really isn't an excuse to having DO or any provider being your sole backup.
  - dmix 7y ago
    Why is everyone talking about backups and not the actual infrastructure?
    - eyeinthepyramid 7y ago
      because that was the biggest issue in the original article, they didn't have backups of their data, so the company was dead. If they had had backups, they could have recreated the infrastructure.
      - z3t4 7y ago
        They did have backups, only problem backups was also on DO. Backing up petabyte of data even on a Gbit connection is slow, and also cost more then a pennies if you for example back it up on AWS. Linus tech tips have a video on youtube explaining the problem.
        PeterisP 7y ago
        The original post literally mentions "all our data (500k rows)" i.e. a data amount so small they could probably back it up to their cellphone every five minutes.
        mort96 7y ago
        It's not really a "backup" if it's just on the same server/data center though. Moving data from one drive to another in the same machine, or from one machine to another in the same building, provides fault tolerance should the main drive/machine die, that's about it. That's true whether the computer(s) is your own or someone else's.
        z3t4 7y ago
        It's all about threat model. And yes "our cloud provider accidentally deletes both our data and our backups" should be in the list.
      - 7y ago
- pbourke 7y ago
  But you don’t need the same architecture replicated across two providers, as your parent post demonstrated. Run Postgres in DO and upload your backups to S3/Azure Storage. This is both cheap and simple to the level that any startup could manage it. If and when it makes sense, you can investigate multi provider replication, but it might just be YAGNI.
- EugeneOZ 7y ago
  No, it's not so difficult task and not even expensive. I don't think every project really needs it, though. But external db backups - they are ridiculously cheap and every project needs them.
- LinuxBender 7y ago
  These are strange times I guess? Maybe I'm a weird one-off, but I have 3 VPS providers for my hobby sites, my own DNS spread across all three (one remains shadow backup) and offline rsnapshot backups that automation can't touch. I can't imagine doing less for a business. In my opinion, the business model and investment plan should account for all of this for at least the first five years.
- jotm 7y ago
  Local backups? A few hard drives and a fiber connection can be cheap. Slow to restore, but not "killed my business" slow
jwr 7y ago
> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better reliability, rent three. I looked at this many times and still check the pricing regularly, and it just doesn't make sense for me to switch from these physical servers to virtualized offerings, not in any foreseeable future.
- geggam 7y ago
  Because I can simply turn off everything and reduce my spend to storage alone. If I turn off that server I still pay for it
- barkingcat 7y ago
  Because DO has their $5 and $10 / month plan, and that's really enough for me.
  - freehunter 7y ago
    That's exactly it for me too. The Hetzner plan costs about twice as much as my monthly DO bill. Not to mention I'm US based and all of my customers are US based so hosting a site in Germany or Finland doesn't make a whole lot of sense.
    - phoobahr 7y ago
      And here I am a Canadian and my client base is Canadian. While not a problem just yet it is likely that PIPEDA/PHIPA would make it complicated to serve data from outside the country. I am not a lawyer but the advice I have been given is murky. It’s much easier to stuck with DO which, at my usage, costs about the same number of $CAD as lightsail would in $USD
    - karussell 7y ago
      Since a year you can use their cloud product: https://www.hetzner.com/cloud https://www.hetzner.com/cloud starting at 2.5€ There are lots of "managed" features still missing but acceptable for a still young product.
EugeneOZ 7y ago
It's because price of your site downtime is insignificant. DO (and many other "bulk vps" services) is only suitable for projects like this. When 1 hour of downtime cost much more than year of hosting, you will be ready to pay for high availability and duplication, replication, backups verification.
codegeek 7y ago
Great point. I always run a separate backup through bash scripts that syncs with S3 bucket even though I also have the DO backups enabled as well. You always want 2 separate offsite backups for any real world production application.
chocolatebunny 7y ago
Cloud or not, you have to trust your vendor to deliver the service you paid them for. That's the whole reason for doing business with them.
- freehunter 7y ago
  The reason for doing business with them is because they say they will deliver the service you paid them for. But if they don't, what then? You can switch to another vendor, but without a backup you're starting from scratch. You should hope they deliver the service you've paid for, but you should never trust that they will. Always plan for failure. That's the entire point of this whole "DO killed my company" saga.
- proyb2 7y ago
  I trust UpCloud more than other VPS provider because the technical support (Regional) has been very approachable and you know "who" they are. Beside, they promise 100% uptime SLA when your business is critical. https://upcloud.com/blog/ https://upcloud.com/blog/
behringer 7y ago
Might I recommend prgmr.com instead? I use both DO and PRGMR (for resiliency).
noir_lord 7y ago
FWIW I do the same I have backups of everything. All that said in close to a decade (actually might be a decade) I've still yet to have any issue with linode (tempting the computer gods here) or their backups. I remember when DO was first starting out and they used to knock their own stuff offline updating router tables. Ever since I have a jaundiced view of their capabilities.
- wiredfool 7y ago
  N=1 Linode backups start to fail when you have more then 3 million files on a drive. When debugging this, they told me the reason, I did a snapshot that worked, and then they turned off backups because they were failing. They didn’t highlight that backups were now off. Restores on the larger machines can take 5+ hours, and they will often report that the restore failed if you are restoring a drive that contains docker’s pipe files. Ymmv, but I’m trying to get off linode.
jotm 7y ago
Do you not have a mirror copy of everything somewhere else? Or is that not possible in your case?
- freehunter 7y ago
  I only run one server, because like the dead company in question, I don't have the money or manpower to do a full high-availability solution. Just a database backup and my site's code in Gitlab, so DR failover is a matter of 'git clone' and 'pg_restore'. It might take me a few hours to get back online and piece my docker containers back together but my customers can absorb a few hours of downtime as long as it's a very rare occasion. If they couldn't, I'd charge them enough to afford a real HA solution.
neovive 7y ago
If you don't mind me asking, what is the preferred method to backup to S3? Is it possible to scp or rsync a mysqldump to S3 or do you install aws tools on DO and run aws s3 cp as a scheduled job?
- freehunter 7y ago
  I use the awscli tools and aws s3 cp. I don't think it's possible to scp or rsync directly to S3, at least last I checked. https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-in...
- slenk 7y ago
  Not OP but I just use shell scripts. You could scp or rsync if you mounted s3 as a drive to your host, but that is not recommended as far as I know
- anderspitman 7y ago
  I recommend looking into restic and/or rclone
- ptype 7y ago
  check out restic which does dedup backups to s3, very easy to use
Alex3917 7y ago
> But it wasn't, because every night I run a PG backup and copy it to AWS S3 Which is probably much easy on DO than AWS. For whatever reason AWS seems to go out of their way to make it as hard as possible to backup your RDS data to a separate AWS account.
PhasmaFelis 7y ago
> I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse. It's not unreasonable to be annoyed when a company you are paying specifically to do the things you're not familiar with, catastrophically fails to do them correctly.
- freehunter 7y ago
  Agreed, but "I'm annoyed" doesn't help your customers. "Don't worry, I'm restoring from last night's backup" sounds a lot nicer. Again, your customer doesn't care what your excuse is, just like the dead company's founder doesn't care what DigitalOcean's excuse was. "Our processes failed" translates directly into "I failed".
PhasmaFelis 7y ago
> I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse. If you're paying someone specifically to make backups for you, you should be able to trust that they've taken every reasonable measure to ensure that backups are actually being made and preserved.
- freehunter 7y ago
  You would think so, but then again you might be wrong. Better to be safe than sorry, no? I expected DO to make the backups I was paying them for and they didn't. Luckily I was making my own backups at the same time. Turns out my backups worked and theirs didn't. If I had just trusted them, I'd be out of business just like the dead company in question. I'm not sure what's so confusing about "don't trust your vendors" but I've had to make this exact same reply way too many times. Don't trust your vendors!
xiaomai 7y ago
I remember getting the e-mail about their backups not having been working for months. I was shocked at how negligent that was. I run my own backups now. (Also their backup offering is so much more cumbersome to use than borg backup that it wasn't a great loss to manage these myself). Still amazed that they didn't notice no one's backups had been running for so long.

dmix 7y ago

The article addresses this in the "Why not just do x?" section. > - Why were you only hosting one JUST ONE cloud provider? > - Why didn't you have backups outside of JUST ONE cloud provider? Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not to have extended downtime. This is much bigger issue than having external backups (which every startup should still do).

zamu 7y ago
Then save it locally as well? There really isn't an excuse to having DO or any provider being your sole backup.
- dmix 7y ago
  Why is everyone talking about backups and not the actual infrastructure?
  - eyeinthepyramid 7y ago
    because that was the biggest issue in the original article, they didn't have backups of their data, so the company was dead. If they had had backups, they could have recreated the infrastructure.
    - z3t4 7y ago
      They did have backups, only problem backups was also on DO. Backing up petabyte of data even on a Gbit connection is slow, and also cost more then a pennies if you for example back it up on AWS. Linus tech tips have a video on youtube explaining the problem.
      - PeterisP 7y ago
        The original post literally mentions "all our data (500k rows)" i.e. a data amount so small they could probably back it up to their cellphone every five minutes.
      - mort96 7y ago
        It's not really a "backup" if it's just on the same server/data center though. Moving data from one drive to another in the same machine, or from one machine to another in the same building, provides fault tolerance should the main drive/machine die, that's about it. That's true whether the computer(s) is your own or someone else's.
        z3t4 7y ago
        It's all about threat model. And yes "our cloud provider accidentally deletes both our data and our backups" should be in the list.
    - 7y ago
pbourke 7y ago
But you don’t need the same architecture replicated across two providers, as your parent post demonstrated. Run Postgres in DO and upload your backups to S3/Azure Storage. This is both cheap and simple to the level that any startup could manage it. If and when it makes sense, you can investigate multi provider replication, but it might just be YAGNI.
EugeneOZ 7y ago
No, it's not so difficult task and not even expensive. I don't think every project really needs it, though. But external db backups - they are ridiculously cheap and every project needs them.
LinuxBender 7y ago
These are strange times I guess? Maybe I'm a weird one-off, but I have 3 VPS providers for my hobby sites, my own DNS spread across all three (one remains shadow backup) and offline rsnapshot backups that automation can't touch. I can't imagine doing less for a business. In my opinion, the business model and investment plan should account for all of this for at least the first five years.
jotm 7y ago
Local backups? A few hard drives and a fiber connection can be cheap. Slow to restore, but not "killed my business" slow

jwr 7y ago

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better reliability, rent three. I looked at this many times and still check the pricing regularly, and it just doesn't make sense for me to switch from these physical servers to virtualized offerings, not in any foreseeable future.

geggam 7y ago
Because I can simply turn off everything and reduce my spend to storage alone. If I turn off that server I still pay for it
barkingcat 7y ago
Because DO has their $5 and $10 / month plan, and that's really enough for me.
- freehunter 7y ago
  That's exactly it for me too. The Hetzner plan costs about twice as much as my monthly DO bill. Not to mention I'm US based and all of my customers are US based so hosting a site in Germany or Finland doesn't make a whole lot of sense.
  - phoobahr 7y ago
    And here I am a Canadian and my client base is Canadian. While not a problem just yet it is likely that PIPEDA/PHIPA would make it complicated to serve data from outside the country. I am not a lawyer but the advice I have been given is murky. It’s much easier to stuck with DO which, at my usage, costs about the same number of $CAD as lightsail would in $USD
  - karussell 7y ago
    Since a year you can use their cloud product: https://www.hetzner.com/cloud https://www.hetzner.com/cloud starting at 2.5€ There are lots of "managed" features still missing but acceptable for a still young product.

EugeneOZ 7y ago

It's because price of your site downtime is insignificant. DO (and many other "bulk vps" services) is only suitable for projects like this. When 1 hour of downtime cost much more than year of hosting, you will be ready to pay for high availability and duplication, replication, backups verification.

codegeek 7y ago

Great point. I always run a separate backup through bash scripts that syncs with S3 bucket even though I also have the DO backups enabled as well. You always want 2 separate offsite backups for any real world production application.

chocolatebunny 7y ago

Cloud or not, you have to trust your vendor to deliver the service you paid them for. That's the whole reason for doing business with them.

freehunter 7y ago
The reason for doing business with them is because they say they will deliver the service you paid them for. But if they don't, what then? You can switch to another vendor, but without a backup you're starting from scratch. You should hope they deliver the service you've paid for, but you should never trust that they will. Always plan for failure. That's the entire point of this whole "DO killed my company" saga.
proyb2 7y ago
I trust UpCloud more than other VPS provider because the technical support (Regional) has been very approachable and you know "who" they are. Beside, they promise 100% uptime SLA when your business is critical. https://upcloud.com/blog/ https://upcloud.com/blog/

behringer 7y ago

Might I recommend prgmr.com instead? I use both DO and PRGMR (for resiliency).

noir_lord 7y ago

FWIW I do the same I have backups of everything. All that said in close to a decade (actually might be a decade) I've still yet to have any issue with linode (tempting the computer gods here) or their backups. I remember when DO was first starting out and they used to knock their own stuff offline updating router tables. Ever since I have a jaundiced view of their capabilities.

wiredfool 7y ago
N=1 Linode backups start to fail when you have more then 3 million files on a drive. When debugging this, they told me the reason, I did a snapshot that worked, and then they turned off backups because they were failing. They didn’t highlight that backups were now off. Restores on the larger machines can take 5+ hours, and they will often report that the restore failed if you are restoring a drive that contains docker’s pipe files. Ymmv, but I’m trying to get off linode.

jotm 7y ago

Do you not have a mirror copy of everything somewhere else? Or is that not possible in your case?

freehunter 7y ago
I only run one server, because like the dead company in question, I don't have the money or manpower to do a full high-availability solution. Just a database backup and my site's code in Gitlab, so DR failover is a matter of 'git clone' and 'pg_restore'. It might take me a few hours to get back online and piece my docker containers back together but my customers can absorb a few hours of downtime as long as it's a very rare occasion. If they couldn't, I'd charge them enough to afford a real HA solution.

neovive 7y ago

If you don't mind me asking, what is the preferred method to backup to S3? Is it possible to scp or rsync a mysqldump to S3 or do you install aws tools on DO and run aws s3 cp as a scheduled job?

freehunter 7y ago
I use the awscli tools and aws s3 cp. I don't think it's possible to scp or rsync directly to S3, at least last I checked. https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-in...
slenk 7y ago
Not OP but I just use shell scripts. You could scp or rsync if you mounted s3 as a drive to your host, but that is not recommended as far as I know
anderspitman 7y ago
I recommend looking into restic and/or rclone
ptype 7y ago
check out restic which does dedup backups to s3, very easy to use

Alex3917 7y ago

> But it wasn't, because every night I run a PG backup and copy it to AWS S3 Which is probably much easy on DO than AWS. For whatever reason AWS seems to go out of their way to make it as hard as possible to backup your RDS data to a separate AWS account.

PhasmaFelis 7y ago

> I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse. It's not unreasonable to be annoyed when a company you are paying specifically to do the things you're not familiar with, catastrophically fails to do them correctly.

freehunter 7y ago
Agreed, but "I'm annoyed" doesn't help your customers. "Don't worry, I'm restoring from last night's backup" sounds a lot nicer. Again, your customer doesn't care what your excuse is, just like the dead company's founder doesn't care what DigitalOcean's excuse was. "Our processes failed" translates directly into "I failed".

PhasmaFelis 7y ago

> I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse. If you're paying someone specifically to make backups for you, you should be able to trust that they've taken every reasonable measure to ensure that backups are actually being made and preserved.

freehunter 7y ago
You would think so, but then again you might be wrong. Better to be safe than sorry, no? I expected DO to make the backups I was paying them for and they didn't. Luckily I was making my own backups at the same time. Turns out my backups worked and theirs didn't. If I had just trusted them, I'd be out of business just like the dead company in question. I'm not sure what's so confusing about "don't trust your vendors" but I've had to make this exact same reply way too many times. Don't trust your vendors!

xiaomai 7y ago

I remember getting the e-mail about their backups not having been working for months. I was shocked at how negligent that was. I run my own backups now. (Also their backup offering is so much more cumbersome to use than borg backup that it wasn't a great loss to manage these myself). Still amazed that they didn't notice no one's backups had been running for so long.