4 ms·

More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely di

More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.

_bxg1 7y ago
Whether it's normally a consideration or not, there are no meaningful barriers in terms of cost or effort, so it's totally realistic to expect it of a tiny startup. Every week there's another article on HN about a tiny business being squished in the gears of a giant, automated platform. In some cases like app stores this is unavoidable, but there are plenty of hosting providers to choose from. People need to learn that this is something that can happen to you in today's world, and take reasonable steps to prepare for it.
- yourbandsucks 7y ago
  And there are a million stories of startups who build the wrong thing, don't achieve product-market fit, etc. You can't dot every I, cross every t and also build a compelling product as a 2 person shop.
  - andrewflnr 7y ago
    Backups aren't "dotting i's and crossing t's", they're fundamental. FFS, just rsync your database directory somewhere.
  - icebraining 7y ago
    Then maybe you shouldn't be building that product with a 2 person shop.
    - yourbandsucks 7y ago
      Sounds good in theory.
jdietrich 7y ago
>getting locked out of your account is usually just not a consideration How many horror stories need to reach the front page of HN before people stop believing this? Getting locked out of your cloud provider is a very common failure mode, with catastrophic effects if you haven't planned for it. To my mind, it should be the first scenario in your disaster recovery plan. Dumping everything to B2 is trivially easy, trivially cheap and gives you substantial protection against total data loss. It also gives you a workable plan for scenarios that might cause a major outage like "we got cut off because of a billing snafu" or "the CTO lost his YubiKey".
- williamdclt 7y ago
  > How many horror stories need to reach the front page of HN before people stop believing this Sounds like the opposite of the survivor bias. I don't believe it's any sort of common (though it does happen), even less that "it should be the first scenario in your disaster recovery plan"
  - kelnos 7y ago
    Even if the stories we hear of account lockouts isn't typical, the absolute number of them that we see -- especially those (like this one) that appear to be locked (and re-locked) by automated processes -- should be cause for concern when setting up a new business on someone else's infrastructure.
  - jdietrich 7y ago
    If you plan for the "all of our cloud infrastructure has failed simultaneously and irreparably" scenario, you get a whole bunch of other disaster scenarios bundled in for free.
Lazare 7y ago
> It’s unrealistic to expect this of a tiny startup. I could not disagree more. There's a right way and a wrong way to do this, it's trivial to do it right, and the risks of doing it wrong are enormous.
bdcravens 7y ago
> It’s unrealistic to expect this of a tiny startup. Then it's unrealistic to trust them with your business.
ufmace 7y ago
I don't know, it seems simple enough to me. I have a server on DO hosting some toy-level projects, and IIRC it took me 15-30 min to set up a daily Cron job to dump the DB, tar it, and send it to S3, with a minimum-privilege account created for the purpose, so that any hacker that got in couldn't corrupt the backups. I'm not a CLI or Linux automation whiz, others could probably do it faster.

_bxg1 7y ago

Whether it's normally a consideration or not, there are no meaningful barriers in terms of cost or effort, so it's totally realistic to expect it of a tiny startup. Every week there's another article on HN about a tiny business being squished in the gears of a giant, automated platform. In some cases like app stores this is unavoidable, but there are plenty of hosting providers to choose from. People need to learn that this is something that can happen to you in today's world, and take reasonable steps to prepare for it.

yourbandsucks 7y ago
And there are a million stories of startups who build the wrong thing, don't achieve product-market fit, etc. You can't dot every I, cross every t and also build a compelling product as a 2 person shop.
- andrewflnr 7y ago
  Backups aren't "dotting i's and crossing t's", they're fundamental. FFS, just rsync your database directory somewhere.
- icebraining 7y ago
  Then maybe you shouldn't be building that product with a 2 person shop.
  - yourbandsucks 7y ago
    Sounds good in theory.

jdietrich 7y ago

>getting locked out of your account is usually just not a consideration How many horror stories need to reach the front page of HN before people stop believing this? Getting locked out of your cloud provider is a very common failure mode, with catastrophic effects if you haven't planned for it. To my mind, it should be the first scenario in your disaster recovery plan. Dumping everything to B2 is trivially easy, trivially cheap and gives you substantial protection against total data loss. It also gives you a workable plan for scenarios that might cause a major outage like "we got cut off because of a billing snafu" or "the CTO lost his YubiKey".

williamdclt 7y ago
> How many horror stories need to reach the front page of HN before people stop believing this Sounds like the opposite of the survivor bias. I don't believe it's any sort of common (though it does happen), even less that "it should be the first scenario in your disaster recovery plan"
- kelnos 7y ago
  Even if the stories we hear of account lockouts isn't typical, the absolute number of them that we see -- especially those (like this one) that appear to be locked (and re-locked) by automated processes -- should be cause for concern when setting up a new business on someone else's infrastructure.
- jdietrich 7y ago
  If you plan for the "all of our cloud infrastructure has failed simultaneously and irreparably" scenario, you get a whole bunch of other disaster scenarios bundled in for free.

Lazare 7y ago

> It’s unrealistic to expect this of a tiny startup. I could not disagree more. There's a right way and a wrong way to do this, it's trivial to do it right, and the risks of doing it wrong are enormous.

bdcravens 7y ago

> It’s unrealistic to expect this of a tiny startup. Then it's unrealistic to trust them with your business.

ufmace 7y ago

I don't know, it seems simple enough to me. I have a server on DO hosting some toy-level projects, and IIRC it took me 15-30 min to set up a daily Cron job to dump the DB, tar it, and send it to S3, with a minimum-privilege account created for the purpose, so that any hacker that got in couldn't corrupt the backups. I'm not a CLI or Linux automation whiz, others could probably do it faster.