3 ms·

I work for a small company and it would be devastating if our database was down for a few hours.

silviogutierrez 5y ago
Unplanned? Probably. But maybe it's not as a dire as you think? Azure Active Directory (auth as a service) went down for a while, globally, and life went on. Same with GCP and occasionally AWS[1] I'm not saying there's no downside, I'm asking against the downside of engineering cost. And that itself carries risk of failure when you go live. It's not guaranteed. [1] AWS is so ubiquitous though, that half the internet would be affected so it makes it less individually noticeable.
- dividedbyzero 5y ago
  So don't host on AWS, wait out their next big outage and then take down your app for that big migration, so you can hide among all the other dead apps. Half joking, of course.
  - silviogutierrez 5y ago
    That's brilliant. Have the PR ready to go and merge/deploy only when that happens. Just host on us-east1 and it'll happen eventually.
jbverschoor 5y ago
I doubt that.. aws is down sometimes too. Card payments are sometimes down, even 112(911) is sometimes down. I hope you have everything redundant, including the CEO
- fullstop 5y ago
  The database has a hot standby and two warm standby instances, redundant hardware, network infrastructure, all of that.
- xcambar 5y ago
  > I hope you have everything redundant, including the CEO Made me laugh, I'll sure reuse it!
hobs 5y ago
OTOH I worked for several small companies that had "business hours" software - if you had to take downtime on the weekend or after hours you'd be looking at impacting <1% of users.
willcipriano 5y ago
In my view that's just saying at some point in the future the company will be devastated. Nothing has 100% uptime.
chefkoch 5y ago
Then hopefully everything is HA in your company.
- fullstop 5y ago
  It absolutely is, BGP with geographically diverse paths, databases, app servers, etc, are all redundant. It's hosted in-house, so there is a cold standby database in AWS which would only be used if, say, an aircraft crashed into our server rooms.
  - jnpatel 5y ago
    Could you share what domain your company is in? Just curious what has such HA requirements
  - ikiris 5y ago
    It always amazes me when someone says they can't ever be down, and then says they only serve from one physical location actively. One of these things is not like the other.
    - fullstop 5y ago
      We have everything in place to run from AWS if needed but do not operate from there because of cost. This goes both ways -- there have been many AWS outages which have not affected us. I hear what you're saying, but we've had only one instance of extended (hours) downtime in the last 20 years.
      - xcambar 5y ago
        The more experienced I become, the more such down to earth solutions seem OMG SO MUCH MORE reasonable than all the bells and whistles of "modern" engineering practices.
      - ohyeshedid 5y ago
        Out of curiosity, what do you use in AWS to keep a running live backup of the data you'd need to migrate in the worst case scenario?

Unplanned? Probably. But maybe it's not as a dire as you think? Azure Active Directory (auth as a service) went down for a while, globally, and life went on. Same with GCP and occasionally AWS[1] I'm not saying there's no downside, I'm asking against the downside of engineering cost. And that itself carries risk of failure when you go live. It's not guaranteed. [1] AWS is so ubiquitous though, that half the internet would be affected so it makes it less individually noticeable.

dividedbyzero 5y ago
So don't host on AWS, wait out their next big outage and then take down your app for that big migration, so you can hide among all the other dead apps. Half joking, of course.
- silviogutierrez 5y ago
  That's brilliant. Have the PR ready to go and merge/deploy only when that happens. Just host on us-east1 and it'll happen eventually.

jbverschoor 5y ago

I doubt that.. aws is down sometimes too. Card payments are sometimes down, even 112(911) is sometimes down. I hope you have everything redundant, including the CEO

fullstop 5y ago
The database has a hot standby and two warm standby instances, redundant hardware, network infrastructure, all of that.
xcambar 5y ago
> I hope you have everything redundant, including the CEO Made me laugh, I'll sure reuse it!

hobs 5y ago

OTOH I worked for several small companies that had "business hours" software - if you had to take downtime on the weekend or after hours you'd be looking at impacting <1% of users.

willcipriano 5y ago

In my view that's just saying at some point in the future the company will be devastated. Nothing has 100% uptime.

chefkoch 5y ago

Then hopefully everything is HA in your company.

fullstop 5y ago
It absolutely is, BGP with geographically diverse paths, databases, app servers, etc, are all redundant. It's hosted in-house, so there is a cold standby database in AWS which would only be used if, say, an aircraft crashed into our server rooms.
- jnpatel 5y ago
  Could you share what domain your company is in? Just curious what has such HA requirements
- ikiris 5y ago
  It always amazes me when someone says they can't ever be down, and then says they only serve from one physical location actively. One of these things is not like the other.
  - fullstop 5y ago
    We have everything in place to run from AWS if needed but do not operate from there because of cost. This goes both ways -- there have been many AWS outages which have not affected us. I hear what you're saying, but we've had only one instance of extended (hours) downtime in the last 20 years.
    - xcambar 5y ago
      The more experienced I become, the more such down to earth solutions seem OMG SO MUCH MORE reasonable than all the bells and whistles of "modern" engineering practices.
    - ohyeshedid 5y ago
      Out of curiosity, what do you use in AWS to keep a running live backup of the data you'd need to migrate in the worst case scenario?