5 ms·

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rathe

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be thought out and weighed against the engineering effort. Don't get me wrong, this is very cool. But I wonder what the engineering cost was. I'd think easily in the hundreds of thousands of dollars.

fullstop 5y ago
I work for a small company and it would be devastating if our database was down for a few hours.
- silviogutierrez 5y ago
  Unplanned? Probably. But maybe it's not as a dire as you think? Azure Active Directory (auth as a service) went down for a while, globally, and life went on. Same with GCP and occasionally AWS[1] I'm not saying there's no downside, I'm asking against the downside of engineering cost. And that itself carries risk of failure when you go live. It's not guaranteed. [1] AWS is so ubiquitous though, that half the internet would be affected so it makes it less individually noticeable.
  - dividedbyzero 5y ago
    So don't host on AWS, wait out their next big outage and then take down your app for that big migration, so you can hide among all the other dead apps. Half joking, of course.
    - silviogutierrez 5y ago
      That's brilliant. Have the PR ready to go and merge/deploy only when that happens. Just host on us-east1 and it'll happen eventually.
- jbverschoor 5y ago
  I doubt that.. aws is down sometimes too. Card payments are sometimes down, even 112(911) is sometimes down. I hope you have everything redundant, including the CEO
  - fullstop 5y ago
    The database has a hot standby and two warm standby instances, redundant hardware, network infrastructure, all of that.
  - xcambar 5y ago
    > I hope you have everything redundant, including the CEO Made me laugh, I'll sure reuse it!
- hobs 5y ago
  OTOH I worked for several small companies that had "business hours" software - if you had to take downtime on the weekend or after hours you'd be looking at impacting <1% of users.
- willcipriano 5y ago
  In my view that's just saying at some point in the future the company will be devastated. Nothing has 100% uptime.
- chefkoch 5y ago
  Then hopefully everything is HA in your company.
  - fullstop 5y ago
    It absolutely is, BGP with geographically diverse paths, databases, app servers, etc, are all redundant. It's hosted in-house, so there is a cold standby database in AWS which would only be used if, say, an aircraft crashed into our server rooms.
    - jnpatel 5y ago
      Could you share what domain your company is in? Just curious what has such HA requirements
    - ikiris 5y ago
      It always amazes me when someone says they can't ever be down, and then says they only serve from one physical location actively. One of these things is not like the other.
      - fullstop 5y ago
        We have everything in place to run from AWS if needed but do not operate from there because of cost. This goes both ways -- there have been many AWS outages which have not affected us. I hear what you're saying, but we've had only one instance of extended (hours) downtime in the last 20 years.
        xcambar 5y ago
        The more experienced I become, the more such down to earth solutions seem OMG SO MUCH MORE reasonable than all the bells and whistles of "modern" engineering practices.
        ohyeshedid 5y ago
        Out of curiosity, what do you use in AWS to keep a running live backup of the data you'd need to migrate in the worst case scenario?
sharadov 5y ago
pg_upgrade with --hard-link option will upgrade the db in mins, I've done a 2TB db in less than a minute. Yes, there is no rollback, but if you do enough testing prior it can be really smooth. Am in the middle of doing this across an environment where we have tens of hosts from a few GB to couple TB.
- zozbot234 5y ago
  You could take a filesystem-level snapshot for rollback.
maccard 5y ago
I think it's always worth questioning both sides. Why is downtime acceptable? People on this site routinely complain about windows needing a restart for system updates while boasting about their Linux servers uptime. People talk about how kubernetes is overkill for many people, but it gives you rolling deployments for your applications out of the box. There's also the "slippery slope" argument. A 0 downtime migration means 0 downtime. A 5 minute migration creeps up to a 30 minutes migration occasionally, and then regression to the mean happens, causing standard migrations to be 30 minutes.
- Daishiman 5y ago
  A lot of 0-downtime migrations end up turning into a months-long ordeal and can still fail. Allowing maintenance windows means you can do things in a much simple manner. Need to take a snapshot of a DB without running into issues with a production system adding data? Sure, go ahead, you just saved two months.
  - hinkley 5y ago
    It's the old can/should argument. The principle of something like zero downtime or switching a vendor out to save 20% may not stack up against the reality in opportunity costs, employee burnout, and customer turnover. My company is reorganizing and so everyone is trying to settle old philosophical debates during The Churn. But everyone is overextended and a number of us have started pushing back. Just the meetings to discuss changing vendors might push the break-even point out 6 months, before you even get into implementation costs. The more time we spend working on infrastructure instead of customer-facing features and bugs, the more likely those customers are to wander off.
- munk-a 5y ago
  > Why is downtime acceptable? At least where I'm working downtime is acceptable because our customer base is composed of professionals in north america that work 9-5 thus taking the DB offline for five minutes at midnight PST has essentially no cost to our client base. To contrast that, spending three months developing a rollout plan costs our company dearly in competitive advantage. I agree that it's always worth examining from both sides but I also think that 0 downtime migration is both a shiny interesting problem and a point of pride for developers that can cause a company to vastly over invest in a complex technical solution to a problem they never had.
jeffbee 5y ago
The problem happens when you estimate it will only take a few hours and at the end of the first hour it's 1% done, or it nears completion and crashes, in a loop. Now what?
deleted 5y ago
[deleted]
rigaspapas 5y ago
A few minutes of downtime wouldn't be that bad, but it would harm the company's prestige. Unlike AWS and Netflix, we are not based on massive consumption, but on a few customers. Losing one good customer can make a difference. We believe it worth the investment of time.
rigaspapas 5y ago
Update: The cost is a good point. It wasn't that high. We actually updated the article to reflect this. It was two engineers, working full-time on it, for three weeks.

fullstop 5y ago

I work for a small company and it would be devastating if our database was down for a few hours.

silviogutierrez 5y ago
Unplanned? Probably. But maybe it's not as a dire as you think? Azure Active Directory (auth as a service) went down for a while, globally, and life went on. Same with GCP and occasionally AWS[1] I'm not saying there's no downside, I'm asking against the downside of engineering cost. And that itself carries risk of failure when you go live. It's not guaranteed. [1] AWS is so ubiquitous though, that half the internet would be affected so it makes it less individually noticeable.
- dividedbyzero 5y ago
  So don't host on AWS, wait out their next big outage and then take down your app for that big migration, so you can hide among all the other dead apps. Half joking, of course.
  - silviogutierrez 5y ago
    That's brilliant. Have the PR ready to go and merge/deploy only when that happens. Just host on us-east1 and it'll happen eventually.
jbverschoor 5y ago
I doubt that.. aws is down sometimes too. Card payments are sometimes down, even 112(911) is sometimes down. I hope you have everything redundant, including the CEO
- fullstop 5y ago
  The database has a hot standby and two warm standby instances, redundant hardware, network infrastructure, all of that.
- xcambar 5y ago
  > I hope you have everything redundant, including the CEO Made me laugh, I'll sure reuse it!
hobs 5y ago
OTOH I worked for several small companies that had "business hours" software - if you had to take downtime on the weekend or after hours you'd be looking at impacting <1% of users.
willcipriano 5y ago
In my view that's just saying at some point in the future the company will be devastated. Nothing has 100% uptime.
chefkoch 5y ago
Then hopefully everything is HA in your company.
- fullstop 5y ago
  It absolutely is, BGP with geographically diverse paths, databases, app servers, etc, are all redundant. It's hosted in-house, so there is a cold standby database in AWS which would only be used if, say, an aircraft crashed into our server rooms.
  - jnpatel 5y ago
    Could you share what domain your company is in? Just curious what has such HA requirements
  - ikiris 5y ago
    It always amazes me when someone says they can't ever be down, and then says they only serve from one physical location actively. One of these things is not like the other.
    - fullstop 5y ago
      We have everything in place to run from AWS if needed but do not operate from there because of cost. This goes both ways -- there have been many AWS outages which have not affected us. I hear what you're saying, but we've had only one instance of extended (hours) downtime in the last 20 years.
      - xcambar 5y ago
        The more experienced I become, the more such down to earth solutions seem OMG SO MUCH MORE reasonable than all the bells and whistles of "modern" engineering practices.
      - ohyeshedid 5y ago
        Out of curiosity, what do you use in AWS to keep a running live backup of the data you'd need to migrate in the worst case scenario?

sharadov 5y ago

pg_upgrade with --hard-link option will upgrade the db in mins, I've done a 2TB db in less than a minute. Yes, there is no rollback, but if you do enough testing prior it can be really smooth. Am in the middle of doing this across an environment where we have tens of hosts from a few GB to couple TB.

zozbot234 5y ago
You could take a filesystem-level snapshot for rollback.

maccard 5y ago

I think it's always worth questioning both sides. Why is downtime acceptable? People on this site routinely complain about windows needing a restart for system updates while boasting about their Linux servers uptime. People talk about how kubernetes is overkill for many people, but it gives you rolling deployments for your applications out of the box. There's also the "slippery slope" argument. A 0 downtime migration means 0 downtime. A 5 minute migration creeps up to a 30 minutes migration occasionally, and then regression to the mean happens, causing standard migrations to be 30 minutes.

Daishiman 5y ago
A lot of 0-downtime migrations end up turning into a months-long ordeal and can still fail. Allowing maintenance windows means you can do things in a much simple manner. Need to take a snapshot of a DB without running into issues with a production system adding data? Sure, go ahead, you just saved two months.
- hinkley 5y ago
  It's the old can/should argument. The principle of something like zero downtime or switching a vendor out to save 20% may not stack up against the reality in opportunity costs, employee burnout, and customer turnover. My company is reorganizing and so everyone is trying to settle old philosophical debates during The Churn. But everyone is overextended and a number of us have started pushing back. Just the meetings to discuss changing vendors might push the break-even point out 6 months, before you even get into implementation costs. The more time we spend working on infrastructure instead of customer-facing features and bugs, the more likely those customers are to wander off.
munk-a 5y ago
> Why is downtime acceptable? At least where I'm working downtime is acceptable because our customer base is composed of professionals in north america that work 9-5 thus taking the DB offline for five minutes at midnight PST has essentially no cost to our client base. To contrast that, spending three months developing a rollout plan costs our company dearly in competitive advantage. I agree that it's always worth examining from both sides but I also think that 0 downtime migration is both a shiny interesting problem and a point of pride for developers that can cause a company to vastly over invest in a complex technical solution to a problem they never had.

jeffbee 5y ago

The problem happens when you estimate it will only take a few hours and at the end of the first hour it's 1% done, or it nears completion and crashes, in a loop. Now what?

deleted 5y ago

[deleted]

rigaspapas 5y ago

A few minutes of downtime wouldn't be that bad, but it would harm the company's prestige. Unlike AWS and Netflix, we are not based on massive consumption, but on a few customers. Losing one good customer can make a difference. We believe it worth the investment of time.

rigaspapas 5y ago

Update: The cost is a good point. It wasn't that high. We actually updated the article to reflect this. It was two engineers, working full-time on it, for three weeks.