4 ms·

I just don’t get all the “just shutdown the db for an hour, you’re not Netflix” comments. If you can do things properly, as an engineer, you absolutely should,

I just don’t get all the “just shutdown the db for an hour, you’re not Netflix” comments. If you can do things properly, as an engineer, you absolutely should, even if your company serves “just” hundreds of thousands instead of hundreds of millions users. It is not like they wrote their own database for that, they just used an open source tool.

mswtk 5y ago
My understanding of doing things "properly" as an engineer, is picking the solution with the right tradeoffs for my use case. If the cost to the business of having some amount of scheduled downtime occasionally is significantly less than the engineering cost of maintaining several 9s worth of availability over major migrations, then I consider the former to be "done right".
perrygeo 5y ago
If you do things properly as an engineer, you've already negotiated and committed to service-level agreements with specific downtime objectives, right? Right? If you've got say 5 minutes of downtime budget per month, do you really think it's a good investment to spend a million dollars of engineering effort and opportunity costs to get database downtime to zero seconds? Or you could use off-the-shelf techniques for orders of magnitude less investment and suffer a few minutes of downtime at worst, well within your SLO. It's an economic decision, not a technical one. If zero downtime at any cost is non-negotiable, well you'd better hire, plan and budget accordingly. And hope those few extra minutes per year are worth the cost.
- coolspot 5y ago
  I agree that it is an economic decision. In my understanding it didn’t cost them millions of dollars to do it zero-downtime. Maybe a $10k-$50k in development/admin/test hours to make it so. Also engineers get better (and happier) when they do challenging tasks!
  - koreth1 5y ago
    > Also engineers get better (and happier) when they do challenging tasks! Better maybe, but happier, I'm not so sure. At my last job I was the main developer in charge of hitting our zero-downtime target for code deploys and system upgrades, and it was a pain in the ass to always have to implement multi-stage migration processes with bidirectional compatibility between adjacent stages for things that would have been very little work if we'd been able to schedule an hour of downtime. The cost was worth it from a business point of view, but it wasn't much fun to actually do the work. Though I guess I agree with the "happier" part too if you're just talking about doing the initial infrastructure and software-architecture work that allows those annoying multi-stage migrations to run seamlessly. I did enjoy writing the migration framework code and figuring out what all the stages would need to be.
  - Cullinet 5y ago
    > Also engineers get better (and happier) when they do challenging tasks! I would rather encourage them to engineer to the limit of ability but deliver to Ed: within the limit of burning out my engineering team, plus some comfortable margin gained from the encouragement given, which should be rewarded. I've talked myself into engineering insufficient time for unforseen downtime tasks and it took me a while to realise that I was wearing my management hat as a founder and not the engineering hat I should have been wearing.

mswtk 5y ago

My understanding of doing things "properly" as an engineer, is picking the solution with the right tradeoffs for my use case. If the cost to the business of having some amount of scheduled downtime occasionally is significantly less than the engineering cost of maintaining several 9s worth of availability over major migrations, then I consider the former to be "done right".

perrygeo 5y ago

If you do things properly as an engineer, you've already negotiated and committed to service-level agreements with specific downtime objectives, right? Right? If you've got say 5 minutes of downtime budget per month, do you really think it's a good investment to spend a million dollars of engineering effort and opportunity costs to get database downtime to zero seconds? Or you could use off-the-shelf techniques for orders of magnitude less investment and suffer a few minutes of downtime at worst, well within your SLO. It's an economic decision, not a technical one. If zero downtime at any cost is non-negotiable, well you'd better hire, plan and budget accordingly. And hope those few extra minutes per year are worth the cost.

coolspot 5y ago
I agree that it is an economic decision. In my understanding it didn’t cost them millions of dollars to do it zero-downtime. Maybe a $10k-$50k in development/admin/test hours to make it so. Also engineers get better (and happier) when they do challenging tasks!
- koreth1 5y ago
  > Also engineers get better (and happier) when they do challenging tasks! Better maybe, but happier, I'm not so sure. At my last job I was the main developer in charge of hitting our zero-downtime target for code deploys and system upgrades, and it was a pain in the ass to always have to implement multi-stage migration processes with bidirectional compatibility between adjacent stages for things that would have been very little work if we'd been able to schedule an hour of downtime. The cost was worth it from a business point of view, but it wasn't much fun to actually do the work. Though I guess I agree with the "happier" part too if you're just talking about doing the initial infrastructure and software-architecture work that allows those annoying multi-stage migrations to run seamlessly. I did enjoy writing the migration framework code and figuring out what all the stages would need to be.
- Cullinet 5y ago
  > Also engineers get better (and happier) when they do challenging tasks! I would rather encourage them to engineer to the limit of ability but deliver to Ed: within the limit of burning out my engineering team, plus some comfortable margin gained from the encouragement given, which should be rewarded. I've talked myself into engineering insufficient time for unforseen downtime tasks and it took me a while to realise that I was wearing my management hat as a founder and not the engineering hat I should have been wearing.