4 ms·

I wonder how much easier software engineering would be if there were a period where things are simply not available. What problems are currently very difficult

I wonder how much easier software engineering would be if there were a period where things are simply not available. What problems are currently very difficult would be made trivial if 6 hours of downtime every Sunday were acceptable? 10PM-4AM EST

yupper32 5y ago
I find it very hard to come up with a use case where a weekly 6 hours of downtime at night EST would be acceptable. US government websites already often do this, and I find that completely unacceptable since these services need to be available to anyone, regardless of work and life schedules. Any website that's international would also suffer greatly for an EST centralized scheduled downtime. Maybe a very localized website that doesn't have much impact on real life?
- endisneigh 5y ago
  Why wouldn’t it be acceptable? People aren’t available all the time either, nor are stores. In fact software is unique in its availability.
  - yupper32 5y ago
    Frankly, because it can be. Weekly scheduled downtime is arbitrary, manufactured, and lazy. Of course, having to schedule the occasional downtime for a database migration is fine. There's probably a few times a year that you'd need to do it if you don't have the bandwidth to do fancy zero-downtime solutions. It's the weekly arbitrary downtime that I'm firmly against.
    - endisneigh 5y ago
      Why is it lazy? When things are up people have to work. Do you believe people should be working all of the time? I think regular downtime is only natural. If you had to choose between 95% availability or 100% availability other than the before mentioned downtime which would you choose?
      - yupper32 5y ago
        Sorry I'm a bit confused by your argument. > When things are up people have to work. That's not true. Monitoring software and on-call rotations are well established things, and wouldn't even go away in your world. Believe it or not, people who run websites already sleep at night. > Do you believe people should be working all of the time? I'm not sure where you got that. Besides, you're already having people in the office in the middle of the night every Sunday already. Or are you saying that the downtime is when your team gets to sleep? > If you had to choose between 95% availability or 100% availability other than the before mentioned downtime which would you choose? Again, not sure what you're getting at. There's only a few reasons why you'd need to schedule a downtime. Database migrations are a good one. All of the reasons should only happen a few times a year, likely. Way better than 52 times a year.
        endisneigh 5y ago
        I’m not making a particular argument. I’m just wondering if having regular downtime is better or worse in terms of reducing engineering complexity.
    - Spooky23 5y ago
      Perhaps. Oftentimes it’s driven by legacy batch processes on mainframes. There’s a wide variety of reasons why it may make sense to carry on in that way.
    - nijave 5y ago
      It's expensive to run things like that. Multiple "9"s are significantly more expensive to engineer than, say, 95% uptime. In general, the correct amount of allowable downtime is when losses due to downtime are greater than the cost of increasing availability (which depends on the system). Should we raise taxes to increase government website availability? Not sure there's a clear cut case and it would definitely depend on each specific website and what utility it provides
maccard 5y ago
Your team is now working 10pm to 4am est every Sunday, and anyone who uses your services in any time zone around the world is without your service. 10pm est is lunch time in Australia. 4am est is working hours in many parts of western Europe. You don't have to be huge to have customers all around the world.
- yifanl 5y ago
  I think the idea is that this world is one where the customer accepts that there may be an outage every Sunday because feature work can be done far more cheaply.
- endisneigh 5y ago
  You’re correct, but if there happened to be an outage during that time now you’d have to have your team working then anyway. The existence of a planned downtime doesn’t necessarily mean deployments, or work, have to happen during that time.
  - yupper32 5y ago
    > but if there happened to be an outage during that time now you’d have to have your team working then anyway. So you're having them work late 52 times a year AND when there's an outage vs only when there's an outage. They're working late WAY more in your plan.
    - endisneigh 5y ago
      The existence of planned downtime doesn’t mean they have to work during said time. I’m not following your reasoning
      - kabdib 5y ago
        Er, you bring the service down so you can do work on it. Somebody is doing that work. I guarantee that you'll have trouble hiring qualified people for a role that includes core hours of "2AM-5AM Sunday mornings" (or whatever other oddball time you've decided). Updating systems is precisely when you may need to have all-hands-on-deck responses. Been there, multiple times.
i_have_an_idea 5y ago
Interestingly, this is unacceptable in the US and probably most of Western Europe. However, I know of more than one bank in Eastern Europe where online banking simply doesn't work after midnight, until about 6am.
- tinus_hn 5y ago
  My Dutch bank often sends me mails about their internet banking being available for a few hours during the night for maintenance. Availability is great but realistically, who’s going to use their bank in the middle of the night? It probably isn’t worth the effort and the risks to avoid the downtime.
legohead 5y ago
Easier sure, but Zero downtime migration is a fun engineering problem (albeit stressful). Just need an employer who is willing to pay for it.
kabdib 5y ago
What part of the world? One service I worked on (with customers around the globe) had about 10 minutes of scheduled downtime a week. Peak user load was, interestingly enough, during our core hours, when there are a lot more hands on deck to handle any issues. So that's the time we chose. Naturally people still complained bitterly about that ten minutes. Getting to zero didn't seem worth the significant engineering effort required. "You're not Amazon, and you're not 911" [emergency number in the US]
doctor_eval 5y ago
Setting aside the other comments, I think it depends on your definition of “easier”. Yes - it is easier in terms of up-front investment to have a scheduled (daily, weekly) downtime. But that often means the downtime is not automated - it’s someone doing the work, often out of hours when they’re not fresh, things can do wrong, etc. And it also means that the duration of the work is limited to the scheduled downtime. Some large upgrades just won’t fit in the allotted time, and you need to schedule an outage. On the other hand, creating a system designed for zero downtime (ie, in-place upgrades, removal of big batch jobs etc) is a lot more of an investment up front, and makes upgrade planning more complex, but ultimately it saves time and effort because you can do all the changes during working hours, and “upgrades” become just how you deploy code - part of the core skill set of your team - and you do them 10 times a day. The main difference that I’ve seen is that the scheduled downtime approach tends to create a much more conservative upgrade cycle, much bigger upgrades, and more risk of an upgrade failing - which means that a pile of features don’t get released. Also, certain customers will push back and demand a ton of documentation and detailed release notes and advanced warning of the upgrades etc - this creates loads of unproductive work for a small company. A continuous deployment model tends to reduce these problems because you’re not externalising your upgrade schedule and you don’t have a single big upgrade that can go bang, so maybe one feature doesn’t get released but everything else does. Having taken both approaches (and even moved a company from scheduled to continuous) I would never go back to a scheduled downtime approach.
tikkabhuna 5y ago
This is very common in the equity trading space. Applications must be up during the trading day (non-fri 8-5) but after that they can be upgraded and restarted easily. Typically there’s strict change management processes that prevent Continuous Delivery and slows the rollout of changes. Often I wish I worked in an industry where I could release during the day.
nijave 5y ago
I think it's still somewhat (fairly?) common for internal facing apps where you can just tell your users "It's going to be down in this window, plan accordingly". For a publicly available app, you'll lose your customers to a competitor with a more advanced maintenance strategy (or even one that happened to pick a NoSQL database where rolling upgrades are trivial)

yupper32 5y ago

I find it very hard to come up with a use case where a weekly 6 hours of downtime at night EST would be acceptable. US government websites already often do this, and I find that completely unacceptable since these services need to be available to anyone, regardless of work and life schedules. Any website that's international would also suffer greatly for an EST centralized scheduled downtime. Maybe a very localized website that doesn't have much impact on real life?

endisneigh 5y ago
Why wouldn’t it be acceptable? People aren’t available all the time either, nor are stores. In fact software is unique in its availability.
- yupper32 5y ago
  Frankly, because it can be. Weekly scheduled downtime is arbitrary, manufactured, and lazy. Of course, having to schedule the occasional downtime for a database migration is fine. There's probably a few times a year that you'd need to do it if you don't have the bandwidth to do fancy zero-downtime solutions. It's the weekly arbitrary downtime that I'm firmly against.
  - endisneigh 5y ago
    Why is it lazy? When things are up people have to work. Do you believe people should be working all of the time? I think regular downtime is only natural. If you had to choose between 95% availability or 100% availability other than the before mentioned downtime which would you choose?
    - yupper32 5y ago
      Sorry I'm a bit confused by your argument. > When things are up people have to work. That's not true. Monitoring software and on-call rotations are well established things, and wouldn't even go away in your world. Believe it or not, people who run websites already sleep at night. > Do you believe people should be working all of the time? I'm not sure where you got that. Besides, you're already having people in the office in the middle of the night every Sunday already. Or are you saying that the downtime is when your team gets to sleep? > If you had to choose between 95% availability or 100% availability other than the before mentioned downtime which would you choose? Again, not sure what you're getting at. There's only a few reasons why you'd need to schedule a downtime. Database migrations are a good one. All of the reasons should only happen a few times a year, likely. Way better than 52 times a year.
      - endisneigh 5y ago
        I’m not making a particular argument. I’m just wondering if having regular downtime is better or worse in terms of reducing engineering complexity.
  - Spooky23 5y ago
    Perhaps. Oftentimes it’s driven by legacy batch processes on mainframes. There’s a wide variety of reasons why it may make sense to carry on in that way.
  - nijave 5y ago
    It's expensive to run things like that. Multiple "9"s are significantly more expensive to engineer than, say, 95% uptime. In general, the correct amount of allowable downtime is when losses due to downtime are greater than the cost of increasing availability (which depends on the system). Should we raise taxes to increase government website availability? Not sure there's a clear cut case and it would definitely depend on each specific website and what utility it provides

maccard 5y ago

Your team is now working 10pm to 4am est every Sunday, and anyone who uses your services in any time zone around the world is without your service. 10pm est is lunch time in Australia. 4am est is working hours in many parts of western Europe. You don't have to be huge to have customers all around the world.

yifanl 5y ago
I think the idea is that this world is one where the customer accepts that there may be an outage every Sunday because feature work can be done far more cheaply.
endisneigh 5y ago
You’re correct, but if there happened to be an outage during that time now you’d have to have your team working then anyway. The existence of a planned downtime doesn’t necessarily mean deployments, or work, have to happen during that time.
- yupper32 5y ago
  > but if there happened to be an outage during that time now you’d have to have your team working then anyway. So you're having them work late 52 times a year AND when there's an outage vs only when there's an outage. They're working late WAY more in your plan.
  - endisneigh 5y ago
    The existence of planned downtime doesn’t mean they have to work during said time. I’m not following your reasoning
    - kabdib 5y ago
      Er, you bring the service down so you can do work on it. Somebody is doing that work. I guarantee that you'll have trouble hiring qualified people for a role that includes core hours of "2AM-5AM Sunday mornings" (or whatever other oddball time you've decided). Updating systems is precisely when you may need to have all-hands-on-deck responses. Been there, multiple times.

i_have_an_idea 5y ago

Interestingly, this is unacceptable in the US and probably most of Western Europe. However, I know of more than one bank in Eastern Europe where online banking simply doesn't work after midnight, until about 6am.

tinus_hn 5y ago
My Dutch bank often sends me mails about their internet banking being available for a few hours during the night for maintenance. Availability is great but realistically, who’s going to use their bank in the middle of the night? It probably isn’t worth the effort and the risks to avoid the downtime.

legohead 5y ago

Easier sure, but Zero downtime migration is a fun engineering problem (albeit stressful). Just need an employer who is willing to pay for it.

kabdib 5y ago

What part of the world? One service I worked on (with customers around the globe) had about 10 minutes of scheduled downtime a week. Peak user load was, interestingly enough, during our core hours, when there are a lot more hands on deck to handle any issues. So that's the time we chose. Naturally people still complained bitterly about that ten minutes. Getting to zero didn't seem worth the significant engineering effort required. "You're not Amazon, and you're not 911" [emergency number in the US]

doctor_eval 5y ago

Setting aside the other comments, I think it depends on your definition of “easier”. Yes - it is easier in terms of up-front investment to have a scheduled (daily, weekly) downtime. But that often means the downtime is not automated - it’s someone doing the work, often out of hours when they’re not fresh, things can do wrong, etc. And it also means that the duration of the work is limited to the scheduled downtime. Some large upgrades just won’t fit in the allotted time, and you need to schedule an outage. On the other hand, creating a system designed for zero downtime (ie, in-place upgrades, removal of big batch jobs etc) is a lot more of an investment up front, and makes upgrade planning more complex, but ultimately it saves time and effort because you can do all the changes during working hours, and “upgrades” become just how you deploy code - part of the core skill set of your team - and you do them 10 times a day. The main difference that I’ve seen is that the scheduled downtime approach tends to create a much more conservative upgrade cycle, much bigger upgrades, and more risk of an upgrade failing - which means that a pile of features don’t get released. Also, certain customers will push back and demand a ton of documentation and detailed release notes and advanced warning of the upgrades etc - this creates loads of unproductive work for a small company. A continuous deployment model tends to reduce these problems because you’re not externalising your upgrade schedule and you don’t have a single big upgrade that can go bang, so maybe one feature doesn’t get released but everything else does. Having taken both approaches (and even moved a company from scheduled to continuous) I would never go back to a scheduled downtime approach.

tikkabhuna 5y ago

This is very common in the equity trading space. Applications must be up during the trading day (non-fri 8-5) but after that they can be upgraded and restarted easily. Typically there’s strict change management processes that prevent Continuous Delivery and slows the rollout of changes. Often I wish I worked in an industry where I could release during the day.

nijave 5y ago

I think it's still somewhat (fairly?) common for internal facing apps where you can just tell your users "It's going to be down in this window, plan accordingly". For a publicly available app, you'll lose your customers to a competitor with a more advanced maintenance strategy (or even one that happened to pick a NoSQL database where rolling upgrades are trivial)