6 ms·

Deployment and maintenance should not happen in the middle of the night. We've known this for years. You design your systems to degrade gracefully, hopefully

by alrs 12y ago

Deployment and maintenance should not happen in the middle of the night. We've known this for years. You design your systems to degrade gracefully, hopefully transparently. Screwing up your engineers' sleep schedules is a terrible idea, so you deploy during the day when people are awake, clear-headed, and on the clock. I'll take a 1PM outage over a 1AM outage any day. If that's bad for business, business should pay for the engineering time to build a more resilient system.

jerhinesmith 12y ago
Is that true? I'm not being sarcastic -- I just have always generally assumed that maintenance was better performed "off-hours". Also, 1pm and 1am aren't the only options. 8am EDT would have been within the work day but still avoided peak usage in the US.
- alrs 12y ago
  8AM EDT is 5AM PDT, which is where web tech can be found in the United States. I'm sure you can find engineers willing to work garbage-man hours, but it's gonna cost you.
  - personZ 12y ago
    8AM EDT is 5AM PDT, which is where web tech can be found in the United States. I'm not sure if you're just being sarcastic, but talent in North America isn't limited to the Pacific Timezone, and such a claim borders on hilarious delusion. I don't say hilarious emotionally or pejoratively, but rather it is actually ha ha funny that anyone would actually believe that. However yes, there are endless loads of top-skill talent that will happily do "maintenance" in the middle of the night. I've done it on occasion, and slept in the next day. Big deal. Half the time I simply took a timeout from some online gaming I'd been doing.
    - alrs 12y ago
      We don't have "loads" of top-skill talent to begin with. The pool of top-skill-talent-with-a-self-esteem-problem is rapidly diminishing.
      - personZ 12y ago
        To do something off hours implies a self-esteem problem? That is absurd. People do stuff off hours in return for something, which might be compensatory "time off", additional pay or bonuses, etc.
        alrs 12y ago
        To do scheduled off-hours work on top of scheduled on-hours work and not get paid for it indicates that the worker has a damaged sense of self-worth.
- Isofarro 12y ago
  Yes, Amazon have a rule to deploy during working hours, during core hours when engineers are around and available should things go wrong. Also, Amazon is one of a number of organisations who don't deploy releases on Friday.
  - bethling 12y ago
    Amazon does not deploy changes during times that they feel can be customer impacting. For North America this means that deployments will be during the night.
- hga 12y ago
  IF you can afford the service interruption, which is true for lots of companies and their internal IT systems, you always follow this policy. All concerned including the users are awake and on their normal sleep and work cycle, if you mess something up you find out immediately from those users, etc. etc.
general_failure 12y ago
This is not how businesses are run. No sane person running a business will take a 1PM outage over 1AM. The whole point of heroku is that you are relying on them to do a lot of the infrastructure systems because you are low on manpower. The resilient system will be built over time and is being worked upon. But to put the blame on heroku's customer for the outage is nonsensical. People do night shifts in so many industries. There's nothing wrong with that, it's just part of the job. My building security guard does a night shift and what he does is way more important and needs more attention than an engineer. Unless you think we should not have security guards in the night.
- alrs 12y ago
  No one with half a backbone should be willing to do scheduled work all day and then during any portion of the night. You need to build a resilient system, and that's gonna cost you. It's a lot less fun than adding features, which is why business people tend to put it off and instead manipulate the tech people into working crazy hours. The 24/7 web culture is dying. Adjust your business plan accordingly.
  - deleted 12y ago
    [deleted]
    - deleted 12y ago
      [deleted]
  - scott_w 12y ago
    You don't have to. You can take the prior or following day off after the overnight work.
    - alrs 12y ago
      I was never presented this option by any of my previous employers.
      - 12y ago
B0Z 12y ago
As a seasoned engineer, I could not agree more with the justifications you provide. As a business owner who's customers are actively engaged with my staff who need servers and resources to be available during the day when, you know, my CUSTOMERS are awake, your recommendation for outage windows would be completely ignored. You schedule outages around the people who pay the bills, not the people who don't.
- alrs 12y ago
  The plan is to never have an outage. Instead be scheduling maintenance that should, at worst, result in some backed up queues or non-functional admin functions. Building a system for graceful degradation costs time and money.
worklogin 12y ago
Number of banks or hospitals alrs has worked at: 0 Hope for the best, and plan for the worst: Build resilient systems when possible, but why risk (or guarantee) outages during the day? OS upgrades, router swapouts, and so on can NOT take place during the day. That would be foolish in most industries. Also realize that "just make systems resilient" is not an easy thing when multi-million dollar transactions are occurring on 30 year old code. If everything were greenfield, it'd be different. But it's not.
- alrs 12y ago
  Always be proving that your infrastructure is resilient. Pain should be felt by stakeholders, not engineering. When 24/7 means millions of dollars, hire a four-shift redundant geo-distributed team.
  - unethical_ban 12y ago
    Thing is, a lot of places are doing that. The downside is the geo-distribution gives us Indian contractors with strange accents, inability to communicate and assert, and overall a huge communications issue by default. Australia needs to step up its subsidies to IT outsourcing firms.
samstave 12y ago
No, that depends on what your traffic curve looks like. For me, I have to do my maint at 12AM PST, because that is when we are in a trough with respect to traffic.
bentaber 12y ago
It's all about reducing the risk. Accidents happen no matter how well you've engineered your maintenance and deployment practices. Do everything you can to reduce the risk, including building a resilient system, AND not taking a major swath of it offline during a high traffic period.
falcolas 12y ago
If you are a company who is using Heroku, you likely do not have the resources (knowledge or money) to set up a system which degrades gracefully when your entire hosting platform goes down. That's a hard, and expensive, contingency to plan for. And even if they did build their service in a way which could tolerate a complete hosting platform failure - degrading a core service during your customer's business hours is enough to make heads roll. It's just stupid.
aiiane 12y ago
For those saying that the parent comment is crazy, or not applicable to "real businesses" - nearly every production push at Google is done during the normal workday hours of the respective development team (which means most of them happen between 9am-5pm PST). The benefit of having the engineering team around to troubleshoot smaller-scale issues before they turn into large-scale outages vastly outweighs the small benefit of potentially moving the extremely rare massive outage to a less busy time of day.
jlawer 12y ago
Why not have engineers in different time zones? 12:30am on EST is 3:30pm AEST (Australian Eastern Standard Time) and 7:30am in London. That way you have skilled people awake all hours should a major issue develop. These people can fix the problems while US based clients are sleeping and non-US clients aren't stuck with waisting a day with less severe faults because the engineers only work 9 to 5 PST.

by alrs 12y ago

jerhinesmith 12y ago

Is that true? I'm not being sarcastic -- I just have always generally assumed that maintenance was better performed "off-hours". Also, 1pm and 1am aren't the only options. 8am EDT would have been within the work day but still avoided peak usage in the US.

alrs 12y ago
8AM EDT is 5AM PDT, which is where web tech can be found in the United States. I'm sure you can find engineers willing to work garbage-man hours, but it's gonna cost you.
- personZ 12y ago
  8AM EDT is 5AM PDT, which is where web tech can be found in the United States. I'm not sure if you're just being sarcastic, but talent in North America isn't limited to the Pacific Timezone, and such a claim borders on hilarious delusion. I don't say hilarious emotionally or pejoratively, but rather it is actually ha ha funny that anyone would actually believe that. However yes, there are endless loads of top-skill talent that will happily do "maintenance" in the middle of the night. I've done it on occasion, and slept in the next day. Big deal. Half the time I simply took a timeout from some online gaming I'd been doing.
  - alrs 12y ago
    We don't have "loads" of top-skill talent to begin with. The pool of top-skill-talent-with-a-self-esteem-problem is rapidly diminishing.
    - personZ 12y ago
      To do something off hours implies a self-esteem problem? That is absurd. People do stuff off hours in return for something, which might be compensatory "time off", additional pay or bonuses, etc.
      - alrs 12y ago
        To do scheduled off-hours work on top of scheduled on-hours work and not get paid for it indicates that the worker has a damaged sense of self-worth.
Isofarro 12y ago
Yes, Amazon have a rule to deploy during working hours, during core hours when engineers are around and available should things go wrong. Also, Amazon is one of a number of organisations who don't deploy releases on Friday.
- bethling 12y ago
  Amazon does not deploy changes during times that they feel can be customer impacting. For North America this means that deployments will be during the night.
hga 12y ago
IF you can afford the service interruption, which is true for lots of companies and their internal IT systems, you always follow this policy. All concerned including the users are awake and on their normal sleep and work cycle, if you mess something up you find out immediately from those users, etc. etc.

general_failure 12y ago

This is not how businesses are run. No sane person running a business will take a 1PM outage over 1AM. The whole point of heroku is that you are relying on them to do a lot of the infrastructure systems because you are low on manpower. The resilient system will be built over time and is being worked upon. But to put the blame on heroku's customer for the outage is nonsensical. People do night shifts in so many industries. There's nothing wrong with that, it's just part of the job. My building security guard does a night shift and what he does is way more important and needs more attention than an engineer. Unless you think we should not have security guards in the night.

alrs 12y ago
No one with half a backbone should be willing to do scheduled work all day and then during any portion of the night. You need to build a resilient system, and that's gonna cost you. It's a lot less fun than adding features, which is why business people tend to put it off and instead manipulate the tech people into working crazy hours. The 24/7 web culture is dying. Adjust your business plan accordingly.
- deleted 12y ago
  [deleted]
  - deleted 12y ago
    [deleted]
- scott_w 12y ago
  You don't have to. You can take the prior or following day off after the overnight work.
  - alrs 12y ago
    I was never presented this option by any of my previous employers.
    - 12y ago

B0Z 12y ago

As a seasoned engineer, I could not agree more with the justifications you provide. As a business owner who's customers are actively engaged with my staff who need servers and resources to be available during the day when, you know, my CUSTOMERS are awake, your recommendation for outage windows would be completely ignored. You schedule outages around the people who pay the bills, not the people who don't.

alrs 12y ago
The plan is to never have an outage. Instead be scheduling maintenance that should, at worst, result in some backed up queues or non-functional admin functions. Building a system for graceful degradation costs time and money.

worklogin 12y ago

Number of banks or hospitals alrs has worked at: 0 Hope for the best, and plan for the worst: Build resilient systems when possible, but why risk (or guarantee) outages during the day? OS upgrades, router swapouts, and so on can NOT take place during the day. That would be foolish in most industries. Also realize that "just make systems resilient" is not an easy thing when multi-million dollar transactions are occurring on 30 year old code. If everything were greenfield, it'd be different. But it's not.

alrs 12y ago
Always be proving that your infrastructure is resilient. Pain should be felt by stakeholders, not engineering. When 24/7 means millions of dollars, hire a four-shift redundant geo-distributed team.
- unethical_ban 12y ago
  Thing is, a lot of places are doing that. The downside is the geo-distribution gives us Indian contractors with strange accents, inability to communicate and assert, and overall a huge communications issue by default. Australia needs to step up its subsidies to IT outsourcing firms.

samstave 12y ago

No, that depends on what your traffic curve looks like. For me, I have to do my maint at 12AM PST, because that is when we are in a trough with respect to traffic.

bentaber 12y ago

It's all about reducing the risk. Accidents happen no matter how well you've engineered your maintenance and deployment practices. Do everything you can to reduce the risk, including building a resilient system, AND not taking a major swath of it offline during a high traffic period.

falcolas 12y ago

If you are a company who is using Heroku, you likely do not have the resources (knowledge or money) to set up a system which degrades gracefully when your entire hosting platform goes down. That's a hard, and expensive, contingency to plan for. And even if they did build their service in a way which could tolerate a complete hosting platform failure - degrading a core service during your customer's business hours is enough to make heads roll. It's just stupid.

aiiane 12y ago

For those saying that the parent comment is crazy, or not applicable to "real businesses" - nearly every production push at Google is done during the normal workday hours of the respective development team (which means most of them happen between 9am-5pm PST). The benefit of having the engineering team around to troubleshoot smaller-scale issues before they turn into large-scale outages vastly outweighs the small benefit of potentially moving the extremely rare massive outage to a less busy time of day.

jlawer 12y ago

Why not have engineers in different time zones? 12:30am on EST is 3:30pm AEST (Australian Eastern Standard Time) and 7:30am in London. That way you have skilled people awake all hours should a major issue develop. These people can fix the problems while US based clients are sleeping and non-US clients aren't stuck with waisting a day with less severe faults because the engineers only work 9 to 5 PST.