5 ms·

This is not how businesses are run. No sane person running a business will take a 1PM outage over 1AM. The whole point of heroku is that you are relying on them

This is not how businesses are run. No sane person running a business will take a 1PM outage over 1AM. The whole point of heroku is that you are relying on them to do a lot of the infrastructure systems because you are low on manpower. The resilient system will be built over time and is being worked upon. But to put the blame on heroku's customer for the outage is nonsensical. People do night shifts in so many industries. There's nothing wrong with that, it's just part of the job. My building security guard does a night shift and what he does is way more important and needs more attention than an engineer. Unless you think we should not have security guards in the night.

alrs 12y ago
No one with half a backbone should be willing to do scheduled work all day and then during any portion of the night. You need to build a resilient system, and that's gonna cost you. It's a lot less fun than adding features, which is why business people tend to put it off and instead manipulate the tech people into working crazy hours. The 24/7 web culture is dying. Adjust your business plan accordingly.
- deleted 12y ago
  [deleted]
  - deleted 12y ago
    [deleted]
- scott_w 12y ago
  You don't have to. You can take the prior or following day off after the overnight work.
  - alrs 12y ago
    I was never presented this option by any of my previous employers.
    - scott_w 12y ago
      That sucks. Whilst we're able to do most work during the 9-5, there's the odd time I'll come in at 5 or the IT manager works a Saturday because we have to take down key infrastructure to work on it. If your working environment is good, it's pretty easy to just say "I'll take tomorrow off and come in to do it on Saturday" without feeling like you're getting taken advantage of.
- samstave 12y ago
  You have no idea what you're talking about.
teacup50 12y ago
Very few professionals do night shifts. Equating a security guard with IT work is insane. The intellectual demands are nowhere near equivalent. For professional examples of "night shifts", we have ER doctors. They do work 12-24 hour shifts -- but they're allotted a room they can sleep in between patients, and they often have 2-7 days between their shifts. You going to pay infrastructure staff what you pay ER doctors for as much on-task work? Are those people going to actually know what to do when you put them on-task? What happens when the IT staff needs to get a developer on-hand to resolve an issue? Wake them up?
- benihana 12y ago
  (for cases like Heroku) When infrastructure staff fuck up, people don't die. There's a massive difference between "make sure a few scripts I wrote earlier work right" and "patch up the guy who can see his guts cause some drunk driver ran a red light at 3 AM" >What happens when the IT staff needs to get a developer on-hand to resolve an issue? Wake them up? Yes. Have you never worked at a place with an on-call dev team?
opendais 12y ago
Eh, the better option would be to run a EU and a US west team. A 10 hour time difference lets you do something like: 10am - 10pm [on call] for US West. 7am - 7pm [on call] for EU [UTC +1] 4 days on / 3 days off. [48 hr on call, 40 hours of work] Its doable. Just expensive to cover 2 jurisdictions and maintain 4 shifts worth of people. No system is ever going to be 100% reliable and cost effective.
- alrs 12y ago
  You're right, totally doable. I should have said 24/7 for the price of 8/5 is going away.
  - opendais 12y ago
    Yep, and paying for 8/5 and getting 24/7 is unreasonable. ;)
- deleted 12y ago
  [deleted]
gbog 12y ago
I have never lived in a place with security guards in the night. I think a big part of freedom and privacy is dead when you have to have security guards. Of topic, sorry.
groby_b 12y ago
Plenty of sane people prefer a 1PM outage over a 1AM. Everybody is in the office. Everybody is wide awake. Of course people do night shifts, but at the same time, we've learned that you're better off fixing things during the day shift. The night shift is an emergency crew. And the idea that a building security guard needs to pay more attention than an engineer is beyond ludicrous.
namecast 12y ago
That's like, three logical fallacies tied together. You've made an argument to authority with your lead-off sentence of "that's not how businessses are run" and a no true scotsman argument with "no sane person running a business", and then tied it up with an absurdo ad reductum analogy about security guards. There are plenty of shops that I've personally worked with, and many more that I've read about here on HN that do daytime maintenance / make potentially breaking changes to production. I'd be willing to bet Heroku does a mix of day and night time scheduled maintenances, with scheduling determined by a reasoned analysis of the risks and potential for an outage. If you think a 1 AM maintenance window helps anything, I'd bet you've never seen the sunlight come up as you staggered out of a datacenter 10 hours after your "midnight maintenance" went south, and no one with an optical light meter to help you trace down a dodgy fiber run was awakee at your transit provider, so you had to wait until the first shops opened to buy your own... ...ok, maybe I'm a little bitter. What I'm getting at is, a 1 PM outage - although more customer facing than a 1 AM outage - will almost definitely be resolved faster, because resources outside of the engineers performing the maintenance are in abundance at 1 PM and not 1 AM. Bonus points: your engineers will not be zombies for the next three days, and that's important both for post-outage work and to make sure everyone stays sharp for the potential "bounce outage" (e.g. your first outage was the core router dying, the second will hit the same week when your new replacement core router's slightly upgraded IOS version causes a BGP flap under certain conditions that don't arise until a good 48 hours after you mail out your customer outage report....) That might sound trivial, but at scale - if you have tons of engineers and are doing tons of maintenance all the time - it's actually really important, at least IMHO. (Quick, take a guess - how many Heroku daytime maintenances do you think went off without a hitch that you've never heard of?) P.S. what time zone are you in? I assume your time zone's 1 AM is the important one, so someone else on the other side of the country is going to either have a late evening or early morning maintenance window..... ;)
zaqokm 12y ago
> No sane person running a business will take a 1PM outage over 1AM. Hey what about us people in Asia who use Heroku ... a early US AM outage is likely to be in our early PMs.
deleted 12y ago
[deleted]
yeukhon 12y ago
I think an important point to note is continuous deployment, twice a day seems logical and minimal to me, even if the deployment is deploying code from yesterday (a 1-day gap).

alrs 12y ago

No one with half a backbone should be willing to do scheduled work all day and then during any portion of the night. You need to build a resilient system, and that's gonna cost you. It's a lot less fun than adding features, which is why business people tend to put it off and instead manipulate the tech people into working crazy hours. The 24/7 web culture is dying. Adjust your business plan accordingly.

deleted 12y ago
[deleted]
- deleted 12y ago
  [deleted]
scott_w 12y ago
You don't have to. You can take the prior or following day off after the overnight work.
- alrs 12y ago
  I was never presented this option by any of my previous employers.
  - scott_w 12y ago
    That sucks. Whilst we're able to do most work during the 9-5, there's the odd time I'll come in at 5 or the IT manager works a Saturday because we have to take down key infrastructure to work on it. If your working environment is good, it's pretty easy to just say "I'll take tomorrow off and come in to do it on Saturday" without feeling like you're getting taken advantage of.
samstave 12y ago
You have no idea what you're talking about.

teacup50 12y ago

Very few professionals do night shifts. Equating a security guard with IT work is insane. The intellectual demands are nowhere near equivalent. For professional examples of "night shifts", we have ER doctors. They do work 12-24 hour shifts -- but they're allotted a room they can sleep in between patients, and they often have 2-7 days between their shifts. You going to pay infrastructure staff what you pay ER doctors for as much on-task work? Are those people going to actually know what to do when you put them on-task? What happens when the IT staff needs to get a developer on-hand to resolve an issue? Wake them up?

benihana 12y ago
(for cases like Heroku) When infrastructure staff fuck up, people don't die. There's a massive difference between "make sure a few scripts I wrote earlier work right" and "patch up the guy who can see his guts cause some drunk driver ran a red light at 3 AM" >What happens when the IT staff needs to get a developer on-hand to resolve an issue? Wake them up? Yes. Have you never worked at a place with an on-call dev team?

opendais 12y ago

Eh, the better option would be to run a EU and a US west team. A 10 hour time difference lets you do something like: 10am - 10pm [on call] for US West. 7am - 7pm [on call] for EU [UTC +1] 4 days on / 3 days off. [48 hr on call, 40 hours of work] Its doable. Just expensive to cover 2 jurisdictions and maintain 4 shifts worth of people. No system is ever going to be 100% reliable and cost effective.

alrs 12y ago
You're right, totally doable. I should have said 24/7 for the price of 8/5 is going away.
- opendais 12y ago
  Yep, and paying for 8/5 and getting 24/7 is unreasonable. ;)
deleted 12y ago
[deleted]

gbog 12y ago

I have never lived in a place with security guards in the night. I think a big part of freedom and privacy is dead when you have to have security guards. Of topic, sorry.

groby_b 12y ago

Plenty of sane people prefer a 1PM outage over a 1AM. Everybody is in the office. Everybody is wide awake. Of course people do night shifts, but at the same time, we've learned that you're better off fixing things during the day shift. The night shift is an emergency crew. And the idea that a building security guard needs to pay more attention than an engineer is beyond ludicrous.

namecast 12y ago

That's like, three logical fallacies tied together. You've made an argument to authority with your lead-off sentence of "that's not how businessses are run" and a no true scotsman argument with "no sane person running a business", and then tied it up with an absurdo ad reductum analogy about security guards. There are plenty of shops that I've personally worked with, and many more that I've read about here on HN that do daytime maintenance / make potentially breaking changes to production. I'd be willing to bet Heroku does a mix of day and night time scheduled maintenances, with scheduling determined by a reasoned analysis of the risks and potential for an outage. If you think a 1 AM maintenance window helps anything, I'd bet you've never seen the sunlight come up as you staggered out of a datacenter 10 hours after your "midnight maintenance" went south, and no one with an optical light meter to help you trace down a dodgy fiber run was awakee at your transit provider, so you had to wait until the first shops opened to buy your own... ...ok, maybe I'm a little bitter. What I'm getting at is, a 1 PM outage - although more customer facing than a 1 AM outage - will almost definitely be resolved faster, because resources outside of the engineers performing the maintenance are in abundance at 1 PM and not 1 AM. Bonus points: your engineers will not be zombies for the next three days, and that's important both for post-outage work and to make sure everyone stays sharp for the potential "bounce outage" (e.g. your first outage was the core router dying, the second will hit the same week when your new replacement core router's slightly upgraded IOS version causes a BGP flap under certain conditions that don't arise until a good 48 hours after you mail out your customer outage report....) That might sound trivial, but at scale - if you have tons of engineers and are doing tons of maintenance all the time - it's actually really important, at least IMHO. (Quick, take a guess - how many Heroku daytime maintenances do you think went off without a hitch that you've never heard of?) P.S. what time zone are you in? I assume your time zone's 1 AM is the important one, so someone else on the other side of the country is going to either have a late evening or early morning maintenance window..... ;)

zaqokm 12y ago

> No sane person running a business will take a 1PM outage over 1AM. Hey what about us people in Asia who use Heroku ... a early US AM outage is likely to be in our early PMs.

deleted 12y ago

[deleted]

yeukhon 12y ago

I think an important point to note is continuous deployment, twice a day seems logical and minimal to me, even if the deployment is deploying code from yesterday (a 1-day gap).