6 ms·

> assumptions about always-on digital services And maybe also always-on humans, which some companies seem to ridiculously expect. I really don't understand th

by dheera 2y ago

> assumptions about always-on digital services And maybe also always-on humans, which some companies seem to ridiculously expect. I really don't understand this obsession with 24/7 uptime for non-critical systems. Requiring your engineers to be always on-call and debug something at 3am is a health hazard and should be treated like one. If a photo-sharing app is down at 3am, I'm sure the users can go to sleep and wait till 10am. This isn't some oxygen life support system. If you have that many users in multiple time zones, then hire people in multiple time zones. Even if TurboTax crashes on 4/15 at 11:35pm and the engineers don't fix it until the next workday, resulting in millions of people not being able to file their taxes, I'm sure the IRS might grumble a lot but would give people an extension. It'll all be good, and everyone will get to sleep .

lukan 2y ago
"If a photo-sharing app is down at 3am, I'm sure the users can go to sleep and wait till 10am" There will be people, who will feel it is critical important to post some pictures at 3 am and they will get stressed, if it is not working (say people preparing an event and the pictures should be online the next day). But .. whether that is worth that engineers must be on call, is a differnt question. I never had a job like this and I know I would never accept it as default for myself.
- dheera 2y ago
  > people preparing an event and the pictures should be online the next day They can use a different platform, print the pictures on paper, there are a million ways to deal with these kind of issues. An earthquake could strike and the event might need to be postponed. If nobody's life or health is at risk, it is not urgent enough to sacrifice someone else's health for it.
  - lukan 2y ago
    "If nobody's life or health is at risk, it is not urgent enough to sacrifice someone else's health for it." I agree. But I think being on call is possible in a healthy way, if there are long enough brakes in between. Occasionally being a night on call is possible. (Well and normal if you have small kids, but having small kids and having to worry about getting an emergency call any minute, that would be draining)
  - em-bee 2y ago
    They can use a different platform not if they already committed to this one. you can't make such a change in a few minutes. limitations are so unusual that nobody would expect them and be prepared. and even if they were known ahead of time, you are also not expecting the kind of situation where you have to upload pictures in the middle of the night.
    - dheera 2y ago
      They are pictures. They can skip them. They can run the event without it. They can print the photos on paper and hand them out. They can hand-draw posters, like everyone did in the 1700s. They can cancel the event. They can postpone the event. This isn't a 911 dispatching system.
      - em-bee 2y ago
        if they are a business then none of these alternatives are acceptable. they could loose customers over this. and surely you are joking with cancelling or postponing an event over this. please try to be realistic. you have to consider that in our society we have come to certain expectations. these may not be ideal, but closing an online service at night is a violation of that. things would look different if everyone did that, but then i'd be the one offering night service and use it to beat the competition.
- genericperson66 2y ago
  It might be 3 am for you but 3 pm for the user
- em-bee 2y ago
  just before going to sleep is the only time i do hobby stuff. in the morning i have to work. putting something off to the next day always means putting it off to the next evening. i don't care how long things take. i care that i can do them right now and then forget about them. if i can't do that then i have to keep the tasks in my mind. i can write them down, but that only helps if i have a habit of doing that. sometimes i remember some important message i need to send colleagues or clients... for that i like the telegram feature where i can send a message with a delay. i can write it now and it will be sent the next morning or whenever i think is a good time. i wish my email client had that feature too.
Dylan16807 2y ago
> Even if TurboTax crashes on 4/15 at 11:35pm and the engineers don't fix it until the next workday, resulting in millions of people not being able to file their taxes, I'm sure the IRS might grumble a lot but would give people an extension. It'll all be good, and everyone will get to sleep . That's way too big of a risk, and way too much stress to put on your customers. For something like tax software, you should have people on call, or even 24/7 staffing, for that specific week. 2% of the year. In general, big release dates or important deadline should often have extra resources. 0-10 days per year. Pay extra for the health hazard, but that doesn't mean don't do it.
- KronisLV 2y ago
  > For something like tax software, you should have people on call, or even 24/7 staffing, for that specific week. In my country, the tax system (EDS, Electronic Declaration System) is down pretty much every single year on the day when tax declaration submissions start. 2020: "SRS: Significantly increasing EDS capacity is expensive and not cost-effective" https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid-butiska-eds-jaudas-palielinasana-ir-darga-un-neatmaksajas.a349466/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid... 2022: "SRS urges not to rush to submit annual income tax returns so as not to overload the EDS" https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid-aicina-nesteigties-ar-gada-ienakumu-deklaracijas-iesniegsanu-lai-neparslogotu-eds.a445591/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid... 2023: "The SRS urges not to rush to submit income tax returns in the first days of March" https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid-aicina-nesteigties-ar-ienakumu-deklaraciju-iesniegsanu-marta-pirmajas-dienas.a498355/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid... 2025: "A virtual queue will be open this year for submitting annual income tax returns to the SRS" https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/28.02.2025-gada-ienakumu-deklaracijas-iesniegsanai-vid-sogad-darbosies-virtuala-rinda.a589798/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/28.... So basically their "solution" for the longest time was to just tell people that it's too expensive to make it have high availability and that they shouldn't use the system on the first days of the period when you can submit the data and eventually just adding a queue in front of the system to manage the concurrent users. It seems that taxes still get handled correctly and that nobody really cares that much. Found this to be an interesting example of going against the established culture of trying to go above and beyond for availability, even if I scoffed at it a few years ago. It definitely wouldn't be horrible to live in a world where a prod outage doesn't mean "Sorry wife, I'm not coming home today, will be stuck in some random war room for hours and then fudge up the groceries massively due to sleep deprivation" but rather "Sorry boss, the system is down, what a bummer. I'll look into it tomorrow at 9 AM." for pretty much anything aside from truly critical and time sensitive systems (e.g. air traffic control, as opposed to your music streaming app).
xtreme 2y ago
If only 10% of Tax software users are affected, IRS may not give an extension and the customers would be cursing you. Probably worth it to keep those extra engineers on call during the tax season.

by dheera 2y ago

lukan 2y ago

"If a photo-sharing app is down at 3am, I'm sure the users can go to sleep and wait till 10am" There will be people, who will feel it is critical important to post some pictures at 3 am and they will get stressed, if it is not working (say people preparing an event and the pictures should be online the next day). But .. whether that is worth that engineers must be on call, is a differnt question. I never had a job like this and I know I would never accept it as default for myself.

dheera 2y ago
> people preparing an event and the pictures should be online the next day They can use a different platform, print the pictures on paper, there are a million ways to deal with these kind of issues. An earthquake could strike and the event might need to be postponed. If nobody's life or health is at risk, it is not urgent enough to sacrifice someone else's health for it.
- lukan 2y ago
  "If nobody's life or health is at risk, it is not urgent enough to sacrifice someone else's health for it." I agree. But I think being on call is possible in a healthy way, if there are long enough brakes in between. Occasionally being a night on call is possible. (Well and normal if you have small kids, but having small kids and having to worry about getting an emergency call any minute, that would be draining)
- em-bee 2y ago
  They can use a different platform not if they already committed to this one. you can't make such a change in a few minutes. limitations are so unusual that nobody would expect them and be prepared. and even if they were known ahead of time, you are also not expecting the kind of situation where you have to upload pictures in the middle of the night.
  - dheera 2y ago
    They are pictures. They can skip them. They can run the event without it. They can print the photos on paper and hand them out. They can hand-draw posters, like everyone did in the 1700s. They can cancel the event. They can postpone the event. This isn't a 911 dispatching system.
    - em-bee 2y ago
      if they are a business then none of these alternatives are acceptable. they could loose customers over this. and surely you are joking with cancelling or postponing an event over this. please try to be realistic. you have to consider that in our society we have come to certain expectations. these may not be ideal, but closing an online service at night is a violation of that. things would look different if everyone did that, but then i'd be the one offering night service and use it to beat the competition.
genericperson66 2y ago
It might be 3 am for you but 3 pm for the user
em-bee 2y ago
just before going to sleep is the only time i do hobby stuff. in the morning i have to work. putting something off to the next day always means putting it off to the next evening. i don't care how long things take. i care that i can do them right now and then forget about them. if i can't do that then i have to keep the tasks in my mind. i can write them down, but that only helps if i have a habit of doing that. sometimes i remember some important message i need to send colleagues or clients... for that i like the telegram feature where i can send a message with a delay. i can write it now and it will be sent the next morning or whenever i think is a good time. i wish my email client had that feature too.

Dylan16807 2y ago

> Even if TurboTax crashes on 4/15 at 11:35pm and the engineers don't fix it until the next workday, resulting in millions of people not being able to file their taxes, I'm sure the IRS might grumble a lot but would give people an extension. It'll all be good, and everyone will get to sleep . That's way too big of a risk, and way too much stress to put on your customers. For something like tax software, you should have people on call, or even 24/7 staffing, for that specific week. 2% of the year. In general, big release dates or important deadline should often have extra resources. 0-10 days per year. Pay extra for the health hazard, but that doesn't mean don't do it.

KronisLV 2y ago
> For something like tax software, you should have people on call, or even 24/7 staffing, for that specific week. In my country, the tax system (EDS, Electronic Declaration System) is down pretty much every single year on the day when tax declaration submissions start. 2020: "SRS: Significantly increasing EDS capacity is expensive and not cost-effective" https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid-butiska-eds-jaudas-palielinasana-ir-darga-un-neatmaksajas.a349466/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid... 2022: "SRS urges not to rush to submit annual income tax returns so as not to overload the EDS" https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid-aicina-nesteigties-ar-gada-ienakumu-deklaracijas-iesniegsanu-lai-neparslogotu-eds.a445591/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid... 2023: "The SRS urges not to rush to submit income tax returns in the first days of March" https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid-aicina-nesteigties-ar-ienakumu-deklaraciju-iesniegsanu-marta-pirmajas-dienas.a498355/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/vid... 2025: "A virtual queue will be open this year for submitting annual income tax returns to the SRS" https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/28.02.2025-gada-ienakumu-deklaracijas-iesniegsanai-vid-sogad-darbosies-virtuala-rinda.a589798/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/ekonomika/28.... So basically their "solution" for the longest time was to just tell people that it's too expensive to make it have high availability and that they shouldn't use the system on the first days of the period when you can submit the data and eventually just adding a queue in front of the system to manage the concurrent users. It seems that taxes still get handled correctly and that nobody really cares that much. Found this to be an interesting example of going against the established culture of trying to go above and beyond for availability, even if I scoffed at it a few years ago. It definitely wouldn't be horrible to live in a world where a prod outage doesn't mean "Sorry wife, I'm not coming home today, will be stuck in some random war room for hours and then fudge up the groceries massively due to sleep deprivation" but rather "Sorry boss, the system is down, what a bummer. I'll look into it tomorrow at 9 AM." for pretty much anything aside from truly critical and time sensitive systems (e.g. air traffic control, as opposed to your music streaming app).

xtreme 2y ago

If only 10% of Tax software users are affected, IRS may not give an extension and the customers would be cursing you. Probably worth it to keep those extra engineers on call during the tax season.