5 ms·
The on call stuff sounds outright dystopian to me, it's not even legal in my country. More places should think about adopting something akin to France's El Khom
by Barrin92 6y ago
The on call stuff sounds outright dystopian to me, it's not even legal in my country. More places should think about adopting something akin to France's El Khomri law's (https://en.wikipedia.org/wiki/Right_to_disconnect https://en.wikipedia.org/wiki/Right_to_disconnect) and ensure that every citizen has enough time to recuperate from work and isn't pestered by their employer 24/7.
- filoleg 6y agoHow are you proposing to run a complex system that essentially powers a big chunk of the entire internet and can go down any second without on-call? If a big selling point of your system is the ability to be on 24/7, on-call is inevitable. But it definitely can be way less gruesome than it currently is at Amazon for a lot of teams right now. For example, my on-call schedule is extremely relaxed and low-stress, it is never a worry I actually have or something I dread ahead of time. But getting rid of it completely is simply impossible for a lot of "always online" services.
- Barrin92 6y ago>How are you proposing to run a complex system that essentially powers a big chunk of the entire internet and can go down any second without on-call? hire more people?
- filoleg 6y agoHire more people to do what? Be on-call? Hiring more people is a great idea for making on-call less stressful, because it allows for much sparser on-call shifts (e.g., one week every 8 months vs. one week every 2 months). But I don't see how hiring more people gets rid of on-call completely. Someone will always have to be on-call. You cannot just hire people who do on-call-only duties and nothing else, you have to be very closely familiar with the code and features being written in order to resolve issues effectively. Which usually means that you need to be someone who actually worked on those things, not just some glorified extra-fancy customer-support worker.
- grumple 6y agoIf the concern is about performance during business hours in China, you should have employees in China. Or hire a night shift.
- smacktoward 6y agoAccording to TFA, programmers at Amazon are required to be on call for a full week at a time. More programmers could mean sparser on call shifts, but it could also mean shorter on call shifts -- i.e. instead of going from one week every two months to one every eight months, go to two or three days every two months. Being effectively at work 24 hours a day for seven days straight sure sounds like a recipe for burnout to me. As TFA notes, these people aren't doctors, the only thing that happens if a glitch takes two hours to fix rather than one is that Amazon loses some money. Amazon naturally would prefer not to lose that money, but its workers presumably would prefer to be able to sleep too.
- devdas 6y agoOn-call will still rotate amongst the developers. They wrote the system, they know how it breaks and can fix it.
- dudul 6y agoI don't understand what's so dystopian about it. Some systems are critical and need to be kept online at all time, or at least have very minimal downtime. The description in the post is not even that bad. A page a week is nothing. And it is clearly state that your workload is reduced during your on call period.
- nighthawk454 6y agoThe dystopianism comes in because the vast majority of teams are decidedly _not_ 27/4 critical. Sure, some teams really run the 1-Click Buy button or the S3 API. Most run some DB behind some API behind another 3 APIs behind 5 other teams that run a widget. In practice, your workload may not be reduced during on call because it's "expected" and you still have to meet your normal deliverables while the team isn't staffed with "extra" people. It also takes a toll on employees to be mentally-available at all times in case some triviality emerges and raises an alarm. You'll find many engineers not willing to leave a small area or even go out for dinner without bringing their laptop and a hotspot, just in case.
- dudul 6y agoYour "in practice" doesn't match my on call experience over the past 15 years. During "on call" workload is reduced (the author of the post even mentions it). There are ways to implement sane on call policies that benefit everyone. A few I've seen: - use a primary and secondary on call so that the primary can still go get groceries, or play in the yard with their kids, etc - overlap on call and work hours. I did that at a job and I was fine with working 6pm - 2am just to cover this time slot. Won't work for everyone, but may be a nice compromise for some. Like everything else, there are ways to do it right, and way to go complete bananas with it.
- nighthawk454 6y ago> Like everything else, there are ways to do it right, and way to go complete bananas with it. Sure, agreed. Sometimes it's implemented well and sometimes it isn't. I've experienced some of both. When it isn't well-implemented and is particularly non-critical, it can feel a bit dystopian to the engineers. If done well and for a good cause, it should be fine.
- un_montagnard 6y agoMy friend (not working at Amazon) was oncall in France in 2019.
- jakebellacera 6y agoWhat was stated seems pretty typical on small teams who manage critical systems. How are critical systems managed otherwise without having to 3x your team (assuming you want 24hr coverage with a 8hr workday)?
- Barrin92 6y ago>How are critical systems managed otherwise without having to 3x your team (assuming you want 24hr coverage with a 8hr workday)? They typically aren't, you need to hire more people then. In anything defined as critical I would suggest that being well rested is also a requirement rather than having a team of people who are so overworked they develop mental illness.
- dudul 6y agoDoesn't make sense financially. You're gonna triple the size of your team, just to make sure that this one time of the week someone will restart the DB? I would much rather bite the bullet, be on call a week a month or so and use the budget for something else.
- Barrin92 6y agoThis is a law to protect the mental and physical well being of citizens and workers, obviously not Amazon's bottom line, that's literally the point. If you treat people like disposable commodities then yes this does not make any sense.
- dudul 6y agoYou know what impact the mental and physical well being of workers? Have their company go under because they had to triple the size of the team just to implement an on call policy. And I'm not talking about Amazon here. Of course Amazon can afford to hire everybody and their mom to cover 24 hours in a day. I'm under the impression that your original comment was really a critic of the practice in general, not just this specific example. Some small companies build products they try to sell to bigger companies. These bigger companies have SLAs, so they need SLAs from their providers to sign a contract. Even at a small startup on call may be necessary to get customers.
- jetti 6y agoWhere I'm at we have an on-call schedule like the one mentioned in the post. It isn't too bad, and we rarely ever get paged. Most of the issues that the on-call person has to deal with, at least on my team, will come up during the normal working hours.