8 ms·
Ask HN: Who else is working/on call over Christmas?
I volunteered this year to be the on call over Christmas and currently I'm dealing with a generator fire. Merry Christmas.
Who else is working through the 25th this year?
- namuol 4y agoMerry Christmas & good luck. I was scheduled to be on call but a lifesaver of a coworker covered for me. Turned out my flights were canceled and I’ll be missing the holiday with my family after all. No active incidents, thankfully.
- ge96 4y agoMeeee (OnCall). But my team is small/I don't really have to do anything (fingers crossed). I do dislike that my phone volume is on, when normally it's off, so every notification is making noise Press 4 to acknowledge.
- OmarIsmail 4y agoYou can make an exception for the pager duty providers number. That will it will always make a sound no matter what.
- ge96 4y agoThanks for the tip
- deleted 4y ago[deleted]
- ksala_ 4y agoI'm not working but oncall. Fingers crossed. I always volunteer (I did a few years in a row) as usually it's quiet (no deployments, people understand it's a holiday) and it's a small price to pay to not feel bad for always trying to sneak out of being oncall for new year's eve :). . Best of luck OP and merry Christmas too!
- doopy1 4y ago[flagged]
- onehair 4y agoOn-call here, but it's mostly because it arrives on a weekend, all I'm required to do is Acknowledge Alerts that are usually false positives, unless some big service disruption occurs
- efficax 4y agoif the alerts are usually false positives you need new alerts!
- donutshop 4y ago^ this
- onehair 4y agoYes, but not so simple. Especially when the alerts are caused by 2 systems down the line and aren't owned by our immediate Team. That and the fact that we're migrating to a better thought out architecture which has been 2 years in the making. Slow pace at big corps..
- efficax 4y agoyou definitely shouldn’t be on call for alerts managed by another team. a lot of smells in the observability story here, i’d be pushing for an overhaul
- qixv 4y agoI’m not on call, but I’m really happy that the guys from home alarm provider are on call. We had a maltunction and they promptly reacted. Cool - and the merry evening could go on. Merry christmas.
- brianwawok 4y agoOwn a little SaaS. Gave everyone the week off between Christmas and New Years. Have been happy to do this 5/6 years we have been in business. Love giving it to my team for all the hard work they do all year. Machine wise k8s is rock solid and I never need to touch it. Client wise, will be a few support tickets…
- loopdoend 4y agoAnd hey, we don’t have to rely on anyone else to work over Xmas.
- deathgripsss 4y agoVery grateful for my clusters right now
- lucb1e 4y ago> Gave everyone the week off between Christmas and New Years. Do you mean additional to their PTO? That's nice, many people feel obliged to take it off for family even if they'd rather do something else with a whole week off and see family on weekends and such.
- brianwawok 4y agoYes, kind of a forced vacation week. I think people come back in Jan feeling less burned out… vs having to sneak some random work days in on Dec 28th or whatever.
- Kon-Peki 4y agoThat’s a nice gesture; hopefully you can keep it up! I once worked at a place that tried to always give close the office two extra days during that span. As long as Christmas wasn’t a Wednesday, they always made it such that you had an extra long weekend twice in a row, and announced it in early November so people could plan well enough in advance. They also, for planning purposes, assumed that there were only 11 months in a year. That is, each person was only expected to do one full month of work total in Nov and Dec. it gave people time to catch up, work on tech debt, learn something, experiment with something, shop, take vacation, etc. I’d say that more often than not it was overall more productive.
- dfir 4y agoon call, in case someone has a major cyber incident. so far, so quiet
- thomascgalvin 4y agoCurrently putting the finishing touches on a proposal that is, for some unfathomable reason, due the 28th of December.
- sho_hn 4y agoThis is my wife, who has to turn in a research paper on 31st. On the flip side, I get to spend Christmas helping to make sure that related academic code doesn't succumb to any stereotypes ... :-)
- m00dy 4y agoRunning a decentralised network. Some nodes not behaving well.
- impetus21 4y agoI am doing a python freelance job. It sucks.
- techie128 4y agoI am always on-call. Pages that fall through, end up paging me. Happy Christmas, New Years everybody and stay safe!
- EvanAnderson 4y agoSorry to hear about your generator fire. Power stuff is usually a major drag because most of us on the IT side can’t do anything about it. I’m on-call ops support for a public safety answering point (a “911 call-center”) this weekend. It’s a regular call rotation and my number was up for this one. It’s usually quiet— except when it isn’t. I’m gonna go on-site to a Customer’s on-prem server room tomorrow morning and swap out a bad drive in a SAN. I’m pretty happy because I know there’s no chance of a “Hey, while you’re here-” event! Empty offices mean no unexpected questions!
- bravetraveler 4y agoI usually volunteer, whether or not I actually end up working varies. Fingers crossed for this year. I'm hopeful, we've done well to improve reliability and coach the junior teams
- preommr 4y agoWorking on my own startup so putting my all into it.
- atdrummond 4y agoBest of luck with the generator. I'm on a hotline, volunteering, for people who might be feeling extra mental stress due to the holidays and might do something they regret. I decided not to risk the travel to see family with the storms and I figured I might as well try to help others who might be dealing with some of the same tough emotions that often plague me this time of year.
- csw-001 4y agoThank you! Have a Merry Christmas.
- sonofhans 4y agoGood on you :)
- hsuduebc2 4y agoThe world always need people like you. Thank you.
- lemper 4y agoyou're doing something good. here's a virtual hug.
- minsight 4y agoYou are doing a ton of good this Christmas. Thank you for your efforts here and there.
- verst 4y agoI'm on call this past week and next week! My on-call is a bit odd because it's both as OSS maintainer deciding whether I need to coordinate a hotfix release, as well as certain functionality within a managed service (PaaS) enabled by the OSS project. Only the latter thing can actually page me, and I don't anticipate being paged. Fingers crossed!
- gnicholas 4y agoI was surprised to find that my startup's contract dev kept working on the 23rd and even 24th. If he's pinging me with questions and sending me new versions to test, I'm going to answer and test stuff out! It's not mission-critical though (it's an manifest v3 transition, which Google has given a reprieve on), so we'll take a few days before getting back to it. The good thing about testing browser extensions is that you can do much of it while just surfing the web! Cheers to everyone else who are clocking in during the holidays!
- jldugger 4y agoOn call, but this is the best time of the year to be on call: none of my coworkers are around to break anything. A gut estimate of my pager load is that 80 percent of pages are caused by bad changes pushed by humans, and another 10 percent are caused by changes pushed by humans exercising latent errors in the system, and only the remaining 10 percent random shit breaking. The main downside is that I also don't have as many people to escalate / help when an outage does occur over the holidays. But that's not as urgent now that say, generator fires are Amazon's problem not mine.
- mulmen 4y ago“Generator fires are highly correlated with Christmas weather” is the best sales pitch I have heard for cloud services.
- toast0 4y agoThis time of year, you do get the pages for things that were always broken, but nobody noticed before, because they only show up when the system has been running without changes for more than two weeks.
- purrcat259 4y agoThis actually happened to us last week in fact. No deployments revealed how a legacy background processor started losing connections to the message queue and gets stuck in a state where it never reconnects. Deployments always cycled the pods before the issue manifested.
- jldugger 4y agoThis is something a (now former) colleague of mine pointed out: that the kubernetes descheduler can enforce a maximum lifetime[0] that sort of forces continual reboots. So if your system cannot tolerate running for a long time continously, this is one method to gracefully restart long running pods. [0]: https://github.com/kubernetes-sigs/descheduler#podlifetime https://github.com/kubernetes-sigs/descheduler#podlifetime
- em-bee 4y agochristmas is the busiest time for one of my customers, and just last week one of the data servers had hardware issues that ended up in everything but the disks being replaced because the hosting provider could not find which component was actually broken. i am busy rebuilding raid since, but already 2 out of 3 raid devices restored. it was touch and go for a while, with a risk of loosing the whole server (because raid kept claiming that disks were faulty when they really weren't) i am already rethinking the architecture to make it more robust and am trying to convince the customer to invest in new servers for that. i am the only one in my role, so i'll be monitoring the rebuild process over the holidays. that said, i don't celebrate christmas anyways, so in a team i'd be the first to volunteer to be on call at this time to allow others to relax, while i'd be taking time off on my own holidays.
- deleted 4y ago[deleted]
- n1b0m 4y agoJust had a pointless alarm go off that resolved after a few minutes. According to the Runbook there isn’t even any action that can be taken!
- RobRivera 4y agononactionable alert? thats a paddlin
- navanchauhan 4y agoI am a volunteer at a research lab right now in another state over my winter break. Had the opportunity to take a break, but I don't know anyone here and won't be here for a long time so thought what the heck
- esfandia 4y agoI am marking exams. My deadline to enter the final grades is December 27th.
- Utkarsh_Mood 4y agoI'm sure you'll get through this!
- bcrosby95 4y agoI'm on call 24/7. One of the downsides of a small company. I go months between needing to do anything though.
- nepthar 4y agoDeadline confusion :)
- thirteenfingers 4y agoMe! First Christmas at my new (fully remote) job on an SRE team and I'm on call through most of this coming week. Not complaining though, I actually asked for this stretch, due to awkward timing with other commitments during other weeks. Fortunately there are other members of the team I can reach if I need them, and family live nearby. I'm just thankful to be gainfully employed. It's pretty quiet at the moment and I'm actually kind of jealous that I'm not facing the kind of excitement you are... I guess I should be careful what I wish for lol. Merry Christmas all!
- piggybox 4y agoI'm on call during the Christmas week 2nd time in a row in my team. Nothing earth shattering so far. There's one issue that could be resolved earlier (by the previous oncall) so it wouldn't cause more delays in other parts... Merry Christmas to all those are on call now.
- geocrasher 4y agoI'm on call every day, but so is the CEO. The startup life :) I am not working this weekend, but I am the next Sunday (the 1st) so that one of my crew can have the day off. I don't celebrate the holidays, so I go out of my way to accommodate those who do, since my schedule is wide open. It's only right, and my crew really appreciates it.
- jejeyyy77 4y agoSerious question - do you have stake in the company the same way your CEO does?
- MPSimmons 4y agoI'm on call. I volunteered. I wasn't scheduled, but the guy who was scheduled, it is his first year with the company, and this is his first job out of college. He was visiting home, and I wanted to make sure he got a chance to go back and have a good time with his family without having to worry about on-call junk.
- x86hacker1010 4y agoYou know, when people go out on a limb for me to do things like that, I never forget them. Cheers
- weakfortress 4y agoIt sounds like your on-call should be structured so that people aren't allowed to be with their family on important holidays. My company does this as well. I find it distasteful and exploitative. Of course, the response is always "just find someone to cover!" You're a good person for doing that. You shouldn't have to. Your family wont be around forever and no board of directors is worth a single minute with your family.
- wyclif 4y agoIt's a lot easier to cover on call if your company is fully remote, or at least the on call staff rotation is remote. Employees tend to resent on call a lot less if they're home with their families.
- jejeyyy77 4y agoThis is so true, and well said. I lost a few family members this year and it made me realize all the bs baggage that comes with employment. Needless to say, I don’t do on call anymore.
- mr90210 4y agoYou are a cool guy!
- ranger207 4y agoI volunteered to be on call while off work both the past week and the next week, since I know it's going to be so quiet. 8 pages over the past week, only one of which was after hours and not caused by people messing with things
- saboot 4y agoFinishing my PhD ... Have a job lined up for February and I've got so much writing left
- vopoylovslov 4y agoWeird flex but ok
- hrrsn 4y agoI’m on call this week (usual rotation, just happened to coincide). Full change blackout means less humans breaking things, and I get to save my annual leave during the mandated company shutdown + get alternate PTO for the public holidays. So far the phone has only rang twice all week. Merry Christmas!
- OmarIsmail 4y agoThis is me. And fingers crossed it’ll be an uneventful few days for us both :)
- anony553923 4y agoWe're a remote company so we have employees that celebrate different holidays. For example, our Chinese employees cover Christmas and we cover them during the Lunar New Year.
- messutied 4y agoI’m on call too. As a solo founder of a small SaaS. I’m curious if there’s a community like a Slack channel for sharing this kind of experiences?
- ungawatkt 4y agoI'm on call for one day, since my team split the holidays to ~1 day per person rather than give someone the whole rotation. Not a bad arrangement imo.
- eveningsteps 4y agoVolunteered to be on call -- here's to hope I won't have to press 1 to ack, 3 to nack, 7 to STFU.
- amerine 4y agoI love carrying the pager during holidays. Nobody breaks anything ;-)
- froderick 4y agoIn my startup experience, the pager never really ends. I’ve got one eye on my alerts inbox, and one eye on John McLane complaining about California. Still, I would rather be doing what I want, the way I want it rather than spending my days paying lip service to who ever might be within earshot. To those that choose the hard path rather than the easy lucrative one, I salute you.
- Gaessaki 4y agoManaging the dispatch of our delivery drivers and monitoring our infra over the holidays. Wish there was some way to bond with others that are on-call while being on-call. It’s lonely with the rest of the company being away. I guess this thread is a close as we get. Best of luck to you and anyone else with generator fires and Happy Holidays to all!
- Hamuko 4y agoI'm never and always on all, since I don't actually have official on-call duty, but if our system breaks, I'm pretty high up on the list of numbers people will start to call for help. Luckily thus far the worst was a 06:00 wake up on one Saturday morning.
- seti0Cha 4y agoOn call, but not too worried because most of our breakages are usage based and not much usage on Christmas. I've been paged on Christmas morning before though, so I know how that feels.
- spike021 4y agoTechnically not because my boss insisted since I started in September I can wait until January. But my teammate left me as a shadow secondary (for learning purposes mostly for daytime pages), so if my team gets paged for anything I'll also get the page. I don't entirely mind at least to be aware of things going on. Fortunately our services are generally fairly quiet compared to my previous jobs'. Hope you all have a quiet few days!
- donohoe 4y agoI am, but from experience it will be uneventful. We didn’t deploy any significant code changes all last week. The few we did do are all minor and very very low risk.
- ghstcode 4y agoI’m on call for a payment provider and have dealt with emergencies basically since the very beginning
- Dopameaner 4y agoYup. on-call, and working through the holidays. Thanks to the frustrating miscalculations of corporate upper-management. And, since I am on h1b as well, cant really quit my job either without changing my personal life drastically. Need to spend another 3 months of leetcoding, and system design to "prep" for another set of interviews. Not looking forward for the 12 hr coding days.
- thayne 4y agoI'm not this year, but I have been on call during Christmas and other holidays before. It is usually pretty quiet, both because there aren't a lot of changes with less people working, and at least for me, there are less customers using the product. But it can still be stressful knowing your celebrations or time with family can be interrupted by an alert. And sometimes you can get unusual alerts, like alerts caused by unusually low traffic, or a bug that only impacts one user, and that user happens to be the only one using a service at midnight on Christmas eve (is that you santa?).
- NaOH 4y agoI own a one-person wholesale bakery. My boss is a jerk. Best to everyone.
- dreamcompiler 4y agoFirefighter/EMT here. Always on call when I'm home. Hoping we don't get a car crash tonight because it's damn cold out there. Turnout gear helps but not enough.
- skrecik 4y agoGood luck, I hope it calmed down a bit OP. Merry Christmas from fellow on-call to you, HN
- VincentLago 4y agoYou are incredible. This is a great example of how to be a decent human. Thank you for sharing this and for creating a place for people to talk about it. I volunteered today to identify, move and secure the community boats on the harbor used by elderly, teens and children that were at risk from floating away with the California King tide rising. My family is a few states away and I couldn’t afford to go. I thought I would feel better just doing some things for other people. I help out 3 women in their 90’s with their technology, will stop by with food for all 3 tomorrow. I want to make the best of it. Thank you for setting a great example. You are an angel.
- deleted 4y ago[deleted]
- bradlys 4y agoDid it last year. I was divorced and had no reason to go anywhere during the holidays. It was my first time ever staying in the Bay Area during Christmas. This year - I don’t have a job because I quit and went traveling and moved to Manhattan. Guess my time spending home at work in the Bay Area wasn’t great. Only thing I’m doing today is going to the gym and taking a yoga class and doing legs if I can get some time at the squat rack. Surprisingly busy yesterday (like busier than I’ve seen it almost any other time!) - we’ll see if busy today too.
- virtuallynathan 4y agoI'm on call for the Netflix CDN, but luckily no one touches anything, so the number of incidents is super low.
- shamiln 4y agoI’m on call, been paged 3 times already for a high CPU on some VMs
- tkiolp4 4y agoI don’t understand the ones that are volunteering being oncall. I don’t think you are making a favor to your colleagues but to the company you work for, and that usually means: the company is being cheap regarding the on call topic. I have zero empathy for companies not handling on call the right way.
- em-bee 4y agowhat is the right way?
- jejeyyy77 4y agoIf a company needs 24/7 coverage they should pay to have that coverage.
- em-bee 4y agowho says they aren't paying? you still need people to volunteer working during holidays
- jejeyyy77 4y agoif they are paying, it's not volunteering. Keep increasing the $ - you'll eventually hit a tipping point where people will fight over it ;)
- em-bee 4y agoit's volunteering to give up you legal holidays for work. in some countries, working on legal holidays automatically qualifies for higher pay, but that aside, i think it is fair to ask for volunteers first in case there are people who don't have family to see over the holidays or are not even celebrating. in fact i get pretty annoyed to not get to work on days that i don't celebrate. i'd rather have different days off that are more meaningful to me. of course, if there are no volunteers of this kind, then other incentives are a good idea, as is a rotation maybe
- IYasha 4y agoI'm sysadmining because there's noone else who can.
- muzani 4y agoOn call this week. I don't celebrate Christmas, so it's become tradition to work end of year while other colleagues take over during Eid.