29 ms·
Tell HN: I salute everyone on call/working support through the holidays
Thank you for keeping systems available and safe. I've been there many times in the past, including having to fly at the last minute to a non-internet-connected data center in NJ to babysit an emergency production bug fix that took the entire holiday to create, install, verify, and monitor.
- devilsAdv0cate 3y ago[dead]
- fuzztester 3y agoRelated: System Administrator Appreciation Day https://www.google.com/search?q=system+administrator+appreciation+day https://www.google.com/search?q=system+administrator+appreci...
- hk-senokr 3y agohttps://news.ycombinator.com/best https://news.ycombinator.com/best
- elmerfud 3y ago[flagged]
- maerF0x0 3y agoYes, absolutely thanks to all who keep our world running when no one is looking. To keep the yule log on Youtube, to keep our christmas tree lights on, to keep a fresh glass of water from the tap, warm natural gas to keep the freezing cold outside etc. Thank you for keeping society ticking away :)
- akomtu 3y agoLet's not confuse on-call firefighters or a water facility staff with the on-call admins that maintain money-making machines monetizing attention of billions. The latter is a net negative on society.
- kridsdale1 3y agoNot a fan of the YouTube Yule Log, I see.
- krallja 3y agoYeah, how dare Netflix provide entertainment on-demand and for cheaper than the other entertainment companies?
- op00to 3y agoI am currently viewing this on ethically sourced rfc1149 (birds gave consent via a scientifically proven “brain electrode interface”), manually decoding packets using an abacus made out of various animal droppings foraged on the forest floor. If I can’t view your content this way, it should not be on the internet.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- oceanplexian 3y agoThat firefighter is probably using YouTube or scrolling through Instagram to unwind while they’re stuck at the station waiting for a call. Just because someone works in entertainment or ads doesn’t mean that the economic puzzle piece they represent isn’t valuable to society.
- kridsdale1 3y agoI’ll give a shout out too to everyone in the military monitoring warning systems and maintaining stance to protect us from being killed while we’re with our families.
- deleted 3y ago[deleted]
- ponector 3y agoHundreds of thousands are fighting in the trenches right now to protect their homes, to protect their families, to protect democracy while politicians went on holiday break without approval of military aid for allied army.
- deleted 3y ago[deleted]
- 15457345234 3y agoYou ever read something and it's just like... yeah, I'm absolutely done with interacting with this userbase forever? Yeah
- maerF0x0 3y agoI'm curious why you say that? I'm taking it as my comment makes you not want to interact with HN?
- 15457345234 3y agoIt was more what I perceived as a weirdly gleeful insincerity in your comment combined with a _lot_ of similarly weirdly gleeful insincerity in the big comment thread about the SMTP smuggling attack, which had a lot of 'GRRRREEEEAATT WORK BY THESE GUYS' type of comments Just too much discordanism seemed to surface all at once and it was exhausting. I know this is 'hacker' news but I always perceived that to mean 'hacker' as in the hacker ethos, not the pejorative 'yeah break shit lol' type of hacker. Sometimes it seems to be all the latter.
- maerF0x0 3y agoas someone who's been part of the community for years, I'd say your perceived version of Hacker is the correct usage here.
- sleazebreeze 3y agoMy wakeup alarm this morning was 9am when OpsGenie let me know I'm on-call today. Praying for peace.
- frakt0x90 3y agoIn a similar vein, I'm grateful for the people who maintain the foundational pieces of our digital world that often go unnoticed like date & time systems.
- isoprophlex 3y agoBig up the on call heroes! Hope you're getting paid well, hope you get no red lights on the bug hotlines.
- chasd00 3y ago> Thank you for keeping systems available and safe. theres that word "safe" again. What systems are dangerous otherwise? Do you mean like traffic lights or something? The API serving ads to your mobile game isn't dangerous.
- dotnet00 3y ago'Safe' in the context of systems can mean hacking attempts, safe from data leaks and other emergencies relative to the system that may arise. It can refer to things that are dangerous for the system itself.
- tecleandor 3y agoOn call till 31st, so please don't hit refresh too much this days ;-)
- Scoundreller 3y agoMe too, but they pay a few bucks an hour to carry the phone so at least that adds up. Ultimate trick is to have a diverse team. Someone that doesn’t care about Christmas but absolutely needs some random day off in March (cool with us!). Someone that celebrates new years some other time.
- kridsdale1 3y agoSpeaking of diversity, if you don’t do Christmas dinner I strongly recommend ordering takeout from a Chinese place on the 25th! There’s lots of happy photos of Chinese chefs and Jewish customers doing a Christmas fist-bump.
- rollcat 3y agoTwo live video productions, including one on the evening of 31st. I managed to push back on last minute infra/workflow changes (:
- mkhnews 3y agoThanks, been there many seasons, and same to you all.
- datpuz 3y agoFor some of us, we look forward to the peace and quiet
- tetha 3y agoYeah, we discourage production changes starting first or second december week, and start freezing changes third december week until it's frozen solid fourth december week until second week of january. December tends to be hell for our customers, so stability should be a priority there. And honestly, no one wants to work on holidays. So lets just wrap everything starting in december, maybe use the third week for some unnoticed issues and then just lay down the tools. Use that time for documentation, or shorter days, quite frankly. That way we minimize the on-call situations occuring. Let's hope it goes well for the engineer this year as well. We have a streak to keep.
- ainiriand 3y agoWe do the same, I work in logistics software and we usually freeze early November up until Christmas.
- JCharante 3y agoE-commerce. Code freeze two months before Christmas. A very chill time indeed.
- bertil 3y agoI think that’s a great policy as it’s clearly intended to help people when they need it, and get people to unplug when it’s valued by their loved ones. _However_ (that part is probably best bookmarked until Jan 2nd), it also betrays that your system is brittle and can be broken by a bad commit. Don’t do it because you want people to grind until Dec 24th at 6 pm. Do it because it’s great the rest of the year, too. I’d recommend you look into (or ask me about) feature flags, alerting, and automated roll-backs. The short version is: there’s a meta-system on top of your release process that can tell (if you are using roll-back not features flags): - commits until xyzsdf are fine; - roll-outs starting from commit abcdef have a 2% error rate, 80% on Android; - revert to xyzsdf, send a message (low-priority, email) to the DevOps on call and the author of abcdef that it happened; - for all commits after abcdef: if there no conflicts with xyzsdf, re-try to roll them out; - if there is a conflict because they were on top or abcdef, send a message (low-priority email) to the authors that there is a conflict. There are more sophisticated versions that can do things like, if you use feature flags, flagging Android users to use the previous version. Another way to do this is to scale who has access to abcdef gradually: say 1% every hour, and revert if you detect issues. All those seem daunting to teams that haven’t worked like this before, but it my experience, they love it very fast.
- lynx23 3y agoI announced a downtime for a smallish GPU Cluster starting from christmas eve just a few hours ago. It is just the perfect time to schedule a day or two of downtime for a system like that. And if IPMI doesn't fail me, I can get a lot of things done without leaving the comfort of my home. I scheduled this without pressure from my boss. It was a totally voluntary decision... While being raised as a Christian, this time of the year is for me more about solstice then about the Christian clelbration. A time to enjoy the comfort of a heated home. A time to celebrate that the days are going to be longer from now on again. A time to reflect on the past year. And all of this is easily done while having a few terminals open and waiting for remote stuff to complete...
- wavemode 3y agoJust barely started my current job too recently to be in the on-call rotation yet. Lucked out! Props to those keeping the wheels turning.
- RationPhantoms 3y agoAs if this week had attempted to take a measure of blood from my body, I'll be on-call next week. Looking forward to all things quiet on the HEP network front.
- 6stringmerc 3y agoAgreed, there are some gigs that just really require support to exist - I know this first-hand from working at a Zoo (very large exotic animal rescue basically). Animals do not take holidays. They need to eat and do animal things in spite of our costumes that day. On the flip side, having worked Cinema on Christmas Day two years I think, there is no amount of Grace and Patience I can give that is enough to those earning their living. Still have a hat and polo. Why? I had to buy them!
- blast 3y agohttps://www.youtube.com/watch?v=zB1T3zgne5Y https://www.youtube.com/watch?v=zB1T3zgne5Y
- bertil 3y agoAlways be kind, and say it’s your fault. If you don’t do it for the sake of the person you are asking for help, do it because it works better. That’s the most practical advice [0] ever given by Hans Rosling [1], the Fact master himself: > In fact, I have the secret to how to get the best help immediately from any customer service, like the phone company or the bank or anything. I have the best line, it always works. You want to know what it is? When I call, I say, “Hello. I am Hans Rosling and I have made a mistake.” People immediately want to help you when you put it this way. You get much more when you don’t offend people. [0]: Unless you are in charge of a developing country’s budget and have to decide between education and healthcare. [1]: https://blog.ted.com/qa_with_hans_ro_1/ https://blog.ted.com/qa_with_hans_ro_1/
- chubs 3y agoI like to append ", unless I'm missing something?" in a similar manner, might be useful in situations where 'made a mistake' doesn't actually make sense :)
- fuzztester 3y agothat dale carnegie book
- chunkymilk 3y ago> Always be kind, and say it’s your fault. I do this with internal teams at work. I've found approaching other teams with issues with their library/framework in a "this could be our mistake" manner really helps in keeping them from getting defensive and stonewalling.
- steve_adams_86 3y agoI do something similar. Hey, I'm pretty sure I'm doing something wrong. Can you help me figure it out? Then be grateful for the help, because it truly isn't granted or a given that people have to drop everything and figure things out for you, even if you work together. And even if the mistake was actually theirs. Gratitude is huge.
- wintogreen74 3y ago
- whalesalad 3y agoMeanwhile a huge number of us (non-religious? introverted kernel compiling cave dwellers?) treat this period no differently than any other week in the year. I'll be here keepin the servers runnin :horns: It's actually my favorite time of the year. Everyone is gone, it is quiet, and I can get shit done.
- kaashif 3y ago> non-religious Or a member of one of the religions that don't celebrate Christmas.
- muzani 3y agoIt's me. But we still have a holiday period at the end of year - normally financial targets are hit and it's a 4 day leave to get 10 days off.
- deleted 3y ago[deleted]
- sneak 3y agoHolidays are special because they’re special, both the winter solstice festival (rebranded for christianity) and the spring equinox one (same deal) can be treated differently for cultural variety by the non-observant. I’m a militant proselytizing atheist raised by a jew and I still have a tree with pretty lights, give presents, and drink and eat some things I only drink/eat once per year (never make homemade eggnog if you ever want to enjoy it guilt free again, you’re basically drinking a megacalorie of heavy cream, yum). It’s fun to celebrate the generic concept of “holiday” - a time that is different from other times. You’re allowed to feel nice about peppermint candy (and/or chocolate gelt, I go for both) at the end of December without bringing the supernatural into the equation. :) \m/
- whalesalad 3y agoOh ya same I love the smell of evergreen wreaths and trees and enjoy partaking in festive activities. A 4K cracklin’ Yule log goes a long way too.
- iddan 3y agoFYI Israelis are not on holiday - our holidays are on whole different dates. Hire Israelis and experience no down time while working with Silicon Valley level talent
- liorsbg 3y agoTrue story
- loloquwowndueo 3y agoJust hope stuff doesn’t break on sabbath, they ain’t touching no computer that day :)
- INTPenis 3y agoThey'd need a goy on-call to flick the switch.
- iddan 3y agoAs I wrote in another comment, most Israelis in tech are not religious and take on call duties during Saturdays and religious holidays
- talldatethrow 3y agoThen why was it a common response after the recent attack that the poor military and police response was due to it being a holiday?
- iddan 3y agoMost Israelis in tech are not religious and take on call duties on Saturday (Sabbath) just like Christians take on call duties on Sundays.
- loloquwowndueo 3y agoI see they also have no sense of humour.
- cco 3y agoI'm not sure if there is something in the water this year, but this week, Dec 18th to Dec 21st (only a partial week), has been our busiest week all time already. Sweating over here trying to make it through the week and praying that it slows at least for the first half of next week.
- yardstick 3y agoJust remember this time of year is often peak vulnerability time. When attackers exploit that teams are at reduced strength and off guard. Slower response times to investigate and fix issues etc.
- timwaagh 3y agoI wouldn't mind honestly. Seems like a good excuse to skip the social obligations.
- smoyer 3y agoI'm on call 12 hours a day and hoping things are very quiet next week. Best wishes to everyone else too!
- acedTrex 3y agoWe've been on freeze for weeks now in preparation for the holiday season.
- er0k 3y agoThanks for the salute, but we also accept cash :)
- mateusfreira 3y agoThanks for all the great work; I hope no one has an outage this holiday and has time to enjoy family and alone. Keep up the good work, folks
- comprev 3y agoI salute those in the startup world - the ones in a team of 5 and they're the only Ops person who always gets paged. Been there, done that.
- itqwertz 3y agoHolidays are excellent times for hackers to take advantage. It’s not just Christmas or other Western holidays, either. Extend this principle to any holiday/world conflict/anniversary of conflict made into holiday/calendar new year and then adjust your time of attack. protip: US companies with offshore groups are usually underfunded, understaffed, and underskilled. Time to see if that disaster recovery environment works! Happy holidays to those who encounter system stress tests. Can’t spell salary without some elements of slavery…
- dallas 3y agoManagers, if you're reading this, and you have engineers/developers "on call" but not contractually (off book), make it so, because it slightly sucks when you're having Christmas drinks but can't enjoy yourself because you might need to drive somewhere and climb up a ladder to tend to a product.
- throwawaaarrgh 3y agoIf somebody asked me to be on call, during a holiday, without overtime, there's no way I'm driving anywhere. If it's worth it to the business, pay me for the time I'm no longer getting to enjoy with my family, or shove it.
- dallas 3y agoYes, would be nice to not need to mentally prepare to make that call.
- ponector 3y agoIs it a US thing to push updates right before holidays and force people to be on call? European companies I've been working on are planing releases till the end of November, first week of December max. All project plans are baked like there is one week in December. While US company tried to force every contractors to work everyday and some weekends too. Are you Indian? You don't celebrate Christmas, you'll be on call. Are you Eastern Slavic? You celebrate Orthodox Christmas in January. You'll be on call.
- JoshTriplett 3y agoNot all breakage happens because of new updates. Sometimes the servers go down, sometimes a bug that already existed gets triggered by something, sometimes there's extra volume over the holidays and the load is too much. Full agreement that you shouldn't push new code in late December, but that won't stop the need for some folks to be on call.
- deleted 3y ago[deleted]
- hatthew 3y agoWe discourage changes starting early/mid november and completely freeze deploys a week before thanksgiving and christmas, but there's still so much that can go wrong. The other commenter summarized it well, but in addition there's also systems that require continuous automatic updates as part of their functionality, and sometimes those automatic updates can start to go haywire. For some systems, the holidays are the most important part of the year, and it's extra crucial for people to be monitoring and ready to respond at a moment's notice.
- erik_seaberg 3y agoThere are also systems that aren’t entirely stable due to resource leaks, and when you stop deploying you might find you need to restart them occasionally.
- crossroadsguy 3y ago
- iancmceachern 3y agoMy uncle used to work for the City. Every holiday season, if it snowed, he would be called away to plow the roads. Here's to those out there plowing the roads so we can get there safe!!
- HankB99 3y agoMy D-I-L is a nurse and will be working the coming weekend. That's her "price" for getting Thanksgiving (in the US) off. We'll schedule as much as possible around her work hours and make sure there's food left for her when she gets home. Many thanks to all of the health care workers who take care of us over the holidays. (Along with all of the others, of course.)
- throwawaaarrgh 3y agoAnd for all those of you trying to hold the world on your shoulders: don't be a hero. If you don't let things fail (that aren't your responsibility), nobody will notice it's at risk of failing, and thus will keep letting you hold it up by yourself.
- TrackerFF 3y agoI worked four years in the military, and three of those I had evening and night shifts during the holidays. Absolutely nothing happened, no activity whatsoever - just babysitting systems deep inside a bunker. Closest I got to new year's eve was watching the fireworks through CCTV. The last year I was on call, which was miles better, but those years definitely cemented my will to get a job with normal work hours.
- Octabrain 3y agoOn call sucks so badly. At this point of my life, I firmly believe that there's not enough amount of money that can compensate the mental suffering it implies. Even more if the company you work for has this mentality of "deal with it" without making improvements, which was my case in the last period I did on call and what made the camel's back to break for me. Nowadays I simply refuse it. For those who are still on the trenches, stay strong, never resígnate yourself to just "deal with it" and thank you.
- hatthew 3y agoTo me, the worst part of being on call is the stress _after_ my shift ends. I understand that it's a necessary part of the job to fix issues that occur during my shift, so I don't really mind it, but it gives me long term issues. I feel anxious whenever I don't have my phone on me, or when I'm far enough into the wilderness to lose my cell signal. Late night when I don't expect to be getting messages from anyone, a random notification can sometimes give me an immediate stomach-drop panic response. Unfortunately I feel like I lucked into this role and if I left I wouldn't be able to find anything anywhere near as good.
- crossroadsguy 3y agoAnd I am not even sure whether you are talking about just day-time on-call or the 24 hours on-call for at least 1 full week to two week stretches or a simple 12 hours on-call you are talking about? In India the Indian managers (and American managers are just fine with it) have made an environment of this barbaric practice of 24x7 on-call handled by just one person. In fact, even when there are US/western counterparts these subhumans projects that they will make sure Indian engineers are on-call even during American daytime. This has been happening at my workplace. They employ all tactics - from fear, intimidation, to try to sweat talk engineers into it with shit like, "Oh, we own it, right? So it's our responsibly to support even when it's night". With that environment it becomes extremely difficult and a pressurised situation for someone like me who simply refuse to even sign up on something like PagerDuty and make it clear that my phone remains silenced and out of my bedroom between 10pm-7am and it really does. I agree with you - there is no amount of money that can put on on-call, definitely not on a night shift on-call.
- enginaar 3y agoparent company got hit by cyberattack yesterday. salute to all IT and InfoSec colleagues working around the clock while everyone else is taking the day off.
- krony 3y agoThank you indeed! While I have fond memories of working holiday pager support early in my career, especially before marriage and kids to cover for those with families, I’m very grateful for those able to cover for all of us now! Cheers to you all
- WesternWind 3y agoThanks. Honestly, all I do is check emails once or twice a day and maybe respond to a DM if there's an emergency, but it's nice to be appreciated.
- mynameisnoone 3y agoFirefighting sucks. I salute all of the ways people ensure that on-call runbooks are thoroughly documented, not shipping code on Fridays, and robust, simplicity-based engineering that reduces problems during and especially afterhours.
- crossroadsguy 3y agoThank you but I wish you were not doing it during your night time. Let someone in their timezone for whom it's daytime do it. Insist on it, refuse being exploited. Take this opportunity to tell your company to hire the time timezones where it's day during your night. If it's not important enough for them then they anyway don't need to you remain awake at night. Don't harm yourself. It's not just losing sleep, that leads to harmful effects in your body that keep showing for long and some damages might not even be reversible. It's not bravery, it's not being a hero as many are putting in comments - it's just plain exploitation if you are awake at night doing this. Your health and wellbeing is not worth any money and definitely not worth someone's shopping cart and checkout page working smoothy over the holidays when your Western colleagues just took off en masse. Not even close. I know a lot of you would be from a third world country like me (yes, that's a thing!). Stay strong and work for this exploitation to end.
- gumby 3y agoMy mum always worked on most of the public holidays because we never really cared about them but it did mean everyone owed her favors (she was a physician and would be on call, at the hospital etc) so she could get someone to cover for when whenever she wanted. Growing up ignoring holidays is mostly great (fly on xmas and everybody feels sorry for you, even though they are the ones working on xmas). But it causes relationship problems bc even when you genuinely try to participate you’re “doing it wrong”.
- starbie 3y agoUsually I choose to work during normal holiday season too, and then go on vacation in mid January when crowds are smaller. Having a family that accepts rescheduling Holidays helps. We've celebrated Thanksgiving, New Year and Christmas on different days before.
- cebert 3y agoI get nervous every New Year’s Eve due to date/time issues. I work on emergency 911 software. In our system each time a 911 Call is created, we create an incident number in the format YYYY-NNNNNNNNNNNN where N is an incrementing number. I was oncall a few years ago when a date time bug was introduced that resulting in numbers being created prematurely by a few hours. As each hour passed more customers in a different time zone called in to report the issue. I was the only person working and was getting hammered with cases. It sounds like an easy isssue to correct, but downstream systems that consume those numbers had already processed them and associated reports and other records with the incidents. I spent the next few months sorting out that mess and helping work with partners to clear out data.
- throwaway2037 3y agonon-internet-connected data center Woah. That sounds wild. If you are allowed to share: What industry / company? I would like to hear more about that setup!
- aorloff 3y agoThe lizard people do not fuck around. And they don't care about your stupid holidays either.
- thatfunkymunki 3y agoIt tends to be a tradition for me to be oncall over at least one of the winter holidays, this year I (again) get to preside over a vendor vs vendor showdown where the only loser is our hardware!
- nemo44x 3y agoIf you’re atheist or of a religion that isn’t relevant then why do you care? There’s no spiritual meaning. It’s just a day off. Get off the fence and own your convictions.
- auntienomen 3y agoWinter holiday celebrations & festivals predate the major world religions. Don't presume there's no meaning for people who don't follow a particular religion.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- BramLovesYams 3y agoHoliday workers are the true heroes.
- deleted 3y ago[deleted]
- cfr2023 3y agoSome people just wake up in the morning wanting to fight. If you're providing support, you might just be the first person they find on a given day. It's crazy to want to fight someone who is trying to help you, but lots of people end up in crazed states when hammering away at projects behind their computer terminals. Taking a step back even further, I think that requiring any form of software support or correspondence with software makers is a completely alien concept. I have done it under 5 times in more than 30 years of using computer software and writing documentation. No doubt that countless people encounter show stopping software issues for which they cannot find alternative tools, workarounds, or workflows, but I find that it's almost always better to just go shopping at these junctures, rather than enter into some kind of chain of correspondence hoping that someone out there in the ether might possibly one day fix your problem.
- doingtheiroming 3y agoI remember playing Unreal for 3 days solid while working an IT service desk in the late '90s. Compaq Deskpro in software mode later upgraded to a Matrox G200 which upended my world. Working Christmas and New Year was a bonus as far as I was concerned. Dodged the family drama and got in plenty of gaming. Also tidied the office...