4 ms·

> I don't know anything about process-payments; it belongs to a completely different team. I wouldn't even know how to check it. I get that the point of the ar

> I don't know anything about process-payments; it belongs to a completely different team. I wouldn't even know how to check it. I get that the point of the article is to sell the value of run books, which are amazing, but are people really going on-call for the first time without any training or knowledge of the services they'll be supporting? Even with perfect run books that seems unwise.

femiagbabiaka 3y ago
Unwise but common. It’s a good sign of an org to stay away from IMO, especially if it means that the people creating services aren’t the people oncall for those services.
jrace 3y ago
I am 1.5years in to a new role, on call for 1.4 years so far... I still get calls about some service or application I have never heard of. Technical debt is real and I lose a little bit of my life every on-call week.
- yodsanklai 3y ago
  Exactly my experience. I remember when I joined my company, after a couple of months I thought my job was pretty demanding but oncall added another level of stress. That being said, it's certainly less stressful now. At least, I can solve a bunch of issues myself, and I'm not lost in the internal tools anymore. And it's pretty rare to be bother in the night. And it's not like we're doctors or something. Worst case, somebody won't access a service for a few hours...
somehnguy 3y ago
I don't think it's that uncommon. When I joined my current org we spent much of my second week going through the apps we are responsible for, etc. But a week wasn't nearly enough to learn everything I needed to know about them. I was soon after added to the on-call rotation and received my first incident shortly after. A few weeks later, repeat. Each time I did what I knew and turned to my team to help with the rest - it has worked out fine so far and I learned so much more by getting hands on with failures compared to passively reading code & looking at diagrams. But I work in e-commerce, not life-critical systems of any sort..
- midasuni 3y ago
  My org is large enough to have a dedicated 24/7 front line team that look after dozens (hundereda) of things. The churn in that team is high that you can’t really give them any significant training on every system. Even a 20 minute video would take weeks to onboard for a system they’ll likely never touch. As far as I’m concerned a run book is there to stop me from being needlessly escalated to at 3am. If there’s a genuine problem then sure, I or another SME need to be contacted. Most of the time there’s a problem it is a reduction in resilience (so no need for out of hours contact), or something out of our control (loss of power in a location, failure of a circuit, in which case the 24 hour team can raise with the appropriate people. I can count on one hand the number of problems worthy of a call out, and that’s in the last 7 years. As such the first entry in my run book is “judging importance”.
tetha 3y ago
I'd say this strongly depends on the kind of error and the quality of the application involved. Like, if the error is something technical - something is upset and refuses to talk with the database now, that's something you can identify and handle with minimal instructions. I'm working on a side-project at work to automatically generate dashboards for these technical foundations of services in a central discoverable place. Or, is the error something functional? Like, payments from Suaheli aren't being taxed right. There, a generic on-call has no chance. But why the hell would this not be caught in tests, or emerge at night out of office ours, and outside of a planned maintenance if you need that? And then you're getting to the quality and if your application is a humble workhorse or a feisty goat. With a lot of our systems, we've pushed it to the point so in doubt you can just restart an instance and it most likely will end up better. This makes on-call somewhat chill - in most cases, you just keep kicking the system one or two times overnight until more knowledgeable people come online in the morning and dump the problem in their lap. This includes out databases, just for the record. On the other hand, I've worked also with enough systems which need like 15 steps to put them back together after a restart."Oh, you need to stop this first, then restart that, then run this jenkins job, then start this first thing, then run this other jenkins job - it'll fail, but that's normal, ..." In such a place, some kind of generic on-call would be impossible. until you get these systems under control.
hosh 3y ago
There was a company I interviewed at which had a particular way of hiring. Although I did not end up working for them, it was something I tried doing when I was involved in hiring for people with ops experience. And that is, being able to troubleshoot complex systems when you have no idea what is going on, under time pressure. Every site is a unique combination of side effects that no one on the internet — or even within the company — have any idea on. Yet there is generally a way to start approaching how to figure out what is going on. Though experience with the components of the system helps a lot.
- jachee 3y ago
  I call it Zero-knowledge Troubleshooting on my resume. It’s amusing how often, when I still worked in an office, that I would overhear part of a discussion, and some detail would tickle my spider sense, without even being sure what the problem being discussed was, and I’d ask a question or two, and it would help unblock the process. That sort of thing is literally the only thing I miss about the office.
  - hosh 3y ago
    Neat term. I might steal that for my resume. Intuition does play a good part of it for me, though I have worked with good troubleshooters who are more methodical and rely less on intuition for these situations. But then again, having these different takes can help the team as a whole to figure things out and fix things.
    - onionisafruit 3y ago
      I’m brushing up my resume today, and I would say there is a good chance this phrase will show up on mine too.
  - macintux 3y ago
    Recent blog post along a similar vein. https://rachelbythebay.com/w/2023/05/30/eng/ https://rachelbythebay.com/w/2023/05/30/eng/
- 3y ago
boruto 3y ago
Not uncommon, We had a bank where they tried on call rota per group instead of team. It was horrible, A ping in at 2 AM meant I had to go through code to triage the issue. No amount of training is enough as features and code gets deployed every day. It was horrible and I left.
throwawaaarrgh 3y ago
> are people really going on-call for the first time without any training or knowledge of the services they'll be supporting? Yes. Very common > Even with perfect run books that seems unwise. Yes
- yodsanklai 3y ago
  Certainly very common in my company. After 1 year in my team, I'm still discovering services we're supporting that I've never heard of! But being oncall doesn't mean that you're supposed to fix every single issue by yourself. You're allowed to ask for help! Monitoring the alerts, taking care of some simple issues (broken CI), trying to fix what you can, and dispatching when you can't. This is usually doable by a new team member.
sokoloff 3y ago
I have taken on-call rotations with no training on the systems I'd be supporting. Even worse, I built and ran the human process that systematically put other devs into that exact same situation. We got good results from the process (as compared to not having the process), but it's obviously not as good as it could possibly be.

femiagbabiaka 3y ago

Unwise but common. It’s a good sign of an org to stay away from IMO, especially if it means that the people creating services aren’t the people oncall for those services.

jrace 3y ago

I am 1.5years in to a new role, on call for 1.4 years so far... I still get calls about some service or application I have never heard of. Technical debt is real and I lose a little bit of my life every on-call week.

yodsanklai 3y ago
Exactly my experience. I remember when I joined my company, after a couple of months I thought my job was pretty demanding but oncall added another level of stress. That being said, it's certainly less stressful now. At least, I can solve a bunch of issues myself, and I'm not lost in the internal tools anymore. And it's pretty rare to be bother in the night. And it's not like we're doctors or something. Worst case, somebody won't access a service for a few hours...

somehnguy 3y ago

I don't think it's that uncommon. When I joined my current org we spent much of my second week going through the apps we are responsible for, etc. But a week wasn't nearly enough to learn everything I needed to know about them. I was soon after added to the on-call rotation and received my first incident shortly after. A few weeks later, repeat. Each time I did what I knew and turned to my team to help with the rest - it has worked out fine so far and I learned so much more by getting hands on with failures compared to passively reading code & looking at diagrams. But I work in e-commerce, not life-critical systems of any sort..

midasuni 3y ago
My org is large enough to have a dedicated 24/7 front line team that look after dozens (hundereda) of things. The churn in that team is high that you can’t really give them any significant training on every system. Even a 20 minute video would take weeks to onboard for a system they’ll likely never touch. As far as I’m concerned a run book is there to stop me from being needlessly escalated to at 3am. If there’s a genuine problem then sure, I or another SME need to be contacted. Most of the time there’s a problem it is a reduction in resilience (so no need for out of hours contact), or something out of our control (loss of power in a location, failure of a circuit, in which case the 24 hour team can raise with the appropriate people. I can count on one hand the number of problems worthy of a call out, and that’s in the last 7 years. As such the first entry in my run book is “judging importance”.

tetha 3y ago

I'd say this strongly depends on the kind of error and the quality of the application involved. Like, if the error is something technical - something is upset and refuses to talk with the database now, that's something you can identify and handle with minimal instructions. I'm working on a side-project at work to automatically generate dashboards for these technical foundations of services in a central discoverable place. Or, is the error something functional? Like, payments from Suaheli aren't being taxed right. There, a generic on-call has no chance. But why the hell would this not be caught in tests, or emerge at night out of office ours, and outside of a planned maintenance if you need that? And then you're getting to the quality and if your application is a humble workhorse or a feisty goat. With a lot of our systems, we've pushed it to the point so in doubt you can just restart an instance and it most likely will end up better. This makes on-call somewhat chill - in most cases, you just keep kicking the system one or two times overnight until more knowledgeable people come online in the morning and dump the problem in their lap. This includes out databases, just for the record. On the other hand, I've worked also with enough systems which need like 15 steps to put them back together after a restart."Oh, you need to stop this first, then restart that, then run this jenkins job, then start this first thing, then run this other jenkins job - it'll fail, but that's normal, ..." In such a place, some kind of generic on-call would be impossible. until you get these systems under control.

hosh 3y ago

There was a company I interviewed at which had a particular way of hiring. Although I did not end up working for them, it was something I tried doing when I was involved in hiring for people with ops experience. And that is, being able to troubleshoot complex systems when you have no idea what is going on, under time pressure. Every site is a unique combination of side effects that no one on the internet — or even within the company — have any idea on. Yet there is generally a way to start approaching how to figure out what is going on. Though experience with the components of the system helps a lot.

jachee 3y ago
I call it Zero-knowledge Troubleshooting on my resume. It’s amusing how often, when I still worked in an office, that I would overhear part of a discussion, and some detail would tickle my spider sense, without even being sure what the problem being discussed was, and I’d ask a question or two, and it would help unblock the process. That sort of thing is literally the only thing I miss about the office.
- hosh 3y ago
  Neat term. I might steal that for my resume. Intuition does play a good part of it for me, though I have worked with good troubleshooters who are more methodical and rely less on intuition for these situations. But then again, having these different takes can help the team as a whole to figure things out and fix things.
  - onionisafruit 3y ago
    I’m brushing up my resume today, and I would say there is a good chance this phrase will show up on mine too.
- macintux 3y ago
  Recent blog post along a similar vein. https://rachelbythebay.com/w/2023/05/30/eng/ https://rachelbythebay.com/w/2023/05/30/eng/
3y ago

boruto 3y ago

Not uncommon, We had a bank where they tried on call rota per group instead of team. It was horrible, A ping in at 2 AM meant I had to go through code to triage the issue. No amount of training is enough as features and code gets deployed every day. It was horrible and I left.

throwawaaarrgh 3y ago

> are people really going on-call for the first time without any training or knowledge of the services they'll be supporting? Yes. Very common > Even with perfect run books that seems unwise. Yes

yodsanklai 3y ago
Certainly very common in my company. After 1 year in my team, I'm still discovering services we're supporting that I've never heard of! But being oncall doesn't mean that you're supposed to fix every single issue by yourself. You're allowed to ask for help! Monitoring the alerts, taking care of some simple issues (broken CI), trying to fix what you can, and dispatching when you can't. This is usually doable by a new team member.

sokoloff 3y ago

I have taken on-call rotations with no training on the systems I'd be supporting. Even worse, I built and ran the human process that systematically put other devs into that exact same situation. We got good results from the process (as compared to not having the process), but it's obviously not as good as it could possibly be.