4 ms·
The problem is, unlike a copilot who is trained on the system, usually on-call people are winging it at best, and after the horse has left the barn. I've been
by cbanek 9y ago
The problem is, unlike a copilot who is trained on the system, usually on-call people are winging it at best, and after the horse has left the barn.
I've been on many on call rotations for many different projects. Some of them have had such great ideas as "you'll learn a lot from being on call, so we'll add you to the rotation (on your first day)."
Sadly, people don't normally document code and emergency procedures nearly as much as they should. Unlike an airliner, there's no emergency manual sitting around. Even a normal manual would be hard to find, and who knows if it's current.
One place I worked had solved this problem by having an actual operations team, a practice that seems to have faded with time. Our operations team was great. They were there in 3 shifts, 24/7, to cover your ass. They knew more about how the service ran than any dev. There were standards for documenting procedures, and if you got the call (there wasn't an on call rotation, you were only called if everyone was clueless and you wrote it), it's because things were really bad. This was before "devops" was minted the silver bullet.
While a dedicated operations team can save you, and your reputation, most organizations would rather not pay that, and push that cost onto the devs, and make them "feel the pain" of their mistakes.