I used to run IT operations for a telecom, hundreds of physical servers, facilities, 24x7 operators. Hey, shit happens. Problems with hardware, facilities, soft

I used to run IT operations for a telecom, hundreds of physical servers, facilities, 24x7 operators. Hey, shit happens. Problems with hardware, facilities, software jobs, COTS software, homegrown software. There’s a lot of benefit to having someone on call in each domain of concern rather than letting problems fester or go unresolved all night, particularly if the issues are customer-facing. It’s easy to say “just build your crap to anticipate every failure mode”, but that’s not realistic for large operations, especially if you aren’t sitting on an Apple-size pile of cash and talent.