3 ms·
always the on-call engineer, me (the team lead), and then my manager and then their manager. The best scheme in my experience is to have a 24/7 fully staffed L
by gaius 8y ago
always the on-call engineer, me (the team lead), and then my manager and then their manager.
The best scheme in my experience is to have a 24/7 fully staffed L1 working shifts for initial response triage then waking the appropriate L2 to deal with the actual problem if it isn’t covered in the runbook. No good waking the DBA if you can’t login to the database because a router has crashed and failed to failover. The next morning the L2 guy updates the runbook so if possible if the same thing recurs L1 can just handle it.
What really doesn’t work is having the on-call support also doing out-of-hours work such as maintenance or whatever. That’s a recipe for disaster. People willing to make the sacrifice of on-call are a precious commodity to be used wisely, or they’ll walk.
Paying only in the event of a callout doesn’t work either, that person has still had to decline other activities in order to make themselves available.
- jedberg 8y agoIf you have 24/7 coverage using run books, perhaps a better system would be to take the money you're spending on the 5ish people you need to do 24/7 coverage, and instead pay two people twice as much to codify everything in the run book so that it happens automatically. Then have the alerts go straight to the L2.
- gaius 8y agoThe infra for that job had far too many moving parts to make that feasible unfortunately. Multiple fibre providers, satellites, microwave relays, etc etc. The L1 guys would be on the phone alot in the event of a serious incident. Even today there still needs to be humans in the loop, I don’t see that changing.