4 ms·

So I guess the author uses a smart phone as a pager, but given his passion for uptime, reliability, latency etc. I wonder if he has experimented with an actual

So I guess the author uses a smart phone as a pager, but given his passion for uptime, reliability, latency etc. I wonder if he has experimented with an actual pager.

dsr_ 12y ago
It turns out that pagers are often unreliable; while I've never had a message not get through eventually, I have frequently seen messages get delayed for hours or even until the pager was power-cycled. On the other hand, if you set up an IMAPS account for each person's alert address, and then have their smartphone use that account to check email, reliability is quite high. (Strangely, sysadmins tend to have wifi in their houses and good data plans, especially when the company pays for it.)
- 0xdeadbeefbabe 12y ago
  Ahhh I thought the medical profession had it figured out, but these unreliable pagers could be why the doctor is always late for the delivery.
apposite 12y ago
We use a variety of escalation techniques. As mentioned down thread, pagers are actually very unreliable. Some SREs carry pagers and mobiles. Most SREs carry phones with escalation via SMS, an actual telephone call from an automated system (after a delay), and/or escalation via a network connection over the data network. Phone calls are way way more reliable than pagers. Unacknowledged pages escalate to a secondary oncaller (e.g. if the oncall is out of range/in a tunnel, under a bus) and tertiary depending on configuration (and then it loops, or falls to another rotation, again depending on configuration). The code and services that do escalations is deliberately and carefully vetted to have minimal overlap with production systems (who's failure they might be alerting people to).
- shackattack 12y ago
  This sort of mirrors how we do things at PagerDuty. Phone may be more reliable than SMS, but every telephony/messaging gateway fails sometimes. We use something like a dozen different phone/SMS gateways to prevent single points of failure, and do end-to-end testing of our SMS providers to check their uptime and latency. Responders can customize their notification methods (push, SMS, phone, and email) and rules, so you can do things like get a lightweight push notification when an alert happens, and then a phone call 2 minutes later if you haven't acknowledged the incident. Teams get escalation timeouts that forward alerts up the chain if the primary hasn't responded after a period of time.

So I guess the author uses a smart phone as a pager, but given his passion for uptime, reliability, latency etc. I wonder if he has experimented with an actual pager.

dsr_ 12y ago

It turns out that pagers are often unreliable; while I've never had a message not get through eventually, I have frequently seen messages get delayed for hours or even until the pager was power-cycled. On the other hand, if you set up an IMAPS account for each person's alert address, and then have their smartphone use that account to check email, reliability is quite high. (Strangely, sysadmins tend to have wifi in their houses and good data plans, especially when the company pays for it.)

0xdeadbeefbabe 12y ago
Ahhh I thought the medical profession had it figured out, but these unreliable pagers could be why the doctor is always late for the delivery.

apposite 12y ago

We use a variety of escalation techniques. As mentioned down thread, pagers are actually very unreliable. Some SREs carry pagers and mobiles. Most SREs carry phones with escalation via SMS, an actual telephone call from an automated system (after a delay), and/or escalation via a network connection over the data network. Phone calls are way way more reliable than pagers. Unacknowledged pages escalate to a secondary oncaller (e.g. if the oncall is out of range/in a tunnel, under a bus) and tertiary depending on configuration (and then it loops, or falls to another rotation, again depending on configuration). The code and services that do escalations is deliberately and carefully vetted to have minimal overlap with production systems (who's failure they might be alerting people to).

shackattack 12y ago
This sort of mirrors how we do things at PagerDuty. Phone may be more reliable than SMS, but every telephony/messaging gateway fails sometimes. We use something like a dozen different phone/SMS gateways to prevent single points of failure, and do end-to-end testing of our SMS providers to check their uptime and latency. Responders can customize their notification methods (push, SMS, phone, and email) and rules, so you can do things like get a lightweight push notification when an alert happens, and then a phone call 2 minutes later if you haven't acknowledged the incident. Teams get escalation timeouts that forward alerts up the chain if the primary hasn't responded after a period of time.