4 ms·

Maybe companies should alert on an increased amount of traffic to their status pages.

kelnos 5y ago
That might be a good idea as a last-resort measure, but if you're only finding out problems because customers are telling you about them (even indirectly through a signal like this), your monitoring and alerting is woefully inadequate.
- hnarn 5y ago
  It’s not last resort, you should do both because you’re alerting on different things.
- szundi 5y ago
  fun fact here that no matter how short the automated system is set up to alert, customers are faster - no one knows how is that possible
  - therein 5y ago
    Yup, that's another reason why it pays off having a direct line of communication to at least some of your top customers. Can't tell you how many times I've had a customer write to me, only to be receiving some automated alerts 30-40 seconds later.
    - lanstin 5y ago
      And the best customers for this are the pickiest annoying ones
  - strictfp 5y ago
    Generally I agree. Humans are much better at recognizing patterns than machines. But as a counterpoint I was once working in a project where our monitoring dashboard showed an anomaly in incoming traffic. It turned out that it was an ISP problem and that we were the first ones to notice according to them. So maybe part of the answer as to why customers are faster is that different monitoring systems are monitoring each other :)
mlazowik 5y ago
I’ve seen alerting based on the number of Twitter mentions too
- colpabar 5y ago
  I think this is how downdetector works, it just looks for tweets that are complaining about something not working.
  - PaulBGD_ 5y ago
    I'm fairly confident they just count a site as down based off the popularity of the page.
scrollaway 5y ago
That's honestly a good idea. Does anyone do that?
- gdubicki 5y ago
  My company does. :)
- cranekam 5y ago
  In my last job our main “is everything okay?” monitoring dashboard had RPS, latency, 500 rate etc and a line for the number of user issues reported per N requests served. We didn’t alert on this but it was a useful way to detect issues that were subtle enough not to trip up our main alerts.
- hnarn 5y ago
  A telco I used to work for did this like two decades ago but even better, they mapped incoming support calls (customers) to stations, and if more than N came in during a certain period for the same DSLAM it triggered some kind of alert. Same thing happens (should happen) when you visit your ISPs website and look for registered downtime — many requests from one zip code from multiple ips should trigger an alarm if the isp is competent.
qubyte 5y ago
Close that loop: Once the traffic goes above a threshold transition it to red!
rozenmd 5y ago
I'm surprised more folks don't monitor their vendors via uptime checkers - worth knowing if you'll be able to get anything done that day.
ksec 5y ago
They should also monitor HN's submission with GitHub mentioned. Judging from historical pattern it is probably faster than monitoring traffic on status page or even twitter.

That might be a good idea as a last-resort measure, but if you're only finding out problems because customers are telling you about them (even indirectly through a signal like this), your monitoring and alerting is woefully inadequate.

hnarn 5y ago
It’s not last resort, you should do both because you’re alerting on different things.
szundi 5y ago
fun fact here that no matter how short the automated system is set up to alert, customers are faster - no one knows how is that possible
- therein 5y ago
  Yup, that's another reason why it pays off having a direct line of communication to at least some of your top customers. Can't tell you how many times I've had a customer write to me, only to be receiving some automated alerts 30-40 seconds later.
  - lanstin 5y ago
    And the best customers for this are the pickiest annoying ones
- strictfp 5y ago
  Generally I agree. Humans are much better at recognizing patterns than machines. But as a counterpoint I was once working in a project where our monitoring dashboard showed an anomaly in incoming traffic. It turned out that it was an ISP problem and that we were the first ones to notice according to them. So maybe part of the answer as to why customers are faster is that different monitoring systems are monitoring each other :)

mlazowik 5y ago

I’ve seen alerting based on the number of Twitter mentions too

colpabar 5y ago
I think this is how downdetector works, it just looks for tweets that are complaining about something not working.
- PaulBGD_ 5y ago
  I'm fairly confident they just count a site as down based off the popularity of the page.

scrollaway 5y ago

That's honestly a good idea. Does anyone do that?

gdubicki 5y ago
My company does. :)
cranekam 5y ago
In my last job our main “is everything okay?” monitoring dashboard had RPS, latency, 500 rate etc and a line for the number of user issues reported per N requests served. We didn’t alert on this but it was a useful way to detect issues that were subtle enough not to trip up our main alerts.
hnarn 5y ago
A telco I used to work for did this like two decades ago but even better, they mapped incoming support calls (customers) to stations, and if more than N came in during a certain period for the same DSLAM it triggered some kind of alert. Same thing happens (should happen) when you visit your ISPs website and look for registered downtime — many requests from one zip code from multiple ips should trigger an alarm if the isp is competent.

qubyte 5y ago

Close that loop: Once the traffic goes above a threshold transition it to red!

rozenmd 5y ago

I'm surprised more folks don't monitor their vendors via uptime checkers - worth knowing if you'll be able to get anything done that day.

ksec 5y ago

They should also monitor HN's submission with GitHub mentioned. Judging from historical pattern it is probably faster than monitoring traffic on status page or even twitter.