6 ms·

From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the se

by Dobbs 10y ago

From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the second paragraph. Too many things can go wrong and you are all around better off outsourcing this to something like Pingdom. You don't have sufficient levels of reliability, you aren't dual homed across twilio and another phone system. Maybe the cause of your outage is that AWS is having issues. Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant.

nhm 10y ago
> outsource to people who obsess over doing this right Completely agree! I often have to fight that "I could just build that myself" mentality, which glosses over the points you made so well.
- avitzurel 10y ago
  This and that! It's the same as "Twitter clone" with just posting messages with 140 char limit and "build a blog in 15 minutes". Alerting over a downed website are is sorta like a glacier, there's so much under the surface, if you just see the surface you're missing out. 1. Multiple locations 2. Multiple check intervals 3. SMS/email provider switch on fail 4. Auto recovery of your checkers 5. Multiple providers with a single storage.
tjholowaychuk 10y ago
Yep, plus most engineering time is worth at minimum $60+/h, which would pay for a year or more with most of these services.
- vacri 10y ago
  On the other hand, it's 'set up once and it just keeps chugging along', and isn't Yet Another SaaS To Manage. Also, if you want a 'proper' ops alerting SaaS, you're looking at something along the lines of $50/user/mo or $15/server/mo, neither of which is trivial.
  - tjholowaychuk 10y ago
    Yeah assuming nothing falls apart with the custom implementation maintenance-wise. Programmers have a hard time focusing on their real goals though, we often re-implement things that really aren't worth the time or money.
IgorPartola 10y ago
My favorite issue recently came up with a Django app of mine which was set up to email me when a request errors out. Turns out, when I switched which server it ran on I misconfigured the email settings and one of the errors was caused due to the inability to send an email. Thankfully it only took a few days to figure this out.
melvinmt 10y ago
> Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant. You make valid points about redundancy and levels of reliability but keep in mind that even Pingdom can go down: http://royal.pingdom.com/2016/10/24/ddos-attack-affects-pingdom http://royal.pingdom.com/2016/10/24/ddos-attack-affects-ping...
- user5994461 10y ago
  Chances are that pingdom won't be down at the same time that your site is down. Diversify to avoid cascading failures ;)
- imtringued 10y ago
  With your own solution you will likely encounter the same problems that pingdom faced including this one. The benefit of a service like pingdom is that they already solved those problems for you or if they haven't you don't have to waste time solving them yourself. It's not very efficient if everyone solves the same problems over and over again.
- dharma1 10y ago
  Use 2 or more providers. Signing up takes a minute or two and there are free alternatives
balaa 10y ago
HN top comments are getting way too predictable.
teddyh 10y ago
We’ve had issues with Pingdom at work. We don’t use them ourselves, but we host web sites, and some customer of ours used Pingdom to monitor their web site hosted on our servers. The customer would complain to us about downtime reported by Pingdom, but we would read the logs and find everything OK, with multiple successful accesses from other people during the time which Pingdom reported our customer’s site as being down. A huge pain.
- snom380 10y ago
  Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.
  - teddyh 10y ago
    If our ISP was down, we would not have had successful accesses from other people at the same time. If some transit ISP was down somewhere between us and Pingdom, well, that’s the Internet for you, eh? Regardless, Pingdom would report us as down, even though we weren’t at fault.
    - dalore 10y ago
      Yes you were down for some of your users. If that's ok for you that's fine. But if I were you I would be calling my ISP and trying to sort out why customers from location X can't access but customers from location Y can. If you're providing a service to your users, and they say that the service is down using pingdom, you should be looking into, not just saying "Works on my machine".
      - teddyh 10y ago
        Why should we be the ones to look into it? It was a random intermittent short-duration fault in the middle if the Internet, at some unknown place on the then-current path between us and Pingdom. Why should not Pingdom be at least equally as obligated to look into it? After all, they’re the ones actually using the failing connection, in order to monitor our and others’ services. But no, Pingdom simply report us as being down, and leave the hard part to us; i.e. the part where we have to explain to our customers that the Pingdom report is actually provably incorrect. I mean, what qualifies as “being up”? If some random link in the middle of the Internet goes down, and you suddenly, for 30 seconds, are unreachable for the few hundred people going through that exact link because it happens to be the best path between those people and your server, can they claim that you have failed to provide adequate uptime? If such a fault happens, are you then responsible to troubleshoot it? I say no. The Internet is the ISP’s responsibility, and the only faults actually meaningful to report to your ISP are the repeatable or long-lasting ones. Small stuff like this is not worth anybody’s time (except ISPs) to go digging into.

by Dobbs 10y ago

nhm 10y ago

> outsource to people who obsess over doing this right Completely agree! I often have to fight that "I could just build that myself" mentality, which glosses over the points you made so well.

avitzurel 10y ago
This and that! It's the same as "Twitter clone" with just posting messages with 140 char limit and "build a blog in 15 minutes". Alerting over a downed website are is sorta like a glacier, there's so much under the surface, if you just see the surface you're missing out. 1. Multiple locations 2. Multiple check intervals 3. SMS/email provider switch on fail 4. Auto recovery of your checkers 5. Multiple providers with a single storage.

tjholowaychuk 10y ago

Yep, plus most engineering time is worth at minimum $60+/h, which would pay for a year or more with most of these services.

vacri 10y ago
On the other hand, it's 'set up once and it just keeps chugging along', and isn't Yet Another SaaS To Manage. Also, if you want a 'proper' ops alerting SaaS, you're looking at something along the lines of $50/user/mo or $15/server/mo, neither of which is trivial.
- tjholowaychuk 10y ago
  Yeah assuming nothing falls apart with the custom implementation maintenance-wise. Programmers have a hard time focusing on their real goals though, we often re-implement things that really aren't worth the time or money.

IgorPartola 10y ago

My favorite issue recently came up with a Django app of mine which was set up to email me when a request errors out. Turns out, when I switched which server it ran on I misconfigured the email settings and one of the errors was caused due to the inability to send an email. Thankfully it only took a few days to figure this out.

melvinmt 10y ago

> Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant. You make valid points about redundancy and levels of reliability but keep in mind that even Pingdom can go down: http://royal.pingdom.com/2016/10/24/ddos-attack-affects-pingdom http://royal.pingdom.com/2016/10/24/ddos-attack-affects-ping...

user5994461 10y ago
Chances are that pingdom won't be down at the same time that your site is down. Diversify to avoid cascading failures ;)
imtringued 10y ago
With your own solution you will likely encounter the same problems that pingdom faced including this one. The benefit of a service like pingdom is that they already solved those problems for you or if they haven't you don't have to waste time solving them yourself. It's not very efficient if everyone solves the same problems over and over again.
dharma1 10y ago
Use 2 or more providers. Signing up takes a minute or two and there are free alternatives

balaa 10y ago

HN top comments are getting way too predictable.

teddyh 10y ago

We’ve had issues with Pingdom at work. We don’t use them ourselves, but we host web sites, and some customer of ours used Pingdom to monitor their web site hosted on our servers. The customer would complain to us about downtime reported by Pingdom, but we would read the logs and find everything OK, with multiple successful accesses from other people during the time which Pingdom reported our customer’s site as being down. A huge pain.

snom380 10y ago
Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.
- teddyh 10y ago
  If our ISP was down, we would not have had successful accesses from other people at the same time. If some transit ISP was down somewhere between us and Pingdom, well, that’s the Internet for you, eh? Regardless, Pingdom would report us as down, even though we weren’t at fault.
  - dalore 10y ago
    Yes you were down for some of your users. If that's ok for you that's fine. But if I were you I would be calling my ISP and trying to sort out why customers from location X can't access but customers from location Y can. If you're providing a service to your users, and they say that the service is down using pingdom, you should be looking into, not just saying "Works on my machine".
    - teddyh 10y ago
      Why should we be the ones to look into it? It was a random intermittent short-duration fault in the middle if the Internet, at some unknown place on the then-current path between us and Pingdom. Why should not Pingdom be at least equally as obligated to look into it? After all, they’re the ones actually using the failing connection, in order to monitor our and others’ services. But no, Pingdom simply report us as being down, and leave the hard part to us; i.e. the part where we have to explain to our customers that the Pingdom report is actually provably incorrect. I mean, what qualifies as “being up”? If some random link in the middle of the Internet goes down, and you suddenly, for 30 seconds, are unreachable for the few hundred people going through that exact link because it happens to be the best path between those people and your server, can they claim that you have failed to provide adequate uptime? If such a fault happens, are you then responsible to troubleshoot it? I say no. The Internet is the ISP’s responsibility, and the only faults actually meaningful to report to your ISP are the repeatable or long-lasting ones. Small stuff like this is not worth anybody’s time (except ISPs) to go digging into.