7 ms·

Why Stashboard and Pingdom Weren't Enough: The Evolution of a Status Page

65 points by zende 14y ago

mahmoudimus 14y ago
This was in response to the feedback we've received and requests to open source the project. The original post was here: http://news.ycombinator.com/item?id=4812222 http://news.ycombinator.com/item?id=4812222 Happy to field any questions :)
- gacba 14y ago
  Great post--I can use this exact setup with my AWS server and was about to start writing something similar. Thanks for saving me loads of time.
  - mahmoudimus 14y ago
    Absolutely! Feel free to shoot an email (in my profile) if you need help setting it up. We're also on irc.freenode.net #balanced
- huhtenberg 14y ago
  Have you looked at anything else but Pingdom? It has nice visuals and free trials, but it's a relative newcomer to the market of the monitoring services and they also have a tendency to keep things simpler than they need to be. Something more mature and enterprisey, like RedAlert, would fit the bill.
  - mjallday 14y ago
    We looked at a few other solutions like ServerDensity etc. The main issue was that we wanted to report across all requests, e.g. there is no single health check endpoint that an external system can hit to determine if everything is OK or not. The crux of the matter is that we needed to parse our Nginx logs for the response code on every single request and calculate the uptime from that.
    - riffraff 14y ago
      I may be missing something, but checking your logs does not fulfill exactly the same function as an external check. I mean, if X% of your requests never make it to the logs cause ELB is messed up/dns is fucked up/disk is full/nginx dies/grasshoppers then you'd happily show that everything is ok, since no new 500 status gets pulled from the logs. So internal and external checks should be compounded, imo.
      - mjallday 14y ago
        That's a great point, we've created an issue to track this on Github - https://github.com/balanced/status.balancedpayments.com/issues/1 https://github.com/balanced/status.balancedpayments.com/issu... PS - Pull requests with solutions get free t-shirts :)
  - amatix 14y ago
    Over 6 years (July 2006 according to archive.org) is a 'relative newcomer'? You're hard to please :) Their API took a long time to arrive, but we've used it for disabling checks during maintenance events pretty effectively. As mentioned in the article, POST/PUT requests are on the list of Things I Want though. Would be good to be able to group a bunch of checks together into a service too.
    - huhtenberg 14y ago
      Pingdom really started to develop into its current form around 2009 I think. Before that it might've existed, but I don't think it was aggressively marketed and developed.
ajsharp 14y ago
Awesome write-up. Really love the technical depth in this post. Slurp looks great too, gonna play around with it.
LiveTheDream 14y ago
Here's a gotcha related to Pingdom -- if your account isn't set up properly, you can run out of SMS credits which means you may not get that critical SMS alert!
- pekk 14y ago
  If only someone would invent a hosted service which sets up your Pingdom account properly for reliable delivery...
- pulledpork 14y ago
  Do they at least ping you to let you know you're out of funds? In an ideal world I'd hope they'd extend you some credit.
  - munger 14y ago
    It's easy to setup correctly. It lets you set a rule like "buy me X credits when I get below Y credits" I think ours is something like buy me 20 credits when we get below 5. Also you can install their iphone app and use push notifications for free instead of or in addition to SMS.
jareau 14y ago
Here's a link to the actual status page: https://status.balancedpayments.com/ https://status.balancedpayments.com/
- mwetzler 14y ago
  I love this!! Scanned the article several times looking for this link. Nice design work! Love the twitter integration.
  - jareau 14y ago
    Thanks! Take the source. Use it at home. Tell your friends.
conroy 14y ago
Author of Stashboard here, love what you guys have done. I'd love to integrate your frontpage design as an optional template. When I wrote Stashboard, monitoring was explicitly not a requirement. Since it has an API, the idea what to write your own tools for monitoring that would update Stashboard in the event of an outage. Pingdom support is something that's been asked for many times, so it might be time to dedicate some time to adding integration.
- mahmoudimus 14y ago
  conroy, thanks for your comment! Can we open an issue on https://github.com/balanced/balanced-api https://github.com/balanced/balanced-api so we can track this? We'd be happy to help with the code contribution as well.

65 points by zende 14y ago

This was in response to the feedback we've received and requests to open source the project. The original post was here: http://news.ycombinator.com/item?id=4812222 http://news.ycombinator.com/item?id=4812222 Happy to field any questions :)

gacba 14y ago
Great post--I can use this exact setup with my AWS server and was about to start writing something similar. Thanks for saving me loads of time.
- mahmoudimus 14y ago
  Absolutely! Feel free to shoot an email (in my profile) if you need help setting it up. We're also on irc.freenode.net #balanced
huhtenberg 14y ago
Have you looked at anything else but Pingdom? It has nice visuals and free trials, but it's a relative newcomer to the market of the monitoring services and they also have a tendency to keep things simpler than they need to be. Something more mature and enterprisey, like RedAlert, would fit the bill.
- mjallday 14y ago
  We looked at a few other solutions like ServerDensity etc. The main issue was that we wanted to report across all requests, e.g. there is no single health check endpoint that an external system can hit to determine if everything is OK or not. The crux of the matter is that we needed to parse our Nginx logs for the response code on every single request and calculate the uptime from that.
  - riffraff 14y ago
    I may be missing something, but checking your logs does not fulfill exactly the same function as an external check. I mean, if X% of your requests never make it to the logs cause ELB is messed up/dns is fucked up/disk is full/nginx dies/grasshoppers then you'd happily show that everything is ok, since no new 500 status gets pulled from the logs. So internal and external checks should be compounded, imo.
    - mjallday 14y ago
      That's a great point, we've created an issue to track this on Github - https://github.com/balanced/status.balancedpayments.com/issues/1 https://github.com/balanced/status.balancedpayments.com/issu... PS - Pull requests with solutions get free t-shirts :)
- amatix 14y ago
  Over 6 years (July 2006 according to archive.org) is a 'relative newcomer'? You're hard to please :) Their API took a long time to arrive, but we've used it for disabling checks during maintenance events pretty effectively. As mentioned in the article, POST/PUT requests are on the list of Things I Want though. Would be good to be able to group a bunch of checks together into a service too.
  - huhtenberg 14y ago
    Pingdom really started to develop into its current form around 2009 I think. Before that it might've existed, but I don't think it was aggressively marketed and developed.

ajsharp 14y ago

Awesome write-up. Really love the technical depth in this post. Slurp looks great too, gonna play around with it.

LiveTheDream 14y ago

Here's a gotcha related to Pingdom -- if your account isn't set up properly, you can run out of SMS credits which means you may not get that critical SMS alert!

pekk 14y ago
If only someone would invent a hosted service which sets up your Pingdom account properly for reliable delivery...
pulledpork 14y ago
Do they at least ping you to let you know you're out of funds? In an ideal world I'd hope they'd extend you some credit.
- munger 14y ago
  It's easy to setup correctly. It lets you set a rule like "buy me X credits when I get below Y credits" I think ours is something like buy me 20 credits when we get below 5. Also you can install their iphone app and use push notifications for free instead of or in addition to SMS.

jareau 14y ago

Here's a link to the actual status page: https://status.balancedpayments.com/ https://status.balancedpayments.com/

mwetzler 14y ago
I love this!! Scanned the article several times looking for this link. Nice design work! Love the twitter integration.
- jareau 14y ago
  Thanks! Take the source. Use it at home. Tell your friends.

conroy 14y ago

Author of Stashboard here, love what you guys have done. I'd love to integrate your frontpage design as an optional template. When I wrote Stashboard, monitoring was explicitly not a requirement. Since it has an API, the idea what to write your own tools for monitoring that would update Stashboard in the event of an outage. Pingdom support is something that's been asked for many times, so it might be time to dedicate some time to adding integration.

mahmoudimus 14y ago
conroy, thanks for your comment! Can we open an issue on https://github.com/balanced/balanced-api https://github.com/balanced/balanced-api so we can track this? We'd be happy to help with the code contribution as well.