5 ms·
Waking up at 2.30am to a total service outage sucks, but at least I know all you bastards are awake and screaming too. That makes me feel slightly better. (We'
by seldo 15y ago
Waking up at 2.30am to a total service outage sucks, but at least I know all you bastards are awake and screaming too. That makes me feel slightly better.
(We're back now. Kudos to Pingdom for noticing and alerting)
- chrischen 15y agoFor a good 20 minutes I didn't know it was an outage and panicked thinking my server mysteriously stopped working, and couldn't even SSH in.
- cperciva 15y agoI've learned that when my SSH session dies, the first thing to do is go to twitter and search for "EC2". People were complaining there within about 60 seconds of the outage starting.
- viraptor 15y agoAlternative monitoring opportunity? Grep a twitter stream for "EC2" and "down" and raise an alert if you get enough hits in 5 min?
- verelo 15y agoLooks like the entire US-East-1 region had connectivity issues. We're a monitoring company, we're watching this very carefully...there is a write up on our blog: http://www.verelo.com/blog/2012/03/15/aws-entire-an-region-down-again/ http://www.verelo.com/blog/2012/03/15/aws-entire-an-region-d...
- mikejarema 15y agoWhat's funny is that this is what I find myself doing instinctually when I encounter an outage or high latencies on ANY service or site. Heroku (recent process startup woes), Google Apps (slowdowns, specifically Gmail), Amazon (when its hammered by traffic to big deals), etc etc. I second the comment above suggesting a "crowdsourced" status app monitoring twitter. Although it's no consolation for service interruptions, it does at least keep you sane knowing the problem is elsewhere.
- jamiesonbecker 15y agoYes, because Twitter never goes down.
- chc 15y agoWhat does it matter if Twitter goes down? The odds of it happening for an extended time right at the beginning of an EC2 outage are rather small, and even in that worst-case scenario, it doesn't really put us in any worse of a position than we're in now.
- mikejarema 15y agoTouché.
- yanivgolan 15y agoAh, pingdom failed to alert us via push got only the email alert (luckily it was exactly when the team is warmed up and ready to hit the keyboard :) 11AM at Israel)
- katzboaz 15y agoI have an IFTTT recipe to get Push notification if you have a server outage http://ifttt.com/recipes/25122 http://ifttt.com/recipes/25122
- yanivgolan 15y agoCool! thanks, adding it now (IFTT rocks)