6 ms·

I guess this is one Reddit outage that can't be blamed on poor scaling

• thezilch 16y ago
  Looks like troubles in only one availability zone.
  • mechanical_fish 16y ago
    That seems to be incorrect. We have problem children in us-east-1b, -1c, and -1d.
    • justinsb 16y ago
      AWS randomize the zones per account, so "your" -1b is not necessarily the same as "my" -1b. I'm only seeing problems in my -1c. Are you seeing 3 zones failing all under the same account? If multiple AZs are down, AWS are going to have some serious explaining to do...
      • jedberg 16y ago
        We (reddit) are seeing failures in all zones.
        • redthrowaway 16y ago
          If memory serves, Amazon's reporting of what zones are experiencing problems has been...optimistic...in the past.
      • nlo 16y ago
        "AWS randomize the zones per account" Interesting. This is news to me. I found some more info here (Google cache since alestic.com is presently unreachable): http://webcache.googleusercontent.com/search?q=cache:0jxzyFj0J7wJ:alestic.com/2009/07/ec2-availability-zones&hl=en&gl=us&strip=1 http://webcache.googleusercontent.com/search?q=cache:0jxzyFj...
      • dangrossman 16y ago
        Amazon RDS's most expensive feature is automatic, instant Multi-AZ failover to protect against this kind of situation. It's not working quite like that, which the AWS status page acknowledges. This is a major failure.
  • justinsb 16y ago
    AWS have now confirmed that this affects multiple availability zones. From the status page: "..impacting EBS volumes in multiple availability zones in the US-EAST-1 region"
    • justincormack 16y ago
      Thats not good. The whole point of multiple AVs is for them to not fail at the same time. Suggests some dependencies that should not be there perhaps, or at least some correlation of something, like software upgrades. Looking for a good explanation of this; one AV going down is not a problem and should not impact anyone who is load balancing.
• StavrosK 16y ago
  Apparently most of their problems are caused by bad EBS writes/performance, or at least so they said a few weeks ago after some particularly bad downtime. It looks like EBS will randomly decide to switch to a few bps of performance from time to time. I would use Amazon for my startup, but these issues really make it hard to justify.
  • muppetman 16y ago
    Slightly offtopic, but wasn't that post by an ex-employee as a comment? Not that the technical aspect of it wasn't fantastic, because it was, but I don't think Reddit said anything publically did they?
    • StavrosK 16y ago
      It was, you are right, I misremembered. valisystem's comments above contains the reference.
      • muppetman 16y ago
        Yes valisystem linked to it. All good, I wasn't sure if I misremembered.
  • valisystem 16y ago
    EBS seems to be the main problem here, I'll cite a former reddit employe (first comment on the blog that talked about EBS problems). I don't work for reddit anymore (as of about a week ago, although I didn't get as much fanfare as raldi did), but I can tell you that they're giving Amazon too much credit here. Amazon's EBSs are a barrel of laughs in terms of performance and reliability and are a constant (and the single largest) source of failure across reddit. reddit's been in talks with Amazon all the way up to CIOs about ways to fix them for nearly a year and they've constantly been making promises that they haven't been keeping, passing us to new people (that "will finally be able to fix it"), and variously otherwise been desperately trying to keep reddit while not actually earning it. Source: http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d... cache: - scroll down for comment - http://webcache.googleusercontent.com/search?q=cache:cfbs-sp10QYJ:www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/+site:reddit.com+downtime+EBS+performance&cd=7&hl=en&ct=clnk&client=ubuntu&source=www.google.comhttp://webcache.googleusercontent.com/search?q=cache:cfbs-sp10QYJ:www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/+site:reddit.com+downtime+EBS+performance&cd=7&hl=en&ct=clnk&client=ubuntu&source=www.google.com http://webcache.googleusercontent.com/search?q=cache:cfbs-sp...
• jedberg 16y ago
  Thankfully, no. :) But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.
  • thirtysixred 16y ago
    SHUT. DOWN. EVERYTHING. Good plan though. :p
    • retube 16y ago
      Reminds me of that scene in Jurassic Park
      • retube 16y ago
        Good lord. What's with all the DVs??
        • retube 16y ago
          Ha. HN really on it's high horse today.
          • eogas 16y ago
            I know right? I made a hilarious joke a little while ago and jedberg yelled at me and everyone downvoted me. And I'll be very surprised if this comment doesn't get downvoted to hell too. EDIT: I also simply greeted jedberg, and a bunch of people thought that was a good reason to downvote. Do people think there's an imminent influx of redditors, and that they have to dissuade them from becoming HNers? I don't think that's the case. EDIT: Fuckin' called it.
    • rmoriz 16y ago
      President Madagascar is doin' cloud biz, too?
    • jedberg 16y ago
• Joakal 16y ago
  How does Reddit display the 'offline' page if it's down?
  • jedberg 16y ago
    The server is still up, so we can serve it right out of the load balancer.
    • wallnutboy 16y ago
      Are you able to enable the 'read-only-mode' using the same method?
• clistctrl 16y ago
  I was to be under the impression a great deal of Reddits issues were linked to Amazon.