5 ms·

AWS is not down. Only US-East. If your app is down, it's only because you don't care about the availability of your service. It's pointless to complain. We've

AWS is not down. Only US-East. If your app is down, it's only because you don't care about the availability of your service. It's pointless to complain. We've all seen before that Amazon can't keep whole regions up. If you rely on a region being up, you will have downtime and it's your fault.

oconnore 14y ago
Robust systems aren't hip. Get back to work and ship, ship, ship.
- narsil 14y ago
  That's how I feel about most hackathons. There isn't enough work done on existing projects, but quick hacks the demonstrate concepts are the norm. It would be great to see a hackathon where the primary purpose was to contribute to an existing open source project or something similar.
haberman 14y ago
According to the AWS status page, only one availability zone within US-East is down, not the whole US-East region. Running a highly-available service exclusively from US-East is a reasonable strategy as long as you're spread across multiple availability zones. I'm not an AWS customer, just reading their docs; please correct me if I'm wrong about any of this.
- dragonstyle 14y ago
  We're across availability zones and definitely ran into an outage across zones tonight.
- sseveran 14y ago
  The idea that a single data center can host a highly available service is just wrong.
beedogs 14y ago
> If your app is down, it's only because you don't care about the availability of your service. That is absolutely absurd. At what point did the common-sense solution to "unacceptable downtime on AWS" become "buy two of everything"?
- NyxWulf 14y ago
  If you operate a service where uptime is critical, two of everything isn't enough. You have to ensure they will be separated geographically and logically so that when things like this happen, you don't lose your systems. We operate systems that sit on the pages of the top e-commerce companies in the world. We have 10 separate segments of clusters. Operating in four AZ's in East, three AZ's in West-1 and two AZ's in West-2. When this outage happened, the servers that were impacted in east were removed from our DNS and within 9 minutes the impact of this event on their sites was eliminated.
  - paulsutter 14y ago
    Thanks for the most constructive post in this whole discussion. Hacker News is a much better place to talk about solutions than just whining. At Quantcast we have physical servers in 14 cities. We use anycast to achieve site failovers in 6 seconds. Downtime for us would impact millions of websites, so we don't have downtime.
    - NyxWulf 14y ago
      That's our next step is automating our fail over. We currently use DNS Made Easy and while you can do that with them it's tougher. We are switching to Dyn, and their Layer 2 failure detection will put our failover in that 6 second range.
      - rdl 14y ago
        DNS based failover in practice will not give you 6 second failover, unlike BGP. Too much of DNS is broken and out of your control. (The trend for regular ISPs is probably improving, except that mobile/carrier DNS is often particularly broken. It would be interesting to do monthly surveys of this.)

oconnore 14y ago

Robust systems aren't hip. Get back to work and ship, ship, ship.

narsil 14y ago
That's how I feel about most hackathons. There isn't enough work done on existing projects, but quick hacks the demonstrate concepts are the norm. It would be great to see a hackathon where the primary purpose was to contribute to an existing open source project or something similar.

haberman 14y ago

According to the AWS status page, only one availability zone within US-East is down, not the whole US-East region. Running a highly-available service exclusively from US-East is a reasonable strategy as long as you're spread across multiple availability zones. I'm not an AWS customer, just reading their docs; please correct me if I'm wrong about any of this.

dragonstyle 14y ago
We're across availability zones and definitely ran into an outage across zones tonight.
sseveran 14y ago
The idea that a single data center can host a highly available service is just wrong.

beedogs 14y ago

> If your app is down, it's only because you don't care about the availability of your service. That is absolutely absurd. At what point did the common-sense solution to "unacceptable downtime on AWS" become "buy two of everything"?

NyxWulf 14y ago
If you operate a service where uptime is critical, two of everything isn't enough. You have to ensure they will be separated geographically and logically so that when things like this happen, you don't lose your systems. We operate systems that sit on the pages of the top e-commerce companies in the world. We have 10 separate segments of clusters. Operating in four AZ's in East, three AZ's in West-1 and two AZ's in West-2. When this outage happened, the servers that were impacted in east were removed from our DNS and within 9 minutes the impact of this event on their sites was eliminated.
- paulsutter 14y ago
  Thanks for the most constructive post in this whole discussion. Hacker News is a much better place to talk about solutions than just whining. At Quantcast we have physical servers in 14 cities. We use anycast to achieve site failovers in 6 seconds. Downtime for us would impact millions of websites, so we don't have downtime.
  - NyxWulf 14y ago
    That's our next step is automating our fail over. We currently use DNS Made Easy and while you can do that with them it's tougher. We are switching to Dyn, and their Layer 2 failure detection will put our failover in that 6 second range.
    - rdl 14y ago
      DNS based failover in practice will not give you 6 second failover, unlike BGP. Too much of DNS is broken and out of your control. (The trend for regular ISPs is probably improving, except that mobile/carrier DNS is often particularly broken. It would be interesting to do monthly surveys of this.)