7 ms·

US-East AWS Connectivity Issues

77 points by fjordan 13y ago

rschmitty 13y ago
Does anyone know why in the world they display a green checkmark with a near invisible little 'i' for this?
- aroch 13y ago
  Because checkmarks make everything look fine...
  - jordanmessina 13y ago
    http://www.youtube.com/watch?v=uriZZ3Slz74 http://www.youtube.com/watch?v=uriZZ3Slz74
  - ceph_ 13y ago
    Non-snarky answer: They want to post that they are investigating an issue, but do not want to comment on the scope of the problem when it is still not fully understood.
- teoruiz 13y ago
  They switch to white-triangle-in-yellow-circle when they find out that there is a real problem. They just switched.
- iota 13y ago
  There are 4 statuses. Green checkmark (status 0) Green checkmark with "info" badge (status 1) Yellow triangle (status 2) Red "do not enter" rectangle (status 3) I suspect that status 0 indicates that they are investigating a problem with the server, and it switches to status 1 once the problem has been confirmed. This is also a good example of poor icon design...they aren't self-explanatory, and so they should not be used.
  - cperciva 13y ago
    What happened to status 2?
    - iota 13y ago
      Good catch. Fixed!
  - rschmitty 13y ago
    Thanks!
o0-0o 13y ago
Down in Manhattan
jolan 13y ago
Amazon is continuing the trend of announcing outages 30 minutes after they start. Just signed up for a support contract since the status page said everything was fine.
- colinbartlett 13y ago
  And by "announcing" you mean indicating everything is a-okay with the green checkmark but putting a tiny footnote next to it.
- acdha 13y ago
  “outage (n): a period when a power supply or other service is not available or when equipment is closed down” A subset of traffic taking longer than normal is not an outage
scrabble 13y ago
So what is the best way to balance a hosted site between Amazon and a separate service? Because these connectivity issues suck.
- bredman 13y ago
  One option would be to use Route 53 weighted round robin (WRR) DNS records and health checks to accomplish this.
NotDaveLane 13y ago
It's region us-east-1c for now, at least from where I'm sitting... I have instances in other us-east datacenters that are fine.
- lreeves 13y ago
  Availability zone letters are specific to your account.
- trevyn 13y ago
  Specific availability zones in a region are mapped per-account, so your east-1c might be my east-1a: "To ensure that resources are distributed across the Availability Zones for a region, we independently map Availability Zones to identifiers for each account. For example, your Availability Zone us-east-1a might not be the same location as us-east-1a for another account. Note that there's no way for you to coordinate Availability Zones between accounts." http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...
  - ceejayoz 13y ago
    I wonder how that works with new zones. I remember us-east-1e being added separately to the original four. Presumably, that one's the same for all accounts that'd already signed up at the time.
  - astrodust 13y ago
    Clever and ridiculous at the same time.
- deleted 13y ago
  [deleted]
frabcus 13y ago
We (ScraperWiki) can still access some of our US East servers. From those, can daisy chain SSH into the ones that are offline. Those servers can't see the world, but are working fine and can see other EC2 instances.
- devy 13y ago
  Hi, can you use port forwarding to get website up on those affected nodes?
jipumarino 13y ago
I got into one of our machines that presented the connectivity issues from another one which was still reachable. It had no external (curl www.google.com) connectivity. Just two minutes ago it started resolving again.
jd007 13y ago
ELBs are also having problems. One of mine is reporting all instances out of service (transient error), then all instances in service, intermittently. But the ELB is never reachable (even when it reports all instances healthy and up). All instances behind this one are reachable, up and running. US-East-1. Some of our other instances are reachable but some are not, same as others have been reporting.
jpea 13y ago
I wonder if it extends beyond Amazon, since my gmail now doesn't pull anything up after 2009, web or IMAP.
rbranson 13y ago
This appears to be connectivity issues entirely to/from the Internet or other EC2 regions from a single availability zone in us-east-1. The intra-AZ networks within us-east-1 have remained available during the event. One of the AZs we use was affected, but no external traffic flows to it. I noticed this because an auto-scale group was trying to bring up instances inside of the affected zone (our us-east-1a) and was unable to contact a server outside of AWS.
- jd007 13y ago
  Not sure if it's the entire AZ. We have some Internet-reachable instances and Internet unreachable instances in the same AZ.
- fjordan 13y ago
  We are able to reach some servers but not all in our us-east AZ and notice it seems to be random.
- cperciva 13y ago
  I'm definitely seeing issues in multiple AZs. It seems to be partly firewall-related, however: I've seen cases where it's hard to get an initial SYN through, but once a TCP connection is established it stays established.
  - fjordan 13y ago
    This, in addition to the increase in traffic we detected directly before, smells of DOS. Also, it is Friday the 13th.
tomweingarten 13y ago
Did anyone else notice a huge spike in incoming network traffic on their EC2 instance immediately before the outage? Roughly 9:55AM EST.
- sadris 13y ago
  Yes
- justinsb 13y ago
  Did it look like a ddos attack, or do you think something went wrong where you were getting traffic meant for other EC2 nodes? I'm not quite sure how you would tell the difference of course...
  - tomweingarten 13y ago
    We didn't get enough data to be able to determine that, but I'd be very curious to hear if someone else did.
- acdha 13y ago
  Nothing extraordinary but there was definitely a healthy spike in one AZ.
sadris 13y ago
Why does this never happen to AWS West? I should really get to migrating over with 3 outages in the past 2 years on US East.
- knodi 13y ago
  It does happen, you just never notice because you don't have instances in US West.
martin_ 13y ago
All of mine just started magically working
- martin_ 13y ago
  I retract that statement! http://shutter.io/img/vs6jjs/raw http://shutter.io/img/vs6jjs/raw
jday 13y ago
this has taken openredis offline: https://twitter.com/openredis https://twitter.com/openredis heroku is also reporting issues: https://status.heroku.com/incidents/554 https://status.heroku.com/incidents/554
aquark 13y ago
I'm getting external monitoring failures that are firing on and off, but have no problem reaching the servers or the site. Interestingly newrelic is reporting the site down at the same time it is reporting a normal level of load on it.
brryant 13y ago
There are definitely issues with network connectivity between AZs as well as public internet connectivity.
knodi 13y ago
Always on a Friday...
- jlgaddis 13y ago
  Not just any Friday... $ date Fri Sep 13 12:07:58 EDT 2013
  - xdissent 13y ago
    Not just any Friday the 13th... http://en.wikipedia.org/wiki/Programmers'_Day http://en.wikipedia.org/wiki/Programmers'_Day
    - TallboyOne 13y ago
      Not just any Friday the 13th Programmer's Day... http://www.holidayinsights.com/other/fortunecookie.htm http://www.holidayinsights.com/other/fortunecookie.htm
TallboyOne 13y ago
Aaand were beck up now.
joe010 13y ago
I've recently moved some of our servers over to Digital Ocean, but I'm still using AWS for DNS since their Route 53 weighted DNS with health checks work as a basic load balancer for our needs. I'm seeing DNS health checks that point at individual servers at Digital Ocean that are showing 0.91 for a status (1 being up and 0 being down. The alarms attached to the health checks keep flipping from "alarm" to "ok" and causing tons of alerts. As of about 15 minutes ago all of my checks started holding steady back at a status of 1 (ok) Good stuff :)
ihaveajob 13y ago
It looks ok now for us (appfluence.com), but even when it was down, our website was still up, only the sync services went offline. And even then, they were accessible from the web server...

77 points by fjordan 13y ago

rschmitty 13y ago

Does anyone know why in the world they display a green checkmark with a near invisible little 'i' for this?

aroch 13y ago
Because checkmarks make everything look fine...
- jordanmessina 13y ago
  http://www.youtube.com/watch?v=uriZZ3Slz74 http://www.youtube.com/watch?v=uriZZ3Slz74
- ceph_ 13y ago
  Non-snarky answer: They want to post that they are investigating an issue, but do not want to comment on the scope of the problem when it is still not fully understood.
teoruiz 13y ago
They switch to white-triangle-in-yellow-circle when they find out that there is a real problem. They just switched.
iota 13y ago
There are 4 statuses. Green checkmark (status 0) Green checkmark with "info" badge (status 1) Yellow triangle (status 2) Red "do not enter" rectangle (status 3) I suspect that status 0 indicates that they are investigating a problem with the server, and it switches to status 1 once the problem has been confirmed. This is also a good example of poor icon design...they aren't self-explanatory, and so they should not be used.
- cperciva 13y ago
  What happened to status 2?
  - iota 13y ago
    Good catch. Fixed!
- rschmitty 13y ago
  Thanks!

o0-0o 13y ago

Down in Manhattan

jolan 13y ago

Amazon is continuing the trend of announcing outages 30 minutes after they start. Just signed up for a support contract since the status page said everything was fine.

colinbartlett 13y ago
And by "announcing" you mean indicating everything is a-okay with the green checkmark but putting a tiny footnote next to it.
acdha 13y ago
“outage (n): a period when a power supply or other service is not available or when equipment is closed down” A subset of traffic taking longer than normal is not an outage

scrabble 13y ago

So what is the best way to balance a hosted site between Amazon and a separate service? Because these connectivity issues suck.

bredman 13y ago
One option would be to use Route 53 weighted round robin (WRR) DNS records and health checks to accomplish this.

NotDaveLane 13y ago

It's region us-east-1c for now, at least from where I'm sitting... I have instances in other us-east datacenters that are fine.

lreeves 13y ago
Availability zone letters are specific to your account.
trevyn 13y ago
Specific availability zones in a region are mapped per-account, so your east-1c might be my east-1a: "To ensure that resources are distributed across the Availability Zones for a region, we independently map Availability Zones to identifiers for each account. For example, your Availability Zone us-east-1a might not be the same location as us-east-1a for another account. Note that there's no way for you to coordinate Availability Zones between accounts." http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...
- ceejayoz 13y ago
  I wonder how that works with new zones. I remember us-east-1e being added separately to the original four. Presumably, that one's the same for all accounts that'd already signed up at the time.
- astrodust 13y ago
  Clever and ridiculous at the same time.
deleted 13y ago
[deleted]

frabcus 13y ago

We (ScraperWiki) can still access some of our US East servers. From those, can daisy chain SSH into the ones that are offline. Those servers can't see the world, but are working fine and can see other EC2 instances.

devy 13y ago
Hi, can you use port forwarding to get website up on those affected nodes?

jipumarino 13y ago

I got into one of our machines that presented the connectivity issues from another one which was still reachable. It had no external (curl www.google.com) connectivity. Just two minutes ago it started resolving again.

jd007 13y ago

ELBs are also having problems. One of mine is reporting all instances out of service (transient error), then all instances in service, intermittently. But the ELB is never reachable (even when it reports all instances healthy and up). All instances behind this one are reachable, up and running. US-East-1. Some of our other instances are reachable but some are not, same as others have been reporting.

jpea 13y ago

I wonder if it extends beyond Amazon, since my gmail now doesn't pull anything up after 2009, web or IMAP.

rbranson 13y ago

This appears to be connectivity issues entirely to/from the Internet or other EC2 regions from a single availability zone in us-east-1. The intra-AZ networks within us-east-1 have remained available during the event. One of the AZs we use was affected, but no external traffic flows to it. I noticed this because an auto-scale group was trying to bring up instances inside of the affected zone (our us-east-1a) and was unable to contact a server outside of AWS.

jd007 13y ago
Not sure if it's the entire AZ. We have some Internet-reachable instances and Internet unreachable instances in the same AZ.
fjordan 13y ago
We are able to reach some servers but not all in our us-east AZ and notice it seems to be random.
cperciva 13y ago
I'm definitely seeing issues in multiple AZs. It seems to be partly firewall-related, however: I've seen cases where it's hard to get an initial SYN through, but once a TCP connection is established it stays established.
- fjordan 13y ago
  This, in addition to the increase in traffic we detected directly before, smells of DOS. Also, it is Friday the 13th.

tomweingarten 13y ago

Did anyone else notice a huge spike in incoming network traffic on their EC2 instance immediately before the outage? Roughly 9:55AM EST.

sadris 13y ago
Yes
justinsb 13y ago
Did it look like a ddos attack, or do you think something went wrong where you were getting traffic meant for other EC2 nodes? I'm not quite sure how you would tell the difference of course...
- tomweingarten 13y ago
  We didn't get enough data to be able to determine that, but I'd be very curious to hear if someone else did.
acdha 13y ago
Nothing extraordinary but there was definitely a healthy spike in one AZ.

sadris 13y ago

Why does this never happen to AWS West? I should really get to migrating over with 3 outages in the past 2 years on US East.

knodi 13y ago
It does happen, you just never notice because you don't have instances in US West.

martin_ 13y ago

All of mine just started magically working

martin_ 13y ago
I retract that statement! http://shutter.io/img/vs6jjs/raw http://shutter.io/img/vs6jjs/raw

jday 13y ago

this has taken openredis offline: https://twitter.com/openredis https://twitter.com/openredis heroku is also reporting issues: https://status.heroku.com/incidents/554 https://status.heroku.com/incidents/554

aquark 13y ago

I'm getting external monitoring failures that are firing on and off, but have no problem reaching the servers or the site. Interestingly newrelic is reporting the site down at the same time it is reporting a normal level of load on it.

brryant 13y ago

There are definitely issues with network connectivity between AZs as well as public internet connectivity.

knodi 13y ago

Always on a Friday...

jlgaddis 13y ago
Not just any Friday... $ date Fri Sep 13 12:07:58 EDT 2013
- xdissent 13y ago
  Not just any Friday the 13th... http://en.wikipedia.org/wiki/Programmers'_Day http://en.wikipedia.org/wiki/Programmers'_Day
  - TallboyOne 13y ago
    Not just any Friday the 13th Programmer's Day... http://www.holidayinsights.com/other/fortunecookie.htm http://www.holidayinsights.com/other/fortunecookie.htm

TallboyOne 13y ago

Aaand were beck up now.

joe010 13y ago

I've recently moved some of our servers over to Digital Ocean, but I'm still using AWS for DNS since their Route 53 weighted DNS with health checks work as a basic load balancer for our needs. I'm seeing DNS health checks that point at individual servers at Digital Ocean that are showing 0.91 for a status (1 being up and 0 being down. The alarms attached to the health checks keep flipping from "alarm" to "ok" and causing tons of alerts. As of about 15 minutes ago all of my checks started holding steady back at a status of 1 (ok) Good stuff :)

ihaveajob 13y ago

It looks ok now for us (appfluence.com), but even when it was down, our website was still up, only the sync services went offline. And even then, they were accessible from the web server...