4 ms·

Ok. If you're receiving errors, and you're NOT using opsworks, please respond. We're using opsworks too and have ~30 servers down. Maybe we all should be lookin

by kacy 11y ago

Ok. If you're receiving errors, and you're NOT using opsworks, please respond. We're using opsworks too and have ~30 servers down. Maybe we all should be looking at the opsworks agent.

codingninja 11y ago
We are getting errors across multiple AWS API's. It's nothing to do with Opsworks itself, rather it appears like there is an internal networking issue. Both SQS and SNS were erroring and now SQS has gone down completly with all requests timing out.
- deleted 11y ago
  [deleted]
- oliyoung 11y ago
  Looks like there's critical infrastructure in us-east-1 that's broken and causing a ripple effect across all of AWS Our platform is entirely hosted in ap-southeast-2 but we've had our EC2 instances deregistered and OpsWorks reporting them terminated where EC2 is showing them active and they're still reachable via SSH
  - frankchn 11y ago
    Yeah, we don't use OpsWorks and had SQS/SNS/SES trouble as well -- thankfully those are not used to serve production traffic. From the set of services affected, it looks like Amazon's internal Kafka-like pub/sub system went down.
warren_s 11y ago
We started getting SQS errors from us-east-1 around 05:08 UTC. We also have a bunch of files on S3 which Cross-Region Replication hasn't yet replicated... I think that depends on SQS as well.
- robotpony 11y ago
  Can confirm. We were also seeing SQS errors around that time. We had some Opsworks instances reboot as well, but only minor outages overall (us-east and us-west).
vonklaus 11y ago
having issues. Not using opsworks. It is not limited to that.
sdrothrock 11y ago
SES errors, no opsworks.
vacri 11y ago
I lost my buildserver - uncontactable, but not terminated. Can't even 'force off'. It had nothing to do with any form of AWS provisioning (manual ansible; a job for Monday), and it's a relatively recent machine (couple of months, t2.medium). Got an email from AWS that the host had degraded, and I noticed the instance was having weird disk issues earlier. "Description: The instance is running on degraded hardware"

by kacy 11y ago

Ok. If you're receiving errors, and you're NOT using opsworks, please respond. We're using opsworks too and have ~30 servers down. Maybe we all should be looking at the opsworks agent.

codingninja 11y ago

We are getting errors across multiple AWS API's. It's nothing to do with Opsworks itself, rather it appears like there is an internal networking issue. Both SQS and SNS were erroring and now SQS has gone down completly with all requests timing out.

deleted 11y ago
[deleted]
oliyoung 11y ago
Looks like there's critical infrastructure in us-east-1 that's broken and causing a ripple effect across all of AWS Our platform is entirely hosted in ap-southeast-2 but we've had our EC2 instances deregistered and OpsWorks reporting them terminated where EC2 is showing them active and they're still reachable via SSH
- frankchn 11y ago
  Yeah, we don't use OpsWorks and had SQS/SNS/SES trouble as well -- thankfully those are not used to serve production traffic. From the set of services affected, it looks like Amazon's internal Kafka-like pub/sub system went down.

warren_s 11y ago

We started getting SQS errors from us-east-1 around 05:08 UTC. We also have a bunch of files on S3 which Cross-Region Replication hasn't yet replicated... I think that depends on SQS as well.

robotpony 11y ago
Can confirm. We were also seeing SQS errors around that time. We had some Opsworks instances reboot as well, but only minor outages overall (us-east and us-west).

vonklaus 11y ago

having issues. Not using opsworks. It is not limited to that.

sdrothrock 11y ago

SES errors, no opsworks.

vacri 11y ago

I lost my buildserver - uncontactable, but not terminated. Can't even 'force off'. It had nothing to do with any form of AWS provisioning (manual ansible; a job for Monday), and it's a relatively recent machine (couple of months, t2.medium). Got an email from AWS that the host had degraded, and I noticed the instance was having weird disk issues earlier. "Description: The instance is running on degraded hardware"