10 ms·

EC2 Maintenance Update II

108 points by jeffbarr 12y ago

thom_nic 12y ago
So this had nothing to do with Shellshock then?
- AznHisoka 12y ago
  they said it was for Xen before last weekend, no? (didnt read the article, assume they said it was for xen)
- davidradcliffe 12y ago
  Correct
- kbenson 12y ago
  Shellshock, being a bash bug (and affecting systems that use bash for shell commands, etc), is OS and/or application level, and the responsibility of the individual customers to handle (as they are responsible for administration of the instances they run). The Xen bug affects how the hardware is virtualized, and this Amazon's responsibility.
captainchaos 12y ago
This explains some weird EC2 stuff we saw yesterday. A good reminder to check up that a box starts up as expected (mounts disks, starts services, etc). It's easy to get lazy about that sort of thing when you're moving fast.
- nnx 12y ago
  Same here, the VM rebooted and restarted all services correctly but the connectivity was failing intermittently. Not fun. Concluded we better just start a new VM... and it worked. Next time there is such scheduled event, I guess being proactive and creating a new VM beforehand is the better solution.
tptacek 12y ago
This is a better announcement than Rackspace's: no spin, direct, to the point.
- ericras 12y ago
  "An Apology": https://community.rackspace.com/general/f/34/t/4341 https://community.rackspace.com/general/f/34/t/4341
  - tptacek 12y ago
    This sentence, early in that post, is where my spin shields went up: Now that this issue has been fully remediated, without any reports of compromised data among our customers, I’d like to explain what happened, and why. [em mine]
  - nthj 12y ago
    You'd think Rackspace's CEO would have at least heard of mail merge, over "Dear Rackspace Customer."
freshflowers 12y ago
"Pay attention to your Inbox and to the alerts on the AWS Management Console." Especially with incidents like these (and other cases where instances are scheduled to be taken down), it really annoys me that AWS doesn't offer any push alerts besides emailing the account owner.
- jeffbarr 12y ago
  What kinds of alerts would work for you? Let me know and I will pass them along to the team.
  - danmactough 12y ago
    I would like to be able to have alerts go right into SNS.
    - dlg 12y ago
      More generally, it would be cool if every account automatically got two free read-only SQS queues: one for events and one for upcoming events. Every known event (startup, terminate, permission change, network error, etc) could be published to the queue. Not high-priority for us but the parent comment sparked the idea (I'm not in devops so maybe this exists via a different mechanism.)
      - jeffbarr 12y ago
        These are both great ideas. I will share them with the team today. Keep them coming!
        mh- 12y ago
        seems like realtime CloudTrail->{SQS,SNS} could satisfy his request and several that I have. (instead of waiting for events to batch from CloudTrail to S3)
    - verelo 12y ago
danmactough 12y ago
We noticed that all of our instances -- in 3 different availability zones in US East -- all cycled at the same time. That was pretty disappointing. Kind of defeated the purpose of having things in different AZs.
- jeffbarr 12y ago
  We did a zone-by-zone reboot. If you want to send me the instance ids I will ask our team to see what happened. You can find my email in my profile.
  - voidlogic 12y ago
    How much delay is between zones? Some peoples services don't come up instantly. Perhaps when zone 3 when down, the users services in zone 1 hadn't finished coming back up.
    - zwily 12y ago
      They did a different zone every day, so ~24 hours.
- kordless 12y ago
  > you may also want to take this opportunity to re-examine your AWS architecture to look for possible ways to make it even more fault-tolerant Basically they won't allow blame to be placed on them for anything they do. According to AWS, it's your fault this happened.
  - akerl_ 12y ago
    Amazon had to perform the maintenance due to XSA 108, and the timetable they had to meet was set by the Xen project. They set up the maintenance to have as little impact as possible by splitting availability zones onto separate days so that people utilizing multiple zones for high-availability would not lose multiple zones at once. Afterwards, they gave a detailed breakdown, linking to the vulnerability and explaining both why they had to perform maintenance and why they could not share more details upfront. They also provided information on how to improve the fault-tolerance of your systems so that future issues like this won't stop your workflow. Amazon isn't placing "fault" on users; they did a pretty stellar job of handling a nasty vulnerability and treating their users as well as possible.
theonewolf 12y ago
@jeffbarr I just wanted to say thanks for not only posting this, but also sticking around in the comments section. Makes you + Amazon seem way more human :-) Also, in case you have any "cloud servers" you want to decommission: https://news.ycombinator.com/item?id=8373394 https://news.ycombinator.com/item?id=8373394
- jeffbarr 12y ago
  I am always happy to help, time and circumstances allowing. Before joining Amazon I earned my living by consulting for startups. I could always tell when they were about to run out of money when they would offer to pay me in servers. This was always the cue to find my next gig.
  - samstave 12y ago
    I would like to request, that if you're going to suggest: "Run instances in two or more Availability Zones." as a fault-tolerant architecture method (which everyone SHOULD do) - can we please have a flag or tag to say that "Traffic between Zone-1-HOST-A and Zone-2-HOST-A" is specifically resiliancy/HA/fault-tolerant connection traffic could benefit from a discount in zone transfer fees? We are moving many hundreds servers between zones every day and sometimes by the hour, specifically to handle all sorts of constraints (Spot, capacity, failure, load, etc)... I have paid quite a lot to zone transfer fees, especially when dealing with AWS network issues or spot price insanity. I would love to classify some traffic as resiliency traffic and be charged differently for that traffic as opposed to general traffic used to service my user-base.
    - snacksmasters 12y ago
      As long as you are using the private IP or public DNS name outbound traffic between EC2 instances in different AZ's in the same region is $.01/GB, inbound is free. Pricing is different for traffic going outside the EC2 network.
    - jeffbarr 12y ago
      There already is a tiered pricing model for this. Data that goes between AZ's is charged at $0.01 / GB. Data that flows from an EC2 instance to the Internet is charged at $0.12 / GB (for up to 10 TB / month, with discounts from there, and there's also 1 GB / month of free traffic). Details are on http://aws.amazon.com/ec2/pricing/ http://aws.amazon.com/ec2/pricing/
amaks 12y ago
No live migration huh?
- toomuchtodo 12y ago
  If you've built your environment properly, you don't require live migration.
  - amaks 12y ago
    That's not true. Hardware where VMs are hosted requires repairs from time to time, so VMs do require migration.
    - toomuchtodo 12y ago
      Stop instance, start instance, land on new hardware (AWS EC2, with EBS storage). If your application requires a hot migration, you've failed, as it can't tolerate failures inherent to a virtualized environment.
lubos 12y ago
One of my instances was scheduled to be rebooted between 22pm-2am. But the reboot has never actually occurred. It's not that I feel left out but did anyone else experience the same?
rubyrescue 12y ago
We had 50 servers reboot and 15... never came back. It was painful. 10 were in one AZ. We're putting servers in more AZs now, but preparing for a reboot and preparing to rebuild a redis slave, 2 zookeepers, 10 cassandras, and an API server are different things.

108 points by jeffbarr 12y ago

thom_nic 12y ago

So this had nothing to do with Shellshock then?

AznHisoka 12y ago
they said it was for Xen before last weekend, no? (didnt read the article, assume they said it was for xen)
davidradcliffe 12y ago
Correct
kbenson 12y ago
Shellshock, being a bash bug (and affecting systems that use bash for shell commands, etc), is OS and/or application level, and the responsibility of the individual customers to handle (as they are responsible for administration of the instances they run). The Xen bug affects how the hardware is virtualized, and this Amazon's responsibility.

captainchaos 12y ago

This explains some weird EC2 stuff we saw yesterday. A good reminder to check up that a box starts up as expected (mounts disks, starts services, etc). It's easy to get lazy about that sort of thing when you're moving fast.

nnx 12y ago
Same here, the VM rebooted and restarted all services correctly but the connectivity was failing intermittently. Not fun. Concluded we better just start a new VM... and it worked. Next time there is such scheduled event, I guess being proactive and creating a new VM beforehand is the better solution.

tptacek 12y ago

This is a better announcement than Rackspace's: no spin, direct, to the point.

ericras 12y ago
"An Apology": https://community.rackspace.com/general/f/34/t/4341 https://community.rackspace.com/general/f/34/t/4341
- tptacek 12y ago
  This sentence, early in that post, is where my spin shields went up: Now that this issue has been fully remediated, without any reports of compromised data among our customers, I’d like to explain what happened, and why. [em mine]
- nthj 12y ago
  You'd think Rackspace's CEO would have at least heard of mail merge, over "Dear Rackspace Customer."

freshflowers 12y ago

"Pay attention to your Inbox and to the alerts on the AWS Management Console." Especially with incidents like these (and other cases where instances are scheduled to be taken down), it really annoys me that AWS doesn't offer any push alerts besides emailing the account owner.

jeffbarr 12y ago
What kinds of alerts would work for you? Let me know and I will pass them along to the team.
- danmactough 12y ago
  I would like to be able to have alerts go right into SNS.
  - dlg 12y ago
    More generally, it would be cool if every account automatically got two free read-only SQS queues: one for events and one for upcoming events. Every known event (startup, terminate, permission change, network error, etc) could be published to the queue. Not high-priority for us but the parent comment sparked the idea (I'm not in devops so maybe this exists via a different mechanism.)
    - jeffbarr 12y ago
      These are both great ideas. I will share them with the team today. Keep them coming!
      - mh- 12y ago
        seems like realtime CloudTrail->{SQS,SNS} could satisfy his request and several that I have. (instead of waiting for events to batch from CloudTrail to S3)
  - verelo 12y ago

danmactough 12y ago

We noticed that all of our instances -- in 3 different availability zones in US East -- all cycled at the same time. That was pretty disappointing. Kind of defeated the purpose of having things in different AZs.

jeffbarr 12y ago
We did a zone-by-zone reboot. If you want to send me the instance ids I will ask our team to see what happened. You can find my email in my profile.
- voidlogic 12y ago
  How much delay is between zones? Some peoples services don't come up instantly. Perhaps when zone 3 when down, the users services in zone 1 hadn't finished coming back up.
  - zwily 12y ago
    They did a different zone every day, so ~24 hours.
kordless 12y ago
> you may also want to take this opportunity to re-examine your AWS architecture to look for possible ways to make it even more fault-tolerant Basically they won't allow blame to be placed on them for anything they do. According to AWS, it's your fault this happened.
- akerl_ 12y ago
  Amazon had to perform the maintenance due to XSA 108, and the timetable they had to meet was set by the Xen project. They set up the maintenance to have as little impact as possible by splitting availability zones onto separate days so that people utilizing multiple zones for high-availability would not lose multiple zones at once. Afterwards, they gave a detailed breakdown, linking to the vulnerability and explaining both why they had to perform maintenance and why they could not share more details upfront. They also provided information on how to improve the fault-tolerance of your systems so that future issues like this won't stop your workflow. Amazon isn't placing "fault" on users; they did a pretty stellar job of handling a nasty vulnerability and treating their users as well as possible.

theonewolf 12y ago

@jeffbarr I just wanted to say thanks for not only posting this, but also sticking around in the comments section. Makes you + Amazon seem way more human :-) Also, in case you have any "cloud servers" you want to decommission: https://news.ycombinator.com/item?id=8373394 https://news.ycombinator.com/item?id=8373394

jeffbarr 12y ago
I am always happy to help, time and circumstances allowing. Before joining Amazon I earned my living by consulting for startups. I could always tell when they were about to run out of money when they would offer to pay me in servers. This was always the cue to find my next gig.
- samstave 12y ago
  I would like to request, that if you're going to suggest: "Run instances in two or more Availability Zones." as a fault-tolerant architecture method (which everyone SHOULD do) - can we please have a flag or tag to say that "Traffic between Zone-1-HOST-A and Zone-2-HOST-A" is specifically resiliancy/HA/fault-tolerant connection traffic could benefit from a discount in zone transfer fees? We are moving many hundreds servers between zones every day and sometimes by the hour, specifically to handle all sorts of constraints (Spot, capacity, failure, load, etc)... I have paid quite a lot to zone transfer fees, especially when dealing with AWS network issues or spot price insanity. I would love to classify some traffic as resiliency traffic and be charged differently for that traffic as opposed to general traffic used to service my user-base.
  - snacksmasters 12y ago
    As long as you are using the private IP or public DNS name outbound traffic between EC2 instances in different AZ's in the same region is $.01/GB, inbound is free. Pricing is different for traffic going outside the EC2 network.
  - jeffbarr 12y ago
    There already is a tiered pricing model for this. Data that goes between AZ's is charged at $0.01 / GB. Data that flows from an EC2 instance to the Internet is charged at $0.12 / GB (for up to 10 TB / month, with discounts from there, and there's also 1 GB / month of free traffic). Details are on http://aws.amazon.com/ec2/pricing/ http://aws.amazon.com/ec2/pricing/

amaks 12y ago

No live migration huh?

toomuchtodo 12y ago
If you've built your environment properly, you don't require live migration.
- amaks 12y ago
  That's not true. Hardware where VMs are hosted requires repairs from time to time, so VMs do require migration.
  - toomuchtodo 12y ago
    Stop instance, start instance, land on new hardware (AWS EC2, with EBS storage). If your application requires a hot migration, you've failed, as it can't tolerate failures inherent to a virtualized environment.

lubos 12y ago

One of my instances was scheduled to be rebooted between 22pm-2am. But the reboot has never actually occurred. It's not that I feel left out but did anyone else experience the same?

rubyrescue 12y ago

We had 50 servers reboot and 15... never came back. It was painful. 10 were in one AZ. We're putting servers in more AZs now, but preparing for a reboot and preparing to rebuild a redis slave, 2 zookeepers, 10 cassandras, and an API server are different things.