12 ms·

AWS issues unavoidable reboot schedules with short notice on many EC2 instances

• robszumski 12y ago
  I'm assuming this is due to fallout from the recent bash security issue?
  • vetrom 12y ago
    Swapping bash should not require reboots. The pending XSA-108 (http://xenbits.xen.org/xsa/ http://xenbits.xen.org/xsa/) however....
    • jagger27 12y ago
      How often do Xen vulnerabilities get embargoed?
      • MertsA 12y ago
        If you look at the public release column you'll see the following date: 2014-10-01 12:00. I'm not sure when the vulnerability was added to the list or what timezone that is referring to but it looks like we'll find out what it is in a couple days.
    • adrr 12y ago
      It has to be that. Can't think of another reason why you need to mass reboot the servers the instances are running on and serious enough they are patching through the weekend. Wonder if its being exploited in the wild. Also none of our C3.2Xlarges are getting rebooted, those use SR-IOV which can virtualize PCIE devices at the hardware level so it probably a network boundary exploit allow you to pull other instance traffic.
    • deleted 12y ago
      [deleted]
  • viraptor 12y ago
    That's unlikely. You can update bash without any restarts. This sounds like an architecture change / global hardware (firmware?) update / kernel upgrade which doesn't allow nice migration to/from the hosts.
  • takeda 12y ago
    There is Xen exploit that will be disclosed on October 1st. Amazon most likely already know what it is.
• Corrado 12y ago
  They say that its fine to reboot but that there is no guarantee that you will land on an updated host. However, AWS does provide a script to run on Windows machines which should tell you if that particular machine has the issue. I took a quick look at the script and deduced that it is indeed a Xen issue.
  • akerl_ 12y ago
    It seems like providing a script that identifies if your system is vulnerable to an embargoed XSA would be a violation of the predisclosure list, since it would basically be pointing at what the issue was?
    • wmf 12y ago
      How does one bit of information (vulnerable / not vulnerable) tell you what the vulnerability is?
      • akerl_ 12y ago
        It's not the results of the script that I'm referring to, it's the contents of the script. If I hand you code that can look at your system and determine something about it, you can look at what the code is doing and identify what it is looking at, which tells you where the vulnerability is. It becomes moderately harder if it's compiled code, but still not very difficult.
        • pfg 12y ago
          Unless the fixed version of Xen exposes some kind of signal to its VMs that says "Hi, I'm not affected!" - which then could be read by this script. I don't know if and how that's possible with Xen though.
        • x0x0 12y ago
          So make the code query an opaque EC2 api, instead of testing the machine. You could still find a machine that is vulnerable and one that isn't and attempt to find out what the difference is, but that's a much harder task.
• deleted 12y ago
  [deleted]
• nraynaud 12y ago
  I love how the amazon support person is absolutely not answering the very precise question.
  • vinceguidry 12y ago
    Heh, I thought the answer was very clear: no.
    • nraynaud 12y ago
      I didn't read "no" in the answer, just circonvolutions around: "it's no but we're mangling it, plus we're not changing our course after your feedback". A clear answer would be: "fuck you".
• quink 12y ago
  EC2 screenshot: http://i.imgur.com/OdCehey.png http://i.imgur.com/OdCehey.png And copy of the email that was sent out to us... ---- Dear Amazon EC2 Customer, One or more of your Amazon EC2 instances are scheduled to be rebooted for required host maintenance. The maintenance will occur sometime during the window provided for each instance. Each instance will experience a clean reboot and will be unavailable while the updates are applied to the underlying host. This generally takes no more than a few minutes to complete. Each instance will return to normal operation after the reboot, and all instance configuration and data will be retained. If you have startup procedures that aren’t automated during your instance boot process, please remember that you will need to log in and run them. We will need to do this maintenance update in the window provided. You will not be able to stop/start or re-launch instances in order to avoid this maintenance update. If you are using Windows Server 2012 R2, please follow the instructions found here: http://aws.amazon.com/windows/2012r2-network-drivers/ http://aws.amazon.com/windows/2012r2-network-drivers/ to ensure that your instance continues to have network connectivity after reboot. This requires that you run a remediation script in order to ensure continued access to your instance. Additional information about Amazon EC2 maintenance can be found at: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/monitoring-instances-status-check_sched.html http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/mo... If you have any questions or concerns, you can contact the AWS Support Team on the community forums and via AWS Premium Support at: http://aws.amazon.com/support http://aws.amazon.com/support. Sincerely, Amazon Web Services To view your instances that are scheduled for reboot, please visit the 'Events' page on the EC2 console: https://console.aws.amazon.com/ec2 https://console.aws.amazon.com/ec2 This message was produced and distributed by Amazon Web Services LLC, 410 Terry Avenue North, Seattle, Washington 98109-5210.
• ecaron 12y ago
  So far I'm 3 for 3 on stopped/started machines that come back no longer having the scheduled reboot event - maybe the problem isn't as annoying as we're thinking? (Edited because I said "reboot" and meant "stop/start")
  • thezilch 12y ago
    The post claims that it could take several hours before their scans identify hosts that were claimed during restarts.
  • rands3311 12y ago
    A reboot won't address it at all (it also keeps you on the same host).. no point in doing that. From the e-mail notification: "You will not be able to stop/start or re-launch instances in order to avoid this maintenance update."
    • ecaron 12y ago
      Sorry, I should have phrased better. I issued a "stop and start". Per the AWS thread: > While executing a stop/start is absolutely fine to do, there is not a guarantee that you will land on an updated host. We are periodically polling for new instances and those impacted will receive new maintenance notifications accordingly. Which I take to mean there's a chance you could land on an updated host. I'll update if any of those machines come back to the Events view.
      • deleted 12y ago
        [deleted]
• akh 12y ago
  This isn't a direct answer, but for others who are coming to this page and need help on where to start, here is some info on what to do now from RightScale's CTO, it also has a little comparison to what happened in Dec 2011 reboot too (just an interesting side note): http://www.rightscale.com/blog/rightscale-news/aws-reboot-substantial-number-ec2-instances http://www.rightscale.com/blog/rightscale-news/aws-reboot-su...
• vacri 12y ago
  There's a possibility that this issue might be related to HVM. All of our AWS systems are on older, non-hvm instance types, and none have been rebooted, and there are no maintenance events listed. Friends who are using newer instances (which are all hvm) are reporting the reboot issues. Anecdata, but a possible explanation.
  • chillericed 12y ago
    this has been true for us so far as well. But I wouldn't rule out that they have more scheduled maintenances that haven't been put out yet.
  • nkozyra 12y ago
    Concur, though also anecdotal.
  • pjl 12y ago
    I don't think the issue is related to HVM. I have instances scheduled for system reboot that are not HVM.
  • Xorlev 12y ago
    We have a lot of non-HVM scheduled for reboot. One of them was our VPN server, scheduled for reboot during our tech demo.
• deleted 12y ago
  [deleted]
• rurounijones 12y ago
  Pretty crappy that you cannot immediately check if an instance has restarted on a patched host or not. I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. Having to wait possibly hours for an email to see if your vm migrated to patched host or not is a terrible solution.
  • iaw 12y ago
    IMO, The problem is that the manager of the cloud product isn't the person that is ordering the reboot. The manager of the cloud product considers communication with customers to mitigate their issues a primary concern. Person ordering the reboot is more 'technically' focused and representative of why engineers have a hard time as managers (eg ignoring the potential ill will this type of action can create).
  • toomuchtodo 12y ago
    > I would have expected Amazon to rush out a tool you can use to check or add a little marker to the dashboard or a simple API to query. Some sort of synchronous option. I would expect Amazon to mark unpatched hosts as bad and not permit new instances to be deployed to them, similar to queue draining. Not cool Amazon.
    • rurounijones 12y ago
      Good point, I wonder if they didn't because that would have crippled capacity due to people rebooting, using up the patched hosts and leaving others without the ability to spin up VMs as it sounds like a significant % of hosts are bad at this point..
    • rands3311 12y ago
      Ideally, that would be great. I doubt that would be an option though due to capacity. Would you rather stop/start your instance and risk a capacity error or have your impacted instances rebooted in 48 hours?
      • 12y ago
• omni 12y ago
  I just got notified that my Elasticache instances are getting restarted, too.
• contingencies 12y ago
  While people may be painting Amazon in a bad light here, the business-level risk of wholly committing to a single infrastructure provider (cloud or otherwise, across multiple 'availability zones' or data centers or countries or continents, or otherwise) is real. There is a clear need for many service authors to work with disparate infrastructure in a cloud provider and platform abstracted manner, and arguably no solid tools for doing it right now. What will the future look like? I believe that a standard, git-like command line tool for infrastructure, sort of a 'brandless ec2' or 'P-abstracted IaaS' version of heroku, will replace all of the current-era providers with a free market for infrastructure based on transparency, real uptime and performance analysis and incident observations by multiple third parties with cryptographic reputation management. Two angles converging on that at http://stani.sh/walter/pfcts http://stani.sh/walter/pfcts and http://ifex-project.org/ http://ifex-project.org/
  • lucian1900 12y ago
    OpenStack has some of those tools and some even have AWS backend shims. I believe Rackspace are one of those involved in its development.
  • irollboozers 12y ago
    Curious, what do large scale companies in the cloud currently do for this?
  • btown 12y ago
    Is Rightscale's multi-cloud offering a step in the direction you're advocating? http://assets.rightscale.com/uploads/pdfs/RightScale-Technical-Overview.pdf http://assets.rightscale.com/uploads/pdfs/RightScale-Technic...
    • contingencies 12y ago
      Kind of. At a glance, it's new, commercial and they gloss over the complexities... therefore I'm skeptical it really works as well as they say it does, and is leaning toward my 'untrustworthy as a long term platform' basket. Though they may have great tools, I believe history shows us that open source is the real way to resolve these very reasonable types of architectural concerns.
• fletchowns 12y ago
  On AWS you should be prepared for an instance to disappear at any time, for any reason. Why is a scheduled reboot such a big deal?
  • omni 12y ago
    It's a big deal because about half of my 100 instances are all going down at roughly the same time. Distribution and replication save you if you have 3 boxes and 1 dies. If all 3 die at the same time, you're still screwed.
    • all_usernames 12y ago
      Each Availability Zone is being rebooted on a different day. Best practices dictate HA clusters with >=1 instance in each AZ. So, in theory well-designed EC2 systems can withstand this without interruption. But yeah, this still sucks.
      • omni 12y ago
        That does seem to be the case, that is a relief.
      • eropple 12y ago
        One thing I'd add: best practices (IMO) dictate HA clusters as you describe, but you get a big boost to survivability by deciding on only using instance stores. Network issues have screwed EBS in the past; EBS is technically neat but very network-sensitive and it's possible to "lose" part of your EBS volume because part of the network goes away (and then your instance faceplants). Instance stores are your friend, and acutely knowing they can disappear in an eyeblink will make you design a better system. One that can survive you having your instances forcibly retired by AWS. :-)
        • toomuchtodo 12y ago
          You use instance stores for persistent data. That instance disappears. Where are you restoring that data from? Either your backups are stale, or you were replicating the data or its underlying filesystem, which means you're still reliant on the network.
• akurilin 12y ago
  Will new instances created now not have to be rebooted? A lot of people on EC2 can probably re-create their vms now instead of having to wait for the reboot, no?
  • alanning 12y ago
    At the time of writing, no. See the linked discussion. The real issue here, as the RightScale guys point out, is that there is no way to reliably provision a patched instance. Although the comment in the discussion forum by the EC2 fleet manager says they are working on a tool that will let us know whether an instance is fixed or not.
• gpjonesii 12y ago
  A couple of points: Autoscaling is your friend. If you're not leveraging it (multiple availability zones), you're doing it wrong. Even single instances can be launched in autoscaling groups with a desired capacity of 1 to ensure that if it falls over, a new one is spun up. Point 2: AWS is likely trying to rotate capacity for updates, which means they need to evict instances. That are running on doms that they need to update/deprecate/etc. The longer your instances are running (or the more specialized the type of instance is), the more likely you'll see an eviction notice. It should be part of a good practice to launch new instances often as new AMIs become available, or as private AMIs are updated for security patches, etc. - at least monthly! Autoscaling and solid config management simplifies this practice greatly. Good Luck!
  • spullara 12y ago
    They are not rotating capacity for updates. They are patching a Xen security issue that will be announced on Oct 1. That is why they are rebooting machines and not forcing moves off of those machines. Otherwise, I agree with the advice.
    • deleted 12y ago
      [deleted]
    • coolbeans01 12y ago
      Could you please confirm or provide evidence for such speculation? I don't see anything about this on google. I heard it is because they are having power issues within their datacenter.
      • jabo 12y ago
        May be this: http://xenbits.xen.org/xsa/ http://xenbits.xen.org/xsa/ XSA-108 2014-10-01 12:00 (Prereleased, but embargoed)
        • coolbeans01 12y ago
          Cool speculation! Anyone with anything concrete? I still heard it is power issues.
• rrggrr 12y ago
  This feels like a critical security or stability update.
  • ianlevesque 12y ago
    XSA-108 perhaps? http://xenbits.xen.org/xsa/ http://xenbits.xen.org/xsa/ AWS is certainly large enough to have access to an embargoed update.
    • akerl_ 12y ago
      The list of people receiving pre-disclosure access is public: http://www.xenproject.org/security-policy.html http://www.xenproject.org/security-policy.html
• monjinan 12y ago
  I don't think a lot of people are really understanding how much of a larger issue this would be for us if AWS didn't patch a major security issue before it was made public. The company has treated me very well over the years, from AWS to retail. My stuff arrives on time, if it doesn't I get reimbursed, most of the time with an extra few bucks for my trouble. The AWS platform is more mature and feature rich than anyone else and keeps getting better. These reboots are going to save a lot of peoples butts, they wouldn't give a 48 hour notice if they had a better option. That being said, I'm very curious to see what info is released next week.
  • dvidsilva 12y ago
    yeah, we had a few minutes downtime but nothing too bad; definitely rather have some few time off than a security threat.
  • fleitz 12y ago
    Yup, fundamentally it's not following best practices vs. a few minutes of downtime. It's a far better call than not forcing the update. Of course all the people who built shoddy cloud presences are going to be screaming blue murder that their 'oh so successful' but 'couldn't be fucked to run in multiple AZs' businesses are going to be down for a few minutes. It reminds me of doing tech support during the dotBomb, every daytrader would phone about how they were losing thousand per second while their internet was down, I'd ask them to switch to their backup connection, and then explain that they had the cheapest residential connection available that's inappropriate for day trading and that if they are making thousands per second they should have a backup connection in place. Then I'd upsell them on a business package and have a guy there in 24 hours.
  • madaxe_again 12y ago
    You get reimbursed?!?! We've been pleading for several months now for a service credit or at least an acknowledgment that they screwed up. Discovered an arcane issue with ARPing to elasticache from within a vpc. Cost us ~$8000 in instances we left running at their request to diagnose, and about the same in man time from our side. Took them 6 weeks to diagnose, too - bloody pathetic. We spend $30k a month with AWS and they treat us like crap. The tech is ok, but their communications and customer service are mindblowingly atrocious. If I can give one bit of advice - don't pay for either their support or an account manager. They don't provide either, and their staff turnover is severe, so your account manager will be a noob to their business every few weeks.
• barkingcat 12y ago
  this might be the xen vulnerability that is as yet undisclosed to the public.
  • emanuelmedina 12y ago
    http://www.cvedetails.com/vulnerability-list/vendor_id-6276/XEN.html http://www.cvedetails.com/vulnerability-list/vendor_id-6276/...
• tszming 12y ago
  I guess if you already put your machines in different availability zones, they will not reboot in the same maintenance window. Actually AWS scheduled maintenance is the best you can get in the market already, because most of the time they allow you to reboot yourself in order to land an updated host (This time is different might be due to some critical security issues). Other providers like Azure / Google, you have no choice.
  • jsolson 12y ago
    Google Compute Engine offers live migration around maintenance events: https://cloud.google.com/compute/docs/instances#onhostmaintenance https://cloud.google.com/compute/docs/instances#onhostmainte... (note: I work on GCE, more or less)
    • tszming 12y ago
      Yes you are right, but GCE does not offer the flexibility that allow user to reboot themselves in their own convenient time. Note: I am not saying the 'live migration' in GCE does not work, I am just saying I've more confident (well...peace of mind, IMO) to shutdown the my own database manually (which is automated and tested), ensure all data are flushed to disk, clients are disconnected gracefully and slave has promoted to master.. and things like that, rather than some black magic like 'live migration' Of course, I am not saying 'live migration' is wrong, but it is just my preference.
      • crb 12y ago
        They demo'd live migration while streaming 1080p video, with no outage. It was described as having the network cable unplugged for a tenth of a second. There's a story from a partner who was testing it, who at the end of the day said "when are you going to live migrate us?", only to be told that Google had moved them six times that day and they hadn't even noticed.
      • morpher 12y ago
        You can configure your instances to automatically shutdown and restart instead of migrating: https://cloud.google.com/compute/docs/instances#onhostmaintenance https://cloud.google.com/compute/docs/instances#onhostmainte... Alternatively, you have a short period (60s) before the migration occurs during which you can lame a service: https://cloud.google.com/compute/docs/metadata#maintenanceevents https://cloud.google.com/compute/docs/metadata#maintenanceev... However, I'll grant that both of these lack the "in your own convenient time" portion of your desire.
• sounds 12y ago
  These instance types are not affected: T1, T2, M2, R3, and HS1 instance types are not affected. http://www.rightscale.com/blog/rightscale-news/aws-reboot-substantial-number-ec2-instances http://www.rightscale.com/blog/rightscale-news/aws-reboot-su...
  • eropple 12y ago
    I'd pour one out for the people with the gigantic I2's that they'll have to reprovision, but I'm pretty sure they can afford their own, they don't need me to pour one out for them.
    • mdellabitta 12y ago
      Are you kidding? I have to run i2s. I can't afford anything else! :)
      • eropple 12y ago
        Well, not once you've bought the I2's, no...
• fjordan 12y ago
  CVE-2014-6271 was posted on the AWS Security Bulletin earlier today: http://aws.amazon.com/security/security-bulletins/CVE_2014_6271_advisory/ http://aws.amazon.com/security/security-bulletins/CVE_2014_6... My guess is that they cannot rely on all users to perform these updates.
  • fjordan 12y ago
    can someone explain the downvotes?
    • rands3311 12y ago
      Already confirmed by AWS its not related. Confirmed in this thread its not related. That issue is an OS level problem, AWS won't touch your OS.
• mustafab 12y ago
  Does anyone think it might be related to the recent bash bug? It even affects dhcp clients which is a case with aws. https://securityblog.redhat.com/2014/09/24/bash-specially-crafted-environment-variables-code-injection-attack/ https://securityblog.redhat.com/2014/09/24/bash-specially-cr...
  • eropple 12y ago
    It is unrelated. This is a Xen patch.
  • bradhe 12y ago
    > Does anyone think it might be related to the recent bash bug That would be...pathetic, at best. There's a Xen bug that's being embargoed 'till 2014-10-01. I think that's more likely.
  • _asummers 12y ago
    The bash bug does not require a system restart. In fact, it can be updated without shutting down the running bash processes, as the vulnerability only affects new shells.
• deleted 12y ago
  [deleted]
• allegory 12y ago
  This is one of the reasons why we fished out for 3x full racks at different DCs and bought our own kit. It's always our schedule.
  • ceejayoz 12y ago
    Yes, but Amazon's schedule involves them having advance, non-public knowledge of major security flaws. In some ways, you've got a false sense of security.
    • allegory 12y ago
      Well the bash vulnerability and then immediate disclose of another vulnerability makes that statement pretty moot. We also have deeply layered non-hetereogenous kit so you have to punch hard to get through it all to something useful.
  • angrybits 12y ago
    Everything else is yours too, like getting an ROI out of the capital your spending. It's nice you have that kind of money laying around to spend on iron instead of advertising (as an example), but not everyone does.
• RossM 12y ago
  I have an instance listed as scheduled for reboot in 2 days but haven't received an email. Without this post I'd have been caught out on Saturday wondering why I'm getting a ton of message queue connection errors. Downtime I can live with, but unreported planned downtime? Not impressed.
  • kordless 12y ago
    Check your spam folder. A few of their mail servers occasionally have blacklisted IPs.
• frozenport 12y ago
  What if Amazon's infrastructure fails during the reboot?
• scottlinux 12y ago
  Not only EC2 instances, but also RDS instances. I received various maintenance email notifications for RDS reboots.
• AngelaT 12y ago
  AWS Reboot FAQs available here: http://www.rightscale.com/blog/rightscale-news/aws-reboot-faqs http://www.rightscale.com/blog/rightscale-news/aws-reboot-fa...