4 ms·

As the title doesn't indicate, this is a pretty big deal. Full host compromise from a guest that always works and doesn't leave any traces. Amazon is claiming i

by tetrep 11y ago

As the title doesn't indicate, this is a pretty big deal. Full host compromise from a guest that always works and doesn't leave any traces. Amazon is claiming it doesn't affect them[0], but they don't give any details as to why. [0]: https://aws.amazon.com/security/security-bulletins/XSAsecurityadvisories-October/ https://aws.amazon.com/security/security-bulletins/XSAsecuri...

barkingcat 11y ago
AWS is part of the advanced notice group. Could it be that they've already been mitigating this behind the scenes before disclosure?
walterbell 11y ago
Cloud providers typically have early access to security fixes and would be patched by now. Amazon has the ability to hot-patch Xen, a feature which is now being developed for general use. This guest escape vulnerability (http://xenbits.xen.org/xsa/advisory-148.html http://xenbits.xen.org/xsa/advisory-148.html) has been present since 2008 and was found by Alibaba, which recently joined the Xen advisory board, http://www.xenproject.org/about/in-the-news/192-xen-project-announces-alibaba-joins-advisory-board.html http://www.xenproject.org/about/in-the-news/192-xen-project-...
- api 11y ago
  I just saw a bunch of our SoftLayer VMs get rebooted (with ahead notification) and they seem to use Xen. They probably got notification too. Looks like my others (Digital Ocean, Vultr) use KVM (according to virt-what). Did anyone see Amazon do reboots? I wonder if Amazon has some method of hot-patching kernel and Xen code under the hood? There are such things, and that would be some interesting secret sauce.
  - walterbell 11y ago
    Here's a talk and wiki page on Xen hot-patching (xsplice), from the Aug 2015 Xen Summit: http://wiki.xen.org/wiki/XSplice http://wiki.xen.org/wiki/XSplice & https://youtu.be/C2R79Ob8KOA https://youtu.be/C2R79Ob8KOA
  - dangrossman 11y ago
    Earlier this year: > Our team has been working around-the-clock to find ways to minimize the impact for those requiring a reboot. We're happy to share that we'll now be able to live-update ‎the vast majority of our older hardware for this Xen Security Advisory. https://aws.amazon.com/premiumsupport/maintenance-2015-03/ https://aws.amazon.com/premiumsupport/maintenance-2015-03/ There's been a few subsequent articles mentioning Amazon's ability to hot-patch Xen without a reboot. They didn't perform a reboot for this vulnerability.
    - samstave 11y ago
      Out of the 1,000+ instances I had running in AWS during that last major reboot -- there were only a handful of hosts affected. Sadly, some of them were the very primary core DBs -- but in general, it was still only a handful. Luckily, many people also converted to HVM over PV already. IIRC it was largely PV instances in EC2 classic... but I can't be sure.
      - sn 11y ago
        Memories are short. What about VENOM https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3456 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3456 which affected both Xen HVM and KVM? Then there was http://xenbits.xen.org/xsa/advisory-110.html http://xenbits.xen.org/xsa/advisory-110.html which is privilege escalation in HVM mode. Regardless of the virtualization technology, you need to keep on top of disclosures. Any public provider running xen should be on the disclosure list by now http://www.xenproject.org/security-policy.html http://www.xenproject.org/security-policy.html
  - some1else 11y ago
    Linode rebooted mine on 21st
- deleted 11y ago
  [deleted]
- zurn 11y ago
  In that case Amazon's wording would be pretty disingenuous - "AWS customers' data and instances are not affected by these issues" when the reality is more like "as far as we know nobody exploited this before we patched"?
  - walterbell 11y ago
    Good point. In theory, they could have had other mitigations in place for this class of vulnerability, which they chose not to share with the GPL Xen community.
  - FireBeyond 11y ago
    Right, "AWS customers' data and instances _ARE_ not affected" (emphasis mine) does not imply "AWS customers' data and instances _WERE_ not affected"
    - walterbell 11y ago
      Breaches are a one-way state change, like death. If customer data WAS affected, then it IS affected, until customers take action such as disclosure, resetting passwords/keys, etc. If customer data was affected, AWS would need to advise customers to take remedial steps. Since they have not issued such advice, is it safe to assume that they had mitigation in place for this attack?
      - ak4g 11y ago
        It could also be the case that they have monitoring systems on the host machines that would have reliably detected this issue being exploited, even before this specific vulnerability was discovered (or perhaps better to say, given the age of the bug, before being discovered by someone that disclosed it to Xen).
  - jerdfelt 11y ago
    It depends on the version of Xen they are running. XSA-148 only affects Xen 3.4 and newer. If AWS is running an older version of Xen, they may never have been affected.
- benmmurphy 11y ago
  without any serious infrastructure you can also hot patch single instructions where the bytes you want to change fall in an 8 byte alignment. though, probably this is not useful for this bug but I know other cloud providers do this for security patches.

by tetrep 11y ago

barkingcat 11y ago

AWS is part of the advanced notice group. Could it be that they've already been mitigating this behind the scenes before disclosure?

walterbell 11y ago

Cloud providers typically have early access to security fixes and would be patched by now. Amazon has the ability to hot-patch Xen, a feature which is now being developed for general use. This guest escape vulnerability (http://xenbits.xen.org/xsa/advisory-148.html http://xenbits.xen.org/xsa/advisory-148.html) has been present since 2008 and was found by Alibaba, which recently joined the Xen advisory board, http://www.xenproject.org/about/in-the-news/192-xen-project-announces-alibaba-joins-advisory-board.html http://www.xenproject.org/about/in-the-news/192-xen-project-...

api 11y ago
I just saw a bunch of our SoftLayer VMs get rebooted (with ahead notification) and they seem to use Xen. They probably got notification too. Looks like my others (Digital Ocean, Vultr) use KVM (according to virt-what). Did anyone see Amazon do reboots? I wonder if Amazon has some method of hot-patching kernel and Xen code under the hood? There are such things, and that would be some interesting secret sauce.
- walterbell 11y ago
  Here's a talk and wiki page on Xen hot-patching (xsplice), from the Aug 2015 Xen Summit: http://wiki.xen.org/wiki/XSplice http://wiki.xen.org/wiki/XSplice & https://youtu.be/C2R79Ob8KOA https://youtu.be/C2R79Ob8KOA
- dangrossman 11y ago
  Earlier this year: > Our team has been working around-the-clock to find ways to minimize the impact for those requiring a reboot. We're happy to share that we'll now be able to live-update ‎the vast majority of our older hardware for this Xen Security Advisory. https://aws.amazon.com/premiumsupport/maintenance-2015-03/ https://aws.amazon.com/premiumsupport/maintenance-2015-03/ There's been a few subsequent articles mentioning Amazon's ability to hot-patch Xen without a reboot. They didn't perform a reboot for this vulnerability.
  - samstave 11y ago
    Out of the 1,000+ instances I had running in AWS during that last major reboot -- there were only a handful of hosts affected. Sadly, some of them were the very primary core DBs -- but in general, it was still only a handful. Luckily, many people also converted to HVM over PV already. IIRC it was largely PV instances in EC2 classic... but I can't be sure.
    - sn 11y ago
      Memories are short. What about VENOM https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3456 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3456 which affected both Xen HVM and KVM? Then there was http://xenbits.xen.org/xsa/advisory-110.html http://xenbits.xen.org/xsa/advisory-110.html which is privilege escalation in HVM mode. Regardless of the virtualization technology, you need to keep on top of disclosures. Any public provider running xen should be on the disclosure list by now http://www.xenproject.org/security-policy.html http://www.xenproject.org/security-policy.html
- some1else 11y ago
  Linode rebooted mine on 21st
deleted 11y ago
[deleted]
zurn 11y ago
In that case Amazon's wording would be pretty disingenuous - "AWS customers' data and instances are not affected by these issues" when the reality is more like "as far as we know nobody exploited this before we patched"?
- walterbell 11y ago
  Good point. In theory, they could have had other mitigations in place for this class of vulnerability, which they chose not to share with the GPL Xen community.
- FireBeyond 11y ago
  Right, "AWS customers' data and instances _ARE_ not affected" (emphasis mine) does not imply "AWS customers' data and instances _WERE_ not affected"
  - walterbell 11y ago
    Breaches are a one-way state change, like death. If customer data WAS affected, then it IS affected, until customers take action such as disclosure, resetting passwords/keys, etc. If customer data was affected, AWS would need to advise customers to take remedial steps. Since they have not issued such advice, is it safe to assume that they had mitigation in place for this attack?
    - ak4g 11y ago
      It could also be the case that they have monitoring systems on the host machines that would have reliably detected this issue being exploited, even before this specific vulnerability was discovered (or perhaps better to say, given the age of the bug, before being discovered by someone that disclosed it to Xen).
- jerdfelt 11y ago
  It depends on the version of Xen they are running. XSA-148 only affects Xen 3.4 and newer. If AWS is running an older version of Xen, they may never have been affected.
benmmurphy 11y ago
without any serious infrastructure you can also hot patch single instructions where the bytes you want to change fall in an 8 byte alignment. though, probably this is not useful for this bug but I know other cloud providers do this for security patches.