9 ms·
Critical Xen bug in PV memory virtualization code
- eli 11y agoLinode apparently was affected but got advance notice and patched all servers over the past week.
- dangrossman 11y agoSame for Rackspace, IBM/Softlayer, and everyone else with a large cloud running Xen. Linode was actually my failover target for some of those clouds to weather the reboots. They offer both Xen and KVM based instances. If you're still running Xen, you can live migrate your instance to KVM and make all your new Linodes default to KVM via a toggle in your account settings.
- upbeatlinux 11y agoThanks - this is good info to have for personal stuff.
- tetrep 11y agoAs the title doesn't indicate, this is a pretty big deal. Full host compromise from a guest that always works and doesn't leave any traces. Amazon is claiming it doesn't affect them[0], but they don't give any details as to why. [0]: https://aws.amazon.com/security/security-bulletins/XSAsecurityadvisories-October/ https://aws.amazon.com/security/security-bulletins/XSAsecuri...
- barkingcat 11y agoAWS is part of the advanced notice group. Could it be that they've already been mitigating this behind the scenes before disclosure?
- walterbell 11y agoCloud providers typically have early access to security fixes and would be patched by now. Amazon has the ability to hot-patch Xen, a feature which is now being developed for general use. This guest escape vulnerability (http://xenbits.xen.org/xsa/advisory-148.html http://xenbits.xen.org/xsa/advisory-148.html) has been present since 2008 and was found by Alibaba, which recently joined the Xen advisory board, http://www.xenproject.org/about/in-the-news/192-xen-project-announces-alibaba-joins-advisory-board.html http://www.xenproject.org/about/in-the-news/192-xen-project-...
- api 11y agoI just saw a bunch of our SoftLayer VMs get rebooted (with ahead notification) and they seem to use Xen. They probably got notification too. Looks like my others (Digital Ocean, Vultr) use KVM (according to virt-what). Did anyone see Amazon do reboots? I wonder if Amazon has some method of hot-patching kernel and Xen code under the hood? There are such things, and that would be some interesting secret sauce.
- walterbell 11y agoHere's a talk and wiki page on Xen hot-patching (xsplice), from the Aug 2015 Xen Summit: http://wiki.xen.org/wiki/XSplice http://wiki.xen.org/wiki/XSplice & https://youtu.be/C2R79Ob8KOA https://youtu.be/C2R79Ob8KOA
- dangrossman 11y agoEarlier this year: > Our team has been working around-the-clock to find ways to minimize the impact for those requiring a reboot. We're happy to share that we'll now be able to live-update the vast majority of our older hardware for this Xen Security Advisory. https://aws.amazon.com/premiumsupport/maintenance-2015-03/ https://aws.amazon.com/premiumsupport/maintenance-2015-03/ There's been a few subsequent articles mentioning Amazon's ability to hot-patch Xen without a reboot. They didn't perform a reboot for this vulnerability.
- samstave 11y ago
- bgirard 11y agoI was wondering what it was. That explains the 'Critical Xen Maintenance' reboot ticket I got from Linode 11 days ago.
- upbeatlinux 11y agoRackspace has been patching their Xen stuff for a little over a week now. It's nice having a multi-region / multi-provider setup otherwise the typical Rackspace reboot window (much like UPS deliveries or the cable guy) would've been a pain in the ass.
- anthony_franco 11y agoWhat's your general setup to achieve the multi-provider redundancy?
- gibsonje 11y agoWe deploy to a single region. This maintenance was a major pain in the ass. We got a notice 72 hours ahead of time (only 1 business day ahead of time) that around 60% of all of our hosts failed live migration and would be rebooted.
- upbeatlinux 11y ago@gibsonje - similar experience here. Except Rackspace gave notice about a week ahead. The day maintenance was scheduled for it was postponed but without notice. Then the reboots happened across 3 days as opposed to 1 day. Glad they are patching but that's not really fanatical support.
- deleted 11y ago[deleted]
- devit 11y agoThe existence of such a catastrophic bug shows that Xen is unfortunately not suitable for use as an hypervisor in a secure system and needs to be replaced. We really need a properly written open source hypervisor: entities using secure hypervisors commercially like Amazon AWS should fund the development of one.
- toomuchtodo 11y agoIs KVM not a "properly written open source hypervisor"?
- esaym 11y agoAs long as you don't need PV...
- nly 11y agoPV = paravirt? Because there are virtio drivers for Windows. KVM qualifies as paravirtualisation to me.
- cthalupa 11y agoKVM is not paravirtualization - it has paravirtualized drivers. A PV VM would use hypercalls instead of regular system calls, would have more context switches due to the lack of protection rings, etc.
- cthalupa 11y agoWhy would you need PV? It's inferior in basically every way to HVM. CPU performance is worse due to missing protection rings on x86_64, PV drivers for disk and network bring HVM performance to PV for those, SR-IOV for networking makes networking faster, other hardware extension make memory access faster... THere's very very very very very few workloads that perform better on PV than HVM.
- esaym 11y ago
- rcconf 11y agoDoes this mean it's a good idea to reset all keys/passwords for any service that was using Xen? I don't know much about Xen, so perhaps someone else can chime in on this. This exploit says it's for PV, and not for HVM. But Xen can apparently run both at the same time. Does this mean even if you were using HVM, and Xen had other guests using PV, you were still exploitable?
- devit 11y agoAnyone who could create Xen PV VMs could trivially take control of the whole machine including of course all your VMs regardless of type.
- deleted 11y ago[deleted]
- akerl_ 11y agoPer AWS, they aren't affected (not shocking, they run very custom Xen), and even so, the info you upload to tarsnap is already encrypted client-side, that being the whole point.
- vox_mollis 11y agoRight, they are not currently affected, because they got advance notice. But it's very likely that Alibaba sat on the vuln for a long time, and Chinese intelligence knew about it. It's prudent to consider all data on ec2 servers compromised and act accordingly.
- walterbell 11y agoToo bad we don't know how Alibaba found this vuln, so the technique can be used to audit the rest of the Xen codebase.
- lsc 11y agoso my speculation (and this is pure speculation) is that they're still running something based on the older rhel5-xen on their paravirt guests, on the general "if it's not broken, don't fuck with it" model. I think that the RHEL5 hypervisor is old enough to not be vulnerable to this problem.
- walterbell 11y agoWould rhel5-xen work on recent-ish servers, let's say 2012?
- lsc 11y agoYeah. AHCI is AHCI.... and if you did have drivers for newer RAID cards and the like, those can be backported with a reasonable number of hours of the skill level amazon can reasonably hire.
- walterbell 11y ago| The code to validate level 2 page table entries is bypassed when | certain conditions are satisfied. This means that a PV guest can | create writeable mappings using super page mappings. | | Such writeable mappings can violate Xen intended invariants for pages | which Xen is supposed to keep read-only. Xen is used by security-focused Qubes, which published an analysis of XSA-148, https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qs...: "The above is a political way of stating the bug is a very critical one. Probably the worst we have seen affecting the Xen hypervisor, ever. Sadly. Admittedly this is subtle bug, because there is no buggy code that could be spotted immediately ... On the other hand, it is really shocking that such a bug has been lurking in the core of the hypervisor for so many years. In our opinion the Xen project should rethink their coding guidelines and try to come up with practices and perhaps additional mechanisms that would not let similar flaws to plague the hypervisor ever again (assert-like mechanisms perhaps?). Otherwise the whole project makes no sense, at least to those who would like to use Xen for security-sensitive work. Specifically, it worries us that, in the last 7 years (i.e. all the time when the bug was sitting there having a good time) so much engineering and development effort has been put into adding all sorts of new features and whatnots, yet no serious effort to improve Xen security effectively. Because there have been, of course, many more security bugs found in Xen over the last years (as the numbering of this XSA suggests)... the bugs in Xen are being found regularly, and this is no good news. For a type-1 hypervisor of the age and maturity of Xen, this simply should not be happening. If it does, it suggests the development process is not prioritizing security."
- bluedino 11y agoWhat's the quote from Theo again? You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes.
- makomk 11y agoThe Qubes commentary on this vulnerability is unintentionally amusing too, it basically boils down to them complaining that the Xen developers failed to spot it for seven years even though there's no obvious way to spot it.
- three14 11y agoThey're arguing that the Xen developers shouldn't code in a way that makes it so hard to spot, not that they should have spotted it as is.
- devit 11y agoYes, the problem is that this terrible code directly copies the untrusted "nl2e" variable from the VM into the extremely critical hardware page table, only doing a broken unclearly written check. Instead, "nl2e" should have a data type preventing such a direct copy, and only allowing to test single bits. The code should then be written to copy bits one by one (for bits where it is appropriate), with a comment for each bit stating why it is safe to copy them. The fact that this is not the case means that none of the other code in Xen can really be trusted to be bug-free, and there is probably no way to fix that without starting over or doing an equivalent amount of rewriting work.
- sn 11y agoCan you name one substantial software project with more than 10 developers that you expect to be bug free?
- arielby 11y ago> This bug might also be considered an argument for the view of ditching of para-virtualized (PV) VMs, and switch to HVMs It's not like Xen HVMs have a better security story than PVMs. The paravirtualization code should probably be more heavily audited than it is already.
- notabot 11y agoThis is not the first time they rant about this sort of things. Of course if they can't stand the code quality of Xen they are always welcome to switch to KVM, virtualbox, bhyve or whatever open source hypervisor they think has the best code quality and security practice. I know it is within their right to write things as they please. But seriously, ranting like this is not very constructive and doesn't move things forward. If Qubes thinks the practices in Xen community are bad, why don't they start a conversation on Xen development mailing list? (edited: typo)
- _delirium 11y agoprgmr.com has a rundown on what affected them: http://blog.prgmr.com/operations/2015/10/29/recent-xsas.html http://blog.prgmr.com/operations/2015/10/29/recent-xsas.html
- PaulHoule 11y agoI think hardware implementation can have security bugs too and it's a lot harder to upgrade the hardware than the software.
- hol1sh1t 11y agoAnyone can explain how to apply those patches? Can't find updated packages in repo
- deleted 11y ago[deleted]