9 ms·

XSA-108 Advisory

227 points by ukandy 12y ago

spydum 12y ago
Ouch, isn't one of the cardinal sins of hypervisors reading memory outside of your allocation? Certainly explains the embargo
aus_ 12y ago
I assume this is why AWS forcefully rebooted many of their VMs recently?
- ra 12y ago
  Yes and Rackspace.
- jeffbarr 12y ago
  http://aws.amazon.com/blogs/aws/ec2-maintenance-update-2/ http://aws.amazon.com/blogs/aws/ec2-maintenance-update-2/
- mukyu 12y ago
  There is no need to assume anything. """ Yesterday we started notifying some of our customers of a timely security and operational update we need to perform on a small percentage (less than 10%) of our EC2 fleet globally. AWS customers know that security and operational excellence are our top two priorities. These updates must be completed by October 1st before the issue is made public as part of an upcoming Xen Security Announcement (XSA). Following security best practices, the details of this update are embargoed until then. The issue in that notice affects many Xen environments, and is not specific to AWS. """ [0] http://aws.amazon.com/blogs/aws/ec2-maintenance-update/ http://aws.amazon.com/blogs/aws/ec2-maintenance-update/
- blacksmith_tb 12y ago
  Softlayer today, also.
eik3_de 12y ago
It will be interesting to see which providers didn't get the embargoed release. So if you get a reboot announcment from your xen vps provider after today 12:00Z, you should list them here.
- ukandy 12y ago
  Pre-disclosure list is at the bottom of this page.. http://www.xenproject.org/security-policy.html http://www.xenproject.org/security-policy.html
  - danielweber 12y ago
    Huh, I have a tiny machine at one of those smaller places, and they are on the list. Good to know the smaller players can build up a reputation for embargoing, too.
    - cthalupa 12y ago
      They spell out the requirements for getting on the list - one of the valid ways to qualify is basically just being a webhost that uses Xen.
    - timoth 12y ago
      Interesting that over half of the companies on the list were added within the last week, if the dates in the page changelog accurately reflect when they were added. If so, perhaps they all suddenly bundled in so they could find out what the embargoed vuln. was.
alexduros 12y ago
interesting link ... from 2010 http://xen.1045712.n5.nabble.com/x2APIC-emulation-for-HVM-guest-td3288786.html http://xen.1045712.n5.nabble.com/x2APIC-emulation-for-HVM-gu...
- mzs 12y ago
  good find: @@ -2189,6 +2190,11 @@ *msr_content = vcpu_vlapic(v)->hw.apic_base_msr; break; + case MSR_IA32_APICBASE_MSR ... MSR_IA32_APICBASE_MSR + 0x3ff: + if ( hvm_x2apic_msr_read(v, msr, msr_content) ) + goto gp_fault; + break; + case MSR_IA32_CR_PAT: *msr_content = v->arch.hvm_vcpu.pat_cr; break; @@ -2296,6 +2302,11 @@ vlapic_msr_set(vcpu_vlapic(v), msr_content); break; + case MSR_IA32_APICBASE_MSR ... MSR_IA32_APICBASE_MSR + 0x3ff: + if ( hvm_x2apic_msr_write(v, msr, msr_content) ) + goto gp_fault; + break; + case MSR_IA32_CR_PAT: if ( !pat_msr_set(&v->arch.hvm_vcpu.pat_cr, msr_content) ) goto gp_fault;
  - diydsp 12y ago
    What is the "..." operator? I have never seen that before. I can't find any references to it. Is that a macro specific to this project? [I checked the post above, but it doesn't match this source code exactly (and doesn't have ... as an operator).]
    - _delirium 12y ago
      It's a GNU extension that lets you define ranges in switch statements' cases: https://gcc.gnu.org/onlinedocs/gcc/Case-Ranges.html https://gcc.gnu.org/onlinedocs/gcc/Case-Ranges.html
  - walterbell 12y ago
    Is this from the Intel Xen+KVM team, crediting the patch to a member of the RedHat KVM team? Since KVM isn't vulnerable to this cross-domain issue, it may be useful to compare with the equivalent code in KVM and/or Linux.
sudowhodoido 12y ago
Well looks like Mr De Raadt was right again: 'x86 virtualization is about basically placing another nearly full kernel, full of new bugs, on top of a nasty x86 architecture which barely has correct page protection. Then running your operating system on the other side of this brand new pile of shit. You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes.' Source: http://marc.info/?l=openbsd-misc&m=119318909016582 http://marc.info/?l=openbsd-misc&m=119318909016582 Personally, I have hope for things like cgroups/jails and MAC/SELinux over virtualization.
- joosters 12y ago
  This is just FUD and in the usual insulting De Raadt style of communicating. 'barely has correct page protection' is just a way of saying 'has correct page protection, but I want to be really snotty about it'. So, highlighting a non-problem. No-one is claiming that virtualization makes a system magically completely secure, but do people actually believe that it makes it less secure? (Compared to, running the same software on the same hardware using a single OS). I don't think so.
  - zorpner 12y ago
    (Compared to, running the same software on the same hardware using a single OS). I don't think so. If you had been running an OpenBSD instance on hardware as a single OS, you would not be vulnerable to having your system's memory read by this hypervisor bug. So... yes.
    - amluto 12y ago
      This is nonsense. A single OpenBSD kernel on bare metal can do whatever it pleases. A Linux kernel on Xen HVM could, due to this bug, do slightly more than it ought to have been able to do. (In this sense, the LLDT bug was worse.)
      - 12y ago
chippy 12y ago
Interesting that the patch is just one characters difference (applied in two locations) 0x3ff to 0xff
- boardwaalk 12y ago
  Seems to me like they should be using constants. Especially since this implies the 0xff/256 is elsewhere as well: elsewhere a 256 rather than 1024 wide window is being used too
  - tarpherder 12y ago
    *0xff/255 Simple but potentially dangerous right? :P
- spacefight 12y ago
  Deliberate? Coincidence?
  - chc 12y ago
    It's pretty certain they deliberately changed both instances of the number. How is that even a question?
- jonknee 12y ago
  From the patch: > While the write path change appears to be purely cosmetic (but still gets done here for consistency), the read side mistake permitted accesses beyond the virtual APIC page. http://xenbits.xen.org/xsa/xsa108.patch http://xenbits.xen.org/xsa/xsa108.patch
- ultramancool 12y ago
  Yep, it seems only to change a small memory range. I'm guessing this means this isn't highly exploitable, but contains at least some risk of leaking private information. Can anyone with more experience with this specific APIC stuff comment? Could you get different data each time or only the same small range?
fndrplayer13 12y ago
Heh, it's been a rough couple of weeks.
asb 12y ago
Can somebody please confirm that it is impossible to boot a HVM system on Linode? The hypervisor on my Linode host certainly supports HVM (according to cat /sys/hypervisor/properties/capabilities). The host Xen is 4.1 and therefore vulnerable in the case that another user could be running HV guests.
- akerl_ 12y ago
  There is no such thing as an HVM Linode server, all Linodes are PV.
  - jedicoffee 12y ago
    Akerl speaks the truth. We're good to go.
  - asb 12y ago
    Thanks for the confirmation. Hoping to see PVH support in the future!
walterbell 12y ago
Thanks to Jan Beulich, the SUSE Xen maintainer in Germany who is credited with finding this x86 HVM vulnerability. It would be helpful if errata announcements included documentation of the static analysis tools, code review process or automated testing techniques which identified the weakness, along with a postmortem of previous audits of relevant code paths. What made it possible for this issue to be identified now, when the issue escaped previous analysis, audits and tests? Such process improvement knowledge is possibly more valuable to the worldwide technical community than any point fix. Heartbleed was discovered by an external party, but this issue which affects the data of millions of users was found by the originating open-source project. Kudos to Jan for finding this cross-domain escalation.
- seanp2k2 12y ago
  Was http://www.brendangregg.com/blog/2014-09-15/the-msrs-of-ec2.html http://www.brendangregg.com/blog/2014-09-15/the-msrs-of-ec2.... the reason they were looking?
  - walterbell 12y ago
    Don't know. A downthread post (https://news.ycombinator.com/item?id=8393911 https://news.ycombinator.com/item?id=8393911) links to the Dec 2010 Intel patch which introduced the bug. It may have originated with a RedHat KVM patch in June 2009, https://lkml.org/lkml/2009/6/29/205 https://lkml.org/lkml/2009/6/29/205
brendangregg 12y ago
I have to wonder if my recent blog post, "The MSRs of EC2", on September 15th has prompted this discovery: http://www.brendangregg.com/blog/2014-09-15/the-msrs-of-ec2.html http://www.brendangregg.com/blog/2014-09-15/the-msrs-of-ec2.... . When I posted that, I could find no examples of MSR usage in EC2 or Xen. I haven't checked after the reboot, but I hope the MSRs I'm using can still be accessed: IA32_MPERF and IA32_APERF (to calculate real CPU MHz); IA32_THERM_STATUS and MSR_TEMPERATURE_TARGET (to calculate CPU temperatures); and MSR_TURBO_RATIO_LIMIT and MSR_TURBO_RATIO_LIMIT1 (to see turbo ratios). I use them in the showboost, cputemp, and cpuhot MSR-based tools: https://github.com/brendangregg/msr-cloud-tools https://github.com/brendangregg/msr-cloud-tools
- mzs 12y ago
  You should email Jan and ask, I'm curious too.
otterley 12y ago
Patches to XenServer (releases 6.0 through 6.2) are now available from Citrix: https://support.citrix.com/article/CTX200218 https://support.citrix.com/article/CTX200218
pjungwir 12y ago
I had "dedicated" AWS instances that were rebooted. A dedicated instance means there is only one guest per box, right? So I'm curious why those had to be rebooted if there is no network-facing vector to this vulnerability. I guess because we could have read from the hypervisor's memory?
- hoov 12y ago
  I believe that "dedicated" means that it's dedicated to your account. So, there can be more than one guest, but those guests are your EC2 instances.
  - pjungwir 12y ago
    Ah, thanks for the clarification! I guess that still means they were rebooting our machines in case the malicious actor was us. :-)
deleted 12y ago
[deleted]
ivank 12y ago
Some impact analysis from the Qubes OS project: https://groups.google.com/forum/#!msg/qubes-devel/HgQ_aWt-EBU/8VWzu2IrQdQJ https://groups.google.com/forum/#!msg/qubes-devel/HgQ_aWt-EB...
mappu 12y ago
The advisory says PV is safe and HVM is dangerous. What about PV-HVM? Got a reasonably long list of VPS providers to submit tickets to.
- liuw 12y ago
  PVHVM is HVM with PV drivers, so it's still HVM. Don't hesitate to submit your tickets.
walterbell 12y ago
Desktop risk analysis from Qubes, http://qubes-os.org http://qubes-os.org via https://groups.google.com/forum/m/#!topic/qubes-devel/HgQ_aWt-EBU https://groups.google.com/forum/m/#!topic/qubes-devel/HgQ_aW... --- This seemingly looks like a serious problem, but if we think a little bit about the practical impact the conclusion might be quite different. First, there are really no secrets or keys in the hypervisor memory that might make a good target for an exploit here. Xen hypervisor does not do encryption, neither it deals with any storage subsystems. Also there is no explicit guest memory content intermixed with the hypervisor code and data. But one place to see pieces of potentially sensitive data are the Xen internal structures where the guest _registers_ are stored whenever the guest execution is interrupted (e.g. because of a trap). These registers might contain e.g. (parts of) keys or other secrets, if the guest was executing some sensitive crypto operation just before it got interrupted. The vulnerability allows to read only a few kB of the hypervisor memory, with only relative addressing from the emulated APIC registers page, whose address is not known to the attacker. Still, for the exactly same systems (same binaries running, same ACPI tables, etc) it's likely that the attacker would be able to guess the address of the APIC page. However, it is much less probable she would be able to predict what Xen structures are located in the adjacent memory. Much less the attacker would be able to control what structure are located there, as there doesn't seem to be many ways of how a malicious HVM might be significantly affecting the layout of the hypervisor heap (e.g. force arch_vcpu structures of interesting domains to appear nearby). Nevertheless, it might happen, by pure coincidence, that an arch_vcpu structure with a content of an interesting VM will just happen to be located adjacently to the emulated APIC page. In that case, the next problem for the attacker would be lack of control and knowledge over the target VM execution: even if the attacker were somehow lucky to find the other VM's register-holding-structure adjacent to the APIC page, it would still be unclear what the target VM was executing at the time it was suspended and so, whether the registers stored in the structure are worthwhile or not. It is thinkable that the attacker might attempt to use some form of a heuristic, such as e.g. "if RIP == X, then RAX likely contains (parts of) the important key", hoping that this specific RIP would signify a specific interesting instruction (e.g. part of some crypto library) being executed while the VM was interrupted, and so the key is to be found in one of the registers. But the attacker's memory reading exploit doesn't offer a comfort of synchronization, so even though the attacker might be so extremely lucky as to find out that *(apic_page + guessed_offset_to_rip) == X (the attacker here assumes the 'guessed_offset_to_rip' is the distance between the APIC page and the address where RIP is stored in the presumable arch_vcpu structure, that presumably is located adjacently), still there is no guarantees that the next read to *(apic_page + guest_offset_to_rax) will return the content of RAX from the same moment that RIP was snapshot (and which the attacker considered interesting). Arguably the attacker might try to fire up the attack continuously, thus increasing chances of success. Assuming this won't cause system to crash due to accessing non-mapped memory, this might sound like a somehow good strategy. However, in case of a desktop system like Qubes OS, the attacker has very limited control over other domains. Unlike as in case of attacking a VM playing a role of a Web server for instance, the attacker probably won't be able to force the target VMs to do lots of repeated crypto operations, neither choose moments when the target VM traps. It seems like exploiting this bug in an IaaS scenario might be more practical, though, as the attacker also has some control of domain creation/termination, so can affect Xen heap to some extent. But on a system like Qubes OS, it seems unlikely. So, are we doomed? We likely are, but probably not because of this bug. ---
comex 12y ago
I wonder if Amazon or someone will take the time to make a ksplice-like system for Xen so that future security upgrades probably won't have to go through such disruptive reboot events. (Or, for that matter, whether they considered making an ad-hoc machine code patch - based on the source patch, it looks like it would probably be doable just by changing a few bytes. I guess it's a bit risky...)
- semenko 12y ago
  Seems a little unlikely, given Ksplice's patents (now Oracle's patents) covering the area. From an older post @ https://news.ycombinator.com/item?id=2791756 https://news.ycombinator.com/item?id=2791756 The first is "Method of finding a safe time to modify code of a running computer program": http://bit.ly/ksplice-1 http://bit.ly/ksplice-1 The second is "Method of determining which computer program functions are changed by an arbitrary source code modification": http://bit.ly/ksplice-2 http://bit.ly/ksplice-2
  - comex 12y ago
    Sigh. That's obnoxious - yet another example of software patents confusing proving that an idea is commercially valuable with inventing it in the first place. Anyone with the requisite skills in reverse engineering, compilers, etc. could have told you that hot patching functions in memory is possible and would take at most a few minutes to notice that this may be unsafe if some suspended thread is sitting in a function prolog. Yet "identifying a portion of executable code to be updated in a running computer program; and determining whether it is safe to modify the executable code of the running computer program without having to restart the running computer program" (an actual claim, not the abstract or title quotes that people tend to misconstrue) is now locked out for the next decade or so.
- soren 12y ago
  There's a paper on live updating Xen to be presented next month at a cloud computing security workshop: https://www.trust.informatik.tu-darmstadt.de/publications/publication-details/?no_cache=1&tx_bibtex_pi1%5Bpub_id%5D=TUD-CS-2014-0918 https://www.trust.informatik.tu-darmstadt.de/publications/pu...
regularfry 12y ago
> A buggy or malicious HVM guest can crash the host Given that writes are no-ops, I don't understand this mechanism. Can someone explain it, please?

227 points by ukandy 12y ago

Ouch, isn't one of the cardinal sins of hypervisors reading memory outside of your allocation? Certainly explains the embargo

aus_ 12y ago

I assume this is why AWS forcefully rebooted many of their VMs recently?

ra 12y ago
Yes and Rackspace.
jeffbarr 12y ago
http://aws.amazon.com/blogs/aws/ec2-maintenance-update-2/ http://aws.amazon.com/blogs/aws/ec2-maintenance-update-2/
mukyu 12y ago
There is no need to assume anything. """ Yesterday we started notifying some of our customers of a timely security and operational update we need to perform on a small percentage (less than 10%) of our EC2 fleet globally. AWS customers know that security and operational excellence are our top two priorities. These updates must be completed by October 1st before the issue is made public as part of an upcoming Xen Security Announcement (XSA). Following security best practices, the details of this update are embargoed until then. The issue in that notice affects many Xen environments, and is not specific to AWS. """ [0] http://aws.amazon.com/blogs/aws/ec2-maintenance-update/ http://aws.amazon.com/blogs/aws/ec2-maintenance-update/
blacksmith_tb 12y ago
Softlayer today, also.

eik3_de 12y ago

It will be interesting to see which providers didn't get the embargoed release. So if you get a reboot announcment from your xen vps provider after today 12:00Z, you should list them here.

ukandy 12y ago
Pre-disclosure list is at the bottom of this page.. http://www.xenproject.org/security-policy.html http://www.xenproject.org/security-policy.html
- danielweber 12y ago
  Huh, I have a tiny machine at one of those smaller places, and they are on the list. Good to know the smaller players can build up a reputation for embargoing, too.
  - cthalupa 12y ago
    They spell out the requirements for getting on the list - one of the valid ways to qualify is basically just being a webhost that uses Xen.
  - timoth 12y ago
    Interesting that over half of the companies on the list were added within the last week, if the dates in the page changelog accurately reflect when they were added. If so, perhaps they all suddenly bundled in so they could find out what the embargoed vuln. was.

alexduros 12y ago

interesting link ... from 2010 http://xen.1045712.n5.nabble.com/x2APIC-emulation-for-HVM-guest-td3288786.html http://xen.1045712.n5.nabble.com/x2APIC-emulation-for-HVM-gu...

mzs 12y ago
good find: @@ -2189,6 +2190,11 @@ *msr_content = vcpu_vlapic(v)->hw.apic_base_msr; break; + case MSR_IA32_APICBASE_MSR ... MSR_IA32_APICBASE_MSR + 0x3ff: + if ( hvm_x2apic_msr_read(v, msr, msr_content) ) + goto gp_fault; + break; + case MSR_IA32_CR_PAT: *msr_content = v->arch.hvm_vcpu.pat_cr; break; @@ -2296,6 +2302,11 @@ vlapic_msr_set(vcpu_vlapic(v), msr_content); break; + case MSR_IA32_APICBASE_MSR ... MSR_IA32_APICBASE_MSR + 0x3ff: + if ( hvm_x2apic_msr_write(v, msr, msr_content) ) + goto gp_fault; + break; + case MSR_IA32_CR_PAT: if ( !pat_msr_set(&v->arch.hvm_vcpu.pat_cr, msr_content) ) goto gp_fault;
- diydsp 12y ago
  What is the "..." operator? I have never seen that before. I can't find any references to it. Is that a macro specific to this project? [I checked the post above, but it doesn't match this source code exactly (and doesn't have ... as an operator).]
  - _delirium 12y ago
    It's a GNU extension that lets you define ranges in switch statements' cases: https://gcc.gnu.org/onlinedocs/gcc/Case-Ranges.html https://gcc.gnu.org/onlinedocs/gcc/Case-Ranges.html
- walterbell 12y ago
  Is this from the Intel Xen+KVM team, crediting the patch to a member of the RedHat KVM team? Since KVM isn't vulnerable to this cross-domain issue, it may be useful to compare with the equivalent code in KVM and/or Linux.

sudowhodoido 12y ago

Well looks like Mr De Raadt was right again: 'x86 virtualization is about basically placing another nearly full kernel, full of new bugs, on top of a nasty x86 architecture which barely has correct page protection. Then running your operating system on the other side of this brand new pile of shit. You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes.' Source: http://marc.info/?l=openbsd-misc&m=119318909016582 http://marc.info/?l=openbsd-misc&m=119318909016582 Personally, I have hope for things like cgroups/jails and MAC/SELinux over virtualization.

joosters 12y ago
This is just FUD and in the usual insulting De Raadt style of communicating. 'barely has correct page protection' is just a way of saying 'has correct page protection, but I want to be really snotty about it'. So, highlighting a non-problem. No-one is claiming that virtualization makes a system magically completely secure, but do people actually believe that it makes it less secure? (Compared to, running the same software on the same hardware using a single OS). I don't think so.
- zorpner 12y ago
  (Compared to, running the same software on the same hardware using a single OS). I don't think so. If you had been running an OpenBSD instance on hardware as a single OS, you would not be vulnerable to having your system's memory read by this hypervisor bug. So... yes.
  - amluto 12y ago
    This is nonsense. A single OpenBSD kernel on bare metal can do whatever it pleases. A Linux kernel on Xen HVM could, due to this bug, do slightly more than it ought to have been able to do. (In this sense, the LLDT bug was worse.)
    - 12y ago

chippy 12y ago

Interesting that the patch is just one characters difference (applied in two locations) 0x3ff to 0xff

boardwaalk 12y ago
Seems to me like they should be using constants. Especially since this implies the 0xff/256 is elsewhere as well: elsewhere a 256 rather than 1024 wide window is being used too
- tarpherder 12y ago
  *0xff/255 Simple but potentially dangerous right? :P
spacefight 12y ago
Deliberate? Coincidence?
- chc 12y ago
  It's pretty certain they deliberately changed both instances of the number. How is that even a question?
jonknee 12y ago
From the patch: > While the write path change appears to be purely cosmetic (but still gets done here for consistency), the read side mistake permitted accesses beyond the virtual APIC page. http://xenbits.xen.org/xsa/xsa108.patch http://xenbits.xen.org/xsa/xsa108.patch
ultramancool 12y ago
Yep, it seems only to change a small memory range. I'm guessing this means this isn't highly exploitable, but contains at least some risk of leaking private information. Can anyone with more experience with this specific APIC stuff comment? Could you get different data each time or only the same small range?

fndrplayer13 12y ago

Heh, it's been a rough couple of weeks.

asb 12y ago

Can somebody please confirm that it is impossible to boot a HVM system on Linode? The hypervisor on my Linode host certainly supports HVM (according to cat /sys/hypervisor/properties/capabilities). The host Xen is 4.1 and therefore vulnerable in the case that another user could be running HV guests.

akerl_ 12y ago
There is no such thing as an HVM Linode server, all Linodes are PV.
- jedicoffee 12y ago
  Akerl speaks the truth. We're good to go.
- asb 12y ago
  Thanks for the confirmation. Hoping to see PVH support in the future!

walterbell 12y ago

Thanks to Jan Beulich, the SUSE Xen maintainer in Germany who is credited with finding this x86 HVM vulnerability. It would be helpful if errata announcements included documentation of the static analysis tools, code review process or automated testing techniques which identified the weakness, along with a postmortem of previous audits of relevant code paths. What made it possible for this issue to be identified now, when the issue escaped previous analysis, audits and tests? Such process improvement knowledge is possibly more valuable to the worldwide technical community than any point fix. Heartbleed was discovered by an external party, but this issue which affects the data of millions of users was found by the originating open-source project. Kudos to Jan for finding this cross-domain escalation.

seanp2k2 12y ago
Was http://www.brendangregg.com/blog/2014-09-15/the-msrs-of-ec2.html http://www.brendangregg.com/blog/2014-09-15/the-msrs-of-ec2.... the reason they were looking?
- walterbell 12y ago
  Don't know. A downthread post (https://news.ycombinator.com/item?id=8393911 https://news.ycombinator.com/item?id=8393911) links to the Dec 2010 Intel patch which introduced the bug. It may have originated with a RedHat KVM patch in June 2009, https://lkml.org/lkml/2009/6/29/205 https://lkml.org/lkml/2009/6/29/205

brendangregg 12y ago

I have to wonder if my recent blog post, "The MSRs of EC2", on September 15th has prompted this discovery: http://www.brendangregg.com/blog/2014-09-15/the-msrs-of-ec2.html http://www.brendangregg.com/blog/2014-09-15/the-msrs-of-ec2.... . When I posted that, I could find no examples of MSR usage in EC2 or Xen. I haven't checked after the reboot, but I hope the MSRs I'm using can still be accessed: IA32_MPERF and IA32_APERF (to calculate real CPU MHz); IA32_THERM_STATUS and MSR_TEMPERATURE_TARGET (to calculate CPU temperatures); and MSR_TURBO_RATIO_LIMIT and MSR_TURBO_RATIO_LIMIT1 (to see turbo ratios). I use them in the showboost, cputemp, and cpuhot MSR-based tools: https://github.com/brendangregg/msr-cloud-tools https://github.com/brendangregg/msr-cloud-tools

mzs 12y ago
You should email Jan and ask, I'm curious too.

otterley 12y ago

Patches to XenServer (releases 6.0 through 6.2) are now available from Citrix: https://support.citrix.com/article/CTX200218 https://support.citrix.com/article/CTX200218

pjungwir 12y ago

I had "dedicated" AWS instances that were rebooted. A dedicated instance means there is only one guest per box, right? So I'm curious why those had to be rebooted if there is no network-facing vector to this vulnerability. I guess because we could have read from the hypervisor's memory?

hoov 12y ago
I believe that "dedicated" means that it's dedicated to your account. So, there can be more than one guest, but those guests are your EC2 instances.
- pjungwir 12y ago
  Ah, thanks for the clarification! I guess that still means they were rebooting our machines in case the malicious actor was us. :-)

deleted 12y ago

[deleted]

ivank 12y ago

Some impact analysis from the Qubes OS project: https://groups.google.com/forum/#!msg/qubes-devel/HgQ_aWt-EBU/8VWzu2IrQdQJ https://groups.google.com/forum/#!msg/qubes-devel/HgQ_aWt-EB...

mappu 12y ago

The advisory says PV is safe and HVM is dangerous. What about PV-HVM? Got a reasonably long list of VPS providers to submit tickets to.

liuw 12y ago
PVHVM is HVM with PV drivers, so it's still HVM. Don't hesitate to submit your tickets.

walterbell 12y ago

Desktop risk analysis from Qubes, http://qubes-os.org http://qubes-os.org via https://groups.google.com/forum/m/#!topic/qubes-devel/HgQ_aWt-EBU https://groups.google.com/forum/m/#!topic/qubes-devel/HgQ_aW... --- This seemingly looks like a serious problem, but if we think a little bit about the practical impact the conclusion might be quite different. First, there are really no secrets or keys in the hypervisor memory that might make a good target for an exploit here. Xen hypervisor does not do encryption, neither it deals with any storage subsystems. Also there is no explicit guest memory content intermixed with the hypervisor code and data. But one place to see pieces of potentially sensitive data are the Xen internal structures where the guest _registers_ are stored whenever the guest execution is interrupted (e.g. because of a trap). These registers might contain e.g. (parts of) keys or other secrets, if the guest was executing some sensitive crypto operation just before it got interrupted. The vulnerability allows to read only a few kB of the hypervisor memory, with only relative addressing from the emulated APIC registers page, whose address is not known to the attacker. Still, for the exactly same systems (same binaries running, same ACPI tables, etc) it's likely that the attacker would be able to guess the address of the APIC page. However, it is much less probable she would be able to predict what Xen structures are located in the adjacent memory. Much less the attacker would be able to control what structure are located there, as there doesn't seem to be many ways of how a malicious HVM might be significantly affecting the layout of the hypervisor heap (e.g. force arch_vcpu structures of interesting domains to appear nearby). Nevertheless, it might happen, by pure coincidence, that an arch_vcpu structure with a content of an interesting VM will just happen to be located adjacently to the emulated APIC page. In that case, the next problem for the attacker would be lack of control and knowledge over the target VM execution: even if the attacker were somehow lucky to find the other VM's register-holding-structure adjacent to the APIC page, it would still be unclear what the target VM was executing at the time it was suspended and so, whether the registers stored in the structure are worthwhile or not. It is thinkable that the attacker might attempt to use some form of a heuristic, such as e.g. "if RIP == X, then RAX likely contains (parts of) the important key", hoping that this specific RIP would signify a specific interesting instruction (e.g. part of some crypto library) being executed while the VM was interrupted, and so the key is to be found in one of the registers. But the attacker's memory reading exploit doesn't offer a comfort of synchronization, so even though the attacker might be so extremely lucky as to find out that *(apic_page + guessed_offset_to_rip) == X (the attacker here assumes the 'guessed_offset_to_rip' is the distance between the APIC page and the address where RIP is stored in the presumable arch_vcpu structure, that presumably is located adjacently), still there is no guarantees that the next read to *(apic_page + guest_offset_to_rax) will return the content of RAX from the same moment that RIP was snapshot (and which the attacker considered interesting). Arguably the attacker might try to fire up the attack continuously, thus increasing chances of success. Assuming this won't cause system to crash due to accessing non-mapped memory, this might sound like a somehow good strategy. However, in case of a desktop system like Qubes OS, the attacker has very limited control over other domains. Unlike as in case of attacking a VM playing a role of a Web server for instance, the attacker probably won't be able to force the target VMs to do lots of repeated crypto operations, neither choose moments when the target VM traps. It seems like exploiting this bug in an IaaS scenario might be more practical, though, as the attacker also has some control of domain creation/termination, so can affect Xen heap to some extent. But on a system like Qubes OS, it seems unlikely. So, are we doomed? We likely are, but probably not because of this bug. ---

comex 12y ago

I wonder if Amazon or someone will take the time to make a ksplice-like system for Xen so that future security upgrades probably won't have to go through such disruptive reboot events. (Or, for that matter, whether they considered making an ad-hoc machine code patch - based on the source patch, it looks like it would probably be doable just by changing a few bytes. I guess it's a bit risky...)

semenko 12y ago
Seems a little unlikely, given Ksplice's patents (now Oracle's patents) covering the area. From an older post @ https://news.ycombinator.com/item?id=2791756 https://news.ycombinator.com/item?id=2791756 The first is "Method of finding a safe time to modify code of a running computer program": http://bit.ly/ksplice-1 http://bit.ly/ksplice-1 The second is "Method of determining which computer program functions are changed by an arbitrary source code modification": http://bit.ly/ksplice-2 http://bit.ly/ksplice-2
- comex 12y ago
  Sigh. That's obnoxious - yet another example of software patents confusing proving that an idea is commercially valuable with inventing it in the first place. Anyone with the requisite skills in reverse engineering, compilers, etc. could have told you that hot patching functions in memory is possible and would take at most a few minutes to notice that this may be unsafe if some suspended thread is sitting in a function prolog. Yet "identifying a portion of executable code to be updated in a running computer program; and determining whether it is safe to modify the executable code of the running computer program without having to restart the running computer program" (an actual claim, not the abstract or title quotes that people tend to misconstrue) is now locked out for the next decade or so.
soren 12y ago
There's a paper on live updating Xen to be presented next month at a cloud computing security workshop: https://www.trust.informatik.tu-darmstadt.de/publications/publication-details/?no_cache=1&tx_bibtex_pi1%5Bpub_id%5D=TUD-CS-2014-0918 https://www.trust.informatik.tu-darmstadt.de/publications/pu...

regularfry 12y ago

> A buggy or malicious HVM guest can crash the host Given that writes are no-ops, I don't understand this mechanism. Can someone explain it, please?