8 ms·

Degraded performance after forced reboot due to AWS instance maintenance

302 points by woliveirajr 9y ago

stedaniels 9y ago
Are we presuming this is the Intel patch being applied?
k-ian 9y ago
for those who don't want to log in, https://i.imgur.com/f76tOvY.png https://i.imgur.com/f76tOvY.png https://i.imgur.com/MhXyT3g.png https://i.imgur.com/MhXyT3g.png
- madez 9y ago
  Even simple .png pictures are not displayed on imgur.com without JavaScript.
  - userbinator 9y ago
    I have no JS and I can see the images fine.
  - Macha 9y ago
    imgur checks your referral header and if it can figure out you've navigated to the image directly (or there's no referral header), it will redirect you to the image viewer + ads page, instead of just displaying the image file. It's a pity, given the reasons imgur started, but it's the cycle of all image hosts I guess.
    - leggomylibro 9y ago
      The site doesn't even perform its basic functions on mobile anymore. Agreed, it's a shame, but such is image hosting.
- phoe-krk 9y ago
  Strange. I see the thread perfectly fine without logging in.
  - ceejayoz 9y ago
    The AWS forums are really weird. If you don't have any AWS Console cookies (incognito window, for example), they load fine, but if they know you've got an AWS account, they require a login.
    - MBCook 9y ago
      Sounds like Google forums.
- vvanders 9y ago
  25% bump in CPU usage from that graph, ouch.
  - acdha 9y ago
    It definitely shows how many people are running close to the limits, too. I have a couple of m1.mediums running ancillary workloads (e.g. rabbitmq) and it added something like ~0.5%. What's more interesting is later on in that thread where a support staffer noted that one of the people reporting problems was running out of memory because they'd apparently been very close to the max and that dropped slightly with the update. I had a problem with two test RDS instances suddenly failing into the invalid parameters state which turned out to be years-old config for the InnoDB pool size being just a little too large after a reboot reduced the available RAM by just enough to matter.
slizard 9y ago
First indications of early KAISER fix testing in the wild..? (which might have also been a test to see how will customers react)
- anarazel 9y ago
  I think this is possibly the xen PV side of the fix. Linux' PTI mitigation isn't enabled for paravirt xen: if (hypervisor_is_type(X86_HYPER_XEN_PV)) { pti_print_if_insecure("disabled on XEN PV."); return; }
  - justryry 9y ago
    Interesting. I took a look and this is the only hypervisor specific piece of code I can find in the patches. I have wondered what the impact would be on hypervisors. Xen seems like they patched it in a way that removes the need for guests to mitigate, but would guests of other hypervisors get hit with the penalty twice in some cases?
    - panarky 9y ago
      > removes the need for guests to mitigate Google says guests need to upgrade. "Compute Engine customers must update their virtual machine operating systems and applications so that their virtual machines are protected from intra-guest attacks and inter-guest attacks that exploit application-level vulnerabilities." "Compute Engine customers should work with their operating system provider(s) to download and install the necessary patches." https://support.google.com/faqs/answer/7622138#gce https://support.google.com/faqs/answer/7622138#gce
    - anarazel 9y ago
      Note that this is only for PV guests, which most people don't use anymore...
macintux 9y ago
"As the notice points out, the update that is being applied is important to maintain the high security and operational aspects for your instances." Yeah, definitely Intel Inside.
- yeukhon 9y ago
  AWS has always told customers they use both stock and custom Intel processors.
soccerdave 9y ago
So this sounds like this Intel bug will not impact performance on HVM instances.
- drvdevd 9y ago
  I don’t know enough about Xen to be sure, but I know typically your dom0 is a Linux kernel even (or especially) with HVM. If it’s been patched and is performing any paging on behalf of the guest, this will indeed affect performance of both paravirt and HVM instances.
- panarky 9y ago
  That's not what it says. Amazon suggests moving to higher-performance instances to offset offset the performance hit from the fix. HVM and m3 instances have higher performance for the same price, but they may also have been degraded by the fix.
  - jamesjoethomas 9y ago
    I don't think that's right, Ctrl-F for "HVM" here: https://xenbits.xen.org/xsa/advisory-254.html https://xenbits.xen.org/xsa/advisory-254.html In the HVM case an attacker can't generate hypervisor addresses because the hypervisor runs in a separate address space, so HVM isn't vulnerable to the most easily exploitable of the disclosed issues.
tuna 9y ago
PV vs HVM woes
phoe-krk 9y ago
If I understand this correctly, paravirtualization in this scenario means that the host kernel is impacted by the KAISER patch's overhead AND the guest kernel is impacted by the KAISER patch overhead, and that these overheads are multiplicative. Could someone more knowledgeable than me explain and/or verify my assumption?
- anarazel 9y ago
  No, I don't think that's correct. KAISER/KPTI isn't enabled on the guest side on xen PV instances: https://news.ycombinator.com/item?id=16065647 https://news.ycombinator.com/item?id=16065647
  - drvdevd 9y ago
    but since the host still has to perform some paging (and possibly for the guest) in paravirt, it should affect guest performance. In this snippet I think that is paravirt guest code so page tables don’t need to be isolated twice. So I think the host kernels’ perf degredation will still affect it.
    - anarazel 9y ago
      > So I think the host kernels’ perf degredation will still affect it. Oh, yes, definitely. I was just responding to "that these overheads are multiplicative." (I suspect additive rather than multiplicative was meant)
      - phoe-krk 9y ago
        No, I actually meant multiplicative. I was worried that the worst case is 30% syscall overhead on virtualization host multiplied by 30% syscall overhead on virtualization guest. Still, even if this mitigation is only enabled on Xen host and not on the guest, can this attack be then used inside the guest's userspace to read memory of the guest's kernel, as opposed to reading memory of the host's kernel?
blaisio 9y ago
Yeah this is due to the recent security bug related to speculative execution on CPUs. It causes a performance hit, and they have to force people who haven't restarted in a long time to restart now because it's around the official disclosure deadline.
justincormack 9y ago
"I have reviewed support case 4743634091 regarding what you're experiencing on your instance. You're correct that what you are seeing is not the same issue as what others are reporting. In the first correspondence from support they correctly pointed out that the kernel in your instance is encountering an out of memory (OOM) condition and made suggestions about how to adjust the configuration within your instance to avoid the OOM processor killer from kicking in. The update that is being applied to instances that have scheduled reboot maintenance can cause slight changes to system resources available to paravirtualized instances, including a small reduction in usable memory. This can cause smaller instances, like m1.medium, that run workloads that were previously just fitting within the usable memory available to the instance to trigger out of memory conditions. Adding a swap file (as no swap is configured in your instance) or reducing the number of processes may resolve the issue on your existing PV instance." ie there is less memory on PV after the fix, and this machine was nearly out and is OOMing. This is not a reflection of any significant performance hit with normal cases.
- Havoc 9y ago
  That's a different person than the one that created the support thread though.
- PuffinBlue 9y ago
  You've mis-read this section - it was answering another commenter who claimed to be having the same issue but was in fact having a different issue as you described in your quote. The underlying problem the update caused was acknowledged by an AWS rep in the thread who suggested moving to an instance with more CPU resources. > If moving to a HVM based AMI is not easy, changing your instance size to m3.medium, which provides more compute than m1.medium at a lower price, may be a workaround.
eob 9y ago
Unsure if this is related, but a few weeks back AWS paravirtualized instances were experiencing extreme clock skew and suffering frequent restarts. Our clocks were skewing about 8 seconds per hour, and the instances were getting swapped out at least once per day, sometimes more. Switching to a HVM instance type resolved the problems.
- kainosnoema 9y ago
  It's related. This was during the early versions of the security patch, which was causing performance and clock skew issues for PV instances (https://twitter.com/kainosnoema/status/941816042103324672 https://twitter.com/kainosnoema/status/941816042103324672). Since then AWS has mostly resolved these, but we've moved all our instances to HVM AMIs as well since they're apparently not affected.
panarky 9y ago
"I guess you are essentially confirming that the instance maintenance was likely to be the reason for the major change to cpu usage, and that the only solution now is to change instance type." "Of course I am not entirely happy about this. I bought a 3-year reserved instance 2 years ago, and now have to hope I can sell the remaining year for a reasonable amount (which may be a stretch given that I am apparently using legacy instance type), and then purchase new reserved instance after upgrading."
- skywhopper 9y ago
  Sounds like all instance types can now be run in HVM mode (news to me, but the AWS tech in the thread made the claim). Converting to HVM is annoying but ultimately not that hard or risky. Since HVM mode appears to resolve much of the performance issue, this seems like a reasonable workaround.
  - gregdunn 9y ago
    >Sounds like all instance types can now be run in HVM mode (news to me, but the AWS tech in the thread made the claim) https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtuali... "All instance types support HVM AMIs." This is a recent change. (I work at AWS, not directly involved with this change and not speaking in any sort of official capacity, but can confirm you can use HVM on m1, etc, now)
- taf2 9y ago
  Or migrate to google since they don’t have a broken discounting billing system
  - RhodesianHunter 9y ago
    I'll stick with the vastly superior offering and swallow the occasional hiccup, but thanks.
    - 9y ago
kreitje 9y ago
We ran into this and finally made the switch this morning to an HVM ami. The performance has been much better. We started seeing issues on Dec 23rd, with HDD reads and it magically went away the night of Jan 1st. The cpu loads remained high though.
newman314 9y ago
Given that Google has been experimenting with POWER, does anyone know if POWER is affected too?
- ghaff 9y ago
  Apparently potentially: Red Hat Security statement https://access.redhat.com/security/vulnerabilities/speculativeexecution https://access.redhat.com/security/vulnerabilities/speculati...
  - newman314 9y ago
    IBM has confirmed that POWER is vulnerable. See https://www.ibm.com/blogs/psirt/potential-impact-processors-power-family/ https://www.ibm.com/blogs/psirt/potential-impact-processors-... Linux patches to drop 1/9, AIX and i to drop 2/12.
- robin_reala 9y ago
  Seems that way: http://tenfourfox.blogspot.se/2018/01/is-powerpc-susceptible-to-spectre-yep.html http://tenfourfox.blogspot.se/2018/01/is-powerpc-susceptible...
amluto 9y ago
Two possibilities: 1. Your distro has PTI/KAISER enabled and the reboot caused you to upgrade your guest kernel. Sucks to be running on an Intel CPU right now... 2. AWS implemented some kind of hypervisor-side mitigation for the VM attack that made VM entries and exits slower. Sucks to be running on an affected CPU (which seems to include AMD, too, but I could be wrong).
- chrisper 9y ago
  I am pretty sure the patch needs to be applied on the host kernel and not the guest.
  - rickycook 9y ago
    from what i understand, it’s both. if you don’t protect the hypervisor, the guest can break through and read it’s ram. if you don’t protect the host, your apps can break into your own OS and see it’s ram. i could be wrong; if i am, clarification would be good!
    - chrisper 9y ago
      Yes it is both now. The host patch protects against vm-to-vm leaks, but you still need to protect your vm against application-to-application leaks. When I wrote the other comment it was at the time I did not have all the information we have now. Like it took me some time to read into it.
      - amluto 9y ago
        The host PTI patch does nothing whatsoever to protect from attacks initiated by a guest. (Well, if host QEMU gets compromised, PTI protects the host kernel a bit.)
  - alpb 9y ago
    This is not true. https://cloud.google.com/compute/docs/security-bulletins https://cloud.google.com/compute/docs/security-bulletins says: > However, all guest operating systems and versions must be patched to protect against this new class of attack regardless of where those systems run.

302 points by woliveirajr 9y ago

Are we presuming this is the Intel patch being applied?

for those who don't want to log in, https://i.imgur.com/f76tOvY.png https://i.imgur.com/f76tOvY.png https://i.imgur.com/MhXyT3g.png https://i.imgur.com/MhXyT3g.png

madez 9y ago
Even simple .png pictures are not displayed on imgur.com without JavaScript.
- userbinator 9y ago
  I have no JS and I can see the images fine.
- Macha 9y ago
  imgur checks your referral header and if it can figure out you've navigated to the image directly (or there's no referral header), it will redirect you to the image viewer + ads page, instead of just displaying the image file. It's a pity, given the reasons imgur started, but it's the cycle of all image hosts I guess.
  - leggomylibro 9y ago
    The site doesn't even perform its basic functions on mobile anymore. Agreed, it's a shame, but such is image hosting.
phoe-krk 9y ago
Strange. I see the thread perfectly fine without logging in.
- ceejayoz 9y ago
  The AWS forums are really weird. If you don't have any AWS Console cookies (incognito window, for example), they load fine, but if they know you've got an AWS account, they require a login.
  - MBCook 9y ago
    Sounds like Google forums.
vvanders 9y ago
25% bump in CPU usage from that graph, ouch.
- acdha 9y ago
  It definitely shows how many people are running close to the limits, too. I have a couple of m1.mediums running ancillary workloads (e.g. rabbitmq) and it added something like ~0.5%. What's more interesting is later on in that thread where a support staffer noted that one of the people reporting problems was running out of memory because they'd apparently been very close to the max and that dropped slightly with the update. I had a problem with two test RDS instances suddenly failing into the invalid parameters state which turned out to be years-old config for the InnoDB pool size being just a little too large after a reboot reduced the available RAM by just enough to matter.

slizard 9y ago

First indications of early KAISER fix testing in the wild..? (which might have also been a test to see how will customers react)

anarazel 9y ago
I think this is possibly the xen PV side of the fix. Linux' PTI mitigation isn't enabled for paravirt xen: if (hypervisor_is_type(X86_HYPER_XEN_PV)) { pti_print_if_insecure("disabled on XEN PV."); return; }
- justryry 9y ago
  Interesting. I took a look and this is the only hypervisor specific piece of code I can find in the patches. I have wondered what the impact would be on hypervisors. Xen seems like they patched it in a way that removes the need for guests to mitigate, but would guests of other hypervisors get hit with the penalty twice in some cases?
  - panarky 9y ago
    > removes the need for guests to mitigate Google says guests need to upgrade. "Compute Engine customers must update their virtual machine operating systems and applications so that their virtual machines are protected from intra-guest attacks and inter-guest attacks that exploit application-level vulnerabilities." "Compute Engine customers should work with their operating system provider(s) to download and install the necessary patches." https://support.google.com/faqs/answer/7622138#gce https://support.google.com/faqs/answer/7622138#gce
  - anarazel 9y ago
    Note that this is only for PV guests, which most people don't use anymore...

macintux 9y ago

"As the notice points out, the update that is being applied is important to maintain the high security and operational aspects for your instances." Yeah, definitely Intel Inside.

yeukhon 9y ago
AWS has always told customers they use both stock and custom Intel processors.

soccerdave 9y ago

So this sounds like this Intel bug will not impact performance on HVM instances.

drvdevd 9y ago
I don’t know enough about Xen to be sure, but I know typically your dom0 is a Linux kernel even (or especially) with HVM. If it’s been patched and is performing any paging on behalf of the guest, this will indeed affect performance of both paravirt and HVM instances.
panarky 9y ago
That's not what it says. Amazon suggests moving to higher-performance instances to offset offset the performance hit from the fix. HVM and m3 instances have higher performance for the same price, but they may also have been degraded by the fix.
- jamesjoethomas 9y ago
  I don't think that's right, Ctrl-F for "HVM" here: https://xenbits.xen.org/xsa/advisory-254.html https://xenbits.xen.org/xsa/advisory-254.html In the HVM case an attacker can't generate hypervisor addresses because the hypervisor runs in a separate address space, so HVM isn't vulnerable to the most easily exploitable of the disclosed issues.

tuna 9y ago

PV vs HVM woes

phoe-krk 9y ago

If I understand this correctly, paravirtualization in this scenario means that the host kernel is impacted by the KAISER patch's overhead AND the guest kernel is impacted by the KAISER patch overhead, and that these overheads are multiplicative. Could someone more knowledgeable than me explain and/or verify my assumption?

anarazel 9y ago
No, I don't think that's correct. KAISER/KPTI isn't enabled on the guest side on xen PV instances: https://news.ycombinator.com/item?id=16065647 https://news.ycombinator.com/item?id=16065647
- drvdevd 9y ago
  but since the host still has to perform some paging (and possibly for the guest) in paravirt, it should affect guest performance. In this snippet I think that is paravirt guest code so page tables don’t need to be isolated twice. So I think the host kernels’ perf degredation will still affect it.
  - anarazel 9y ago
    > So I think the host kernels’ perf degredation will still affect it. Oh, yes, definitely. I was just responding to "that these overheads are multiplicative." (I suspect additive rather than multiplicative was meant)
    - phoe-krk 9y ago
      No, I actually meant multiplicative. I was worried that the worst case is 30% syscall overhead on virtualization host multiplied by 30% syscall overhead on virtualization guest. Still, even if this mitigation is only enabled on Xen host and not on the guest, can this attack be then used inside the guest's userspace to read memory of the guest's kernel, as opposed to reading memory of the host's kernel?

blaisio 9y ago

Yeah this is due to the recent security bug related to speculative execution on CPUs. It causes a performance hit, and they have to force people who haven't restarted in a long time to restart now because it's around the official disclosure deadline.

justincormack 9y ago

"I have reviewed support case 4743634091 regarding what you're experiencing on your instance. You're correct that what you are seeing is not the same issue as what others are reporting. In the first correspondence from support they correctly pointed out that the kernel in your instance is encountering an out of memory (OOM) condition and made suggestions about how to adjust the configuration within your instance to avoid the OOM processor killer from kicking in. The update that is being applied to instances that have scheduled reboot maintenance can cause slight changes to system resources available to paravirtualized instances, including a small reduction in usable memory. This can cause smaller instances, like m1.medium, that run workloads that were previously just fitting within the usable memory available to the instance to trigger out of memory conditions. Adding a swap file (as no swap is configured in your instance) or reducing the number of processes may resolve the issue on your existing PV instance." ie there is less memory on PV after the fix, and this machine was nearly out and is OOMing. This is not a reflection of any significant performance hit with normal cases.

Havoc 9y ago
That's a different person than the one that created the support thread though.
PuffinBlue 9y ago
You've mis-read this section - it was answering another commenter who claimed to be having the same issue but was in fact having a different issue as you described in your quote. The underlying problem the update caused was acknowledged by an AWS rep in the thread who suggested moving to an instance with more CPU resources. > If moving to a HVM based AMI is not easy, changing your instance size to m3.medium, which provides more compute than m1.medium at a lower price, may be a workaround.

eob 9y ago

Unsure if this is related, but a few weeks back AWS paravirtualized instances were experiencing extreme clock skew and suffering frequent restarts. Our clocks were skewing about 8 seconds per hour, and the instances were getting swapped out at least once per day, sometimes more. Switching to a HVM instance type resolved the problems.

kainosnoema 9y ago
It's related. This was during the early versions of the security patch, which was causing performance and clock skew issues for PV instances (https://twitter.com/kainosnoema/status/941816042103324672 https://twitter.com/kainosnoema/status/941816042103324672). Since then AWS has mostly resolved these, but we've moved all our instances to HVM AMIs as well since they're apparently not affected.

panarky 9y ago

"I guess you are essentially confirming that the instance maintenance was likely to be the reason for the major change to cpu usage, and that the only solution now is to change instance type." "Of course I am not entirely happy about this. I bought a 3-year reserved instance 2 years ago, and now have to hope I can sell the remaining year for a reasonable amount (which may be a stretch given that I am apparently using legacy instance type), and then purchase new reserved instance after upgrading."

skywhopper 9y ago
Sounds like all instance types can now be run in HVM mode (news to me, but the AWS tech in the thread made the claim). Converting to HVM is annoying but ultimately not that hard or risky. Since HVM mode appears to resolve much of the performance issue, this seems like a reasonable workaround.
- gregdunn 9y ago
  >Sounds like all instance types can now be run in HVM mode (news to me, but the AWS tech in the thread made the claim) https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualization_types.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtuali... "All instance types support HVM AMIs." This is a recent change. (I work at AWS, not directly involved with this change and not speaking in any sort of official capacity, but can confirm you can use HVM on m1, etc, now)
taf2 9y ago
Or migrate to google since they don’t have a broken discounting billing system
- RhodesianHunter 9y ago
  I'll stick with the vastly superior offering and swallow the occasional hiccup, but thanks.
  - 9y ago

kreitje 9y ago

We ran into this and finally made the switch this morning to an HVM ami. The performance has been much better. We started seeing issues on Dec 23rd, with HDD reads and it magically went away the night of Jan 1st. The cpu loads remained high though.

newman314 9y ago

Given that Google has been experimenting with POWER, does anyone know if POWER is affected too?

ghaff 9y ago
Apparently potentially: Red Hat Security statement https://access.redhat.com/security/vulnerabilities/speculativeexecution https://access.redhat.com/security/vulnerabilities/speculati...
- newman314 9y ago
  IBM has confirmed that POWER is vulnerable. See https://www.ibm.com/blogs/psirt/potential-impact-processors-power-family/ https://www.ibm.com/blogs/psirt/potential-impact-processors-... Linux patches to drop 1/9, AIX and i to drop 2/12.
robin_reala 9y ago
Seems that way: http://tenfourfox.blogspot.se/2018/01/is-powerpc-susceptible-to-spectre-yep.html http://tenfourfox.blogspot.se/2018/01/is-powerpc-susceptible...

amluto 9y ago

Two possibilities: 1. Your distro has PTI/KAISER enabled and the reboot caused you to upgrade your guest kernel. Sucks to be running on an Intel CPU right now... 2. AWS implemented some kind of hypervisor-side mitigation for the VM attack that made VM entries and exits slower. Sucks to be running on an affected CPU (which seems to include AMD, too, but I could be wrong).

chrisper 9y ago
I am pretty sure the patch needs to be applied on the host kernel and not the guest.
- rickycook 9y ago
  from what i understand, it’s both. if you don’t protect the hypervisor, the guest can break through and read it’s ram. if you don’t protect the host, your apps can break into your own OS and see it’s ram. i could be wrong; if i am, clarification would be good!
  - chrisper 9y ago
    Yes it is both now. The host patch protects against vm-to-vm leaks, but you still need to protect your vm against application-to-application leaks. When I wrote the other comment it was at the time I did not have all the information we have now. Like it took me some time to read into it.
    - amluto 9y ago
      The host PTI patch does nothing whatsoever to protect from attacks initiated by a guest. (Well, if host QEMU gets compromised, PTI protects the host kernel a bit.)
- alpb 9y ago
  This is not true. https://cloud.google.com/compute/docs/security-bulletins https://cloud.google.com/compute/docs/security-bulletins says: > However, all guest operating systems and versions must be patched to protect against this new class of attack regardless of where those systems run.