4 ms·

Would love to know if anyone else had data on: * Impact on M5/C5 instances over similar time period, any difference with the Nitro hypervisor? * Were Dedicate

Would love to know if anyone else had data on: * Impact on M5/C5 instances over similar time period, any difference with the Nitro hypervisor? * Were Dedicated instances (https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst...) patched as well? * Other examples of software that adapted batching performance automatically with increase in call latency.

taf2 9y ago
We had a lot of m5 and c5 servers randomly die. It was as if someone was running chaos monkey from Netflix in our VPCs...
- otterley 9y ago
  Likewise. Can you reach out to me privately? I'd love to have independent corroboration.
- _msw_ 9y ago
  Could you send a list of instance IDs and timeframes where you saw this?
cthalupa 9y ago
Not able to answer your questions, but a comment on the article - >During this same time period, we saw additional CPU increases on our PV instances that had been previously upgraded. This seems to imply some level of HVM patching was occurring on these PV instances around the same time that all pure-HVM instances were patched This is likely due to Vixen: https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00436.html https://lists.xenproject.org/archives/html/xen-devel/2018-01... >.... Instead of trying to make a KPTI-like approach work for Xen PV, it seems reasonable to run a copy of Xen within an HVM (or PVH) domU ..... >.... all PV instances in EC2 are using this .... So the initial bump after the reboot would have been the shim hypervisor which mitigates Vixen. The secondary bump, and bump the native HVM instances saw, would have been the Spectre related stuff. Based on https://aws.amazon.com/security/security-bulletins/AWS-2018-013/ https://aws.amazon.com/security/security-bulletins/AWS-2018-... - guessing Intel microcode updates

taf2 9y ago

We had a lot of m5 and c5 servers randomly die. It was as if someone was running chaos monkey from Netflix in our VPCs...

otterley 9y ago
Likewise. Can you reach out to me privately? I'd love to have independent corroboration.
_msw_ 9y ago
Could you send a list of instance IDs and timeframes where you saw this?

cthalupa 9y ago

Not able to answer your questions, but a comment on the article - >During this same time period, we saw additional CPU increases on our PV instances that had been previously upgraded. This seems to imply some level of HVM patching was occurring on these PV instances around the same time that all pure-HVM instances were patched This is likely due to Vixen: https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00436.html https://lists.xenproject.org/archives/html/xen-devel/2018-01... >.... Instead of trying to make a KPTI-like approach work for Xen PV, it seems reasonable to run a copy of Xen within an HVM (or PVH) domU ..... >.... all PV instances in EC2 are using this .... So the initial bump after the reboot would have been the shim hypervisor which mitigates Vixen. The secondary bump, and bump the native HVM instances saw, would have been the Spectre related stuff. Based on https://aws.amazon.com/security/security-bulletins/AWS-2018-013/ https://aws.amazon.com/security/security-bulletins/AWS-2018-... - guessing Intel microcode updates