4 ms·

Re: NUMA, I had thought it was fairly straightforward to peg VMs to particular NUMA nodes (or declare CPU or memory affinity) in VMware. I agree GCP does it w

Re: NUMA, I had thought it was fairly straightforward to peg VMs to particular NUMA nodes (or declare CPU or memory affinity) in VMware. I agree GCP does it well (better than the other public clouds!), but I've also seen this done well on-prem.

dijit 8y ago
I have too, which is why I singled out hyper-converged. In hyper converged environments the storage and the compute live together on a service mesh and thus some processing is done on the hypervisor to manage storage, those processes don't have an affinity (in the case of vmware for example) so a guest VM core gets suspended for a couple of hundred clock ticks sometimes.
- _msw_ 8y ago
  Hi! I'm from the EC2 engineering team. This is a super challenging problem for general purpose virtualization stacks. EC2 has been working to avoid any guest VM interruptions. With the Nitro hypervisor, there are no management tasks that share CPUs with guest workloads. See more here, including data from a real customer with real-time requirements running on EC2 under the Nitro hypervisor: https://youtu.be/e8DVmwj3OEs?t=1796 https://youtu.be/e8DVmwj3OEs?t=1796
- parasubvert 8y ago
  That's a good point, I have seen cases where we've separated a portion of nodes for VSAN or ScaleIO out to a separate cluster to avoid this behaviour on latency sensitive workloads. These nodes don't need a ton of RAM and you'd better have good E-W bandwidth for them, but it's always a tradeoff...
_msw_ 8y ago
Hi! I'm from the EC2 team. This is a great tool to help uncover if CPU and memory NUMA within a VM aligns to the physical host or not: https://www.cl.cam.ac.uk/research/srg/netos/projects/ipc-bench/ https://www.cl.cam.ac.uk/research/srg/netos/projects/ipc-ben... Paper from the USENIX 2012 conference: http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.pdf http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.... EC2 instances have been NUMA optimized since we launched our CC1 instances in 2010. I encourage you to try ipc-bench on your cloud provider of choice, if CPU and NUMA affinity is important to your workload.

dijit 8y ago

I have too, which is why I singled out hyper-converged. In hyper converged environments the storage and the compute live together on a service mesh and thus some processing is done on the hypervisor to manage storage, those processes don't have an affinity (in the case of vmware for example) so a guest VM core gets suspended for a couple of hundred clock ticks sometimes.

_msw_ 8y ago
Hi! I'm from the EC2 engineering team. This is a super challenging problem for general purpose virtualization stacks. EC2 has been working to avoid any guest VM interruptions. With the Nitro hypervisor, there are no management tasks that share CPUs with guest workloads. See more here, including data from a real customer with real-time requirements running on EC2 under the Nitro hypervisor: https://youtu.be/e8DVmwj3OEs?t=1796 https://youtu.be/e8DVmwj3OEs?t=1796
parasubvert 8y ago
That's a good point, I have seen cases where we've separated a portion of nodes for VSAN or ScaleIO out to a separate cluster to avoid this behaviour on latency sensitive workloads. These nodes don't need a ton of RAM and you'd better have good E-W bandwidth for them, but it's always a tradeoff...

_msw_ 8y ago

Hi! I'm from the EC2 team. This is a great tool to help uncover if CPU and memory NUMA within a VM aligns to the physical host or not: https://www.cl.cam.ac.uk/research/srg/netos/projects/ipc-bench/ https://www.cl.cam.ac.uk/research/srg/netos/projects/ipc-ben... Paper from the USENIX 2012 conference: http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.pdf http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.... EC2 instances have been NUMA optimized since we launched our CC1 instances in 2010. I encourage you to try ipc-bench on your cloud provider of choice, if CPU and NUMA affinity is important to your workload.