5 ms·

Google Compute Engine MPI Latency

37 points by gpoort 13y ago

montecarl 13y ago
This isn't that interesting or surprising. 100-200 microseconds is the latency of all ethernet that I have ever seen. Infiniband or other high performance networks can achieve about 10-100x lower latency, but are very expensive. Infiniband switches and cards can be double the cost of a cluster.
- sargun 13y ago
  One of the things that Google is also working on is PCIe switching, and using that to get rid of having to the ethernet encapsulation and conversion. This allows for significantly lower latencies <5 μs.
  - montecarl 13y ago
    That sounds awesome. Do you have any links to describe this research? I would be interested in the technology if it stands to lower the cost of low latency networks.
- gpoort 13y ago
  AWS is still using ethernet (10 Gigabit on the highest end) and can achieve ~2x better performance
  - montecarl 13y ago
    I have no experience with 10 gigabit ethernet. However, it seems that there are some lower latency (down to sub millisecond) switching fabrics.
- wmf 13y ago
  Just for comparison, on non-virtualized 10GbE I think we got ~30 us. I would guess the extra latency in the cloud is caused by the hypervisor. Also, last I checked 16 Gbps Infiniband was cheaper than 10 Gbps Ethernet but people don't want to use verbs.
- rb12345 13y ago
  This is slightly off-topic, but... There's plenty of references to physical Infiniband cards being allocated to a guest VM. What I haven't seen though is a hypervisor-level Infiniband vswitch. I wonder if that could help with some of the latency issues; for example, inter-VM memory access could be done via host-level shared memory. Ideally the vswitch could handle inter-host links too, saving the need for manual assignment of cards to VMs.
  - wmf 13y ago
    Mellanox did some experiments a few years ago that showed that DMAing to the NIC and back was actually faster than going through dom0 in software. Things may have changed since then, although it's probably irrelevant since the people who want to use RDMA and the people who want to use VMs don't overlap much.
codemac 13y ago
To me this is where the "software defined networking" type of virtualization can really make an impact. The network performance of a known cluster of virtualized instances could be extremely quick if you just lie, and say your packet went through a network, when really you just pass a pointer in the hypervisor.. I assume this has already been done, but at almost 200 microseconds, you know it hasn't been done in these experiments.
- montecarl 13y ago
  That is only the case for networking inside of a single physical machine. Most HPC MPI use cases span many machines, for which this is typical latency of ethernet.
  - codemac 13y ago
    In cases where I've used OpenMPI it was spanning machines as well as being multiple processes on the same box. The goal was making that interchangeable (in my use case). I imagine google's compute engine isn't all on separate machines, but utilizes VMs heavily.. although it's probably a bad idea to put one customer's VM's on all the same box. That's a long way of saying "of course you're right, I guess my thought doesn't contribute as much as I thought" :)
    - montecarl 13y ago
      You are also right of course. Intra-machine latency is quite important. Many problems can be decomposed in to smaller parallel parts that can be done per machine.
- ori_b 13y ago
  That's an interesting hack for MPI, but I don't think it would be worth the complexity of implementing it. There are two possible use cases here: One: You're testing multiple VMs on one box to simulate a network. This is a sane use case, but a bit of overhead from network latency isn't going to kill you. It definitely isn't worth the effort of messing around with faking your network. Two: You're trying to wring whatever performance you can out of a single box. In this case, you can ditch the VMs, and just run your MPI nodes as threads or processes on the local machine, using far faster and more efficient transports than TCP. Also, in this specific case, note that the instances are not necessarily on the same machine.

37 points by gpoort 13y ago

This isn't that interesting or surprising. 100-200 microseconds is the latency of all ethernet that I have ever seen. Infiniband or other high performance networks can achieve about 10-100x lower latency, but are very expensive. Infiniband switches and cards can be double the cost of a cluster.

sargun 13y ago
One of the things that Google is also working on is PCIe switching, and using that to get rid of having to the ethernet encapsulation and conversion. This allows for significantly lower latencies <5 μs.
- montecarl 13y ago
  That sounds awesome. Do you have any links to describe this research? I would be interested in the technology if it stands to lower the cost of low latency networks.
gpoort 13y ago
AWS is still using ethernet (10 Gigabit on the highest end) and can achieve ~2x better performance
- montecarl 13y ago
  I have no experience with 10 gigabit ethernet. However, it seems that there are some lower latency (down to sub millisecond) switching fabrics.
wmf 13y ago
Just for comparison, on non-virtualized 10GbE I think we got ~30 us. I would guess the extra latency in the cloud is caused by the hypervisor. Also, last I checked 16 Gbps Infiniband was cheaper than 10 Gbps Ethernet but people don't want to use verbs.
rb12345 13y ago
This is slightly off-topic, but... There's plenty of references to physical Infiniband cards being allocated to a guest VM. What I haven't seen though is a hypervisor-level Infiniband vswitch. I wonder if that could help with some of the latency issues; for example, inter-VM memory access could be done via host-level shared memory. Ideally the vswitch could handle inter-host links too, saving the need for manual assignment of cards to VMs.
- wmf 13y ago
  Mellanox did some experiments a few years ago that showed that DMAing to the NIC and back was actually faster than going through dom0 in software. Things may have changed since then, although it's probably irrelevant since the people who want to use RDMA and the people who want to use VMs don't overlap much.

codemac 13y ago

To me this is where the "software defined networking" type of virtualization can really make an impact. The network performance of a known cluster of virtualized instances could be extremely quick if you just lie, and say your packet went through a network, when really you just pass a pointer in the hypervisor.. I assume this has already been done, but at almost 200 microseconds, you know it hasn't been done in these experiments.

montecarl 13y ago
That is only the case for networking inside of a single physical machine. Most HPC MPI use cases span many machines, for which this is typical latency of ethernet.
- codemac 13y ago
  In cases where I've used OpenMPI it was spanning machines as well as being multiple processes on the same box. The goal was making that interchangeable (in my use case). I imagine google's compute engine isn't all on separate machines, but utilizes VMs heavily.. although it's probably a bad idea to put one customer's VM's on all the same box. That's a long way of saying "of course you're right, I guess my thought doesn't contribute as much as I thought" :)
  - montecarl 13y ago
    You are also right of course. Intra-machine latency is quite important. Many problems can be decomposed in to smaller parallel parts that can be done per machine.
ori_b 13y ago
That's an interesting hack for MPI, but I don't think it would be worth the complexity of implementing it. There are two possible use cases here: One: You're testing multiple VMs on one box to simulate a network. This is a sane use case, but a bit of overhead from network latency isn't going to kill you. It definitely isn't worth the effort of messing around with faking your network. Two: You're trying to wring whatever performance you can out of a single box. In this case, you can ditch the VMs, and just run your MPI nodes as threads or processes on the local machine, using far faster and more efficient transports than TCP. Also, in this specific case, note that the instances are not necessarily on the same machine.