5 ms·

This isn't that interesting or surprising. 100-200 microseconds is the latency of all ethernet that I have ever seen. Infiniband or other high performance netwo

This isn't that interesting or surprising. 100-200 microseconds is the latency of all ethernet that I have ever seen. Infiniband or other high performance networks can achieve about 10-100x lower latency, but are very expensive. Infiniband switches and cards can be double the cost of a cluster.

sargun 13y ago
One of the things that Google is also working on is PCIe switching, and using that to get rid of having to the ethernet encapsulation and conversion. This allows for significantly lower latencies <5 μs.
- montecarl 13y ago
  That sounds awesome. Do you have any links to describe this research? I would be interested in the technology if it stands to lower the cost of low latency networks.
gpoort 13y ago
AWS is still using ethernet (10 Gigabit on the highest end) and can achieve ~2x better performance
- montecarl 13y ago
  I have no experience with 10 gigabit ethernet. However, it seems that there are some lower latency (down to sub millisecond) switching fabrics.
wmf 13y ago
Just for comparison, on non-virtualized 10GbE I think we got ~30 us. I would guess the extra latency in the cloud is caused by the hypervisor. Also, last I checked 16 Gbps Infiniband was cheaper than 10 Gbps Ethernet but people don't want to use verbs.
rb12345 13y ago
This is slightly off-topic, but... There's plenty of references to physical Infiniband cards being allocated to a guest VM. What I haven't seen though is a hypervisor-level Infiniband vswitch. I wonder if that could help with some of the latency issues; for example, inter-VM memory access could be done via host-level shared memory. Ideally the vswitch could handle inter-host links too, saving the need for manual assignment of cards to VMs.
- wmf 13y ago
  Mellanox did some experiments a few years ago that showed that DMAing to the NIC and back was actually faster than going through dom0 in software. Things may have changed since then, although it's probably irrelevant since the people who want to use RDMA and the people who want to use VMs don't overlap much.

sargun 13y ago

One of the things that Google is also working on is PCIe switching, and using that to get rid of having to the ethernet encapsulation and conversion. This allows for significantly lower latencies <5 μs.

montecarl 13y ago
That sounds awesome. Do you have any links to describe this research? I would be interested in the technology if it stands to lower the cost of low latency networks.

gpoort 13y ago

AWS is still using ethernet (10 Gigabit on the highest end) and can achieve ~2x better performance

montecarl 13y ago
I have no experience with 10 gigabit ethernet. However, it seems that there are some lower latency (down to sub millisecond) switching fabrics.

wmf 13y ago

Just for comparison, on non-virtualized 10GbE I think we got ~30 us. I would guess the extra latency in the cloud is caused by the hypervisor. Also, last I checked 16 Gbps Infiniband was cheaper than 10 Gbps Ethernet but people don't want to use verbs.

rb12345 13y ago

This is slightly off-topic, but... There's plenty of references to physical Infiniband cards being allocated to a guest VM. What I haven't seen though is a hypervisor-level Infiniband vswitch. I wonder if that could help with some of the latency issues; for example, inter-VM memory access could be done via host-level shared memory. Ideally the vswitch could handle inter-host links too, saving the need for manual assignment of cards to VMs.

wmf 13y ago
Mellanox did some experiments a few years ago that showed that DMAing to the NIC and back was actually faster than going through dom0 in software. Things may have changed since then, although it's probably irrelevant since the people who want to use RDMA and the people who want to use VMs don't overlap much.