3 ms·

Do a ping test between two machines in your local network. Turns out it’s very hard to get sub millisecond speeds. Especially sub 500 microseconds. the kernel

Do a ping test between two machines in your local network. Turns out it’s very hard to get sub millisecond speeds. Especially sub 500 microseconds. the kernel will buffer and queue packets vs sending them out asap. So there’s delay added at your host level and networking layer. The. You’ve got the speed of light traversing that distance. All this adds latency.

londons_explore 5y ago
But something like a filesystem should probably be built on RDMA to get the kernel mostly out-of-the-way. Then you should see performance like this: https://www.microway.com/knowledge-center-articles/performance-characteristics-of-common-network-fabrics/ https://www.microway.com/knowledge-center-articles/performan... And the speed of light is 1000 feet per microsecond, so even the largest data centers won't add more than a handful of microseconds.
jeffbee 5y ago
I just put a crossover cable between a Mac mini and a random Linux box and with no tuning the median ping is 180 microseconds.
- BenjiWiebe 5y ago
  Now add some routing and switching... Or is there no routing between the storage and the application servers in a data center?
  - londons_explore 5y ago
    1.3 microseconds per switch is typical on 10GigE ethernet. I doubt your message goes through more than 10 switches even in amazons data center.
    - jeffbee 5y ago
      RTTs are extremely low in EC2, and I doubt EC2 is wasting any fibers on a mere 10ge link.
pgwhalen 5y ago
> Turns out it’s very hard to get sub millisecond speeds. Especially sub 500 microseconds. I have to contest this a bit. The company I work for doesn't put much effort into latency from what I can tell, and sees ~200us between servers in the same data center, and ~400us to another data center 12 miles away.
dataangel 5y ago
Uh… no. PCI-E bus is about 900ns on each end, switch adds 300ns. You can get under 3us ping pong, you’re off by orders of magnitude. Your sense is probably coming from bad OS stacks. Kernel bypass like Amazon uses achieves this easily.

londons_explore 5y ago

But something like a filesystem should probably be built on RDMA to get the kernel mostly out-of-the-way. Then you should see performance like this: https://www.microway.com/knowledge-center-articles/performance-characteristics-of-common-network-fabrics/ https://www.microway.com/knowledge-center-articles/performan... And the speed of light is 1000 feet per microsecond, so even the largest data centers won't add more than a handful of microseconds.

jeffbee 5y ago

I just put a crossover cable between a Mac mini and a random Linux box and with no tuning the median ping is 180 microseconds.

BenjiWiebe 5y ago
Now add some routing and switching... Or is there no routing between the storage and the application servers in a data center?
- londons_explore 5y ago
  1.3 microseconds per switch is typical on 10GigE ethernet. I doubt your message goes through more than 10 switches even in amazons data center.
  - jeffbee 5y ago
    RTTs are extremely low in EC2, and I doubt EC2 is wasting any fibers on a mere 10ge link.

pgwhalen 5y ago

> Turns out it’s very hard to get sub millisecond speeds. Especially sub 500 microseconds. I have to contest this a bit. The company I work for doesn't put much effort into latency from what I can tell, and sees ~200us between servers in the same data center, and ~400us to another data center 12 miles away.

dataangel 5y ago

Uh… no. PCI-E bus is about 900ns on each end, switch adds 300ns. You can get under 3us ping pong, you’re off by orders of magnitude. Your sense is probably coming from bad OS stacks. Kernel bypass like Amazon uses achieves this easily.