5 ms·

Do you mean 1-2ms?

by gtirloni 2y ago

Do you mean 1-2ms?

eqvinox 2y ago
No, 1-2us is correct for that — in a datacenter, with cut-through switching.
- gtirloni 2y ago
  That's really impressive. I need to update myself on this topic. Thanks.
  - mickg10 2y ago
    In reality - with decent switches at 25g - and no fec - node to node is reliably under 300ns (0.3 us)
    - davekeck 2y ago
      Out of curiosity, how is that measured across machines? (The first thing that comes to my mind would be to use an oscilloscope with two probes, one to each machine, but I’m guessing that’s not it.)
      - toast0 2y ago
        Measure the round trip and divide by two for the approximate one way time. It'd be really neat to measure the time it takes for a packet to travel in one direction, but it's somewhere between hard and impossible[1]; a very short path has less room to be asymetric though. [1] If the clocks are synchronized, you can measure send time on one end, and receive time on the other. But synchronizing clocks involves estimating the time it takes for signals to pass im each direction, typically assuming each direction takes half the round trip.
        pkhuong 2y ago
        You can use something like White Rabbit (https://en.wikipedia.org/wiki/White_Rabbit_Project https://en.wikipedia.org/wiki/White_Rabbit_Project) to keep clocks in sync. That still involves estimates, but a dedicated time sync network can do things like make sure all the cables are the same length.
        namibj 2y ago
        Copper white rabbit is special, it uses the same wire in both directions (1000BASE-T phy with added carrier phase lock to and from outside clocks).
    - deleted 2y ago
      [deleted]
    - znyboy 2y ago
      Considering that 300 light-nanoseconds is about 90m, getting a response (or even just one-way) in that time is essentially running right at the limits of physics/causality.
      - deleted 2y ago
        [deleted]
- jiggawatts 2y ago
  Meanwhile the best network I’ve ever benchmarked was AWS and measured about 55µs for a round trip! What on earth are you using that gets you down to single digits!?
  - deleted 2y ago
    [deleted]
  - crest 2y ago
    I assume 1-3 hops of modern switches without congestion. Given 100Gb/s lanes these numbers are possible if you get all the bottlenecks out of the way. The moment you hit a deep queue the latency explodes.
    - jiggawatts 2y ago
      So, are you talking about theoretical latencies here based on bandwidths and cable lengths, or actual measured latencies end-to-end between hosts? I know that "in principle" the physics of the cabling allows single digit microseconds, but I've never seen it anywhere near that low even with cross-over cables with zero switches in-path!
      - eqvinox 2y ago
        You need high bandwidth links (time to get the entire packet across starts to matter), run on bare metal (or have very well working HW virtualisation support), and tune NIC parameters and OS processing appropriately. But it's practically achievable. Switches in these scenarios (e.g. 25GE DC targeted) are pretty predictable and add <1μs (unless misconfigured)
        jiggawatts 2y ago
        > But it's practically achievable. I've never seen this in practice. Maaaaybe with Infiniband and custom-written apps that use a proprietary SDK. I'd love to see references to actual benchmarks.
  - 2y ago

Do you mean 1-2ms?

eqvinox 2y ago

No, 1-2us is correct for that — in a datacenter, with cut-through switching.

gtirloni 2y ago
That's really impressive. I need to update myself on this topic. Thanks.
- mickg10 2y ago
  In reality - with decent switches at 25g - and no fec - node to node is reliably under 300ns (0.3 us)
  - davekeck 2y ago
    Out of curiosity, how is that measured across machines? (The first thing that comes to my mind would be to use an oscilloscope with two probes, one to each machine, but I’m guessing that’s not it.)
    - toast0 2y ago
      Measure the round trip and divide by two for the approximate one way time. It'd be really neat to measure the time it takes for a packet to travel in one direction, but it's somewhere between hard and impossible[1]; a very short path has less room to be asymetric though. [1] If the clocks are synchronized, you can measure send time on one end, and receive time on the other. But synchronizing clocks involves estimating the time it takes for signals to pass im each direction, typically assuming each direction takes half the round trip.
      - pkhuong 2y ago
        You can use something like White Rabbit (https://en.wikipedia.org/wiki/White_Rabbit_Project https://en.wikipedia.org/wiki/White_Rabbit_Project) to keep clocks in sync. That still involves estimates, but a dedicated time sync network can do things like make sure all the cables are the same length.
        namibj 2y ago
        Copper white rabbit is special, it uses the same wire in both directions (1000BASE-T phy with added carrier phase lock to and from outside clocks).
  - deleted 2y ago
    [deleted]
  - znyboy 2y ago
    Considering that 300 light-nanoseconds is about 90m, getting a response (or even just one-way) in that time is essentially running right at the limits of physics/causality.
    - deleted 2y ago
      [deleted]
jiggawatts 2y ago
Meanwhile the best network I’ve ever benchmarked was AWS and measured about 55µs for a round trip! What on earth are you using that gets you down to single digits!?
- deleted 2y ago
  [deleted]
- crest 2y ago
  I assume 1-3 hops of modern switches without congestion. Given 100Gb/s lanes these numbers are possible if you get all the bottlenecks out of the way. The moment you hit a deep queue the latency explodes.
  - jiggawatts 2y ago
    So, are you talking about theoretical latencies here based on bandwidths and cable lengths, or actual measured latencies end-to-end between hosts? I know that "in principle" the physics of the cabling allows single digit microseconds, but I've never seen it anywhere near that low even with cross-over cables with zero switches in-path!
    - eqvinox 2y ago
      You need high bandwidth links (time to get the entire packet across starts to matter), run on bare metal (or have very well working HW virtualisation support), and tune NIC parameters and OS processing appropriately. But it's practically achievable. Switches in these scenarios (e.g. 25GE DC targeted) are pretty predictable and add <1μs (unless misconfigured)
      - jiggawatts 2y ago
        > But it's practically achievable. I've never seen this in practice. Maaaaybe with Infiniband and custom-written apps that use a proprietary SDK. I'd love to see references to actual benchmarks.
- 2y ago