7 ms·

Splitting the Ping

120 points by symisc_devel 6y ago

datastoat 6y ago
As the article explains, latency and clock sync go hand in hand. Here's a blog post [1] that goes further into clock sync, contrasting NTP and hardware-based systems. The company behind the blog post says that their solution is available as a managed service on Azure and GCP, but I've never looked out for it. [1] https://www.ticktocknetworks.com/tick-tock-the-clock-runs-wild/ https://www.ticktocknetworks.com/tick-tock-the-clock-runs-wi...
Ono-Sendai 6y ago
It's basically impossible to measure the one-way latency, without external information in the form of clock synchronisation done externally. See http://twistedoakstudios.com/blog/Post2353_when-one-way-latency-doesnt-matter http://twistedoakstudios.com/blog/Post2353_when-one-way-late... This fact is also the basis for special relativity (different observers may choose different simultaneity conventions - for example Einstein synchronisation)
- bentcorner 6y ago
  You should be able to tell when one side of the link degrades though, correct? You'd have known time deltas for the link each way (whether its accurate doesn't really matter) and when the round trip time changes you should be able to tell which side that occurred on. (I'm interested in this problem because my cable internet reliably (!) degrades during the workday)
  - slaymaker1907 6y ago
    The problem is physics can't even distinguish whether light travels instantaneously in direction x, but at c/2 going in the opposite direction. There is no known way to synchronize clocks or come up with a clever apparatus to prove that light moves at the same speed in every direction. You need to make some strong assumptions in order to try and measure asymmetrical latency on a network. I believe the method you are proposing is to measure RTT when RTT differs depending on the starting node. The difficult problem is determining how much of your RTT time comes from node A to node B as opposed to node B to node A. I believe you need to at least have some sort of oracle on both sides which can synchronize time without using the network path you are trying to measure (I think you can do this if you have node C which both A and B have a fast connection to relative to each other, this implies that the delay between A and B is significantly longer than the speed of light delay).
- datastoat 6y ago
  To be precise, it's impossible to measure one-way latency between a pair of nodes without external information. But if you have a mesh of nodes, the story is different [1]. If you have N nodes and hence N unknown clock offsets, and if you have ping data from N(N-1) pairs, you can do better. [1] https://www.usenix.org/conference/nsdi18/presentation/geng https://www.usenix.org/conference/nsdi18/presentation/geng
  - Ono-Sendai 6y ago
    Well I think we need to distinguish between practical latency measurements on earth, where it's mostly a matter of rejecting dodgy data / outliers etc.., and detecting systematic latency bias that is theoretically impossible to detect. If you could get around this problem just by using a network of N nodes, then you could do stuff like measuring an absolute reference frame, thereby violating special relativity.
statstutor 6y ago
The article says "Now that we have a synced clock", but surely the clock synchronisation would also be unable to resolve any asymmetry in the ping and the synchronisation would inevitably be impacted by this. (Or, does NTP have a solution to this?) [Edited to add: from https://en.wikipedia.org/wiki/Network_Time_Protocol https://en.wikipedia.org/wiki/Network_Time_Protocol: "Asymmetric routes... can cause errors of 100 ms or more."]
- benjojo12 6y ago
  The point was to say, if you are in a DC environment and have ~1ms access to a Stratum 1 NTP server, then you can use that, if you are on DSL/DOCSIS etc, then it's likely required to use GNSS/PPS sources
- jtsiskin 6y ago
  Yes, NTP clock sync assumes symmetric delays, so you have the exact same problem. However they use the closest NTP time server, so assuming the NTP server owners correctly are able to sync their own servers, the offset is probably close. Unless the asymmetric delay is on the path which both the ping and the NTP synchronization take
  - toast0 6y ago
    > Unless the asymmetric delay is on the path which both the ping and the NTP synchronization take Assuming the NTP server is not on the LAN using a GPS (or other radio) reference, a large part of the asymmetry is on the poster's VDSL last mile hop. At least on my bonded VDSL2 line, it seems there's significant asymmetry; I found with regular ntpd I needed to delay incoming packets by around 18 ms to get a good match between a GPS receiver and network servers; my round trip time to the PPPoE concentrator is about 21 ms, and I was able to find a few NTP servers close enough that round trip to them was about 22 ms; not much room for additional asymmetry there.
gerdesj 6y ago
"However there is a common assumption on this latency number. Is that you can divide it in half to get the time it takes to send data in one direction" So you ssh the other end and ping in the other direction. I tend to use mtr instead. Nowadays latency to internet services are within the sort of latencies we used to require on site. OK I do understand that not all internet connections are equal. I live in a fairly rural part of the UK - a small town of roughly 25,000 odd people. The UK is a fairly small country and fairly densely populated and quite rich, so in general: internets are fairly reasonable in comparison to the rest of the world. Not world beating but overall pretty decent. If I ping say www.google.com (yes I know it's a funky address with random "distance") from home on my FTTC connection I see something like 15 to 20ms returns. I can ssh the office and use a 1GB link and get 8ms returns. My office PC is on the end of three 1GBs-1 hops in the building itself and it's a crappy old ex-customer hand me down (I run Arch Linux, I don't need whatever W10 requires). Both of those links are via the same ISP. My home PC test is via wifi! I'm not going to bother going too much further with this now but if I was bothered (and I will be one day!), I would start my pinging n stuff at my internet facing router. It wasn't that long ago that we (my company, about 10 years ago) accepted a condition of contract that required a site latency of 30ms end to end (ignoring hosts - the latency was directly measured without normal IT involvement.) 30ms!
pxx 6y ago
Not entirely related, but a fun and interesting tangent: There's actually no way that we know of to measure the "one-way" speed of light, as the specific synchronization that you use (and this post uses to do its calculation) assumes that the speed of light is the same in both directions. For all we know, light travels infinitely quickly in one direction and at c/2 in the return direction. https://en.wikipedia.org/wiki/One-way_speed_of_light https://en.wikipedia.org/wiki/One-way_speed_of_light recent-ish video about this: https://www.youtube.com/watch?v=pTn6Ewhb27k https://www.youtube.com/watch?v=pTn6Ewhb27k
- bitsen 6y ago
  That’s the stupidest argument I’ve ever heard from someone attempting to be a physicist. The gear experiment is the simplest answer. With a gear whose diameter spans the distance between the light source and receiver, and where the speed of rotation is controlled by an atomic clock, the gear could have a small hole through it. If light sent from one side to the other while the gear is spinning is too slow, it will not make it through the gear. The size of the bike through the gear will dictate the speed, because the gear itself measures duration and distance in a single direction.
  - quiescant_dodo 6y ago
    I think you missed the point. Measuring one-way speed of light requires perfectly synchronized clocks. You cannot do this over any distance, as it requires the speed of light to propagate between the clocks. So you must synchronize the clocks when they are together. But then you must move at least one clock, which subjects the clocks to unknown time dilation. We can't even assume that the time dilation from movement X is the same as from movement -X, even to impossibly-precise measurements and controls. The _premise_ of the question about one-way speed of light is un-divorceable from time dilation. If you _assume_ that the time dilation is the exact same, you are also _assuming_ that the amount of time light takes from A-to-B is exactly half of A-to-B-to-A. Einstein himself articulated this problem. He's certainly known for more than "attempting to be a physicist".
    - 6y ago
dec0dedab0de 6y ago
This reminded me of the buffer bloat[0] problem esr was talking about a few years ago. I thought I read a paper where he was getting hardware made for the purpose of using gps for time so it could be accurate enough to measure latency. I just googled and I realized it wasn't a paper, it was a talk I went to in 2012 that I forgot about[1] I wonder if anything came of it, does anyone know? [0] https://hn.algolia.com/?q=bufferbloat https://hn.algolia.com/?q=bufferbloat (it made the front page more than once) [1] https://youtu.be/1b17ggwkR60 https://youtu.be/1b17ggwkR60
- drothlis 6y ago
  This "not quite 5 minute guide to making an NTP server" is from 2014: https://ava.upuaut.net/?p=726 https://ava.upuaut.net/?p=726 So GPS boards for Raspberry Pi were cheap off-the-shelf hardware by 2014, at least.
jedimastert 6y ago
An excellent video about why it's impossible to measure the speed of light in one direction https://www.youtube.com/watch?v=pTn6Ewhb27k https://www.youtube.com/watch?v=pTn6Ewhb27k
nullserver 6y ago
https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles Email can’t go father then 500 miles
great_wubwub 6y ago
Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Middle devices are generally routers or firewalls, and their job is to route and firewall, not to respond to a packet as quickly as it comes in. Transit traffic is far more important than processing control plane packets. Devices can add several msec or more in latency by sticking ICMP echo requests and the like in a low-priority queue and getting around to responding eventually. This will dwarf any gains produced by "the best NTP server". And no, setting QoS bits on the packet will not help.
- tyingq 6y ago
  Wouldn't some ICMP messages be important to send back quickly? Like "Fragmentation Needed"?
  - swinglock 6y ago
    No more important than other ICMP and not important enough to prioritize it as that would risk denial of service. Answering within milliseconds is fast enough.
  - hansel_der 6y ago
    not really. icmp itself is a datagramm protocol i.e. there is no delivery guarantee and hence it's kinda optional (in ipv4 that is, on the other hand ipv6 trades in some robustness and places a higher emphasis on working icmp)
- bogomipz 6y ago
  >"Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Middle devices are generally routers or firewalls, and their job is to route and firewall, not to respond to a packet as quickly as it comes in." That's not correct. A box in the middle of the network path by definition doesn't respond at all to the ping request since it's not addressed to them. It's simply forwards the IP packet that encapsulates the ICMP echo request towards its destination. As forwarding happens in the data plane this doesn't involve the control plane at all. A router with no QoS will forward IP datagrams at the same rate whether they encapsulate TCP, UDP or ICMP.
IgorPartola 6y ago
I thought NTP specifically did account for transit latency. Not sure why I made that assumption, but if it’s not true, how can I ever trust my clock to be correct?
- detaro 6y ago
  As the article says, it does account for it. It just can't account for unknown constant asymmetric latency. > how can I ever trust my clock to be correct? what does it mean for your clock to be "correct"? If you need your time to be precise to more than a few 100ms you probably shouldn't be getting it from random NTP servers over unknown connections, but for most people that's an acceptable error.
swinglock 6y ago
This protocol and software exists, it's called OWAMP.
- chlbny 6y ago
  OWAMP is even a standardized protocol (RFC 4656). For the record, there is another tool that can perform such kind of one-way measurement called https://github.com/heistp/irtt https://github.com/heistp/irtt .
deleted 6y ago
[deleted]
jlgaddis 6y ago
If you've got clock synchronization between two hosts, there's OWAMP, a.k.a. RFC4656 [0]. The Minimum-Pairs Protocol [1] eliminates the need for clock synchronization but requires (at least) 3 hosts under your control, plus the "uncooperative" fourth node: > The minimum-pairs (or MP) is an active measurement protocol to estimate in real-time the smaller of the forward and reverse one-way network delays (OWDs).[1] It is designed to work in hostile environments, where a set of three network nodes can estimate an upper-bound OWDs between themselves and a fourth untrusted node. All four nodes must cooperate, though honest cooperation from the fourth node is not required. The objective is to conduct such estimates without involving the untrusted nodes in clock synchronization, and in a manner more accurate than simply half the Round-Trip Time (RTT). -- [0]: https://tools.ietf.org/html/rfc4656 https://tools.ietf.org/html/rfc4656 [1]: https://en.wikipedia.org/wiki/Minimum-Pairs_Protocol https://en.wikipedia.org/wiki/Minimum-Pairs_Protocol

120 points by symisc_devel 6y ago

As the article explains, latency and clock sync go hand in hand. Here's a blog post [1] that goes further into clock sync, contrasting NTP and hardware-based systems. The company behind the blog post says that their solution is available as a managed service on Azure and GCP, but I've never looked out for it. [1] https://www.ticktocknetworks.com/tick-tock-the-clock-runs-wild/ https://www.ticktocknetworks.com/tick-tock-the-clock-runs-wi...

Ono-Sendai 6y ago

It's basically impossible to measure the one-way latency, without external information in the form of clock synchronisation done externally. See http://twistedoakstudios.com/blog/Post2353_when-one-way-latency-doesnt-matter http://twistedoakstudios.com/blog/Post2353_when-one-way-late... This fact is also the basis for special relativity (different observers may choose different simultaneity conventions - for example Einstein synchronisation)

bentcorner 6y ago
You should be able to tell when one side of the link degrades though, correct? You'd have known time deltas for the link each way (whether its accurate doesn't really matter) and when the round trip time changes you should be able to tell which side that occurred on. (I'm interested in this problem because my cable internet reliably (!) degrades during the workday)
- slaymaker1907 6y ago
  The problem is physics can't even distinguish whether light travels instantaneously in direction x, but at c/2 going in the opposite direction. There is no known way to synchronize clocks or come up with a clever apparatus to prove that light moves at the same speed in every direction. You need to make some strong assumptions in order to try and measure asymmetrical latency on a network. I believe the method you are proposing is to measure RTT when RTT differs depending on the starting node. The difficult problem is determining how much of your RTT time comes from node A to node B as opposed to node B to node A. I believe you need to at least have some sort of oracle on both sides which can synchronize time without using the network path you are trying to measure (I think you can do this if you have node C which both A and B have a fast connection to relative to each other, this implies that the delay between A and B is significantly longer than the speed of light delay).
datastoat 6y ago
To be precise, it's impossible to measure one-way latency between a pair of nodes without external information. But if you have a mesh of nodes, the story is different [1]. If you have N nodes and hence N unknown clock offsets, and if you have ping data from N(N-1) pairs, you can do better. [1] https://www.usenix.org/conference/nsdi18/presentation/geng https://www.usenix.org/conference/nsdi18/presentation/geng
- Ono-Sendai 6y ago
  Well I think we need to distinguish between practical latency measurements on earth, where it's mostly a matter of rejecting dodgy data / outliers etc.., and detecting systematic latency bias that is theoretically impossible to detect. If you could get around this problem just by using a network of N nodes, then you could do stuff like measuring an absolute reference frame, thereby violating special relativity.

statstutor 6y ago

The article says "Now that we have a synced clock", but surely the clock synchronisation would also be unable to resolve any asymmetry in the ping and the synchronisation would inevitably be impacted by this. (Or, does NTP have a solution to this?) [Edited to add: from https://en.wikipedia.org/wiki/Network_Time_Protocol https://en.wikipedia.org/wiki/Network_Time_Protocol: "Asymmetric routes... can cause errors of 100 ms or more."]

benjojo12 6y ago
The point was to say, if you are in a DC environment and have ~1ms access to a Stratum 1 NTP server, then you can use that, if you are on DSL/DOCSIS etc, then it's likely required to use GNSS/PPS sources
jtsiskin 6y ago
Yes, NTP clock sync assumes symmetric delays, so you have the exact same problem. However they use the closest NTP time server, so assuming the NTP server owners correctly are able to sync their own servers, the offset is probably close. Unless the asymmetric delay is on the path which both the ping and the NTP synchronization take
- toast0 6y ago
  > Unless the asymmetric delay is on the path which both the ping and the NTP synchronization take Assuming the NTP server is not on the LAN using a GPS (or other radio) reference, a large part of the asymmetry is on the poster's VDSL last mile hop. At least on my bonded VDSL2 line, it seems there's significant asymmetry; I found with regular ntpd I needed to delay incoming packets by around 18 ms to get a good match between a GPS receiver and network servers; my round trip time to the PPPoE concentrator is about 21 ms, and I was able to find a few NTP servers close enough that round trip to them was about 22 ms; not much room for additional asymmetry there.

gerdesj 6y ago

"However there is a common assumption on this latency number. Is that you can divide it in half to get the time it takes to send data in one direction" So you ssh the other end and ping in the other direction. I tend to use mtr instead. Nowadays latency to internet services are within the sort of latencies we used to require on site. OK I do understand that not all internet connections are equal. I live in a fairly rural part of the UK - a small town of roughly 25,000 odd people. The UK is a fairly small country and fairly densely populated and quite rich, so in general: internets are fairly reasonable in comparison to the rest of the world. Not world beating but overall pretty decent. If I ping say www.google.com (yes I know it's a funky address with random "distance") from home on my FTTC connection I see something like 15 to 20ms returns. I can ssh the office and use a 1GB link and get 8ms returns. My office PC is on the end of three 1GBs-1 hops in the building itself and it's a crappy old ex-customer hand me down (I run Arch Linux, I don't need whatever W10 requires). Both of those links are via the same ISP. My home PC test is via wifi! I'm not going to bother going too much further with this now but if I was bothered (and I will be one day!), I would start my pinging n stuff at my internet facing router. It wasn't that long ago that we (my company, about 10 years ago) accepted a condition of contract that required a site latency of 30ms end to end (ignoring hosts - the latency was directly measured without normal IT involvement.) 30ms!

pxx 6y ago

Not entirely related, but a fun and interesting tangent: There's actually no way that we know of to measure the "one-way" speed of light, as the specific synchronization that you use (and this post uses to do its calculation) assumes that the speed of light is the same in both directions. For all we know, light travels infinitely quickly in one direction and at c/2 in the return direction. https://en.wikipedia.org/wiki/One-way_speed_of_light https://en.wikipedia.org/wiki/One-way_speed_of_light recent-ish video about this: https://www.youtube.com/watch?v=pTn6Ewhb27k https://www.youtube.com/watch?v=pTn6Ewhb27k

bitsen 6y ago
That’s the stupidest argument I’ve ever heard from someone attempting to be a physicist. The gear experiment is the simplest answer. With a gear whose diameter spans the distance between the light source and receiver, and where the speed of rotation is controlled by an atomic clock, the gear could have a small hole through it. If light sent from one side to the other while the gear is spinning is too slow, it will not make it through the gear. The size of the bike through the gear will dictate the speed, because the gear itself measures duration and distance in a single direction.
- quiescant_dodo 6y ago
  I think you missed the point. Measuring one-way speed of light requires perfectly synchronized clocks. You cannot do this over any distance, as it requires the speed of light to propagate between the clocks. So you must synchronize the clocks when they are together. But then you must move at least one clock, which subjects the clocks to unknown time dilation. We can't even assume that the time dilation from movement X is the same as from movement -X, even to impossibly-precise measurements and controls. The _premise_ of the question about one-way speed of light is un-divorceable from time dilation. If you _assume_ that the time dilation is the exact same, you are also _assuming_ that the amount of time light takes from A-to-B is exactly half of A-to-B-to-A. Einstein himself articulated this problem. He's certainly known for more than "attempting to be a physicist".
  - 6y ago

dec0dedab0de 6y ago

This reminded me of the buffer bloat[0] problem esr was talking about a few years ago. I thought I read a paper where he was getting hardware made for the purpose of using gps for time so it could be accurate enough to measure latency. I just googled and I realized it wasn't a paper, it was a talk I went to in 2012 that I forgot about[1] I wonder if anything came of it, does anyone know? [0] https://hn.algolia.com/?q=bufferbloat https://hn.algolia.com/?q=bufferbloat (it made the front page more than once) [1] https://youtu.be/1b17ggwkR60 https://youtu.be/1b17ggwkR60

drothlis 6y ago
This "not quite 5 minute guide to making an NTP server" is from 2014: https://ava.upuaut.net/?p=726 https://ava.upuaut.net/?p=726 So GPS boards for Raspberry Pi were cheap off-the-shelf hardware by 2014, at least.

jedimastert 6y ago

An excellent video about why it's impossible to measure the speed of light in one direction https://www.youtube.com/watch?v=pTn6Ewhb27k https://www.youtube.com/watch?v=pTn6Ewhb27k

nullserver 6y ago

https://web.mit.edu/jemorris/humor/500-miles https://web.mit.edu/jemorris/humor/500-miles Email can’t go father then 500 miles

great_wubwub 6y ago

Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Middle devices are generally routers or firewalls, and their job is to route and firewall, not to respond to a packet as quickly as it comes in. Transit traffic is far more important than processing control plane packets. Devices can add several msec or more in latency by sticking ICMP echo requests and the like in a low-priority queue and getting around to responding eventually. This will dwarf any gains produced by "the best NTP server". And no, setting QoS bits on the packet will not help.

tyingq 6y ago
Wouldn't some ICMP messages be important to send back quickly? Like "Fragmentation Needed"?
- swinglock 6y ago
  No more important than other ICMP and not important enough to prioritize it as that would risk denial of service. Answering within milliseconds is fast enough.
- hansel_der 6y ago
  not really. icmp itself is a datagramm protocol i.e. there is no delivery guarantee and hence it's kinda optional (in ipv4 that is, on the other hand ipv6 trades in some robustness and places a higher emphasis on working icmp)
bogomipz 6y ago
>"Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Middle devices are generally routers or firewalls, and their job is to route and firewall, not to respond to a packet as quickly as it comes in." That's not correct. A box in the middle of the network path by definition doesn't respond at all to the ping request since it's not addressed to them. It's simply forwards the IP packet that encapsulates the ICMP echo request towards its destination. As forwarding happens in the data plane this doesn't involve the control plane at all. A router with no QoS will forward IP datagrams at the same rate whether they encapsulate TCP, UDP or ICMP.

IgorPartola 6y ago

I thought NTP specifically did account for transit latency. Not sure why I made that assumption, but if it’s not true, how can I ever trust my clock to be correct?

detaro 6y ago
As the article says, it does account for it. It just can't account for unknown constant asymmetric latency. > how can I ever trust my clock to be correct? what does it mean for your clock to be "correct"? If you need your time to be precise to more than a few 100ms you probably shouldn't be getting it from random NTP servers over unknown connections, but for most people that's an acceptable error.

swinglock 6y ago

This protocol and software exists, it's called OWAMP.

chlbny 6y ago
OWAMP is even a standardized protocol (RFC 4656). For the record, there is another tool that can perform such kind of one-way measurement called https://github.com/heistp/irtt https://github.com/heistp/irtt .

deleted 6y ago

[deleted]

jlgaddis 6y ago

If you've got clock synchronization between two hosts, there's OWAMP, a.k.a. RFC4656 [0]. The Minimum-Pairs Protocol [1] eliminates the need for clock synchronization but requires (at least) 3 hosts under your control, plus the "uncooperative" fourth node: > The minimum-pairs (or MP) is an active measurement protocol to estimate in real-time the smaller of the forward and reverse one-way network delays (OWDs).[1] It is designed to work in hostile environments, where a set of three network nodes can estimate an upper-bound OWDs between themselves and a fourth untrusted node. All four nodes must cooperate, though honest cooperation from the fourth node is not required. The objective is to conduct such estimates without involving the untrusted nodes in clock synchronization, and in a manner more accurate than simply half the Round-Trip Time (RTT). -- [0]: https://tools.ietf.org/html/rfc4656 https://tools.ietf.org/html/rfc4656 [1]: https://en.wikipedia.org/wiki/Minimum-Pairs_Protocol https://en.wikipedia.org/wiki/Minimum-Pairs_Protocol