4 ms·

Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Midd

Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Middle devices are generally routers or firewalls, and their job is to route and firewall, not to respond to a packet as quickly as it comes in. Transit traffic is far more important than processing control plane packets. Devices can add several msec or more in latency by sticking ICMP echo requests and the like in a low-priority queue and getting around to responding eventually. This will dwarf any gains produced by "the best NTP server". And no, setting QoS bits on the packet will not help.

tyingq 6y ago
Wouldn't some ICMP messages be important to send back quickly? Like "Fragmentation Needed"?
- swinglock 6y ago
  No more important than other ICMP and not important enough to prioritize it as that would risk denial of service. Answering within milliseconds is fast enough.
- hansel_der 6y ago
  not really. icmp itself is a datagramm protocol i.e. there is no delivery guarantee and hence it's kinda optional (in ipv4 that is, on the other hand ipv6 trades in some robustness and places a higher emphasis on working icmp)
bogomipz 6y ago
>"Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Middle devices are generally routers or firewalls, and their job is to route and firewall, not to respond to a packet as quickly as it comes in." That's not correct. A box in the middle of the network path by definition doesn't respond at all to the ping request since it's not addressed to them. It's simply forwards the IP packet that encapsulates the ICMP echo request towards its destination. As forwarding happens in the data plane this doesn't involve the control plane at all. A router with no QoS will forward IP datagrams at the same rate whether they encapsulate TCP, UDP or ICMP.
- zaphirplane 6y ago
  The destination presumably in the parent post is the network device
  - bogomipz 6y ago
    Please read what I quoted from the parent where they explicitly state that "middle boxes" deprioritize pings. They are actually making this distinction between middle and end device. So no, not "presumably" at all. The exact opposite.
walrus01 6y ago
Absolutely this. One of the first thing that an ISP's NOC will tell a business customer, when they're complaining about ICMP loss or high latency to some intermediate-hop seen in a traceroute, is to test against the ICMP loss and latency/jitter to an endpoint destination. Preferably to an endpoint destination that is some sort of server which isn't firewalling/rate limiting answering ICMP, and not a firewall or router. You can see what looks like terrible loss and jitter to intermediate hops in a traceroute, but traffic to your your VoIP server that is 12.3ms away on the far side of those hops and several AS-to-AS adjacencies, might average 0.00% loss over multi week periods, with less than 0.3ms of jitter range. Customers will also be advised to use other tools that can measure success/failure and RTT answer time of other services and daemons running on the endpoint they're testing against (at OSI layers 4-7), such as the various ways of measuring DNS lookup query time, or time to curl a sample file from an httpd over TLS1.2/1.3, and plot that on a multi day/week chart. For their own protection routers and other network equipment with packets flowing through them deprioritize answering ICMP.
- fragmede 6y ago
  Which is frustrating enough, but I’ve heard rumors that AWS’ (and other cloud) providers sometimes do weird things with routing, so ICMP doesn’t even take the same path as TCP traffic. Other times ICMP is just blocked somewhere along the way. I recommend TCPPing or similar for doing investigations.
  - notyourday 6y ago
    Just wait until you see the fun they have with edge termination where they terminate ICMP at the edge while letting TCP/UDP into the core.
rwmj 6y ago
Do they actually do this? It sounds like it would be more effort for a device to deeply inspect the packet and sort them into queues (and to what end exactly?), rather than simply forward as fast as possible based on the destination address.
- pfortuny 6y ago
  I don’t know but that seems a pretty simple feature to implement in hardware.
- sneak 6y ago
  GP is referring to when you ping the router in the path itself, not when you send pings through it. "Forward as fast as possible" is indeed what they do. Respond to pings (to itself) that may be in those packets is further down the list.
- parliament32 6y ago
  You don't need to do any DPI because ICMP is a layer 3 protocol. It's in the same layer as looking up source/destination addresses.
yencabulator 6y ago
Oh, it's even worse. I've observed bottlenecked ISPs prioritizing ping, where ICMP Echo latency and packet loss are less than the same for TCP & UDP packets (and I mean packets not inside the TCP stream). I guess the squeaky wheel got the grease, and that stopped people using ping results to complain about their slow internet connection.

tyingq 6y ago

Wouldn't some ICMP messages be important to send back quickly? Like "Fragmentation Needed"?

swinglock 6y ago
No more important than other ICMP and not important enough to prioritize it as that would risk denial of service. Answering within milliseconds is fast enough.
hansel_der 6y ago
not really. icmp itself is a datagramm protocol i.e. there is no delivery guarantee and hence it's kinda optional (in ipv4 that is, on the other hand ipv6 trades in some robustness and places a higher emphasis on working icmp)

bogomipz 6y ago

>"Everybody is talking about clock accuracy and totally missing that devices in the middle of the network path do not care about responding quickly to pings. Middle devices are generally routers or firewalls, and their job is to route and firewall, not to respond to a packet as quickly as it comes in." That's not correct. A box in the middle of the network path by definition doesn't respond at all to the ping request since it's not addressed to them. It's simply forwards the IP packet that encapsulates the ICMP echo request towards its destination. As forwarding happens in the data plane this doesn't involve the control plane at all. A router with no QoS will forward IP datagrams at the same rate whether they encapsulate TCP, UDP or ICMP.

zaphirplane 6y ago
The destination presumably in the parent post is the network device
- bogomipz 6y ago
  Please read what I quoted from the parent where they explicitly state that "middle boxes" deprioritize pings. They are actually making this distinction between middle and end device. So no, not "presumably" at all. The exact opposite.

walrus01 6y ago

Absolutely this. One of the first thing that an ISP's NOC will tell a business customer, when they're complaining about ICMP loss or high latency to some intermediate-hop seen in a traceroute, is to test against the ICMP loss and latency/jitter to an endpoint destination. Preferably to an endpoint destination that is some sort of server which isn't firewalling/rate limiting answering ICMP, and not a firewall or router. You can see what looks like terrible loss and jitter to intermediate hops in a traceroute, but traffic to your your VoIP server that is 12.3ms away on the far side of those hops and several AS-to-AS adjacencies, might average 0.00% loss over multi week periods, with less than 0.3ms of jitter range. Customers will also be advised to use other tools that can measure success/failure and RTT answer time of other services and daemons running on the endpoint they're testing against (at OSI layers 4-7), such as the various ways of measuring DNS lookup query time, or time to curl a sample file from an httpd over TLS1.2/1.3, and plot that on a multi day/week chart. For their own protection routers and other network equipment with packets flowing through them deprioritize answering ICMP.

fragmede 6y ago
Which is frustrating enough, but I’ve heard rumors that AWS’ (and other cloud) providers sometimes do weird things with routing, so ICMP doesn’t even take the same path as TCP traffic. Other times ICMP is just blocked somewhere along the way. I recommend TCPPing or similar for doing investigations.
- notyourday 6y ago
  Just wait until you see the fun they have with edge termination where they terminate ICMP at the edge while letting TCP/UDP into the core.

rwmj 6y ago

Do they actually do this? It sounds like it would be more effort for a device to deeply inspect the packet and sort them into queues (and to what end exactly?), rather than simply forward as fast as possible based on the destination address.

pfortuny 6y ago
I don’t know but that seems a pretty simple feature to implement in hardware.
sneak 6y ago
GP is referring to when you ping the router in the path itself, not when you send pings through it. "Forward as fast as possible" is indeed what they do. Respond to pings (to itself) that may be in those packets is further down the list.
parliament32 6y ago
You don't need to do any DPI because ICMP is a layer 3 protocol. It's in the same layer as looking up source/destination addresses.

yencabulator 6y ago

Oh, it's even worse. I've observed bottlenecked ISPs prioritizing ping, where ICMP Echo latency and packet loss are less than the same for TCP & UDP packets (and I mean packets not inside the TCP stream). I guess the squeaky wheel got the grease, and that stopped people using ping results to complain about their slow internet connection.