4 ms·

Traceroute is problematic and sometimes lies. Paths aren't always symmetric. Sometimes routers are too busy to answer ICMP. And MPLS/encapsulating/tunneling

by mlyle 3y ago

Traceroute is problematic and sometimes lies. Paths aren't always symmetric. Sometimes routers are too busy to answer ICMP. And MPLS/encapsulating/tunneling is opaque. On the other hand, real world network engineers rely a whole lot on traceroute from their machines and from looking glasses (and mtr is wonderful).

creatonez 3y ago
The author doesn't like mtr either: > The example given is that a handful of users running MTR (do not get me started on this bastard program) can actually hit this rate limit. This is an outstanding example because I have seen something similar in practice. > Consider what that would look like, and how common it would be: If you have a NOC full of people who think they know what they're doing, but don't, that only enhances the probability that everyone is trying to troubleshoot on their own instead of doing a screenshare and coordinating their efforts - thus, you have six guys running MTR to the same IP. > If they hit that rate limit, what do they see? Nodes suddenly not responding! Randomly, in fact - sometimes responding, sometimes not! That means it's not just a hop that doesn't respond to traceroutes, but packet loss! Wow! We found the problem! > Of course, if four of them hit Ctrl+C, the PL would mysteriously vanish. Huh! Weird! Well, it must be an intermittent issue in squints at resolved hostname Hurricane Electric's network. I'm sure they have a flapping port they haven't noticed (lol.) Just send them a trouble ticket!
westurner 3y ago
Is there any guarantee that there's a stable path for the duration of the traceroute packet series; FWIU, IP routes can change in the middle of a traceroute? From "Gping – ping, but with a graph" https://news.ycombinator.com/item?id=36549005 https://news.ycombinator.com/item?id=36549005 : > Scapy has a 3d visualization of one traceroute sequence with vpython. In college, I remember modifying it to run multiple traceroutes and then overlaying all of the routes; and wondering whether a given route is even stable through a complete traceroute packet sequence. https://scapy.readthedocs.io/en/latest/usage.html#tcp-traceroute-2 https://scapy.readthedocs.io/en/latest/usage.html#tcp-tracer... It's trivial to modify packet TTLs with e.g. iptables. Virtual circuit: https://en.wikipedia.org/wiki/Virtual_circuit https://en.wikipedia.org/wiki/Virtual_circuit
- mlyle 3y ago
  mtr, etc, notices changes. Sure, you can create bogus traceroutes once things enter your network. I had 10 extra hops for awhile for luls.
- toast0 3y ago
  There's no guarantees on path during a trace; after all, a connection could drop or come up during a trace, and routing may change as a result. We're packet switched, not circuit switched, so there's nothing out of spec if you route differently. However, tcp is sensitive to getting packets out of order, and most multipacket udp applications aren't a big fan either. So it's common to ensure a packet with a given 5-tuple {ip dest, ip source, protocol, protocol port dest, port source} will route the same, including traveling over the same port in a multi-port bundle, unless that next hop goes down or a better next hop comes up. That said, many traceroute implementations default to using a new port for each probe, and depending on the networks between you and what you're probing, you might see a variety of paths in a single short probe. When that happens, you really don't get much information. Of course, the return path for the packets is likely to be different and isn't easy to know; if you control both ends, you obviously can; and if the destination network has a good looking glass, you can get some idea, but detailed traces aren't going to happen. All that said, parts of the rant are accurate. Routers and hosts don't have to participate and many don't or are rate limited. Intermediate latency is often not very helpful. Where a trace ends only tells you were the trace ends, not where the packets left the network (possibly at the destination). Having useful diagnostic information doesn't help if you can't get in touch with someone who will use the information to fix the problem. Being able to trace from multiple vantage points helps make sense of things. If you're experiencing intermittent behavior like some tcp connections between two hosts work great and some don't, and most of the network involved participates in tracing and you have a good contact, you can do specific tracing and narrow down the routers they need to check to find the problem; sometimes that helps time to resolution a lot. If you don't have a cooperative network, it's better to invest in making multiple connections and using the best one as measured; because that gives you agency to 'fix' network problems.
  - westurner 3y ago

by mlyle 3y ago

creatonez 3y ago

The author doesn't like mtr either: > The example given is that a handful of users running MTR (do not get me started on this bastard program) can actually hit this rate limit. This is an outstanding example because I have seen something similar in practice. > Consider what that would look like, and how common it would be: If you have a NOC full of people who think they know what they're doing, but don't, that only enhances the probability that everyone is trying to troubleshoot on their own instead of doing a screenshare and coordinating their efforts - thus, you have six guys running MTR to the same IP. > If they hit that rate limit, what do they see? Nodes suddenly not responding! Randomly, in fact - sometimes responding, sometimes not! That means it's not just a hop that doesn't respond to traceroutes, but packet loss! Wow! We found the problem! > Of course, if four of them hit Ctrl+C, the PL would mysteriously vanish. Huh! Weird! Well, it must be an intermittent issue in squints at resolved hostname Hurricane Electric's network. I'm sure they have a flapping port they haven't noticed (lol.) Just send them a trouble ticket!

westurner 3y ago

Is there any guarantee that there's a stable path for the duration of the traceroute packet series; FWIU, IP routes can change in the middle of a traceroute? From "Gping – ping, but with a graph" https://news.ycombinator.com/item?id=36549005 https://news.ycombinator.com/item?id=36549005 : > Scapy has a 3d visualization of one traceroute sequence with vpython. In college, I remember modifying it to run multiple traceroutes and then overlaying all of the routes; and wondering whether a given route is even stable through a complete traceroute packet sequence. https://scapy.readthedocs.io/en/latest/usage.html#tcp-traceroute-2 https://scapy.readthedocs.io/en/latest/usage.html#tcp-tracer... It's trivial to modify packet TTLs with e.g. iptables. Virtual circuit: https://en.wikipedia.org/wiki/Virtual_circuit https://en.wikipedia.org/wiki/Virtual_circuit

mlyle 3y ago
mtr, etc, notices changes. Sure, you can create bogus traceroutes once things enter your network. I had 10 extra hops for awhile for luls.
toast0 3y ago
There's no guarantees on path during a trace; after all, a connection could drop or come up during a trace, and routing may change as a result. We're packet switched, not circuit switched, so there's nothing out of spec if you route differently. However, tcp is sensitive to getting packets out of order, and most multipacket udp applications aren't a big fan either. So it's common to ensure a packet with a given 5-tuple {ip dest, ip source, protocol, protocol port dest, port source} will route the same, including traveling over the same port in a multi-port bundle, unless that next hop goes down or a better next hop comes up. That said, many traceroute implementations default to using a new port for each probe, and depending on the networks between you and what you're probing, you might see a variety of paths in a single short probe. When that happens, you really don't get much information. Of course, the return path for the packets is likely to be different and isn't easy to know; if you control both ends, you obviously can; and if the destination network has a good looking glass, you can get some idea, but detailed traces aren't going to happen. All that said, parts of the rant are accurate. Routers and hosts don't have to participate and many don't or are rate limited. Intermediate latency is often not very helpful. Where a trace ends only tells you were the trace ends, not where the packets left the network (possibly at the destination). Having useful diagnostic information doesn't help if you can't get in touch with someone who will use the information to fix the problem. Being able to trace from multiple vantage points helps make sense of things. If you're experiencing intermittent behavior like some tcp connections between two hosts work great and some don't, and most of the network involved participates in tracing and you have a good contact, you can do specific tracing and narrow down the routers they need to check to find the problem; sometimes that helps time to resolution a lot. If you don't have a cooperative network, it's better to invest in making multiple connections and using the best one as measured; because that gives you agency to 'fix' network problems.
- westurner 3y ago