3 ms·

Copying the per-hop loss indicator from mtr is a bad decision in my opinion. It's always been a source of incorrect diagnosis of network issues. The only loss t

Copying the per-hop loss indicator from mtr is a bad decision in my opinion. It's always been a source of incorrect diagnosis of network issues. The only loss that matters is end to end.

FujiApple 3y ago
You are right that showing packet loss for intermediate hops is a frequent source of confusion. Rather than leave it out, I added a status column which shows different statuses for intermediate hops (blue if the hop responds to less than 100% of probes and brown if it responds to 0%) vs the target hop (which show amber and red respectively). Where this breaks down is when dealing with ECMP for UDP & TCP tracing, as a given hop (ttl) may represent the target for a given round of tracing but not for the next. The mistake, imho, is to associate _any_ data with a hop (ttl) rather than the hop in the context of a tracing flow. That is why Trippy had a number of features aimed at helping with ECMP, such as Paris and Dublin tracing, and the ability to filter tracing by unique flow id. I've covered these quite a bit in the 0.8.0 [0] and 0.9.0 [1] release notes if you want to know more. [0] https://github.com/fujiapple852/trippy/releases/tag/0.8.0 https://github.com/fujiapple852/trippy/releases/tag/0.8.0 [1] https://github.com/fujiapple852/trippy/releases/tag/0.9.0 https://github.com/fujiapple852/trippy/releases/tag/0.9.0
- geraldhh 3y ago
  > Rather than leave it out, I added a status column but why?
deleted 3y ago
[deleted]
lathiat 3y ago
That is not entirely true. Sure it’s not a 100% reliable signal as routing is asymmetric but it also often isn’t and also gives you an idea of which point to ask first at least. If the packet loss starts at your wifi router, or your ISPs router. Or the next hop after you ISP. That all gives you a bit of an idea where the problem likely is. I solve problems like that all the time.
- commandersaki 3y ago
  I was a network engineer for over a decade in hosting datacenter environments and would get false reports of packet loss to various destinations because people would use MTR and say they see loss on the path. If a packet takes the path A -> B -> C and pings to B have 50% loss but pings to C have 0% loss, then the path is perfectly fine. The only way to reliably isolate packet loss to a hop on the path is to have a destination for testing where packets pass through that hop and is in its bailiwick which doesn't perform rate limiting or policing of ICMP traffic.
  - FujiApple 3y ago
    Something I intend to add to Trippy, but have not got around to it yet; is to codify the "If a packet takes the path A -> B -> C and pings to B have 50% loss but pings to C have 0% loss, then the path is perfectly fine" idea and use that to produce more meaningful headline status information to the user. How would you codify this?
    - commandersaki 3y ago
      I would love for there to be a useful indicator to the user to say if loss or latency is an issue. Being able to indicate cascading loss (e.g. path A->B->C->D) shows loss at B, C, and D, is worth bubbling up to the user to say there might be real issues. Also any indication of loss at D is also an issue. Trying to reconcile these scenarios with the UI matters, but I don't think there's an easy way. What I think is more important than UI that is sorely needed is documentation / users guide explaining how to read and understand these indicators. I know documentation is usually overlooked by users first trying out a program, but having it documented and explained can be used as a reference to point to a user that is misunderstanding the tool. I found that MTR didn't have this much needed documentation / reference that people would easily misunderstand the tool and it was a herculean effort to correct them. I would also like to point out that a 0% loss indicator at the destination isn't reliable either if the packets are spaced out with enough slack. One of my goto when testing packet loss of a link I've brought up is to smash a destination host with a ping flood, e.g. ping -c 100 -f 1.1.1.1. By inundating the link it helps provide a clear indicator if there is loss somewhere on the path (usually the first mile or the last). Cloudflare speedtest now has a packet loss tester that floods 1000 packets, although I'm not sure if it does it over an unreliable transport or not.
mrngm 3y ago
At NANOG 62, there was a presentation "A practical guide to (correctly) troubleshooting with traceroute", by Richard Steenbergen (slides [pdf]: https://archive.nanog.org/sites/default/files/tuesday_steenbergen_troublshootingtraceroute_62.49.pdf https://archive.nanog.org/sites/default/files/tuesday_steenb..., talk [video]: https://www.youtube.com/watch?v=WL0ZTcfSvB4 https://www.youtube.com/watch?v=WL0ZTcfSvB4). It also mentions that isolated hops that show increased latency or loss are most likely throttling on the device. However, if that latency or loss persists on further hops, that indicates a problem. Another issue with traceroutes is that is usually doesn't account for asymmetry in the return path. What I would find interesting to see is something like isoping/splitping (see this blog post https://blog.benjojo.co.uk/post/ping-with-loss-latency-split https://blog.benjojo.co.uk/post/ping-with-loss-latency-split), to account for that asymmetry. Regarding the tool trippy itself: awesome visualizations!
- commandersaki 3y ago
  Yep it's a great presentation. I like to always say that traceroute gives you the approximate path a packet will take, whereas ping is for end to end measurement and loss. I'm personally not a big fan of combining the two tools.
- tptacek 3y ago
  This is a cool talk, thanks for posting it. One cute thing tools like `trippy` could do (from the talk) would be the reverse lookup on the peer /30 address for all the intermediate hops. I don't know if `trippy` does this (I've installed and played with it but not carefully), but you could also color-code latency spikes that persist into future hops.

Copying the per-hop loss indicator from mtr is a bad decision in my opinion. It's always been a source of incorrect diagnosis of network issues. The only loss that matters is end to end.

FujiApple 3y ago

You are right that showing packet loss for intermediate hops is a frequent source of confusion. Rather than leave it out, I added a status column which shows different statuses for intermediate hops (blue if the hop responds to less than 100% of probes and brown if it responds to 0%) vs the target hop (which show amber and red respectively). Where this breaks down is when dealing with ECMP for UDP & TCP tracing, as a given hop (ttl) may represent the target for a given round of tracing but not for the next. The mistake, imho, is to associate _any_ data with a hop (ttl) rather than the hop in the context of a tracing flow. That is why Trippy had a number of features aimed at helping with ECMP, such as Paris and Dublin tracing, and the ability to filter tracing by unique flow id. I've covered these quite a bit in the 0.8.0 [0] and 0.9.0 [1] release notes if you want to know more. [0] https://github.com/fujiapple852/trippy/releases/tag/0.8.0 https://github.com/fujiapple852/trippy/releases/tag/0.8.0 [1] https://github.com/fujiapple852/trippy/releases/tag/0.9.0 https://github.com/fujiapple852/trippy/releases/tag/0.9.0

geraldhh 3y ago
> Rather than leave it out, I added a status column but why?

deleted 3y ago

[deleted]

lathiat 3y ago

That is not entirely true. Sure it’s not a 100% reliable signal as routing is asymmetric but it also often isn’t and also gives you an idea of which point to ask first at least. If the packet loss starts at your wifi router, or your ISPs router. Or the next hop after you ISP. That all gives you a bit of an idea where the problem likely is. I solve problems like that all the time.

commandersaki 3y ago
I was a network engineer for over a decade in hosting datacenter environments and would get false reports of packet loss to various destinations because people would use MTR and say they see loss on the path. If a packet takes the path A -> B -> C and pings to B have 50% loss but pings to C have 0% loss, then the path is perfectly fine. The only way to reliably isolate packet loss to a hop on the path is to have a destination for testing where packets pass through that hop and is in its bailiwick which doesn't perform rate limiting or policing of ICMP traffic.
- FujiApple 3y ago
  Something I intend to add to Trippy, but have not got around to it yet; is to codify the "If a packet takes the path A -> B -> C and pings to B have 50% loss but pings to C have 0% loss, then the path is perfectly fine" idea and use that to produce more meaningful headline status information to the user. How would you codify this?
  - commandersaki 3y ago
    I would love for there to be a useful indicator to the user to say if loss or latency is an issue. Being able to indicate cascading loss (e.g. path A->B->C->D) shows loss at B, C, and D, is worth bubbling up to the user to say there might be real issues. Also any indication of loss at D is also an issue. Trying to reconcile these scenarios with the UI matters, but I don't think there's an easy way. What I think is more important than UI that is sorely needed is documentation / users guide explaining how to read and understand these indicators. I know documentation is usually overlooked by users first trying out a program, but having it documented and explained can be used as a reference to point to a user that is misunderstanding the tool. I found that MTR didn't have this much needed documentation / reference that people would easily misunderstand the tool and it was a herculean effort to correct them. I would also like to point out that a 0% loss indicator at the destination isn't reliable either if the packets are spaced out with enough slack. One of my goto when testing packet loss of a link I've brought up is to smash a destination host with a ping flood, e.g. ping -c 100 -f 1.1.1.1. By inundating the link it helps provide a clear indicator if there is loss somewhere on the path (usually the first mile or the last). Cloudflare speedtest now has a packet loss tester that floods 1000 packets, although I'm not sure if it does it over an unreliable transport or not.

mrngm 3y ago

At NANOG 62, there was a presentation "A practical guide to (correctly) troubleshooting with traceroute", by Richard Steenbergen (slides [pdf]: https://archive.nanog.org/sites/default/files/tuesday_steenbergen_troublshootingtraceroute_62.49.pdf https://archive.nanog.org/sites/default/files/tuesday_steenb..., talk [video]: https://www.youtube.com/watch?v=WL0ZTcfSvB4 https://www.youtube.com/watch?v=WL0ZTcfSvB4). It also mentions that isolated hops that show increased latency or loss are most likely throttling on the device. However, if that latency or loss persists on further hops, that indicates a problem. Another issue with traceroutes is that is usually doesn't account for asymmetry in the return path. What I would find interesting to see is something like isoping/splitping (see this blog post https://blog.benjojo.co.uk/post/ping-with-loss-latency-split https://blog.benjojo.co.uk/post/ping-with-loss-latency-split), to account for that asymmetry. Regarding the tool trippy itself: awesome visualizations!

commandersaki 3y ago
Yep it's a great presentation. I like to always say that traceroute gives you the approximate path a packet will take, whereas ping is for end to end measurement and loss. I'm personally not a big fan of combining the two tools.
tptacek 3y ago
This is a cool talk, thanks for posting it. One cute thing tools like `trippy` could do (from the talk) would be the reverse lookup on the peer /30 address for all the intermediate hops. I don't know if `trippy` does this (I've installed and played with it but not carefully), but you could also color-code latency spikes that persist into future hops.