4 ms·
I no longer have access to it, but I wrote a tool to find these types of problems at my last job. It didn't seem generally applicable enough to try to get it op
by toast0 6y ago
I no longer have access to it, but I wrote a tool to find these types of problems at my last job. It didn't seem generally applicable enough to try to get it open sourced (and I didn't want to polish it enough for that either).
The key insight is that LACP is almost always configured to use do a hash of { Source IP, Dest IP, Protocol, Source Port, Dest Port } so that packets from each TCP and UDP flow will always be sent on the same individual link. (this is directional though, so it may go from peer A to peer B on cable X and from B to A on cable Y).
So the way to confirm a broken link is to connect a bunch of UDP flows (on different ports) between peer A and peer B, send data, and measure loss (and/or delay!). If you see zero loss / uniform delay, either none of your flows cross the broken link or the problem isn't a broken link or the rate of issues is too low to detect. Once you've found a broken flow, you can use a 'paris traceroute' tool to confirm the IP routers it's between. Paris traceroute holds the UDP source and destination ports fixed, so the route on LACP should say the same. I contributed support for this in mtr, but I'm not sure if it still works; if you see 100% packet loss with mtr in fixed udp port mode, send me an email (address in profile) and I'll ask you for data and try to debug.
Once you narrow down to the two routers the link is between, it should be easy enough to confirm. Usually through link quality counters, if not, through just pulling links and seeing if things get better or worse.
If you have a long network path, and most links are LACP, the total number of possible paths between two peers gets large, and there's a chance that you might not be able to survey them all from a pair of servers; so you may have to try a few different hosts.
You can find packet loss, but also congestion/buffering this way. In an ideal world, all the link error counters would be collected and anomalies would be addressed, but it seems that doesn't always happen.
- pwarner 6y agoYeah we get major problems in a top 2 cloud provider where our on prem to cloud link dropped packets. We narrowed it down with iperf to the packet loss only happening on some ephemeral source ports. Ports were always ok, or always slow. Destroyed and recreated the cloud gateways and all was well. Should say another engineer figured it out. Cloud provider tried to blame our side. They did not excel at operations...
- toast0 6y agoYep, I wrote the tool to help narrow down network problems at our bare metal hosting provider. Our system used lots of long lasting TCP connections, 1:1 between systems; persistent packet loss was problematic. For some reason, they wouldn't let me have access to router link stats ;)
- ramesh213 6y agoIt will be grateful if you can share how you established root cause.
- pabs3 6y agoSide note: paris-traceroute is unmaintained and no longer builds with recent versions of GCC: https://bugs.debian.org/957663 https://bugs.debian.org/957663 https://bugs.debian.org/970290 https://bugs.debian.org/970290
- toast0 6y agoIf that's the case, definitely use mtr (at least V0.87, but newer is probably better) --udp --localport X --port Y (where you've found X and Y using a tool you wrote that's like what I described).