4 ms·
Correct me if I'm wrong but the SNMP retransmit counters are just that: a count of retransmits the host sent. Raw retransmit counts are often just a vague indic
by cranekam 5y ago
Correct me if I'm wrong but the SNMP retransmit counters are just that: a count of retransmits the host sent. Raw retransmit counts are often just a vague indication that something's up — the host is retransmiting, either because there's loss on the path or the receiver is overloaded. But given a host can talk over many paths to many other hosts a raw count isn't specific enough to be useful.
The system Facebook (which predated eBPF via a custom ftrace event) produced, effectively, tuples of `(src_ip, src_port, dst_ip, dst_port, src_container, dst_container, ...)` and aggregated them over all hosts. This allowed counting retransmits by, say, receiving host. If there's one host that has a bad cable and is receiving retransmits from 1000 clients we may not see that signal in simple counters on the clients — for them it's just a tiny bump in the overall retransmit rate. But if we aggregate by receiving host the bad guy will stand out like a sore thumb. Same thing for all the hosts in a rack, or all hosts reachable over a given router interface, or whatever else you want. One of my common workflows when facing a bump in general errors (e.g. timeouts to the cache layer) was to quickly try grouping retransmits by a few dimensions to see if one particular combination of hosts stood out as sending or receiving more retransmits.
tl;dr: the SNMP data is one-dimensional. FB's system allowed aggregating and querying by many dimensions. This is really useful when there are thousands of machines talking to each other over many network paths.