4 ms·

> TCP retransmits were also sampled in detail. Hosts usually collects SNMP metrics, which includes TCP retransmits and more. Do you know what SNMP was lacking

> TCP retransmits were also sampled in detail. Hosts usually collects SNMP metrics, which includes TCP retransmits and more. Do you know what SNMP was lacking compared to eBPF? What I can think of is more dimensions in eBPF's case.

ikiris 5y ago
you're comparing apples and pumpkins. SNMP is a query mechanism, eBPF is the sampling mechanism.
- hintymad 5y ago
  Aha! I was too removed from the infrastructure then. All I knew was that SNMP metrics showed up in our telemetry system, and we got a set of standard metrics to look into. Our platform team took care of having sampling agent in place. I didn't know the protocol was about querying instead of sampling.
cranekam 5y ago
Correct me if I'm wrong but the SNMP retransmit counters are just that: a count of retransmits the host sent. Raw retransmit counts are often just a vague indication that something's up — the host is retransmiting, either because there's loss on the path or the receiver is overloaded. But given a host can talk over many paths to many other hosts a raw count isn't specific enough to be useful. The system Facebook (which predated eBPF via a custom ftrace event) produced, effectively, tuples of `(src_ip, src_port, dst_ip, dst_port, src_container, dst_container, ...)` and aggregated them over all hosts. This allowed counting retransmits by, say, receiving host. If there's one host that has a bad cable and is receiving retransmits from 1000 clients we may not see that signal in simple counters on the clients — for them it's just a tiny bump in the overall retransmit rate. But if we aggregate by receiving host the bad guy will stand out like a sore thumb. Same thing for all the hosts in a rack, or all hosts reachable over a given router interface, or whatever else you want. One of my common workflows when facing a bump in general errors (e.g. timeouts to the cache layer) was to quickly try grouping retransmits by a few dimensions to see if one particular combination of hosts stood out as sending or receiving more retransmits. tl;dr: the SNMP data is one-dimensional. FB's system allowed aggregating and querying by many dimensions. This is really useful when there are thousands of machines talking to each other over many network paths.
paulfurtado 5y ago
Doing it with eBPF gives you a hook for each retransmit, this makes it possible to know the exact connection, process, and network interface that hit the retransmit and allows you measure things like "100s of hosts are hitting 1000s of retransmits to 10.0.0.56:443", which 10.0.0.56's netstat metrics may not clearly indicate. It gets more interesting if you break things down by VM hosts, racks, rows, data centers, etc. If you go deeper with the eBPF tracing, you can also determine which code path the retransmit occurred on, which may or many not be interesting.
Hikikomori 5y ago
eBPF can give detailed stats and TCP state information on a per connection basis (flow), much more powerful than TCP stats you can grab with SNMP that are aggregated.
takeda 5y ago
eBPF similarly to DTrace allows you to view internals of your OS. With SNMP you have fixed number of metrics that you can view, while with eBPF you can create new ones. You could implement SNMP daemon that uses eBPF to get the data, and perhaps that will happen in the future if it already didn't.
hintymad 5y ago
Thanks for all the answers. I learned so much!

ikiris 5y ago

you're comparing apples and pumpkins. SNMP is a query mechanism, eBPF is the sampling mechanism.

hintymad 5y ago
Aha! I was too removed from the infrastructure then. All I knew was that SNMP metrics showed up in our telemetry system, and we got a set of standard metrics to look into. Our platform team took care of having sampling agent in place. I didn't know the protocol was about querying instead of sampling.

cranekam 5y ago

Correct me if I'm wrong but the SNMP retransmit counters are just that: a count of retransmits the host sent. Raw retransmit counts are often just a vague indication that something's up — the host is retransmiting, either because there's loss on the path or the receiver is overloaded. But given a host can talk over many paths to many other hosts a raw count isn't specific enough to be useful. The system Facebook (which predated eBPF via a custom ftrace event) produced, effectively, tuples of `(src_ip, src_port, dst_ip, dst_port, src_container, dst_container, ...)` and aggregated them over all hosts. This allowed counting retransmits by, say, receiving host. If there's one host that has a bad cable and is receiving retransmits from 1000 clients we may not see that signal in simple counters on the clients — for them it's just a tiny bump in the overall retransmit rate. But if we aggregate by receiving host the bad guy will stand out like a sore thumb. Same thing for all the hosts in a rack, or all hosts reachable over a given router interface, or whatever else you want. One of my common workflows when facing a bump in general errors (e.g. timeouts to the cache layer) was to quickly try grouping retransmits by a few dimensions to see if one particular combination of hosts stood out as sending or receiving more retransmits. tl;dr: the SNMP data is one-dimensional. FB's system allowed aggregating and querying by many dimensions. This is really useful when there are thousands of machines talking to each other over many network paths.

paulfurtado 5y ago

Doing it with eBPF gives you a hook for each retransmit, this makes it possible to know the exact connection, process, and network interface that hit the retransmit and allows you measure things like "100s of hosts are hitting 1000s of retransmits to 10.0.0.56:443", which 10.0.0.56's netstat metrics may not clearly indicate. It gets more interesting if you break things down by VM hosts, racks, rows, data centers, etc. If you go deeper with the eBPF tracing, you can also determine which code path the retransmit occurred on, which may or many not be interesting.

Hikikomori 5y ago

eBPF can give detailed stats and TCP state information on a per connection basis (flow), much more powerful than TCP stats you can grab with SNMP that are aggregated.

takeda 5y ago

eBPF similarly to DTrace allows you to view internals of your OS. With SNMP you have fixed number of metrics that you can view, while with eBPF you can create new ones. You could implement SNMP daemon that uses eBPF to get the data, and perhaps that will happen in the future if it already didn't.

hintymad 5y ago

Thanks for all the answers. I learned so much!