4 ms·

Is there any way to get to statistics about number of package drops, retransmits, etc from the application level? We are running jvm (netty client and tomcat s

by spockz 5y ago

Is there any way to get to statistics about number of package drops, retransmits, etc from the application level? We are running jvm (netty client and tomcat server) applications in K8s and are experiencing p99 delays for some requests. Server side, application metrics have a p99 of 8ms, but clients experience a p99 of 2000ms. In this same interval other requests continue happily. I do suspects either scheduling somehow, or something in the network. Any suggestions on how to figure out how to detect where this happens? (Putting wireshark in between would be a last resort.)

jeffbee 5y ago
Better than wireshark would be a BPF program. For established sockets you can get retransmission stats out of the kernel with the `ss` tool, but that leaves you no visibility into transient sockets. Switching to QUIC or other user-space protocol would give you the best observability.
tptacek 5y ago
Yes, there are a bunch of different network statistics, which you can scrape for instance out of proc (see the "net" subdirectory, and, particularly, "net/snmp"; it's trivially parseable). A more ordinary way to come at this would be to hook up a Prometheus node_exporter and just look at this stuff in Grafana.
bigcat123 5y ago
xxpor 5y ago
If it's almost exactly 2000 ms, it sounds like a retransmission timer, i.e. packet loss.
- jeffbee 5y ago
  More specifically, it's probably a lost SYN. 2s is too long to be a single retransmission on an established flow.
- spockz 5y ago
  Thanks. I’ll look into that. It actually appears to be multiples of 2000 + some additional processing times.
toast0 5y ago
netstat -s gives protocol statistics, some of which might be useful, you might also have ethernet interface statistics somewhere in case the interface is dropping packets (on FreeBSD, interface stats for nice drivers will be in sysctl dev.X where X is the driver name; some drivers have better data than others, but I haven't debugged Linux issues enough to find the same data there) Either way, tcpdump on the host or the client (sometimes you need dumps from both) should tell the tale. You probably don't need or want wireshark in between the peers, a capture from either can be loaded at your leasure. If you suspect a non-application issue, you can do -s 64 or so and not capture most of the application data, but still have all the TCP headers. pcaps aren't always exactly the truth about what's on the wire, but they're pretty close. Personally, I get a lot of mileage out of Wireshark, so every problem report looks like a pcap. If the server is really busy, the pcaps get too big, and you have to do things like short bursts of capturing and hope you see the issue in the burst, or sampling, but if your server isn't very busy, you can do things like pcap all the packets, and ask for problem reports to include client ip and port number and then you can find it in the big pcap a lot easier.
- tptacek 5y ago
  Wireshark, tcpdump, and pcap tools in general are probably way overkill for basic network statistics issues.
  - toast0 5y ago
    Overkill is the best kind of kill! Client and server disagree on p99 times by a lot and there's enough reports to actually look into it says there's probably something interesting going on, and pcap will tell you what it is. And you'll almost always see at least a few other things that might be worth fixing while you're looking at pcaps; some of which might actually be fixable. Sometimes you even get lucky and can see what traffic was going on before the mysterious gap in packets, and make a good guess at what's blocking your rx queues or whatever. Stats are nice too, of course.
namibj 5y ago
If it's usiong kernel TCP, switch to a TCP congestion control that uses tail loss probes. If you have enough control and the network you control supports it, consider L4S TCP technology or at least normal (old-style) ECN-based congestion control.

by spockz 5y ago

jeffbee 5y ago

Better than wireshark would be a BPF program. For established sockets you can get retransmission stats out of the kernel with the `ss` tool, but that leaves you no visibility into transient sockets. Switching to QUIC or other user-space protocol would give you the best observability.

tptacek 5y ago

Yes, there are a bunch of different network statistics, which you can scrape for instance out of proc (see the "net" subdirectory, and, particularly, "net/snmp"; it's trivially parseable). A more ordinary way to come at this would be to hook up a Prometheus node_exporter and just look at this stuff in Grafana.

bigcat123 5y ago

xxpor 5y ago

If it's almost exactly 2000 ms, it sounds like a retransmission timer, i.e. packet loss.

jeffbee 5y ago
More specifically, it's probably a lost SYN. 2s is too long to be a single retransmission on an established flow.
spockz 5y ago
Thanks. I’ll look into that. It actually appears to be multiples of 2000 + some additional processing times.

toast0 5y ago

netstat -s gives protocol statistics, some of which might be useful, you might also have ethernet interface statistics somewhere in case the interface is dropping packets (on FreeBSD, interface stats for nice drivers will be in sysctl dev.X where X is the driver name; some drivers have better data than others, but I haven't debugged Linux issues enough to find the same data there) Either way, tcpdump on the host or the client (sometimes you need dumps from both) should tell the tale. You probably don't need or want wireshark in between the peers, a capture from either can be loaded at your leasure. If you suspect a non-application issue, you can do -s 64 or so and not capture most of the application data, but still have all the TCP headers. pcaps aren't always exactly the truth about what's on the wire, but they're pretty close. Personally, I get a lot of mileage out of Wireshark, so every problem report looks like a pcap. If the server is really busy, the pcaps get too big, and you have to do things like short bursts of capturing and hope you see the issue in the burst, or sampling, but if your server isn't very busy, you can do things like pcap all the packets, and ask for problem reports to include client ip and port number and then you can find it in the big pcap a lot easier.

tptacek 5y ago
Wireshark, tcpdump, and pcap tools in general are probably way overkill for basic network statistics issues.
- toast0 5y ago
  Overkill is the best kind of kill! Client and server disagree on p99 times by a lot and there's enough reports to actually look into it says there's probably something interesting going on, and pcap will tell you what it is. And you'll almost always see at least a few other things that might be worth fixing while you're looking at pcaps; some of which might actually be fixable. Sometimes you even get lucky and can see what traffic was going on before the mysterious gap in packets, and make a good guess at what's blocking your rx queues or whatever. Stats are nice too, of course.

namibj 5y ago

If it's usiong kernel TCP, switch to a TCP congestion control that uses tail loss probes. If you have enough control and the network you control supports it, consider L4S TCP technology or at least normal (old-style) ECN-based congestion control.