5 ms·

TCP Tracepoints have arrived in Linux

325 points by brendangregg 9y ago

gregdunn 9y ago
Fantastic! I'm particularly excited about the retransmit tracepoints, as troubleshooting some of these issues on systems that can't afford the tcpdump overhead has caused me significant headache in the past.
- fulafel 9y ago
  I'm sure you found these too, but your comment made me want to refresh my memory: For older kernels, there is a ftrace way for tracing retransmits: http://www.brendangregg.com/blog/2014-09-06/linux-ftrace-tcp-retransmit-tracing.html http://www.brendangregg.com/blog/2014-09-06/linux-ftrace-tcp... The oldest are SNMP style counters used by "netstat -s" / "sar -n ETCP" for all retransmits on the system, if you don't need per flow stats. Ten there's an application level getsockopt TCP_INFO, detailed at: https://linuxgazette.net/136/pfeiffer.html https://linuxgazette.net/136/pfeiffer.html
chrissnell 9y ago
Can someone explain how this differs from tcpdump and what advantages/disadvantages this new capability has?
- brendangregg 9y ago
  tcpdump shows on-the-wire packet details only, and works by instrumenting all send/receive events. These tracepoints can expose a lot more information: kernel state (including congestion control metrics), process IDs, application stacks, etc, and do so with much lower overhead by instrumenting lower-frequency events only. This makes it much more suitable for production use. We'll be able to collect TCP session information 24x7 with negligible overhead, whereas tcpdump/libpcap would be too expensive in our environment (Netflix).
LamaOfRuin 9y ago
So does this make it easier for Linux to have a version of Little Snitch?
- pantalaimon 9y ago
  There is https://github.com/evilsocket/opensnitch https://github.com/evilsocket/opensnitch Haven't used it though.
- brendangregg 9y ago
  I haven't used Little Snitch, but if it's a firewall then yes, this work is related to the eBPF work in Linux, which has been used for custom firewalls. There's a lot of work in this space, so I don't know where best to start. One example is cilium: https://github.com/cilium/cilium https://github.com/cilium/cilium
  - LamaOfRuin 9y ago
    Aha, that looks great. Yes, it's a firewall/network monitor, which monitors/controls network traffic tied to specific applications. That does appear to be the level 7 piece in cilium that, based on my naive reading of older attempts at Little Snitch-alike projects, was difficult, unreliable, and costly to do. Now I can confirm my excitement about these developments, even if I don't really understand them. :)
    - rkeene2 9y ago
      A "Little Snitch"-like system for Linux would be pretty easy to develop without these changes as well. I've thought about building such a tool, but never really had the need or saw the point over my existing workflows. I did once (over 10 years ago now) develop a tool that would, upon receiving a packet from a new source, perform an OS fingerprint on it and if it turned out to be Microsoft Windows, block it once the fingerprinting process returned... :-) This tool actually just fired off a script (JIT compiled) for every packet that matched pre-defined criteria and that script could return a verdict on what to do with that packet, which could involve inspecting userspace connections trivially
anonymousDan 9y ago
So does this make it easy to retrieve the current window size for a connection efficiently?
ksec 9y ago
All these work, does it mean Netflix is abandoning BSD?
- brendangregg 9y ago
  No. I always make sure I put Linux and BSD logos on my slidedecks when introducing Netflix. Linux on the cloud, BSD on the CDN. And also, the TCP tracepoint kernel commits was really done by multiple other companies. :)
  - trasz 9y ago
    Also, Netflix folks have just committed a „TCP Black Box” to FreeBSD-CURRENT, which seems to be something very similar in purpose.
indigodaddy 9y ago
Any potential security issues here? Could this be potentially exploited? I guess you could ask that about anything, but I'm asking.
jwbensley 9y ago
@brendangregg - superb work! Do you (or anyone else here) know if something like this exists already, or is in the pipeline, for raw sockets in Linux?

325 points by brendangregg 9y ago

Fantastic! I'm particularly excited about the retransmit tracepoints, as troubleshooting some of these issues on systems that can't afford the tcpdump overhead has caused me significant headache in the past.

fulafel 9y ago
I'm sure you found these too, but your comment made me want to refresh my memory: For older kernels, there is a ftrace way for tracing retransmits: http://www.brendangregg.com/blog/2014-09-06/linux-ftrace-tcp-retransmit-tracing.html http://www.brendangregg.com/blog/2014-09-06/linux-ftrace-tcp... The oldest are SNMP style counters used by "netstat -s" / "sar -n ETCP" for all retransmits on the system, if you don't need per flow stats. Ten there's an application level getsockopt TCP_INFO, detailed at: https://linuxgazette.net/136/pfeiffer.html https://linuxgazette.net/136/pfeiffer.html

chrissnell 9y ago

Can someone explain how this differs from tcpdump and what advantages/disadvantages this new capability has?

brendangregg 9y ago
tcpdump shows on-the-wire packet details only, and works by instrumenting all send/receive events. These tracepoints can expose a lot more information: kernel state (including congestion control metrics), process IDs, application stacks, etc, and do so with much lower overhead by instrumenting lower-frequency events only. This makes it much more suitable for production use. We'll be able to collect TCP session information 24x7 with negligible overhead, whereas tcpdump/libpcap would be too expensive in our environment (Netflix).

LamaOfRuin 9y ago

So does this make it easier for Linux to have a version of Little Snitch?

pantalaimon 9y ago
There is https://github.com/evilsocket/opensnitch https://github.com/evilsocket/opensnitch Haven't used it though.
brendangregg 9y ago
I haven't used Little Snitch, but if it's a firewall then yes, this work is related to the eBPF work in Linux, which has been used for custom firewalls. There's a lot of work in this space, so I don't know where best to start. One example is cilium: https://github.com/cilium/cilium https://github.com/cilium/cilium
- LamaOfRuin 9y ago
  Aha, that looks great. Yes, it's a firewall/network monitor, which monitors/controls network traffic tied to specific applications. That does appear to be the level 7 piece in cilium that, based on my naive reading of older attempts at Little Snitch-alike projects, was difficult, unreliable, and costly to do. Now I can confirm my excitement about these developments, even if I don't really understand them. :)
  - rkeene2 9y ago
    A "Little Snitch"-like system for Linux would be pretty easy to develop without these changes as well. I've thought about building such a tool, but never really had the need or saw the point over my existing workflows. I did once (over 10 years ago now) develop a tool that would, upon receiving a packet from a new source, perform an OS fingerprint on it and if it turned out to be Microsoft Windows, block it once the fingerprinting process returned... :-) This tool actually just fired off a script (JIT compiled) for every packet that matched pre-defined criteria and that script could return a verdict on what to do with that packet, which could involve inspecting userspace connections trivially

anonymousDan 9y ago

So does this make it easy to retrieve the current window size for a connection efficiently?

ksec 9y ago

All these work, does it mean Netflix is abandoning BSD?

brendangregg 9y ago
No. I always make sure I put Linux and BSD logos on my slidedecks when introducing Netflix. Linux on the cloud, BSD on the CDN. And also, the TCP tracepoint kernel commits was really done by multiple other companies. :)
- trasz 9y ago
  Also, Netflix folks have just committed a „TCP Black Box” to FreeBSD-CURRENT, which seems to be something very similar in purpose.

indigodaddy 9y ago

Any potential security issues here? Could this be potentially exploited? I guess you could ask that about anything, but I'm asking.

jwbensley 9y ago

@brendangregg - superb work! Do you (or anyone else here) know if something like this exists already, or is in the pipeline, for raw sockets in Linux?