5 ms·

Tracing HTTP Requests with tcpflow

109 points by ananthakumaran 4y ago

nhoughto 4y ago
This looks handy, better ergonomics than strace to just hook into http requests of a running process. I wonder how it works with TLS if it’s working at the socket level..
- youngtaff 4y ago
  I’m guessing it doesn’t work with TLS as it’d need the keys to decrypt the traffic
  - LinuxBender 4y ago
    I recall seeing a thread somewhere saying tcpflow would not add this capability and they point people to ssldump [1][2] and even that has some limitations. [1] - https://github.com/adulau/ssldump https://github.com/adulau/ssldump [2] - https://linux.die.net/man/1/ssldump https://linux.die.net/man/1/ssldump
  - fmajid 4y ago
    With TLS 1.3 and Perfect Forward Safety, even knowing the secret key is no longer enough. Otherwise, ssldump is a tool that would dump sessions if you knew the server private key and were somehow able to force negotiating a non-PFS TLS 1.2 cipher suite.
- xtreak29 4y ago
  There seems to be an issue open for this https://github.com/simsong/tcpflow/issues/58 https://github.com/simsong/tcpflow/issues/58
imiric 4y ago
For a tool that understands TCP and HTTP conversations, the filter syntax is awfully similar to tcpdump's, and having to use `grep -A 15` to filter specific requests seems clunky. It's good knowing this tool exists, but I think I'll stick to tcpdump and Wireshark. In Wireshark it's trivial to use the `http.request` filter to do this, and following the TCP conversation with decoded bodies, or specifying a TLS cert, is equally simple.
- denysvitali 4y ago
  For a CLI Wireshark, use tshark
ystad 4y ago
A thought. I would usually start with my application logs. Reverting to a tcp tool for a http app problem seems cobtrived
- 0x445442 4y ago
  Yeah the app looked like it was delegating to another service and needed a bearer token for that. The app logs should have showed this straight away. Furthermore, I’ve been burned enough to know that if I have a service that’s dependent on other services then I setup tasks in my service to ping the health of those services periodically and log warnings/errors if they’re unreachable. This allows me to divert blame as quickly as possible if alerted in the middle of the night.
jimmyed 4y ago
There is also netpeek[1] which has better filtering capabilities and UX overall. It supports ngrep like bpf filters which we found useful. 1: https://github.com/darshanime/netpeek https://github.com/darshanime/netpeek
_ZeD_ 4y ago
in this space I have successfully used mitmproxy, charles and, with the best experience, fiddler mitmproxy.org www.charlesproxy.com https://www.telerik.com/fiddler/fiddler-classic https://www.telerik.com/fiddler/fiddler-classic
- rwestergren 4y ago
  +1 for Fiddler, though it doesn't sound like the author was in a position to forward traffic to an HTTP proxy at the time in prod
thegabez 4y ago
This assumes you have shell and those tools installed via your Dockerfile. Best practice is to have a multi stage build and just have the working binary, keeping the image as light as possible. So when do you make a fat image with debug tools and when do you keep it skinny?
- mozman 4y ago
  Or you’re running on a VM and can install these tools for debugging and then blow it away when you’re done
29athrowaway 4y ago
There's also termshark, a TUI for tshark inspired by Wireshark. You also have mitmproxy which can be useful if TLS is involved.
aftbit 4y ago
Just because I know them better, I would have used `netstat -putln` and `tcpdump -i lo -n -A port 8000` to do the same thing. I'll take a look at tcpflow and ss though - it's always nice to know more tools.

This looks handy, better ergonomics than strace to just hook into http requests of a running process. I wonder how it works with TLS if it’s working at the socket level..

youngtaff 4y ago
I’m guessing it doesn’t work with TLS as it’d need the keys to decrypt the traffic
- LinuxBender 4y ago
  I recall seeing a thread somewhere saying tcpflow would not add this capability and they point people to ssldump [1][2] and even that has some limitations. [1] - https://github.com/adulau/ssldump https://github.com/adulau/ssldump [2] - https://linux.die.net/man/1/ssldump https://linux.die.net/man/1/ssldump
- fmajid 4y ago
  With TLS 1.3 and Perfect Forward Safety, even knowing the secret key is no longer enough. Otherwise, ssldump is a tool that would dump sessions if you knew the server private key and were somehow able to force negotiating a non-PFS TLS 1.2 cipher suite.
xtreak29 4y ago
There seems to be an issue open for this https://github.com/simsong/tcpflow/issues/58 https://github.com/simsong/tcpflow/issues/58

imiric 4y ago

For a tool that understands TCP and HTTP conversations, the filter syntax is awfully similar to tcpdump's, and having to use `grep -A 15` to filter specific requests seems clunky. It's good knowing this tool exists, but I think I'll stick to tcpdump and Wireshark. In Wireshark it's trivial to use the `http.request` filter to do this, and following the TCP conversation with decoded bodies, or specifying a TLS cert, is equally simple.

denysvitali 4y ago
For a CLI Wireshark, use tshark

ystad 4y ago

A thought. I would usually start with my application logs. Reverting to a tcp tool for a http app problem seems cobtrived

0x445442 4y ago
Yeah the app looked like it was delegating to another service and needed a bearer token for that. The app logs should have showed this straight away. Furthermore, I’ve been burned enough to know that if I have a service that’s dependent on other services then I setup tasks in my service to ping the health of those services periodically and log warnings/errors if they’re unreachable. This allows me to divert blame as quickly as possible if alerted in the middle of the night.

jimmyed 4y ago

There is also netpeek[1] which has better filtering capabilities and UX overall. It supports ngrep like bpf filters which we found useful. 1: https://github.com/darshanime/netpeek https://github.com/darshanime/netpeek

_ZeD_ 4y ago

in this space I have successfully used mitmproxy, charles and, with the best experience, fiddler mitmproxy.org www.charlesproxy.com https://www.telerik.com/fiddler/fiddler-classic https://www.telerik.com/fiddler/fiddler-classic

rwestergren 4y ago
+1 for Fiddler, though it doesn't sound like the author was in a position to forward traffic to an HTTP proxy at the time in prod

thegabez 4y ago

This assumes you have shell and those tools installed via your Dockerfile. Best practice is to have a multi stage build and just have the working binary, keeping the image as light as possible. So when do you make a fat image with debug tools and when do you keep it skinny?

mozman 4y ago
Or you’re running on a VM and can install these tools for debugging and then blow it away when you’re done

29athrowaway 4y ago

There's also termshark, a TUI for tshark inspired by Wireshark. You also have mitmproxy which can be useful if TLS is involved.

aftbit 4y ago

Just because I know them better, I would have used `netstat -putln` and `tcpdump -i lo -n -A port 8000` to do the same thing. I'll take a look at tcpflow and ss though - it's always nice to know more tools.