12 ms·

tcpdump is amazing (2016)

317 points by sandebert 4y ago

roeles 4y ago
Packet capture on Linux has something that I wish I could get in more systems: kernel timestamping. This increases the accuracy of the timestamps a lot, since you lose the context switch to userspace. I wish the same thing was available for i2c and uart reads, for example.
foobarbecue 4y ago
What's the [Berke]?
- Reubensson 4y ago
  I suppose it should be a link to some resource about Berkeley Packet Filters (https://en.m.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.m.wikipedia.org/wiki/Berkeley_Packet_Filter).
- kitd 4y ago
  I'd imagine it's the Berkeley Packet Filter which underpins tcpdump, but he got distracted half way through? Filter rules syntax is here: https://biot.com/capstats/bpf.html https://biot.com/capstats/bpf.html
  - fstokesman 4y ago
    She* - this is Julia Evans blog.
    - kitd 4y ago
      Oops, apologies to her.
dTal 4y ago
Shoutout to the lesser-known tcpflow, which reassembles the complete contents of a TCP connection by paying attention to sequence numbers. Very handy!
- RickHull 4y ago
  I used tcpflow extensively about 17 jobs ago. I had a Linux box on a hub segment with Windows servers running a .NET application, used for realtime capture and monitoring application traffic in a test environment. I had an independent implementation of the framing protocol and message format written in Ruby, and if my decoder choked, then it was either my problem or their problem. Early on, it was always my problem, but after about 6 weeks, it was their problem 70% of the time. Great way to uncover subtle bugs, at the upfront cost of a 2nd independent implementation.
mr_mitm 4y ago
Obligatory advice that you shouldn't run tcpdump, tshark or wireshark as root or with sudo. Instead, set up least privileges (e.g. with capabilities) or at least use the tcpdump flag `-Z` (may be enabled by default at compile time in your distribution). Debian has a group called "wireshark" for this use case by default.
- mattrighetti 4y ago
  Why is that?
  - mr_mitm 4y ago
    These tools are parsing untrusted input (tshark and wireshark are parsing a very large number of different protocols), and considering these are all written in a memory unsafe language (C), they are at a high risk of remote command execution vulnerabilities. And since they don't actually require root permissions, just permission to access raw sockets and a few other network related things, it's simply best practices to use as little privileges as possible and only as much as necessary.
    - rixrax 4y ago
      Sometimes when no other options exist, and I need to capture untrusted traffic as root, I use 'tcpdump -i eth0 -s0 -n -w some.pcap' to save the packets to a .pcap. Just recording the frames shouldn't (invoke any parsing code although I have never actually checked). Then I open the .pcap with wire/tshark or even tcpdump in some safer sandbox.
      - hamburglar 4y ago
        You can also send the pcap to stdout and pipe it to stdin of a tshark running as non-root. And if you provide the right capture filters on tcpdump you can even have the tcpdump in an ssh session piping to a local tshark so you don’t even have to have tshark on the server.
netfortius 4y ago
Somehow related - can't praise enough ntopng [1]. I've used it not only in the more traditional manner, as active traffic analysis, but also as forensic tool, by passing it traffic capture files, and obtaining clear relationships and contributing end points apps and security issues. Highly recommend it. Edit: Forgot the nprobe [2] (same author), allowing flow data creation and export, expecially where L3 flow capable devices are not usable (e.g. intra-VLAN) [1] https://www.ntop.org/products/traffic-analysis/ntop/ https://www.ntop.org/products/traffic-analysis/ntop/ [2] https://www.ntop.org/products/netflow/nprobe/ https://www.ntop.org/products/netflow/nprobe/
- bhgtopt 4y ago
  Nice
  - 4ggr0 4y ago
    Just as a heads up if you wonder why your comment is gray and has a negative score. HN really doesn't like or wants comments saying "agree", "nice", "+1" etc., because they add nothing to the conversation. Just upvote a comment/post if you agree or like it :)
    - deleted 4y ago
      [deleted]
- zX41ZdbW 4y ago
  It is especially great, because it allows fast analytics on historical data with ClickHouse: https://www.ntop.org/ntop/historical-traffic-analysis-at-scale-using-clickhouse-with-ntopng/ https://www.ntop.org/ntop/historical-traffic-analysis-at-sca...
- seanw444 4y ago
  Kind of an unfortunate name. At first I thought it was "NToPNG" and was slightly confused.
- sneak 4y ago
_8j50 4y ago
You know what else is neat? netsh on windows. Let's you do circular size limited packet captures and associate every packet with the originating/destination process. But yes, tcpdump and strace/ltrace are the ultimate tool for "I have no idea what the problem is, let me find out exactly what is happening" when troubleshooting.
- EvanAnderson 4y ago
  I used to use netsh to capture on Windows boxes but the workflow for getting the data into pcap format was awful. Lately I’ve been using RawCap[0]. It’s a single EXE that works on stock versions of Windows and produces pcap files. It has no capture-time filtering options, which is a bummer, but it’s quicker for me to get data to look at than the netsh / convert to pcap workflow. [0] https://www.netresec.com/?page=Blog&month=2020-01&post=RawCap-Redux https://www.netresec.com/?page=Blog&month=2020-01&post=RawCa...
- hamburglar 4y ago
  > tcpdump and strace/ltrace are the ultimate tool for "I have no idea what the problem is, let me find out exactly what is happening" when troubleshooting. And the ability to do that type of troubleshooting when the shit hits the fan is becoming more and more rare. I regularly run into devs, even so-called “senior” devs, these days who don’t know how it’s useful to look at packets or system calls and whose minds are blown that anyone would know details that far down the stack. It would be depressing if it wasn’t so good for job security.
- pjc50 4y ago
  Huh, I never had any idea that netsh existed.
  - _8j50 4y ago
    That's what you use for managing the windows firewall, setting up tunnels/port forwarding too. Pretty nifty for a windows cli tool.
soheil 4y ago
Yeah but it's pretty useless for tracking what https requests get made or pretty much anything that uses encryption. You can filter by https GET requests but you won't be able to see what's inside them.
- jdsnape 4y ago
  you shouldn't even be able to see the method (GET/POST etc.) as that is inside the encrypted TLS body. You will be able to see timings and approximate sizes, as well as potentially the hostname from the SNI but that's about it.
- hoherd 4y ago
  Sure, but https is not the only protocol. tcpdump is also pretty useless at debugging OpenGL. https is only one protocol. There are hundreds or thousands of other protocols that tcpdump works fine with. Some of these protocols are underneath https, including tcp, which is in the name of the tool. So while tcpdump may not function as a tlsdump and can't see into your tls connection, it does a lot of other things great. You may as well complain that your screwdriver is no good for cutting wood. Clearly you need a different tool.
- MAGZine 4y ago
  i used to use tcpdump all of the of time for local development. with everything moving to https, sometimes even local environments, it's basically been useless now however.
- password4321 4y ago
  SSLKEYLOGFILE can help.
  - LinuxBender 4y ago
    Also SSLDump [1] [1] - https://linux.die.net/man/1/ssldump https://linux.die.net/man/1/ssldump
- benburwell 4y ago
  I wrote about how I decrypt TLS streams in Wireshark for debugging purposes here: https://www.benburwell.com/posts/intercepting-golang-tls-with-wireshark/ https://www.benburwell.com/posts/intercepting-golang-tls-wit... It won't apply to every situation, but might provide some building blocks you can adapt to your environment.
stonepresto 4y ago
You don't need wireshark to view packet data, use -X and -XX
- ye-olde-sysrq 4y ago
  this is my favorite use of tcpdump, as a "just show me basic prints" live version of wireshark, which is like 90% of the time mostly what I want. Usually when I'm debugging stuff and am reaching for wireshark/tcpdump, I'm mostly just trying to figure out if traffic is hitting the right ports at all, and then only sometimes do I even care about metadata (packet size / count etc) and it's very rare I care about actual payload (not that you can even usually see it easily nowadays anyway since everything is TLS). recent example: used tcpdump -X to debug why my wireguard setup wasn't working, since with UDP you just kind of get a shrug as to why or at what hop the udp packet got lost or filtered. So I ssh'd to all the middle boxes and just tcpdumped the UDP ports I was interested in.
PenguinCoder 4y ago
tcpdump is amazing. Question though; does anyone know why an ACK flag is displayed as a . instead of as A? I've not been able to find the reason for that.
- tptacek 4y ago
  Probably because it would be visually noisy to do anything else, since ACK is set on almost every segment.
pm2222 4y ago
Highly recommend documentation by F5 it's a goldmine, e.g. K2289: Using advanced tcpdump filters https://support.f5.com/csp/article/K2289 https://support.f5.com/csp/article/K2289
choult 4y ago
What a nice coincidence - I used it for debugging a UDP connection earlier today. > sudo tcpdump -n udp port 8125 -v -X -i lo
nimrody 4y ago
Is there a way to place the BPF filter after IP packet reassembly? Otherwise, matching for TCP port, etc. fails to capture fragments which do not have the TCP header and the resulting file is missing some data.
- toast0 4y ago
  I'm not sure, but you can probably just capture all ip fragments and do a second pass with something later? When I was running a webserver with worldwide audience and 40+ Gbps traffic (most of that was our apk though), I saw no more than a couple fragments per second; unless there was some UDP reflection DDoS going on and I was getting fragments from that. Lots of high profile sites don't even accept ip fragmentation because it's too costly to deal with.
- johnnny 4y ago
  TCP tries really hard to avoid fragmentation, by setting the DF flag and reducing the MSS when it sees a PTB error. Of course that sometimes fails, so if you still have TCP fragmented segments, the next best thing is to filter by source/destination address, saving to a PCAP file, then run tshark on that file with the "-2" flag which does packet reassembly. (I don't know if tshark can be made to do on-the-fly reassembly, that would require keeping a buffer of un-reassembled fragments until the rest of the packets are seen.)
- tptacek 4y ago
  The direct answer to your question is "no", because tcpdump isn't stateful. It reads fragments one at a time, prints them, and forgets about them.
bojanbabic 4y ago
Seems like OP fell in love with tshark
eddieroger 4y ago
tcpdump is the kind of tool I'm glad is in my toolbox, but I hate when I have to get it out. Not because it's hard to use, and this article / Julia does a great a job at explaining the basics, but every time I've had to use it in the past was because something was really wrong and none of the partner teams I've had to work with either knew how to troubleshoot or had the tools to do it. If you run any kind of network connected infra, it's worth the investment in learning how tcpdump and Wireshark work.
- lbriner 4y ago
  I've realised this recently reading various stories about what seem like amazing debug stories but boiled down to understanding how to use tcpdump or wireshark or some other profiling/logging tool but they are amazing because most devs simply don't know how to use them (or even if to use them). This is probably part of the reason why Google is super fast and most software is super slow. Same for IT teams. I should be able to assume that any answer to a network question includes the relevant info from the wireshark trace not just, "I can't see anything wrong with it"!
  - eddieroger 4y ago
    It's not even that the best debug stories boil down to understanding and using tcpdump, but there's a real, giant divide between people who know how to troubleshoot and those who don't. Slinging code is relatively easy if you know the language, but odds are your app talks to a database, or makes a web call, or uses memory, and knowing how you interact at that layer is the difference between good and bad experiences, and uptime or not when things fail. I'm no electrician, but I know how to check my circuit box when something in my house won't power on, and then how to swap power cables, etc.
- waynesonfire 4y ago
  absolutely. the tcpdump is as simple as possible and no simpler.
scottlamb 4y ago
> I mentioned really briefly that tcpdump lets you save pcap files. This is awesome because literally every network analysis tool in the universe understands pcap files. pcap files are like freshly baked chocolate chip cookies. Everybody loves them. OMG, yes, very well put. When I get a bug report with a pcap file I'm happy because I know I'll be able to see exactly what happened. Speaking of which: for one of my libraries, I want to make a diagnostic tool that replays an interaction. My library mostly operates at the TCP level (also some UDP), so I need to reconstruct the TCP flows in my tool to feed to my library. Either I need an easy-to-use Rust library to do that directly from pcap files [1] or some format that represents bytes moving over the flow (like sets of lines with a timestamp, flow id, and pretty hexdump of the bytes) with a tool that produces it from pcap. This seems like something that should exist? Wireshark's "Analyze > Follow > TCP Stream"’s "Save As…", "entire stream", "hex dump" is kind of what I want, but it doesn't have timestamps, and it doesn't have a way to put everything (multiple flows, UDP packets also) in one file. Seems like there should be something under "Export Packet Dissections" [2] but I haven't found quite the right thing there either. [1] https://crates.io/crates/pnet https://crates.io/crates/pnet looks promising but it wasn't as obvious as I hoped how to plug it in for what I want. [2] https://www.wireshark.org/docs/wsug_html_chunked/ChIOExportSection.html#idp7 https://www.wireshark.org/docs/wsug_html_chunked/ChIOExportS...
- hacb 4y ago
  Now I know that some people get pcap files with bug report and I'm jealous
- vdm 4y ago
  https://tcpreplay.appneta.com https://tcpreplay.appneta.com
  - scottlamb 4y ago
    Does that help? I don't want to replay the packets on a network. Instead, I want to feed them to my library without having to deal with reconstructing the TCP stream from the packets myself. I don't see anything for that in that suite. tcpflow (mentioned in another thread here) seems much closer in that it does reconstruct TCP flows from pcap, but it doesn't timestamp stuff (it'd be nice to have an idea if some client->server data or some server->client data came first as well as just relate to timeouts), and I'd prefer to put everything in one file (both directions of TCP data, UDP packets). Maybe I "just" need to figure out that Rust library I mentioned in the grandparent, and maybe create my own intermediate format that has just the data I want for my library. (I can discard TCP retransmissiony stuff, MACs, etc. to focus on what a library sees through the kernel socket interfaces.)
krupan 4y ago
I recently discovered netsniff-ng and it's really great too. It seems to recognize more protocols than tcpdump, like PTP. I use them both now.
iso1631 4y ago
I used tcpdump a fair bit, with -T rtp, to pull our RTP sequence numbers, which I could then look for any discontinuities. It's a great way of finding shitty network devices like Juniper SRXs which just randomly and silently drop packets in the middle of flows.
deleted 4y ago
[deleted]
dang 4y ago
Discussed at the time: Tcpdump is amazing - https://news.ycombinator.com/item?id=11302992 https://news.ycombinator.com/item?id=11302992 - March 2016 (102 comments)

317 points by sandebert 4y ago

Packet capture on Linux has something that I wish I could get in more systems: kernel timestamping. This increases the accuracy of the timestamps a lot, since you lose the context switch to userspace. I wish the same thing was available for i2c and uart reads, for example.

foobarbecue 4y ago

What's the [Berke]?

Reubensson 4y ago
I suppose it should be a link to some resource about Berkeley Packet Filters (https://en.m.wikipedia.org/wiki/Berkeley_Packet_Filter https://en.m.wikipedia.org/wiki/Berkeley_Packet_Filter).
kitd 4y ago
I'd imagine it's the Berkeley Packet Filter which underpins tcpdump, but he got distracted half way through? Filter rules syntax is here: https://biot.com/capstats/bpf.html https://biot.com/capstats/bpf.html
- fstokesman 4y ago
  She* - this is Julia Evans blog.
  - kitd 4y ago
    Oops, apologies to her.

dTal 4y ago

Shoutout to the lesser-known tcpflow, which reassembles the complete contents of a TCP connection by paying attention to sequence numbers. Very handy!

RickHull 4y ago
I used tcpflow extensively about 17 jobs ago. I had a Linux box on a hub segment with Windows servers running a .NET application, used for realtime capture and monitoring application traffic in a test environment. I had an independent implementation of the framing protocol and message format written in Ruby, and if my decoder choked, then it was either my problem or their problem. Early on, it was always my problem, but after about 6 weeks, it was their problem 70% of the time. Great way to uncover subtle bugs, at the upfront cost of a 2nd independent implementation.

mr_mitm 4y ago

Obligatory advice that you shouldn't run tcpdump, tshark or wireshark as root or with sudo. Instead, set up least privileges (e.g. with capabilities) or at least use the tcpdump flag `-Z` (may be enabled by default at compile time in your distribution). Debian has a group called "wireshark" for this use case by default.

mattrighetti 4y ago
Why is that?
- mr_mitm 4y ago
  These tools are parsing untrusted input (tshark and wireshark are parsing a very large number of different protocols), and considering these are all written in a memory unsafe language (C), they are at a high risk of remote command execution vulnerabilities. And since they don't actually require root permissions, just permission to access raw sockets and a few other network related things, it's simply best practices to use as little privileges as possible and only as much as necessary.
  - rixrax 4y ago
    Sometimes when no other options exist, and I need to capture untrusted traffic as root, I use 'tcpdump -i eth0 -s0 -n -w some.pcap' to save the packets to a .pcap. Just recording the frames shouldn't (invoke any parsing code although I have never actually checked). Then I open the .pcap with wire/tshark or even tcpdump in some safer sandbox.
    - hamburglar 4y ago
      You can also send the pcap to stdout and pipe it to stdin of a tshark running as non-root. And if you provide the right capture filters on tcpdump you can even have the tcpdump in an ssh session piping to a local tshark so you don’t even have to have tshark on the server.

netfortius 4y ago

Somehow related - can't praise enough ntopng [1]. I've used it not only in the more traditional manner, as active traffic analysis, but also as forensic tool, by passing it traffic capture files, and obtaining clear relationships and contributing end points apps and security issues. Highly recommend it. Edit: Forgot the nprobe [2] (same author), allowing flow data creation and export, expecially where L3 flow capable devices are not usable (e.g. intra-VLAN) [1] https://www.ntop.org/products/traffic-analysis/ntop/ https://www.ntop.org/products/traffic-analysis/ntop/ [2] https://www.ntop.org/products/netflow/nprobe/ https://www.ntop.org/products/netflow/nprobe/

bhgtopt 4y ago
Nice
- 4ggr0 4y ago
  Just as a heads up if you wonder why your comment is gray and has a negative score. HN really doesn't like or wants comments saying "agree", "nice", "+1" etc., because they add nothing to the conversation. Just upvote a comment/post if you agree or like it :)
  - deleted 4y ago
    [deleted]
zX41ZdbW 4y ago
It is especially great, because it allows fast analytics on historical data with ClickHouse: https://www.ntop.org/ntop/historical-traffic-analysis-at-scale-using-clickhouse-with-ntopng/ https://www.ntop.org/ntop/historical-traffic-analysis-at-sca...
seanw444 4y ago
Kind of an unfortunate name. At first I thought it was "NToPNG" and was slightly confused.
sneak 4y ago

_8j50 4y ago

You know what else is neat? netsh on windows. Let's you do circular size limited packet captures and associate every packet with the originating/destination process. But yes, tcpdump and strace/ltrace are the ultimate tool for "I have no idea what the problem is, let me find out exactly what is happening" when troubleshooting.

EvanAnderson 4y ago
I used to use netsh to capture on Windows boxes but the workflow for getting the data into pcap format was awful. Lately I’ve been using RawCap[0]. It’s a single EXE that works on stock versions of Windows and produces pcap files. It has no capture-time filtering options, which is a bummer, but it’s quicker for me to get data to look at than the netsh / convert to pcap workflow. [0] https://www.netresec.com/?page=Blog&month=2020-01&post=RawCap-Redux https://www.netresec.com/?page=Blog&month=2020-01&post=RawCa...
hamburglar 4y ago
> tcpdump and strace/ltrace are the ultimate tool for "I have no idea what the problem is, let me find out exactly what is happening" when troubleshooting. And the ability to do that type of troubleshooting when the shit hits the fan is becoming more and more rare. I regularly run into devs, even so-called “senior” devs, these days who don’t know how it’s useful to look at packets or system calls and whose minds are blown that anyone would know details that far down the stack. It would be depressing if it wasn’t so good for job security.
pjc50 4y ago
Huh, I never had any idea that netsh existed.
- _8j50 4y ago
  That's what you use for managing the windows firewall, setting up tunnels/port forwarding too. Pretty nifty for a windows cli tool.

soheil 4y ago

Yeah but it's pretty useless for tracking what https requests get made or pretty much anything that uses encryption. You can filter by https GET requests but you won't be able to see what's inside them.

jdsnape 4y ago
you shouldn't even be able to see the method (GET/POST etc.) as that is inside the encrypted TLS body. You will be able to see timings and approximate sizes, as well as potentially the hostname from the SNI but that's about it.
hoherd 4y ago
Sure, but https is not the only protocol. tcpdump is also pretty useless at debugging OpenGL. https is only one protocol. There are hundreds or thousands of other protocols that tcpdump works fine with. Some of these protocols are underneath https, including tcp, which is in the name of the tool. So while tcpdump may not function as a tlsdump and can't see into your tls connection, it does a lot of other things great. You may as well complain that your screwdriver is no good for cutting wood. Clearly you need a different tool.
MAGZine 4y ago
i used to use tcpdump all of the of time for local development. with everything moving to https, sometimes even local environments, it's basically been useless now however.
password4321 4y ago
SSLKEYLOGFILE can help.
- LinuxBender 4y ago
  Also SSLDump [1] [1] - https://linux.die.net/man/1/ssldump https://linux.die.net/man/1/ssldump
benburwell 4y ago
I wrote about how I decrypt TLS streams in Wireshark for debugging purposes here: https://www.benburwell.com/posts/intercepting-golang-tls-with-wireshark/ https://www.benburwell.com/posts/intercepting-golang-tls-wit... It won't apply to every situation, but might provide some building blocks you can adapt to your environment.

stonepresto 4y ago

You don't need wireshark to view packet data, use -X and -XX

ye-olde-sysrq 4y ago
this is my favorite use of tcpdump, as a "just show me basic prints" live version of wireshark, which is like 90% of the time mostly what I want. Usually when I'm debugging stuff and am reaching for wireshark/tcpdump, I'm mostly just trying to figure out if traffic is hitting the right ports at all, and then only sometimes do I even care about metadata (packet size / count etc) and it's very rare I care about actual payload (not that you can even usually see it easily nowadays anyway since everything is TLS). recent example: used tcpdump -X to debug why my wireguard setup wasn't working, since with UDP you just kind of get a shrug as to why or at what hop the udp packet got lost or filtered. So I ssh'd to all the middle boxes and just tcpdumped the UDP ports I was interested in.

PenguinCoder 4y ago

tcpdump is amazing. Question though; does anyone know why an ACK flag is displayed as a . instead of as A? I've not been able to find the reason for that.

tptacek 4y ago
Probably because it would be visually noisy to do anything else, since ACK is set on almost every segment.

pm2222 4y ago

Highly recommend documentation by F5 it's a goldmine, e.g. K2289: Using advanced tcpdump filters https://support.f5.com/csp/article/K2289 https://support.f5.com/csp/article/K2289

choult 4y ago

What a nice coincidence - I used it for debugging a UDP connection earlier today. > sudo tcpdump -n udp port 8125 -v -X -i lo

nimrody 4y ago

Is there a way to place the BPF filter after IP packet reassembly? Otherwise, matching for TCP port, etc. fails to capture fragments which do not have the TCP header and the resulting file is missing some data.

toast0 4y ago
I'm not sure, but you can probably just capture all ip fragments and do a second pass with something later? When I was running a webserver with worldwide audience and 40+ Gbps traffic (most of that was our apk though), I saw no more than a couple fragments per second; unless there was some UDP reflection DDoS going on and I was getting fragments from that. Lots of high profile sites don't even accept ip fragmentation because it's too costly to deal with.
johnnny 4y ago
TCP tries really hard to avoid fragmentation, by setting the DF flag and reducing the MSS when it sees a PTB error. Of course that sometimes fails, so if you still have TCP fragmented segments, the next best thing is to filter by source/destination address, saving to a PCAP file, then run tshark on that file with the "-2" flag which does packet reassembly. (I don't know if tshark can be made to do on-the-fly reassembly, that would require keeping a buffer of un-reassembled fragments until the rest of the packets are seen.)
tptacek 4y ago
The direct answer to your question is "no", because tcpdump isn't stateful. It reads fragments one at a time, prints them, and forgets about them.

bojanbabic 4y ago

Seems like OP fell in love with tshark

eddieroger 4y ago

tcpdump is the kind of tool I'm glad is in my toolbox, but I hate when I have to get it out. Not because it's hard to use, and this article / Julia does a great a job at explaining the basics, but every time I've had to use it in the past was because something was really wrong and none of the partner teams I've had to work with either knew how to troubleshoot or had the tools to do it. If you run any kind of network connected infra, it's worth the investment in learning how tcpdump and Wireshark work.

lbriner 4y ago
I've realised this recently reading various stories about what seem like amazing debug stories but boiled down to understanding how to use tcpdump or wireshark or some other profiling/logging tool but they are amazing because most devs simply don't know how to use them (or even if to use them). This is probably part of the reason why Google is super fast and most software is super slow. Same for IT teams. I should be able to assume that any answer to a network question includes the relevant info from the wireshark trace not just, "I can't see anything wrong with it"!
- eddieroger 4y ago
  It's not even that the best debug stories boil down to understanding and using tcpdump, but there's a real, giant divide between people who know how to troubleshoot and those who don't. Slinging code is relatively easy if you know the language, but odds are your app talks to a database, or makes a web call, or uses memory, and knowing how you interact at that layer is the difference between good and bad experiences, and uptime or not when things fail. I'm no electrician, but I know how to check my circuit box when something in my house won't power on, and then how to swap power cables, etc.
waynesonfire 4y ago
absolutely. the tcpdump is as simple as possible and no simpler.

scottlamb 4y ago

> I mentioned really briefly that tcpdump lets you save pcap files. This is awesome because literally every network analysis tool in the universe understands pcap files. pcap files are like freshly baked chocolate chip cookies. Everybody loves them. OMG, yes, very well put. When I get a bug report with a pcap file I'm happy because I know I'll be able to see exactly what happened. Speaking of which: for one of my libraries, I want to make a diagnostic tool that replays an interaction. My library mostly operates at the TCP level (also some UDP), so I need to reconstruct the TCP flows in my tool to feed to my library. Either I need an easy-to-use Rust library to do that directly from pcap files [1] or some format that represents bytes moving over the flow (like sets of lines with a timestamp, flow id, and pretty hexdump of the bytes) with a tool that produces it from pcap. This seems like something that should exist? Wireshark's "Analyze > Follow > TCP Stream"’s "Save As…", "entire stream", "hex dump" is kind of what I want, but it doesn't have timestamps, and it doesn't have a way to put everything (multiple flows, UDP packets also) in one file. Seems like there should be something under "Export Packet Dissections" [2] but I haven't found quite the right thing there either. [1] https://crates.io/crates/pnet https://crates.io/crates/pnet looks promising but it wasn't as obvious as I hoped how to plug it in for what I want. [2] https://www.wireshark.org/docs/wsug_html_chunked/ChIOExportSection.html#idp7 https://www.wireshark.org/docs/wsug_html_chunked/ChIOExportS...

hacb 4y ago
Now I know that some people get pcap files with bug report and I'm jealous
vdm 4y ago
https://tcpreplay.appneta.com https://tcpreplay.appneta.com
- scottlamb 4y ago
  Does that help? I don't want to replay the packets on a network. Instead, I want to feed them to my library without having to deal with reconstructing the TCP stream from the packets myself. I don't see anything for that in that suite. tcpflow (mentioned in another thread here) seems much closer in that it does reconstruct TCP flows from pcap, but it doesn't timestamp stuff (it'd be nice to have an idea if some client->server data or some server->client data came first as well as just relate to timeouts), and I'd prefer to put everything in one file (both directions of TCP data, UDP packets). Maybe I "just" need to figure out that Rust library I mentioned in the grandparent, and maybe create my own intermediate format that has just the data I want for my library. (I can discard TCP retransmissiony stuff, MACs, etc. to focus on what a library sees through the kernel socket interfaces.)

krupan 4y ago

I recently discovered netsniff-ng and it's really great too. It seems to recognize more protocols than tcpdump, like PTP. I use them both now.

iso1631 4y ago

I used tcpdump a fair bit, with -T rtp, to pull our RTP sequence numbers, which I could then look for any discontinuities. It's a great way of finding shitty network devices like Juniper SRXs which just randomly and silently drop packets in the middle of flows.

deleted 4y ago

[deleted]

dang 4y ago

Discussed at the time: Tcpdump is amazing - https://news.ycombinator.com/item?id=11302992 https://news.ycombinator.com/item?id=11302992 - March 2016 (102 comments)