4 ms·
>> Also if you understand how to reason about the overhead of using tcpdump ("below 2 MB/s is always ok"?), I would REALLY REALLY LOVE TO KNOW. Please tell me.
by DyslexicAtheist 11y ago
>> Also if you understand how to reason about the overhead of using tcpdump ("below 2 MB/s is always ok"?), I would REALLY REALLY LOVE TO KNOW. Please tell me.
I think when Dick Sites from Google says he has a rule of "stay below 1% of overhead" when analyzing traffic on datacenter nodes he wants you to select the right tool for the job. In the context of tcpdump you can run it with so many options that makes it very powerful. But that power is dangerous in the hand of a novice user. A simple error in how you run it (maybe missing filters or too wide an address space) can cause you to shoot well above the theoretical 1% limit. But that's not the tools fault IMO.
Anyway this seems more theoretical, because in practice I'd prefer a hardware based network tap and analyze that without creating any risk to the live traffic
- duaneb 11y ago> Anyway this seems more theoretical, because in practice I'd prefer a hardware based network tap and analyze that without creating any risk to the live traffic I'll ask amazon to install it when I need it...? They probably won't mind.
- DyslexicAtheist 11y agoUnless I misunderstood her ... I think the argument she makes is analyzing traffic in a datacenter and she talks about analyzing network traffic on a scale. Also she mentions Dick Sites who in his talk mentions datacenter technologies and not how to debug as an end user in some cloud.
- duaneb 11y agoSure. I'm just pointing out that it's becoming increasingly rare to be able to replace a given use case of tcpdump with a physical tap, so you can't always sidestep the nastiness of calculating overhead—especially on a live system.
- DyslexicAtheist 11y agothinking about it further I wouldn't want even a 1% fluctuation caused by analysis since it distorts the data captured in statistics. And as a datacenter guy who hasn't written the software I might not know the sensitivity level that some of my software is depending upon (timeout, max-latency, etc) which could spin it well above 1% not because tcpdump but because of a domino effect. So if it would be my datacenter I'd have a rule of no analysis of non-tap data.
- windowsworkstoo 11y ago>I'd have a rule of no analysis of non-tap data. This is a ridiculous rule, you are cutting off your nose to spite your face. The saying "perfect is the enemy of good" comes to mind. To your above point >maybe missing filters or too wide an address space There is merit in just capturing a shitload more than you need (provided of course, you're not trying to cap a full 10Gbit) because it is often the filters that are the cause of any "performance" issues, however you define that.
- DyslexicAtheist 11y agothe rule depends on context which I failed to mention. I agree with you in context of non critical infrastructure. Though I'm not talking about web apps in AWS but places where you still want to have your own datacenter like in a banking or telecoms, or insurance or medical environments. EDIT: whether you allow any access to a node for whatever purpose other than the software that was meant to run on that node would probably depend on what damage is done if that node goes down. If the damage is a blip in statistic and you can live with that fine but that's not always the case
- windowsworkstoo 11y agoYou're still overthinking things and dealing in hypotheticals it sounds like to me. Almost the only time you would want a physical tap is if you need a permanent tap capturing everything over a long term - often for the purpose of running through an IDS/IPS, an even then SPAN/RSPAN/ERSPAN works pretty well. Even in those industries you mention, most of the other time you are doing a capture is to troubleshoot something, so you don't need to run things for a long time, nor does any "perf issue", which is overstated, probably matter, since things are possibly already half way to fucked. And the compliance argument doesn't hold either vis-a-vis installing tcpdump - your processes and policies would be written as such to allow for debugging (or should be)
- cbhl 11y agoIf you had 100 computers, <1% overhead could mean "capture every packet going over a 1 Gbit network interface going to only 1 computer, even if it has a 100% overhead on that computer".
- deleted 11y ago[deleted]
- mortenlarsen 11y agoUsing something like "time tcpdump -c 1000 -w file.pcap ..." which stops after 1000 packets is always a good idea. It helps catch a too greedy BPF expression (PCAP filter), before using to many resources. Also always output to a file first, and then you can check the output afterwards with "tcpdump -nr file.pcap | less".
- drewg123 11y agoThe -c option is indeed a good idea. However, one of the worst things that tcpdump does is to put the NIC into promisc mode. On a physical NIC, this can be VERY expensive and may involve bouncing the link (behind your back) and dropping packets. At the very least, it can wreak havoc with steering filters on some NICs. To prevent this, use the -p option to prevent tcpdump from putting the NIC into promisc mode. Another issue with tcpdump on an endstation is caused by stateless offloads like checksum offload and offloads like TSO on the send side, and GRO / LRO on the receive side. Because the BPF filters are applied between the network stack and the device driver, you may noticed tcpdump / wireshark complaining about bad checksums on transmit -- this is likely due to checksum offload. And you may see gigantic (way larger than MTU) sized frames. This is due to GRO/LRO on receive, and TSO on transmit. You can disable stateless offloads (ethtool -K on linux, ifconfig on bsd), but that will slow the entire system down.