5 ms·

What's wrong with pcap filters?

76 points by luu 12y ago

jsnell 12y ago
So, a couple of extra anecdotes from after I published that post. I linked that post to our internal chat, and our support manager was all "Oh wow, that VLAN section totally explains why those filters at customer X two weeks ago didn't work". This is 4 years into trying to make sure that everyone using the filters to configure our system knows about this pitfall. They knew about a couple of failure modes in the list, but not all of them :-( An hour after that I had a discussion with someone who knows the language very well, and we were puzzled about why the silly "look for a specific TCP option" filter was compiling to something that looked like it wouldn't work with ipv6. Turns out that while "tcp" works with IPv4 and IPv6, "tcp[]" works only on IPv4. He knew it but had forgotten, while I'd never parsed the documentation correctly to know about it. (There's a comment in the code generator suggesting that this is maybe not how things should work).
orf 12y ago
Sounds like a more succinct language that compiles down to (unreadable) pcap filters could be made, no?
- _lce0 12y ago
  call it spcap .. the .io is available ;)
  - bostik 12y ago
    I know that's a low-brow joke, but I hope you know the history of S-prefixing a complex beast for a "simple" approach is not exactly encouraging. SMTP - Simple Mail Transfer Protocol. Simple does not rule out hideously complex. SNMP - Simple Network Management Protocol. Let's not got there.. I shiver at the thought of a Simple Packet Capture Language. It might just end up looking like someone summoning the Ancient Ones. EDIT: ditched the word "rewrite". It's not exactly suitable here.
    - bch 12y ago
      Can't believe SOAP (simple object access protocol) didn't lead your list...
    - bd7lx 12y ago
      LOL there is also "Very Simple Control Protocol" here: http://www.vscp.org/ http://www.vscp.org/
    - lmm 12y ago
      Don't forget SBT - Simple Build Tool, the most hideously complex build tool I've ever had to use.
- breakingcups 12y ago
  How about pcap++? Compiler would be invoked with pcappp
lmz 12y ago
Unrelated: the previous post on that blog about Finnish election candidate-matching websites is quite fascinating: https://www.snellman.net/blog/archive/2015-05-11-okcupid-for-voting-the-finnish-election-engines/ https://www.snellman.net/blog/archive/2015-05-11-okcupid-for...
est 12y ago
For me, the only problem is ss, tcpdump and pcap all have different syntax which is PITA to remember which is which.
- tptacek 12y ago
  tcpdump and libpcap have different syntaxes? When did that start? tcpdump's filtering language is libpcap.
  - est 12y ago
    Sorry, I mean wireshark. You can use a filter like these in wireshark GUI: ip.addr==127.0.0.1 But not in tcpdump $ sudo tcpdump 'ip.addr==127.0.0.1' syntax error. This works in tcpdump sudo tcpdump 'host 127.0.0.1' But not in wireshark GUI
    - matthiasl 12y ago
      It's a bit more complicated than that. Wireshark allows two different filtering languages in two different places, used at different times. Your first example ("ip.addr=127.0.0.1") is a display filter. You enter those in the text field at the top of the GUI. Your third example, 'host 127.0.0.1' is a 'capture filter'. You can enter capture filters in Capture/options.
      - est 12y ago
        I can understand that, but why do we invent these two different DSLs? I suppose they have a very large portion of overlap!
        matthiasl 12y ago
        There are different trade-offs for the capture filter language and the display filter language. The capture filter language considers short and bounded runtime (no loops) to be paramount. Roughly, when you're capturing, it's important that your filters eat a limited amount of resources. Juha talks about this in the article. The display filter language abandons the careful runtime limits in favour of being more powerful. You can go higher up in the stack, you can use regexes, etc. That's acceptable because you're usually doing it offline. The conclusion of Juha's article is, roughly, "the libpcap DSL is sometimes frustrating, but I'm not aware of a clearly better alternative". I agree. I'd love it if someone invented something prettier but still solid.
lukego 12y ago
We have three main use cases for pcap filters (based on pflua) in Snabb Switch: First is ad-hoc filtering in applications. "If this is an IPv6 control message for the address of interest then put the packet on this queue." The filter notation is convenient for this and if the filters run as fast as hand-written code then why not use them. Second is end-user configuration. For example we have a "snabbnfv traffic" program that connects a 10G NIC to several QEMU virtual machines with Virtio-net. We support defining stateful packet filters by pcap filter expressions. This is a powerful tool that is already documented and familiar so it is easy for us to use in the interface. It is a power tool though and you can easily write an expression that does not do what you mean. (Happens to me too.) Third is as an intermediate language. For example, we translate OpenStack "Security Group" rules (ACLs) into pcap filter expressions for the snabbnfv traffic process. This allows OpenStack to provide a simple and limited interface to the user and for us to map that onto our more general mechanism underneath. I see this as future proof in terms of supporting more front-ends besides OpenStack and also supporting extensions for special needs e.g. matching on exotic protocol headers. So I am pretty happy with the current state of affairs. I do think there is a lot of potential for pflua to move things forward and I am looking forward to seeing more projects pick it up so that we can cooperate on this. If you are tempted to adopt pcap filters then check out pflua from Igalia: https://fosdem.org/2015/schedule/event/packet_filtering_pflua/ https://fosdem.org/2015/schedule/event/packet_filtering_pflu...
k_sze 11y ago
Wow. Thankfully I have never had to deal with pcap myself. That vlan stuff just looks completely b0rken. The semantics of `vlan` spilling over an `or` or parentheses reeks of basic language design failure.

76 points by luu 12y ago

So, a couple of extra anecdotes from after I published that post. I linked that post to our internal chat, and our support manager was all "Oh wow, that VLAN section totally explains why those filters at customer X two weeks ago didn't work". This is 4 years into trying to make sure that everyone using the filters to configure our system knows about this pitfall. They knew about a couple of failure modes in the list, but not all of them :-( An hour after that I had a discussion with someone who knows the language very well, and we were puzzled about why the silly "look for a specific TCP option" filter was compiling to something that looked like it wouldn't work with ipv6. Turns out that while "tcp" works with IPv4 and IPv6, "tcp[]" works only on IPv4. He knew it but had forgotten, while I'd never parsed the documentation correctly to know about it. (There's a comment in the code generator suggesting that this is maybe not how things should work).

orf 12y ago

Sounds like a more succinct language that compiles down to (unreadable) pcap filters could be made, no?

_lce0 12y ago
call it spcap .. the .io is available ;)
- bostik 12y ago
  I know that's a low-brow joke, but I hope you know the history of S-prefixing a complex beast for a "simple" approach is not exactly encouraging. SMTP - Simple Mail Transfer Protocol. Simple does not rule out hideously complex. SNMP - Simple Network Management Protocol. Let's not got there.. I shiver at the thought of a Simple Packet Capture Language. It might just end up looking like someone summoning the Ancient Ones. EDIT: ditched the word "rewrite". It's not exactly suitable here.
  - bch 12y ago
    Can't believe SOAP (simple object access protocol) didn't lead your list...
  - bd7lx 12y ago
    LOL there is also "Very Simple Control Protocol" here: http://www.vscp.org/ http://www.vscp.org/
  - lmm 12y ago
    Don't forget SBT - Simple Build Tool, the most hideously complex build tool I've ever had to use.
breakingcups 12y ago
How about pcap++? Compiler would be invoked with pcappp

lmz 12y ago

Unrelated: the previous post on that blog about Finnish election candidate-matching websites is quite fascinating: https://www.snellman.net/blog/archive/2015-05-11-okcupid-for-voting-the-finnish-election-engines/ https://www.snellman.net/blog/archive/2015-05-11-okcupid-for...

est 12y ago

For me, the only problem is ss, tcpdump and pcap all have different syntax which is PITA to remember which is which.

tptacek 12y ago
tcpdump and libpcap have different syntaxes? When did that start? tcpdump's filtering language is libpcap.
- est 12y ago
  Sorry, I mean wireshark. You can use a filter like these in wireshark GUI: ip.addr==127.0.0.1 But not in tcpdump $ sudo tcpdump 'ip.addr==127.0.0.1' syntax error. This works in tcpdump sudo tcpdump 'host 127.0.0.1' But not in wireshark GUI
  - matthiasl 12y ago
    It's a bit more complicated than that. Wireshark allows two different filtering languages in two different places, used at different times. Your first example ("ip.addr=127.0.0.1") is a display filter. You enter those in the text field at the top of the GUI. Your third example, 'host 127.0.0.1' is a 'capture filter'. You can enter capture filters in Capture/options.
    - est 12y ago
      I can understand that, but why do we invent these two different DSLs? I suppose they have a very large portion of overlap!
      - matthiasl 12y ago
        There are different trade-offs for the capture filter language and the display filter language. The capture filter language considers short and bounded runtime (no loops) to be paramount. Roughly, when you're capturing, it's important that your filters eat a limited amount of resources. Juha talks about this in the article. The display filter language abandons the careful runtime limits in favour of being more powerful. You can go higher up in the stack, you can use regexes, etc. That's acceptable because you're usually doing it offline. The conclusion of Juha's article is, roughly, "the libpcap DSL is sometimes frustrating, but I'm not aware of a clearly better alternative". I agree. I'd love it if someone invented something prettier but still solid.

lukego 12y ago

We have three main use cases for pcap filters (based on pflua) in Snabb Switch: First is ad-hoc filtering in applications. "If this is an IPv6 control message for the address of interest then put the packet on this queue." The filter notation is convenient for this and if the filters run as fast as hand-written code then why not use them. Second is end-user configuration. For example we have a "snabbnfv traffic" program that connects a 10G NIC to several QEMU virtual machines with Virtio-net. We support defining stateful packet filters by pcap filter expressions. This is a powerful tool that is already documented and familiar so it is easy for us to use in the interface. It is a power tool though and you can easily write an expression that does not do what you mean. (Happens to me too.) Third is as an intermediate language. For example, we translate OpenStack "Security Group" rules (ACLs) into pcap filter expressions for the snabbnfv traffic process. This allows OpenStack to provide a simple and limited interface to the user and for us to map that onto our more general mechanism underneath. I see this as future proof in terms of supporting more front-ends besides OpenStack and also supporting extensions for special needs e.g. matching on exotic protocol headers. So I am pretty happy with the current state of affairs. I do think there is a lot of potential for pflua to move things forward and I am looking forward to seeing more projects pick it up so that we can cooperate on this. If you are tempted to adopt pcap filters then check out pflua from Igalia: https://fosdem.org/2015/schedule/event/packet_filtering_pflua/ https://fosdem.org/2015/schedule/event/packet_filtering_pflu...

k_sze 11y ago

Wow. Thankfully I have never had to deal with pcap myself. That vlan stuff just looks completely b0rken. The semantics of `vlan` spilling over an `or` or parentheses reeks of basic language design failure.