3 ms·

Matching Binary Patterns with Erlang

51 points by StreamBright 6y ago

dnautics 6y ago
It's even more powerful when you can have a length in your header and match the binary length to the value inside the binary The major advantage to all this is that by being declarative you aren't going to mess up with binary codecs by getting lost in piles if statements to validate your binary lengths and prevent strange packets from messing up your system.
- jlouis 6y ago
  And optimization can happen.
  - dnautics 6y ago
    true but for 98% of internet-protocolley things the network's gonna be the bottleneck, at least, IMO.
senderista 6y ago
Here’s a toy bloom filter I wrote a decade ago with this syntax: https://github.com/senderista/erlbloom https://github.com/senderista/erlbloom
imglorp 6y ago
Wow, that's my NTP code they're pointing to! https://github.com/mnp/erlang-ntp https://github.com/mnp/erlang-ntp AMA? Erlang seems optimized for several problems and network packet mangling is one of them.
- StreamBright 6y ago
  Yeah, thanks for your code, it helped me greatly. >>Erlang seems optimized for several problems and network packet mangling is one of them. The only question is why other languages (claim to be low-level) did not adopt this.
  - imglorp 6y ago
    Good to hear! And the blog post is quality hands-on learning material, like the sibling comment is asking about. We know why Erlang does it well: its original narrow focus of switching packets and insane uptime. This meant it didn't prioritize other niceties, like strings for example. So it made sense to hard wire bit vector dissection and construction. I think most other platforms have richer types and would rather use the generic type stack. Also, most general-purpose languages hide all the networking inside libraries so most users can just do foo = get($url) and not worry about it.
    - StreamBright 6y ago
      Yeah, Joe got that right. And a whole lot more.
- zests 6y ago
  I want to get some hands on experience with networking and am looking for a side project. Do you have any suggestions or know of any projects that I can use for inspiration?
  - imglorp 6y ago
    It doesn't take much to get started. A good start for reading is the concepts around IP, especially how protocols can encapsulate each other, also known as layers. Skim [1] maybe. The definitive reference used to be the W. Richard Stevens books cited there. Next pick a fun, simple protocol you're interested in. Consider ICMP, Echo, DNS, or if you like NTP, see OP's fine writeup to repeat the experiments. Skim the IETF RFC [2] for your protocol, especially the bitmap diagram. Install Wireshark and play around dissecting packets on your network, noting how it shows you the layers and bitfields as above. The most important step is to write some toy code for a client, server, or peer for your fun protocol. As above Erlang makes it simple to take apart packets directly referring to the bitfield diagrams, but most stacks should have a packet inspection library you can mess with. Wireshark will help watch your work. Another good thing to learn is routing. If you have a BSD or Linux box or a Raspberry Pi, learn some IPTables or BPF [3]. Play around with writing your own firewall rules on your box. Play with how the OS sees networking, ie netstat, arp, and route outputs. 1. https://en.wikipedia.org/wiki/Internet_protocol_suite https://en.wikipedia.org/wiki/Internet_protocol_suite 2. https://ietf.org/standards/rfcs https://ietf.org/standards/rfcs 3. http://www.brendangregg.com/ebpf.html http://www.brendangregg.com/ebpf.html
    - StreamBright 6y ago
      The next protocol is DNS! :) It is a bit tricky but I have the working parser for the header. Looking into how I could parse qname with that octet thing going on. https://github.com/n0hup/elafi/blob/master/apps/dnscache/lib/dnscache/server.ex#L27 https://github.com/n0hup/elafi/blob/master/apps/dnscache/lib... It will be running on raspberry nodes (kind of replacing Pihole).
      - imglorp 6y ago

51 points by StreamBright 6y ago

It's even more powerful when you can have a length in your header and match the binary length to the value inside the binary The major advantage to all this is that by being declarative you aren't going to mess up with binary codecs by getting lost in piles if statements to validate your binary lengths and prevent strange packets from messing up your system.

jlouis 6y ago
And optimization can happen.
- dnautics 6y ago
  true but for 98% of internet-protocolley things the network's gonna be the bottleneck, at least, IMO.

senderista 6y ago

Here’s a toy bloom filter I wrote a decade ago with this syntax: https://github.com/senderista/erlbloom https://github.com/senderista/erlbloom

imglorp 6y ago

Wow, that's my NTP code they're pointing to! https://github.com/mnp/erlang-ntp https://github.com/mnp/erlang-ntp AMA? Erlang seems optimized for several problems and network packet mangling is one of them.

StreamBright 6y ago
Yeah, thanks for your code, it helped me greatly. >>Erlang seems optimized for several problems and network packet mangling is one of them. The only question is why other languages (claim to be low-level) did not adopt this.
- imglorp 6y ago
  Good to hear! And the blog post is quality hands-on learning material, like the sibling comment is asking about. We know why Erlang does it well: its original narrow focus of switching packets and insane uptime. This meant it didn't prioritize other niceties, like strings for example. So it made sense to hard wire bit vector dissection and construction. I think most other platforms have richer types and would rather use the generic type stack. Also, most general-purpose languages hide all the networking inside libraries so most users can just do foo = get($url) and not worry about it.
  - StreamBright 6y ago
    Yeah, Joe got that right. And a whole lot more.
zests 6y ago
I want to get some hands on experience with networking and am looking for a side project. Do you have any suggestions or know of any projects that I can use for inspiration?
- imglorp 6y ago
  It doesn't take much to get started. A good start for reading is the concepts around IP, especially how protocols can encapsulate each other, also known as layers. Skim [1] maybe. The definitive reference used to be the W. Richard Stevens books cited there. Next pick a fun, simple protocol you're interested in. Consider ICMP, Echo, DNS, or if you like NTP, see OP's fine writeup to repeat the experiments. Skim the IETF RFC [2] for your protocol, especially the bitmap diagram. Install Wireshark and play around dissecting packets on your network, noting how it shows you the layers and bitfields as above. The most important step is to write some toy code for a client, server, or peer for your fun protocol. As above Erlang makes it simple to take apart packets directly referring to the bitfield diagrams, but most stacks should have a packet inspection library you can mess with. Wireshark will help watch your work. Another good thing to learn is routing. If you have a BSD or Linux box or a Raspberry Pi, learn some IPTables or BPF [3]. Play around with writing your own firewall rules on your box. Play with how the OS sees networking, ie netstat, arp, and route outputs. 1. https://en.wikipedia.org/wiki/Internet_protocol_suite https://en.wikipedia.org/wiki/Internet_protocol_suite 2. https://ietf.org/standards/rfcs https://ietf.org/standards/rfcs 3. http://www.brendangregg.com/ebpf.html http://www.brendangregg.com/ebpf.html
  - StreamBright 6y ago
    The next protocol is DNS! :) It is a bit tricky but I have the working parser for the header. Looking into how I could parse qname with that octet thing going on. https://github.com/n0hup/elafi/blob/master/apps/dnscache/lib/dnscache/server.ex#L27 https://github.com/n0hup/elafi/blob/master/apps/dnscache/lib... It will be running on raspberry nodes (kind of replacing Pihole).
    - imglorp 6y ago