9 ms·

Capturing Millions of Packets per Second in Linux without Third-Party Libraries

249 points by andreygrehov 10y ago

cm3 10y ago
In case you're wondering about the different linux kernel bypass mechanisms, here's the relevant slide from a recent talk: https://lh3.googleusercontent.com/TO1UdUicn1wuF4jIAhskikO6MLaQgUuurORG_l9Zxr_L_XRvwvVhuZmF5vWTGGkauRFVT2daOOM=w1439-h1079-no https://lh3.googleusercontent.com/TO1UdUicn1wuF4jIAhskikO6ML... Sorry I haven't found the actual slides yet, that's why it's a photo from someone who took it while attending the talk.
revelation 10y ago
This must be the 10th blog post to land on HN on the same topic, and they all walk through the same steps and all use the same hardware (ixgbe), which is by the way a hard prerequisite to make much of these strategies effective. In any case, stop reinventing the wheel, just use a library purpose-made: http://dpdk.org/ http://dpdk.org/
- mianos 10y ago
  Plus it also supports even lower level drivers for a bunch of cards (some are VM virtualised, such as the Intel em), as well as AF_PACKET, oh, and pcap.
  - qb45 10y ago
    And OP reimplemented the subset he actually needed in half KLOC.
- rdtsc 10y ago
  Depending what you are doing, with latencies (or throughput) in that range, sticking a black box library in there right away might not be the best idea always. Doing what the author did is also a way to learn how things work. Eventually the library might be the answer, but if I had to do what they did, I would do it by hand first as well.
  - Jweb_Guru 10y ago
    I doubt there's a single real use case out there for whom DPDK isn't fast enough, but custom hardware isn't warranted.
- bogomipz 10y ago
  Can you elaborate on what you mean by the Intel NICs/drivers being a hard prerequisite here?
  - revelation 10y ago
    Lots of the low latency options require driver and Hardware cooperation, busy polling, BQL, essentially all of the ethtool options, even the IRQ affinity. Intel has been a driving force behind many kernel networking improvements but they naturally don't care for other manufacturers, so they implement a little bit of kernel infrastructure and put the rest into their drivers.
    - bogomipz 10y ago
      Understood, agreed. Thanks for the clarification.
- lukego 10y ago
  I am a Snabb hacker and I see things differently. Ethernet I/O is fundamentally a simple problem, DPDK is taking the industry in the wrong direction, and application developers should fight back. Ethernet I/O is simple at heart. You have an array of pointer+length packets that you want to send, an array of pointer+length buffers where you want to receive, and some configuration like "hash across these 10 rings" or "pick a ring based on VLAN-ID." This should not be more work than, say, a JSON parser. (However, if you aren't vigilant you could easily make it as complex as a C++ parser.) DPDK has created a direct vector for hardware vendors to ship code into applications. Hardware vendors have specific interests: they want to differentiate themselves with complicated features, they want to get their product out the door quickly even if that means throwing bodies at a complicated implementation, and they want to optimize for the narrow cases that will look good on their marketing literature. They are happy for their complicated proprietary interfaces to propagate throughout the software ecosystem. They also focus their support on their big customers via account teams and aren't really bothered about independent developers or people on non-mainstream platforms. Case in point: We want to run Snabb on Mellanox NICs. If we adopt the vendor ecosystem then we are buying into four (!) large software ecosystems: Linux kernel (mlx5 driver), Mellanox OFED (control plane), DPDK (data plane built on OFED+kernel), and Mellanox firmware tools (mostly non-open-source, strangely licensed, distributed as binaries that only work on a few distros). In practice it will be our problem to make sure these all play nice together and that will be a challenge e.g. in a container environment where we don't have control over which kernel is used. We also have to accept the engineering trade-offs that the vendor engineering team has made which in this case seems to include special optimizations to game benchmarks [1]. I say forget that for a joke. Instead we have done a bunch more work up front to first successfully lobby the vendor to release their driver API [2] and then to write a stand-alone driver of our own [3] that does not depend on anything else (kernel, ofed, dpdk, etc). This is around 1 KLOC of Lua code when all is said and done. I would love to hear from other people who want to join the ranks of self-sufficient application developers. Honestly our ConnectX driver has been a lot of work but it should be much easier for the next guy/gal to build on our experience. If you needed a JSON parser you would not look for a 100 KLOC implementation full of weird vendor extensions, so why do that for an ethernet driver? [1] http://dpdk.org/ml/archives/dev/2016-September/046705.html http://dpdk.org/ml/archives/dev/2016-September/046705.html [2] http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf http://www.mellanox.com/related-docs/user_manuals/Ethernet_A... [3] https://github.com/snabbco/snabb/blob/mellanox/src/apps/mellanox/connectx4.lua https://github.com/snabbco/snabb/blob/mellanox/src/apps/mell...
- kristianov 10y ago
  This. The author does not even address why they are not using DPDK. Are they using AMD server CPUs?
- _pmf_ 10y ago
  Because sometimes you do not want to replace one abstraction with another if the very point is to remove a layer.
Damogran6 10y ago
There's still not a lot of oomph left over to do anything with the traffic...or is that not the point of the exercise? You're not going to be comparing it, or writing it to disk at these levels. This traffic is a bit above the levels I've dealt with, but I've seen Cloud Datacenter levels of traffic that, as far as I know, you can't practically log/monitor/IPS/SIEM...or am I misinformed?
- nmjohn 10y ago
  > you can't practically log/monitor/IPS/SIEM...or am I misinformed? It depends the hardware you're using (specifically router), but using netflow / sflow / ipfix [0] you can get pretty high visibility even for high bandwidth networks. This only gets you "metadata" and not a full packet capture - but for monitoring and the like, the metadata can be far more useful. I'm not entirely sure what level of traffic you're talking about, but I know it's possible with the right hardware to use netflow with 100GbE links without having to sample (ie: Recording flows for every packet, not 1 in every n packets) [0]: Good sflow vs. netflow beakdown: http://networkengineering.stackexchange.com/a/1335 http://networkengineering.stackexchange.com/a/1335
- usefulcat 10y ago
  You probably could with more cores. For example, the newest Xeon E5-2697A has 16 cores and can be made to run at 3+GHz continuously.
bsder 10y ago
That's great, but how do you then get that many packets to disk so that you can do something with them? Presumably you need flash drives, and probably an append-only filesystem?
- signa11 10y ago
  > but how do you then get that many packets to disk it may be possible to do disk i/o at that high rate e.g. with pci-e or a dedicated appliance for dumping the entire stream. but you would running out of storage pretty fast. for example, a quick back-of-the-envelope calculation, where you dump packet stream from 4x10gbps cards with minimal 84b size (on ethernet), show that you would exhaust the storage in approx. 4.5 minutes :)
  - akira2501 10y ago
    > where you dump packet stream from 4x10gbps cards with minimal 84b size (on ethernet), show that you would exhaust the storage in approx. 4.5 minutes I don't understand your relations here, or the size of the storage you're considering. I did this: 3.5 million packets per second * 84 bytes per packet * 4 interfaces == 101.606 terabytes/day. Or, at the raw interface rate: 10 Gb/s * 4 interfaces == 432 terabytes/day.
    - signa11 10y ago
      > or the size of the storage whoopsie, i was considering 1.2tb of storage... > 3.5 million packets per second it is actually, 14.88 Mpps.
  - bsder 10y ago
    40 Gigabits per second is roughly 4 Gigabytes per second. 4 Gigabytes per second times 86400 seconds per day is 345,600 Gigabytes per day. Roughly: 345 Terabytes per day. Large, but not stupidly so.
    - signa11 10y ago
      > 40 Gigabits per second is roughly 4 Gigabytes per second. it should be 5GB/s right ? :)
pavel_odintsov 10y ago
Thanks for resurrecting my article! :) It was originally wrote for one russian site and I do not have spare time to translate it.
user5994461 10y ago
What's the point of capturing and storing 40 Gb/s of network traffic?
- detaro 10y ago
  Same as it is on slower speeds. Analytics, debugging, security monitoring, ... Of course you're going to try to avoid storing all traffic, but to decide what's interesting it has at least to be captured first. And on big sites, 40 Gb/s is still only an already random-sampled or pre-filtered subset of all traffic.
- feld 10y ago
  analysis, IDS, etc. The inability to analyze traffic at this rate is a serious problem. How do you study it to see how protocols can be improved? A lab environment cannot compare to real world traffic. How do you detect attacks (not DoS!) if it's hidden in a link operating at this capacity?
  - user5994461 10y ago
    Even if you can capture the traffic at wire speed, the CPU doesn't have the power to analyse the stream. I thought that traffic analysers had to be done with FPGA/ASIC because of that.
    - feld 10y ago
      My manager did his thesis on this. Endace NICs, split traffic up and send to a cluster of IDS servers. Allows you to actually do line rate analysis. No need for FPGA/ASIC.
- zxv 10y ago
  when monitoring a network, and faced with choices of where to tap the network, a tap which captures a wider view can be advantageous. For instance, capturing at a WAN link can provide a better view of attackers.
hacknat 10y ago
If anyone is interested I wrote a lock-free, c/c++ free, implementation of an AF_PACKET socket abstraction in Golang: https://github.com/nathanjsweet/zsocket https://github.com/nathanjsweet/zsocket I haven't implemented fan-out at all, but if anybody is interested in adding it, I'd happily apply they're pull-request.

249 points by andreygrehov 10y ago

cm3 10y ago

In case you're wondering about the different linux kernel bypass mechanisms, here's the relevant slide from a recent talk: https://lh3.googleusercontent.com/TO1UdUicn1wuF4jIAhskikO6MLaQgUuurORG_l9Zxr_L_XRvwvVhuZmF5vWTGGkauRFVT2daOOM=w1439-h1079-no https://lh3.googleusercontent.com/TO1UdUicn1wuF4jIAhskikO6ML... Sorry I haven't found the actual slides yet, that's why it's a photo from someone who took it while attending the talk.

revelation 10y ago

This must be the 10th blog post to land on HN on the same topic, and they all walk through the same steps and all use the same hardware (ixgbe), which is by the way a hard prerequisite to make much of these strategies effective. In any case, stop reinventing the wheel, just use a library purpose-made: http://dpdk.org/ http://dpdk.org/

mianos 10y ago
Plus it also supports even lower level drivers for a bunch of cards (some are VM virtualised, such as the Intel em), as well as AF_PACKET, oh, and pcap.
- qb45 10y ago
  And OP reimplemented the subset he actually needed in half KLOC.
rdtsc 10y ago
Depending what you are doing, with latencies (or throughput) in that range, sticking a black box library in there right away might not be the best idea always. Doing what the author did is also a way to learn how things work. Eventually the library might be the answer, but if I had to do what they did, I would do it by hand first as well.
- Jweb_Guru 10y ago
  I doubt there's a single real use case out there for whom DPDK isn't fast enough, but custom hardware isn't warranted.
bogomipz 10y ago
Can you elaborate on what you mean by the Intel NICs/drivers being a hard prerequisite here?
- revelation 10y ago
  Lots of the low latency options require driver and Hardware cooperation, busy polling, BQL, essentially all of the ethtool options, even the IRQ affinity. Intel has been a driving force behind many kernel networking improvements but they naturally don't care for other manufacturers, so they implement a little bit of kernel infrastructure and put the rest into their drivers.
  - bogomipz 10y ago
    Understood, agreed. Thanks for the clarification.
lukego 10y ago
I am a Snabb hacker and I see things differently. Ethernet I/O is fundamentally a simple problem, DPDK is taking the industry in the wrong direction, and application developers should fight back. Ethernet I/O is simple at heart. You have an array of pointer+length packets that you want to send, an array of pointer+length buffers where you want to receive, and some configuration like "hash across these 10 rings" or "pick a ring based on VLAN-ID." This should not be more work than, say, a JSON parser. (However, if you aren't vigilant you could easily make it as complex as a C++ parser.) DPDK has created a direct vector for hardware vendors to ship code into applications. Hardware vendors have specific interests: they want to differentiate themselves with complicated features, they want to get their product out the door quickly even if that means throwing bodies at a complicated implementation, and they want to optimize for the narrow cases that will look good on their marketing literature. They are happy for their complicated proprietary interfaces to propagate throughout the software ecosystem. They also focus their support on their big customers via account teams and aren't really bothered about independent developers or people on non-mainstream platforms. Case in point: We want to run Snabb on Mellanox NICs. If we adopt the vendor ecosystem then we are buying into four (!) large software ecosystems: Linux kernel (mlx5 driver), Mellanox OFED (control plane), DPDK (data plane built on OFED+kernel), and Mellanox firmware tools (mostly non-open-source, strangely licensed, distributed as binaries that only work on a few distros). In practice it will be our problem to make sure these all play nice together and that will be a challenge e.g. in a container environment where we don't have control over which kernel is used. We also have to accept the engineering trade-offs that the vendor engineering team has made which in this case seems to include special optimizations to game benchmarks [1]. I say forget that for a joke. Instead we have done a bunch more work up front to first successfully lobby the vendor to release their driver API [2] and then to write a stand-alone driver of our own [3] that does not depend on anything else (kernel, ofed, dpdk, etc). This is around 1 KLOC of Lua code when all is said and done. I would love to hear from other people who want to join the ranks of self-sufficient application developers. Honestly our ConnectX driver has been a lot of work but it should be much easier for the next guy/gal to build on our experience. If you needed a JSON parser you would not look for a 100 KLOC implementation full of weird vendor extensions, so why do that for an ethernet driver? [1] http://dpdk.org/ml/archives/dev/2016-September/046705.html http://dpdk.org/ml/archives/dev/2016-September/046705.html [2] http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf http://www.mellanox.com/related-docs/user_manuals/Ethernet_A... [3] https://github.com/snabbco/snabb/blob/mellanox/src/apps/mellanox/connectx4.lua https://github.com/snabbco/snabb/blob/mellanox/src/apps/mell...
kristianov 10y ago
This. The author does not even address why they are not using DPDK. Are they using AMD server CPUs?
_pmf_ 10y ago
Because sometimes you do not want to replace one abstraction with another if the very point is to remove a layer.

Damogran6 10y ago

There's still not a lot of oomph left over to do anything with the traffic...or is that not the point of the exercise? You're not going to be comparing it, or writing it to disk at these levels. This traffic is a bit above the levels I've dealt with, but I've seen Cloud Datacenter levels of traffic that, as far as I know, you can't practically log/monitor/IPS/SIEM...or am I misinformed?

nmjohn 10y ago
> you can't practically log/monitor/IPS/SIEM...or am I misinformed? It depends the hardware you're using (specifically router), but using netflow / sflow / ipfix [0] you can get pretty high visibility even for high bandwidth networks. This only gets you "metadata" and not a full packet capture - but for monitoring and the like, the metadata can be far more useful. I'm not entirely sure what level of traffic you're talking about, but I know it's possible with the right hardware to use netflow with 100GbE links without having to sample (ie: Recording flows for every packet, not 1 in every n packets) [0]: Good sflow vs. netflow beakdown: http://networkengineering.stackexchange.com/a/1335 http://networkengineering.stackexchange.com/a/1335
usefulcat 10y ago
You probably could with more cores. For example, the newest Xeon E5-2697A has 16 cores and can be made to run at 3+GHz continuously.

bsder 10y ago

That's great, but how do you then get that many packets to disk so that you can do something with them? Presumably you need flash drives, and probably an append-only filesystem?

signa11 10y ago
> but how do you then get that many packets to disk it may be possible to do disk i/o at that high rate e.g. with pci-e or a dedicated appliance for dumping the entire stream. but you would running out of storage pretty fast. for example, a quick back-of-the-envelope calculation, where you dump packet stream from 4x10gbps cards with minimal 84b size (on ethernet), show that you would exhaust the storage in approx. 4.5 minutes :)
- akira2501 10y ago
  > where you dump packet stream from 4x10gbps cards with minimal 84b size (on ethernet), show that you would exhaust the storage in approx. 4.5 minutes I don't understand your relations here, or the size of the storage you're considering. I did this: 3.5 million packets per second * 84 bytes per packet * 4 interfaces == 101.606 terabytes/day. Or, at the raw interface rate: 10 Gb/s * 4 interfaces == 432 terabytes/day.
  - signa11 10y ago
    > or the size of the storage whoopsie, i was considering 1.2tb of storage... > 3.5 million packets per second it is actually, 14.88 Mpps.
- bsder 10y ago
  40 Gigabits per second is roughly 4 Gigabytes per second. 4 Gigabytes per second times 86400 seconds per day is 345,600 Gigabytes per day. Roughly: 345 Terabytes per day. Large, but not stupidly so.
  - signa11 10y ago
    > 40 Gigabits per second is roughly 4 Gigabytes per second. it should be 5GB/s right ? :)

pavel_odintsov 10y ago

Thanks for resurrecting my article! :) It was originally wrote for one russian site and I do not have spare time to translate it.

user5994461 10y ago

What's the point of capturing and storing 40 Gb/s of network traffic?

detaro 10y ago
Same as it is on slower speeds. Analytics, debugging, security monitoring, ... Of course you're going to try to avoid storing all traffic, but to decide what's interesting it has at least to be captured first. And on big sites, 40 Gb/s is still only an already random-sampled or pre-filtered subset of all traffic.
feld 10y ago
analysis, IDS, etc. The inability to analyze traffic at this rate is a serious problem. How do you study it to see how protocols can be improved? A lab environment cannot compare to real world traffic. How do you detect attacks (not DoS!) if it's hidden in a link operating at this capacity?
- user5994461 10y ago
  Even if you can capture the traffic at wire speed, the CPU doesn't have the power to analyse the stream. I thought that traffic analysers had to be done with FPGA/ASIC because of that.
  - feld 10y ago
    My manager did his thesis on this. Endace NICs, split traffic up and send to a cluster of IDS servers. Allows you to actually do line rate analysis. No need for FPGA/ASIC.
zxv 10y ago
when monitoring a network, and faced with choices of where to tap the network, a tap which captures a wider view can be advantageous. For instance, capturing at a WAN link can provide a better view of attackers.

hacknat 10y ago

If anyone is interested I wrote a lock-free, c/c++ free, implementation of an AF_PACKET socket abstraction in Golang: https://github.com/nathanjsweet/zsocket https://github.com/nathanjsweet/zsocket I haven't implemented fan-out at all, but if anybody is interested in adding it, I'd happily apply they're pull-request.