6 ms·

Achieving 100Gbps intrusion prevention on a single server

157 points by kungfudoi 6y ago

cryptofistMonk 6y ago
Reading "FGPA" in the fourth paragraph confused me more than it should have.
- 90red 6y ago
  I was going to comment something on the lines of that, I don't have any prior experience with the technology so my question might be stupid but does the FGPA take the place of a dedicated machine like a server where it handles the loads?
anticristi 6y ago
What a beast! I love the FPGA-first design logic.
serialx 6y ago
This is just awesome. I wish there are open and arduino like dev env for FPGAs.
- shakna 6y ago
  The ICE40 is about as close to this as you can get. A lot of open source tools work with them, and you can find boards being sold by companies that generally specialise in the Arduino ecosystem. Unfortunately not open hardware, but that's unlikely to happen anytime soon. You'd never want to build anything production-quality with one, but they're decent enough for hobbyists.
- ryanmjacobs 6y ago
  shameless plug... https://webfpga.com https://webfpga.com isn't bad if you're looking to get your feet wet with FPGA dev
davidbrennerjr 6y ago
Snort3 will offload work to a second core only when the first core is overloaded, for the most part Snort3 uses only 1 core.
teleforce 6y ago
What a lovely fond memories of having to research and use the very first several versions of Snort IDS and its first IPS (Hogwash) about 20 years ago [1],[2]. I'd like to test this new system on the cheaper Xilinx ULtraScale+ FPGA using the provided software [3], but not sure will it even work with different FPGA set up with minimum change [4]? Another thing is that it will be interesting to test and compare it with eBPF based IPS system bypassing the kernel without the need for FPGA? It seems that for Suricata IDS/IPS it has been proposed but no performance metrics are provided of the effort [5]. [1] https://www.snort.org https://www.snort.org [2] http://hogwash.sourceforge.net/docs/overview.html http://hogwash.sourceforge.net/docs/overview.html [3] https://github.com/cmu-snap/pigasus https://github.com/cmu-snap/pigasus [4] https://www.xilinx.com/products/intellectual-property/cmac.html https://www.xilinx.com/products/intellectual-property/cmac.h... [5] https://cdn2.hubspot.net/hubfs/6344338/Resources/Stamus_WP_Intro_to_eBPF_and_XDP_in_Suricata_Online.pdf https://cdn2.hubspot.net/hubfs/6344338/Resources/Stamus_WP_I...
- gjulianm 6y ago
  At those speeds the problem is that the CPUs are not fast enough. Say you need to process these 25 Mpps (and that's being generous, the maximum packet rate at 100Gbps is 162Mpps), in a single core at 3,6 GHz: you have 144 clock cycles per packet. A cache miss or a branch predictor miss will already eat quite a lot of the available time for that packet. Even if you split the workload among multiple cores you're still pretty limited in what you can do. And if you need to forward those packets, you might be close to the PCIe 3 x16 maximum efective bandwidth. In my company we're developing traffic capture/analysis software at 100Gbps (which is orders of magnitude faster than IDS/IPS) and in order to achieve those speeds we need fast processors with a lot of cores and quite a lot of RAM, interact directly with the NIC buffers (we use DPDK now, we previously worked with modified drivers) avoiding the kernel entirely, and we have to limit the tasks per packet a lot to the point that flow state management is pretty difficult, and TCP reassembly looks impossible. I don't see a software IDS/IPS system getting anywhere close to the performance of an FPGA.
hnaccy 6y ago
>At the start of the pipeline we can afford to run lots of memory-cheap filters in parallel. Only a subset of incoming packets make it past these filters, so we need less of the more memory intensive filters running in parallel behind them to achieve the desired line rate. What would happen if you manufactured lots of packets to trigger the expensive filters?
- mr__y 6y ago
  >What would happen if you manufactured lots of packets to trigger the expensive filters? then you would effectively DoS/DDoS the IPS. Now depending on how the system as a whole works it could be an efficient way to get through with a different attack that would normally be detected/blocked by IPS.
- rwmj 6y ago
  I was wondering this too. I guess the trick would be how would you (the attacker) know which filters are being used / which are expensive? If they're only using the Snort standard filters, that would be one thing, but Cloudflare or similar services which might actually use this hardware would probably also have a completely custom rule set. So could you somehow detect experimentally what packets trigger the expensive rules? Perhaps some kind of fuzzing attack could do that.
  - justinclift 6y ago
    Might be the kind of thing that a side channel or timing attack could figure out?
gjulianm 6y ago
It's a very interesting article! Achieving 25Mpps throughput with all of those tasks to do per packet is amazing, even more with the headroom they have in resources. However, I miss more discussion on how to increase the number of concurrent flows, which would be the more complex part. Even with no rules, they only can process 500k concurrent flows, which is nowhere near enough for a 100Gbps network. For reference, in 10Gbps networks we're seeing between four and six million concurrent flows, depending on how much is that network exposed to the Internet.
- touisteur 6y ago
  Still very interesting for internal networks and middleboxes. If you're trying to bring 10GbE to the end-user, you'll need to filter closer. And in a closed or somehow isolated network, you're less likely to go over the 10k connections threshold, I think. Especially if the high bandwidth (>1GbE) is used primarily for network file access.
londons_explore 6y ago
> The reassembler is responsible for ordering TCP packets. It needs to do this at line rate while maintaining state for 100K flows. 100k flows at a line rate of 100Gbps, suggests the average flow is 1Mbps. I find that hard to believe - A typical home or office computer might have hundreds of TCP connections open at any given point in time, yet barely be downloading anything. Most of those connections are just sitting idle. Perhaps this hardware only bothers with keeping the 100k most active flows in the reassembler... In which case the obvious attack is simply to send some packets out of order by a few seconds to make sure you're out of the most-active reassembly window?
- stefan_ 6y ago
  Ding ding ding. Not only is the number of flows limited by the BRAM which is going to be impossible/very expensive to scale, it will drop particularly OOO flows: If the OOO Engine detects that BRAM capacity for OOO flows exceeds 90% of its maximum capacity, it drops the flow with the longest linked list
  - londons_explore 6y ago
    I mean you could potentially drop out of order packets that can't be reassembled to force them to be retransmitted in-order? That way, you'd reduce performance of some flows, but perhaps that's a small enough proportion of flows that it doesn't matter? The state you keep per flow also only needs to be the length of the longest pattern, so if your patterns are short (eg. banning rude words, <script> tags, etc.), it isn't much state at all... I would assume in most cases it's just a few bytes.
- ashika 6y ago
  > the average flow is 1Mbps. I find that hard to believe hard to believe? I can confirm - tcp sessions with average flows of >1Mbps do exist. you are right in identifying a weakness of the system but incorrect as to its significance. A drag racer cannot turn, because it has been optimized for something else.
crispyambulance 6y ago
Just some really basic questions since I don't understand all the jargon in the article, can someone help me? * They're talking about 100Gbps ethernet traffic. Where does one even see that line rate? It seems those rates would only occur at carriers, optical transport, or in very large data-centers. * Intrusion detection, AFAIK, would mean that the packets have to be parsed and inspected. But most traffic is encrypted, right? How far beyond typical firewall capability can you get with this stuff if you can only look at the unencrypted part of the packet? * Not sure exactly what "intrusion" means in this context? What are they looking for? What are some common rules that get applied? Every protocol is going to be wildly different and working at an application level. Don't you need specific application experts to even think about writing such specialized rules? * There are commercial products which, I think, do this stuff from Gigamon and Netscout. These products have special FPGA-based switches that have many 10G, 40G or 100G ports, how does the tech in the article differ? What's the use case?
- nix23 6y ago
  Sorry to be that harsh, but you can read everything in the net. For what a IDS is, start with that article it alone answers 3 of your questions: https://en.wikipedia.org/wiki/Intrusion_detection_system https://en.wikipedia.org/wiki/Intrusion_detection_system
  - crispyambulance 6y ago
    No offense taken. The wikipedia article makes some of the jargon clearer, but it still doesn't explain what is actually meant by intrusion detection, or even an example of a rule that's more than what a firewall would enforce. It talks about statistics and machine learning being used to apply rules. I am asking what kinds of rules? Keywords in the payload? Peculiar GET requests? Connections from Kazakhstan between midnight and 4am? Wouldn't rules like that have lots of false positives and piss off legit users? In particular, I am curious about encrypted traffic (eg TLS/https/ssh). Wikipedia says IDS's can't work on encrypted traffic. "...Encrypted packets are not processed by most intrusion detection devices..." But most traffic IS encrypted. So where do you ever have a situation where you have 100Gbps on a line, unencrypted? What's the use-case?
louwrentius 6y ago
TLDR: How they did it? Use an FPGA to offload the pattern matching. Very interesting though.
- tleb_ 6y ago
  Did you read the article? Your TLDR is misleading.
snehesht 6y ago
Github Repo - https://github.com/cmu-snap/pigasus https://github.com/cmu-snap/pigasus
Quarrel 6y ago
Interesting. So, Snort 3.0 uses Intel’s Hyperscan library, as per the article. Intel acquired this when they bought Sensory Networks, Inc. Guess what SN did before they moved to software? They made and sold hardware that essentially implemented Hyperscan in FPGAs (and SRAM, on PCI boards), for IDS & virus scanning at speed. They even had a patch to add support to Snort for it. SN eventually moved to a pure software model during the 2008 downturn (which smashed a bunch of their customers plans) and ultimately sold to Intel. End of the day it is a real bitch to do right, and it was very hard at 10Gbps, and the problems are essentially identical now (only with more stuff over https). I have no idea if they'd use them, but Intel also acquired a heap of patents on doing exactly this using the techniques that Hyperscan implements, and while they're probably happy with you doing it in software, they're probably more diligent in fighting off hardware adversaries.
- ianhowson 6y ago
  Don't forget about the I-Series! If you're still interested in hardware, there's the NVIDIA BlueField 2 and its regex accelerator: https://www.nvidia.com/en-us/networking/products/data-processing-unit/ https://www.nvidia.com/en-us/networking/products/data-proces...

157 points by kungfudoi 6y ago

cryptofistMonk 6y ago

Reading "FGPA" in the fourth paragraph confused me more than it should have.

90red 6y ago
I was going to comment something on the lines of that, I don't have any prior experience with the technology so my question might be stupid but does the FGPA take the place of a dedicated machine like a server where it handles the loads?

anticristi 6y ago

What a beast! I love the FPGA-first design logic.

serialx 6y ago

This is just awesome. I wish there are open and arduino like dev env for FPGAs.

shakna 6y ago
The ICE40 is about as close to this as you can get. A lot of open source tools work with them, and you can find boards being sold by companies that generally specialise in the Arduino ecosystem. Unfortunately not open hardware, but that's unlikely to happen anytime soon. You'd never want to build anything production-quality with one, but they're decent enough for hobbyists.
ryanmjacobs 6y ago
shameless plug... https://webfpga.com https://webfpga.com isn't bad if you're looking to get your feet wet with FPGA dev

davidbrennerjr 6y ago

Snort3 will offload work to a second core only when the first core is overloaded, for the most part Snort3 uses only 1 core.

teleforce 6y ago

What a lovely fond memories of having to research and use the very first several versions of Snort IDS and its first IPS (Hogwash) about 20 years ago [1],[2]. I'd like to test this new system on the cheaper Xilinx ULtraScale+ FPGA using the provided software [3], but not sure will it even work with different FPGA set up with minimum change [4]? Another thing is that it will be interesting to test and compare it with eBPF based IPS system bypassing the kernel without the need for FPGA? It seems that for Suricata IDS/IPS it has been proposed but no performance metrics are provided of the effort [5]. [1] https://www.snort.org https://www.snort.org [2] http://hogwash.sourceforge.net/docs/overview.html http://hogwash.sourceforge.net/docs/overview.html [3] https://github.com/cmu-snap/pigasus https://github.com/cmu-snap/pigasus [4] https://www.xilinx.com/products/intellectual-property/cmac.html https://www.xilinx.com/products/intellectual-property/cmac.h... [5] https://cdn2.hubspot.net/hubfs/6344338/Resources/Stamus_WP_Intro_to_eBPF_and_XDP_in_Suricata_Online.pdf https://cdn2.hubspot.net/hubfs/6344338/Resources/Stamus_WP_I...

gjulianm 6y ago
At those speeds the problem is that the CPUs are not fast enough. Say you need to process these 25 Mpps (and that's being generous, the maximum packet rate at 100Gbps is 162Mpps), in a single core at 3,6 GHz: you have 144 clock cycles per packet. A cache miss or a branch predictor miss will already eat quite a lot of the available time for that packet. Even if you split the workload among multiple cores you're still pretty limited in what you can do. And if you need to forward those packets, you might be close to the PCIe 3 x16 maximum efective bandwidth. In my company we're developing traffic capture/analysis software at 100Gbps (which is orders of magnitude faster than IDS/IPS) and in order to achieve those speeds we need fast processors with a lot of cores and quite a lot of RAM, interact directly with the NIC buffers (we use DPDK now, we previously worked with modified drivers) avoiding the kernel entirely, and we have to limit the tasks per packet a lot to the point that flow state management is pretty difficult, and TCP reassembly looks impossible. I don't see a software IDS/IPS system getting anywhere close to the performance of an FPGA.

hnaccy 6y ago

>At the start of the pipeline we can afford to run lots of memory-cheap filters in parallel. Only a subset of incoming packets make it past these filters, so we need less of the more memory intensive filters running in parallel behind them to achieve the desired line rate. What would happen if you manufactured lots of packets to trigger the expensive filters?

mr__y 6y ago
>What would happen if you manufactured lots of packets to trigger the expensive filters? then you would effectively DoS/DDoS the IPS. Now depending on how the system as a whole works it could be an efficient way to get through with a different attack that would normally be detected/blocked by IPS.
rwmj 6y ago
I was wondering this too. I guess the trick would be how would you (the attacker) know which filters are being used / which are expensive? If they're only using the Snort standard filters, that would be one thing, but Cloudflare or similar services which might actually use this hardware would probably also have a completely custom rule set. So could you somehow detect experimentally what packets trigger the expensive rules? Perhaps some kind of fuzzing attack could do that.
- justinclift 6y ago
  Might be the kind of thing that a side channel or timing attack could figure out?

gjulianm 6y ago

It's a very interesting article! Achieving 25Mpps throughput with all of those tasks to do per packet is amazing, even more with the headroom they have in resources. However, I miss more discussion on how to increase the number of concurrent flows, which would be the more complex part. Even with no rules, they only can process 500k concurrent flows, which is nowhere near enough for a 100Gbps network. For reference, in 10Gbps networks we're seeing between four and six million concurrent flows, depending on how much is that network exposed to the Internet.

touisteur 6y ago
Still very interesting for internal networks and middleboxes. If you're trying to bring 10GbE to the end-user, you'll need to filter closer. And in a closed or somehow isolated network, you're less likely to go over the 10k connections threshold, I think. Especially if the high bandwidth (>1GbE) is used primarily for network file access.

londons_explore 6y ago

> The reassembler is responsible for ordering TCP packets. It needs to do this at line rate while maintaining state for 100K flows. 100k flows at a line rate of 100Gbps, suggests the average flow is 1Mbps. I find that hard to believe - A typical home or office computer might have hundreds of TCP connections open at any given point in time, yet barely be downloading anything. Most of those connections are just sitting idle. Perhaps this hardware only bothers with keeping the 100k most active flows in the reassembler... In which case the obvious attack is simply to send some packets out of order by a few seconds to make sure you're out of the most-active reassembly window?

stefan_ 6y ago
Ding ding ding. Not only is the number of flows limited by the BRAM which is going to be impossible/very expensive to scale, it will drop particularly OOO flows: If the OOO Engine detects that BRAM capacity for OOO flows exceeds 90% of its maximum capacity, it drops the flow with the longest linked list
- londons_explore 6y ago
  I mean you could potentially drop out of order packets that can't be reassembled to force them to be retransmitted in-order? That way, you'd reduce performance of some flows, but perhaps that's a small enough proportion of flows that it doesn't matter? The state you keep per flow also only needs to be the length of the longest pattern, so if your patterns are short (eg. banning rude words, <script> tags, etc.), it isn't much state at all... I would assume in most cases it's just a few bytes.
ashika 6y ago
> the average flow is 1Mbps. I find that hard to believe hard to believe? I can confirm - tcp sessions with average flows of >1Mbps do exist. you are right in identifying a weakness of the system but incorrect as to its significance. A drag racer cannot turn, because it has been optimized for something else.

crispyambulance 6y ago

Just some really basic questions since I don't understand all the jargon in the article, can someone help me? * They're talking about 100Gbps ethernet traffic. Where does one even see that line rate? It seems those rates would only occur at carriers, optical transport, or in very large data-centers. * Intrusion detection, AFAIK, would mean that the packets have to be parsed and inspected. But most traffic is encrypted, right? How far beyond typical firewall capability can you get with this stuff if you can only look at the unencrypted part of the packet? * Not sure exactly what "intrusion" means in this context? What are they looking for? What are some common rules that get applied? Every protocol is going to be wildly different and working at an application level. Don't you need specific application experts to even think about writing such specialized rules? * There are commercial products which, I think, do this stuff from Gigamon and Netscout. These products have special FPGA-based switches that have many 10G, 40G or 100G ports, how does the tech in the article differ? What's the use case?

nix23 6y ago
Sorry to be that harsh, but you can read everything in the net. For what a IDS is, start with that article it alone answers 3 of your questions: https://en.wikipedia.org/wiki/Intrusion_detection_system https://en.wikipedia.org/wiki/Intrusion_detection_system
- crispyambulance 6y ago
  No offense taken. The wikipedia article makes some of the jargon clearer, but it still doesn't explain what is actually meant by intrusion detection, or even an example of a rule that's more than what a firewall would enforce. It talks about statistics and machine learning being used to apply rules. I am asking what kinds of rules? Keywords in the payload? Peculiar GET requests? Connections from Kazakhstan between midnight and 4am? Wouldn't rules like that have lots of false positives and piss off legit users? In particular, I am curious about encrypted traffic (eg TLS/https/ssh). Wikipedia says IDS's can't work on encrypted traffic. "...Encrypted packets are not processed by most intrusion detection devices..." But most traffic IS encrypted. So where do you ever have a situation where you have 100Gbps on a line, unencrypted? What's the use-case?

louwrentius 6y ago

TLDR: How they did it? Use an FPGA to offload the pattern matching. Very interesting though.

tleb_ 6y ago
Did you read the article? Your TLDR is misleading.

snehesht 6y ago

Github Repo - https://github.com/cmu-snap/pigasus https://github.com/cmu-snap/pigasus

Quarrel 6y ago

Interesting. So, Snort 3.0 uses Intel’s Hyperscan library, as per the article. Intel acquired this when they bought Sensory Networks, Inc. Guess what SN did before they moved to software? They made and sold hardware that essentially implemented Hyperscan in FPGAs (and SRAM, on PCI boards), for IDS & virus scanning at speed. They even had a patch to add support to Snort for it. SN eventually moved to a pure software model during the 2008 downturn (which smashed a bunch of their customers plans) and ultimately sold to Intel. End of the day it is a real bitch to do right, and it was very hard at 10Gbps, and the problems are essentially identical now (only with more stuff over https). I have no idea if they'd use them, but Intel also acquired a heap of patents on doing exactly this using the techniques that Hyperscan implements, and while they're probably happy with you doing it in software, they're probably more diligent in fighting off hardware adversaries.

ianhowson 6y ago
Don't forget about the I-Series! If you're still interested in hardware, there's the NVIDIA BlueField 2 and its regex accelerator: https://www.nvidia.com/en-us/networking/products/data-processing-unit/ https://www.nvidia.com/en-us/networking/products/data-proces...