6 ms·

Achieving 100Gbps intrusion prevention on a single server (2020)

92 points by harporoeder 4y ago

egberts1 4y ago
"The slow path can’t take advantage of constant time operations, but fortunately is less often used as most packets arrive in order. It’s also used when inserting new flows." Going to be in a world of hurt once a slow-path DDoS packets come rolling in.
- reilly3000 4y ago
  I really only operate on L7 most of the time, so pardon me if this is a dumb question: is there some form of TCP rate limiting that could happen at the switch level? WAF? Can policy be applied for the slow path? Is there a way to protect against that type of DDOS you raised?
  - megous 4y ago
    Switches don't care about TCP.
    - fuckstick 4y ago
      You are wrong. https://en.wikipedia.org/wiki/Multilayer_switch https://en.wikipedia.org/wiki/Multilayer_switch See for an intro.
      - iwillbenice 4y ago
        Respectfully, you are incorrect. Switches, in the technical sense and accepted terminology, refers to Layer 2 processing. Routing is Layer 3. And so on. MLS is just a combined product offering multiple Layers of processing.
        hackmiester 4y ago
        I think this argument can never be won. "Layer 3 switch" is common terminology. But "switching" strictly speaking is a Layer 2 action. But sometimes we say that a switch is "switching packets at Layer 3" when it is doing a hardware action in response to IP layer information. We could go back and forth all day. So let's all be reasonable if possible.
        numpad0 4y ago
        I believe “L3 switch” and routers(L3-L7) are distinguished by architecture; L2 and L3 switches employ non-programmable “packet switching fabric” ASIC with CPU acting as a control system, while routers are generally a general purpose computer optionally with non-Turing-complete ASICs for faster packet processing. Expectations of a “switch” is therefore that it’s not a dual core PowerPC box with 24-96 GbE ports on PCIe, running outdated Linux Kernel, and that it can’t do what such a bare metal box could do.
        hackmiester 4y ago
        I'm not sure what is non-programmable about the packet switching fabric. It can be programmed to switch packets, which is what we want the device to do. It can also be programmed to route packets, which is done at the same rate as switching them (usually line rate). So we can call this "layer 3 switching" because it is the same process as the L2 switching but it is happening at L3. That's what a L3 switch is and does. It can't do the same as a box with general purpose CPU, but it can do the thing you bought it for (routing) at line rate (hence the comparison to switching).
        iwillbenice 4y ago
        fuckstick 4y ago
        https://www.cisco.com/c/en/us/products/collateral/switches/business-350-series-managed-switches/datasheet-c78-744156.html https://www.cisco.com/c/en/us/products/collateral/switches/b... There’s more to switching than Bestbuy home equipment. I was going to say Netgear, but even they offer some layer 3 switches I think.
      - reilly3000 4y ago
        From reading that I see L3 switching uses "specialized ASICs with the help of content-addressable memory" and later on features "flow accounting". I'm supposing those primitives could make a rate limiter. I'm refusing to go down this rabbit hole though. If I'm wrong I'm sure I'll hear about it by tomorrow :)
  - hackmiester 4y ago
    Yes, switches can do this with policing. The hardware can inspect into higher layers in (certain, specific) ways.
  - marcus0x62 4y ago
    Switches (and other network devices) can cause TCP senders to rate limit their transmissions, and can do so without even being aware of TCP per se. This[0] page does a good job of explaining it in detail, but the gist of it is that TCP implementations try to match their transmission rate to the capacity of the underlying network by maintaining a congestion window: a sender starts sending slowly, gradually ramping up its transmit rate as long as no packet loss is observed, but quickly and drastically reduces its transmit rate when packet loss occurs. This is behavior is manipulated by non-targeted congestion control schemes like Random Early Detect and targeted schemes like traffic policing and shaping. Beyond consuming slow path resources, you might also be interested in SYN cookies to mitigate state-table exhaustion attacks. Note that UDP, by itself, does not respond to loss in the same way. Any congestion control would have to be implemented at the application level, although policing / shaping are frequently applied to UDP traffic with good effect. One minor nitpick, though: these tools are generally, but not always, effective at mitigating single-source DoS attacks. For volumetric DDoS, they don’t really work at all. For those, you need a provider, like Cloudflare or Akamai who can divert the malicious traffic away from your network. By the time the end network receives those packets, the damage is almost always done - it really doesn’t matter if you drop or forward at that point, if 100% of your Internet connection is filled with malicious traffic. [0] - https://witestlab.poly.edu/blog/tcp-congestion-control-basics/ https://witestlab.poly.edu/blog/tcp-congestion-control-basic...
- tptacek 4y ago
  This seems like a whole pattern of attacks that can be defeated with a single counter and random-early-drop.
  - egberts1 4y ago
    Nope. Need an array of counters, a big big array.
    - tptacek 4y ago
      Because...
      - egberts1 4y ago
        You want people to know how to generate these slow-path DDoS, do ya? Been there, on both ends, ... within a network accelerator lab.
        tptacek 4y ago
        I spent several years as the lead on what was at the time the most widely deployed DDOS product in the core of the Internet; I do not know what you are talking about. Could you maybe be more specific about why this particular DOS attack requires extreme statekeeping from middleboxes? I don't think it does.
        egberts1 4y ago
        Like. I said, within a network accelerator lab (or in laymen terms, an artificial DDoS whitelab environment for testing middleboxes).
      - kortilla 4y ago
        “with hundreds of thousands of concurrent flows ”
- cobertos 4y ago
  Isn't an IDS meant to be used in a LAN though? If you can craft a packet that special and hit this LAN device with it, wouldn't they have _way_ bigger problems than a dos?
  - ilyt 4y ago
    If you need 100Gbps of IDS you probably are running that in front of something internet facing.
    - getcrunk 4y ago
      Large corporate or college campus?
throw0101c 4y ago
For multi-server situations, using Arista's Tap Aggregation (TapAgg) architecture is a popular solution for splitting up network packets after doing a optical fibre tap on the network connection: * https://www.arista.com/en/solutions/tap-aggregation-with-danz https://www.arista.com/en/solutions/tap-aggregation-with-dan... The hosts would run software like Suricata, which is multi-threaded, and so can take advantage of many cores. (Until recently (3.0?) Snort was single-threaded.)
wastedimage 4y ago
This is a cool idea like the potato powered clock. There are so many holes here. Let me just pick one. They don't seem to account for reassembly issues which is a huge problem and vastly multiplies your problem space depending on how you implement the solution. What the fuck am I going on about you ask? Think of it like this: sig: abc Traffic a[1] b[2] c[3] where the packets are properly ordered in 1 2 3 order. Simple fragmentation could be sending them out of order - I believe this paper accounts for that. What if instead you send a[1] b[2] b[2] c[3]? Windows assembles this one way (depending on the version), linux another, bsd another. It's super fun. Then what if you send c[3] b[2] c[3] a[1] b[2]. One could argue, "hey d*ckhead we're going to normalize the traffic first" the problem is what is normal? Stevens had tons of good work on this. Some systems have a 'normalization' standard that's similar to how their network gear works. Also I find the fact that they say 'all the patterns' must be matched for the sig to fire. Does that include an or? Are they breaking the or down into sub detectors or something? The 10,000 signature thing is also kind of fake as the number of signatures constantly grows like the number of amazing taylor swift songs. All in all these authors need to go read the old breakingpoint test standards, or ixia, or nss, or really anyone.
- comex 4y ago
  The article does in fact mention both reassembly and out-of-order packets.
- insanitybit 4y ago
  Reassembly and ordering is one of the major focus points of this research. They talk very much about this and the performance implications. You can literally "ctrl + f 'reassembly'" ?? This all has very real world applications like with Corelight.
  - wastedimage 4y ago
    My point was they only address the most simplistic case.
- yencabulator 4y ago
  > One could argue, "hey d*ckhead we're going to normalize the traffic first" the problem is what is normal? This is why this kind of IPS integrates with a firewall well. Two decades back, my team built very fast for its day firewall that would only let assembled-and-refragmented fragments through. There was no confused ordering past the firewall, and no scenario of IDS/IPS and victim defragmenting differently.
hsbauauvhabzb 4y ago
I’m yet to see a signature matching system with defaults which are actually good. Don’t get me wrong, the ability to add regex filters are infinitely valuable for custom or emergency mitigations, but I think largely current systems blocking heuristics which look like attacks aren’t solving much. That being said, I’m interested to see anomaly detection engines that learn data patterns and flag potentially malicious traffic.
- viraptor 4y ago
  I don't believe reasonable defaults are possible for that use case. I mean, some anomalies are sure interesting. But I don't care about people trying to exploit MS exchange on my Linux boxes... Everyone runs something different, so unless a lot of profiles are provided that you can turn on/off, what would a reasonable default even be?
  - hsbauauvhabzb 4y ago
    That’s my point. Though, maybe there is something to be said for blocking things like OWA vulnerabilities as compared to generic xss/sqli detection rules.
EricE 4y ago
General purpose CPUs seem to have peaked - purpose built hardware is definitely on the rise. It worked well for GPUs (FPUs used to be external too) so I guess it shouldn't be a surprise. Mainframe on the desktop!
defrost 4y ago
For completeness, this is a (very good) November 2020 (two years ago) write up on a paper [1] with a video presentation (19 minutes) [2]. [1] https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf [2] https://www.youtube.com/watch?v=KCUCUMNPMl0 https://www.youtube.com/watch?v=KCUCUMNPMl0
xwowsersx 4y ago
Direct link to the PDF https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf (not suggesting skipping the intro post as it had some good context/color)
dang 4y ago
Discussed at the time: Achieving 100Gbps intrusion prevention on a single server - https://news.ycombinator.com/item?id=25108392 https://news.ycombinator.com/item?id=25108392 - Nov 2020 (35 comments)

92 points by harporoeder 4y ago

"The slow path can’t take advantage of constant time operations, but fortunately is less often used as most packets arrive in order. It’s also used when inserting new flows." Going to be in a world of hurt once a slow-path DDoS packets come rolling in.

reilly3000 4y ago
I really only operate on L7 most of the time, so pardon me if this is a dumb question: is there some form of TCP rate limiting that could happen at the switch level? WAF? Can policy be applied for the slow path? Is there a way to protect against that type of DDOS you raised?
- megous 4y ago
  Switches don't care about TCP.
  - fuckstick 4y ago
    You are wrong. https://en.wikipedia.org/wiki/Multilayer_switch https://en.wikipedia.org/wiki/Multilayer_switch See for an intro.
    - iwillbenice 4y ago
      Respectfully, you are incorrect. Switches, in the technical sense and accepted terminology, refers to Layer 2 processing. Routing is Layer 3. And so on. MLS is just a combined product offering multiple Layers of processing.
      - hackmiester 4y ago
        I think this argument can never be won. "Layer 3 switch" is common terminology. But "switching" strictly speaking is a Layer 2 action. But sometimes we say that a switch is "switching packets at Layer 3" when it is doing a hardware action in response to IP layer information. We could go back and forth all day. So let's all be reasonable if possible.
        numpad0 4y ago
        I believe “L3 switch” and routers(L3-L7) are distinguished by architecture; L2 and L3 switches employ non-programmable “packet switching fabric” ASIC with CPU acting as a control system, while routers are generally a general purpose computer optionally with non-Turing-complete ASICs for faster packet processing. Expectations of a “switch” is therefore that it’s not a dual core PowerPC box with 24-96 GbE ports on PCIe, running outdated Linux Kernel, and that it can’t do what such a bare metal box could do.
        hackmiester 4y ago
        I'm not sure what is non-programmable about the packet switching fabric. It can be programmed to switch packets, which is what we want the device to do. It can also be programmed to route packets, which is done at the same rate as switching them (usually line rate). So we can call this "layer 3 switching" because it is the same process as the L2 switching but it is happening at L3. That's what a L3 switch is and does. It can't do the same as a box with general purpose CPU, but it can do the thing you bought it for (routing) at line rate (hence the comparison to switching).
        iwillbenice 4y ago
      - fuckstick 4y ago
        https://www.cisco.com/c/en/us/products/collateral/switches/business-350-series-managed-switches/datasheet-c78-744156.html https://www.cisco.com/c/en/us/products/collateral/switches/b... There’s more to switching than Bestbuy home equipment. I was going to say Netgear, but even they offer some layer 3 switches I think.
    - reilly3000 4y ago
      From reading that I see L3 switching uses "specialized ASICs with the help of content-addressable memory" and later on features "flow accounting". I'm supposing those primitives could make a rate limiter. I'm refusing to go down this rabbit hole though. If I'm wrong I'm sure I'll hear about it by tomorrow :)
- hackmiester 4y ago
  Yes, switches can do this with policing. The hardware can inspect into higher layers in (certain, specific) ways.
- marcus0x62 4y ago
  Switches (and other network devices) can cause TCP senders to rate limit their transmissions, and can do so without even being aware of TCP per se. This[0] page does a good job of explaining it in detail, but the gist of it is that TCP implementations try to match their transmission rate to the capacity of the underlying network by maintaining a congestion window: a sender starts sending slowly, gradually ramping up its transmit rate as long as no packet loss is observed, but quickly and drastically reduces its transmit rate when packet loss occurs. This is behavior is manipulated by non-targeted congestion control schemes like Random Early Detect and targeted schemes like traffic policing and shaping. Beyond consuming slow path resources, you might also be interested in SYN cookies to mitigate state-table exhaustion attacks. Note that UDP, by itself, does not respond to loss in the same way. Any congestion control would have to be implemented at the application level, although policing / shaping are frequently applied to UDP traffic with good effect. One minor nitpick, though: these tools are generally, but not always, effective at mitigating single-source DoS attacks. For volumetric DDoS, they don’t really work at all. For those, you need a provider, like Cloudflare or Akamai who can divert the malicious traffic away from your network. By the time the end network receives those packets, the damage is almost always done - it really doesn’t matter if you drop or forward at that point, if 100% of your Internet connection is filled with malicious traffic. [0] - https://witestlab.poly.edu/blog/tcp-congestion-control-basics/ https://witestlab.poly.edu/blog/tcp-congestion-control-basic...
tptacek 4y ago
This seems like a whole pattern of attacks that can be defeated with a single counter and random-early-drop.
- egberts1 4y ago
  Nope. Need an array of counters, a big big array.
  - tptacek 4y ago
    Because...
    - egberts1 4y ago
      You want people to know how to generate these slow-path DDoS, do ya? Been there, on both ends, ... within a network accelerator lab.
      - tptacek 4y ago
        I spent several years as the lead on what was at the time the most widely deployed DDOS product in the core of the Internet; I do not know what you are talking about. Could you maybe be more specific about why this particular DOS attack requires extreme statekeeping from middleboxes? I don't think it does.
        egberts1 4y ago
        Like. I said, within a network accelerator lab (or in laymen terms, an artificial DDoS whitelab environment for testing middleboxes).
    - kortilla 4y ago
      “with hundreds of thousands of concurrent flows ”
cobertos 4y ago
Isn't an IDS meant to be used in a LAN though? If you can craft a packet that special and hit this LAN device with it, wouldn't they have _way_ bigger problems than a dos?
- ilyt 4y ago
  If you need 100Gbps of IDS you probably are running that in front of something internet facing.
  - getcrunk 4y ago
    Large corporate or college campus?

throw0101c 4y ago

For multi-server situations, using Arista's Tap Aggregation (TapAgg) architecture is a popular solution for splitting up network packets after doing a optical fibre tap on the network connection: * https://www.arista.com/en/solutions/tap-aggregation-with-danz https://www.arista.com/en/solutions/tap-aggregation-with-dan... The hosts would run software like Suricata, which is multi-threaded, and so can take advantage of many cores. (Until recently (3.0?) Snort was single-threaded.)

wastedimage 4y ago

This is a cool idea like the potato powered clock. There are so many holes here. Let me just pick one. They don't seem to account for reassembly issues which is a huge problem and vastly multiplies your problem space depending on how you implement the solution. What the fuck am I going on about you ask? Think of it like this: sig: abc Traffic a[1] b[2] c[3] where the packets are properly ordered in 1 2 3 order. Simple fragmentation could be sending them out of order - I believe this paper accounts for that. What if instead you send a[1] b[2] b[2] c[3]? Windows assembles this one way (depending on the version), linux another, bsd another. It's super fun. Then what if you send c[3] b[2] c[3] a[1] b[2]. One could argue, "hey d*ckhead we're going to normalize the traffic first" the problem is what is normal? Stevens had tons of good work on this. Some systems have a 'normalization' standard that's similar to how their network gear works. Also I find the fact that they say 'all the patterns' must be matched for the sig to fire. Does that include an or? Are they breaking the or down into sub detectors or something? The 10,000 signature thing is also kind of fake as the number of signatures constantly grows like the number of amazing taylor swift songs. All in all these authors need to go read the old breakingpoint test standards, or ixia, or nss, or really anyone.

comex 4y ago
The article does in fact mention both reassembly and out-of-order packets.
insanitybit 4y ago
Reassembly and ordering is one of the major focus points of this research. They talk very much about this and the performance implications. You can literally "ctrl + f 'reassembly'" ?? This all has very real world applications like with Corelight.
- wastedimage 4y ago
  My point was they only address the most simplistic case.
yencabulator 4y ago
> One could argue, "hey d*ckhead we're going to normalize the traffic first" the problem is what is normal? This is why this kind of IPS integrates with a firewall well. Two decades back, my team built very fast for its day firewall that would only let assembled-and-refragmented fragments through. There was no confused ordering past the firewall, and no scenario of IDS/IPS and victim defragmenting differently.

hsbauauvhabzb 4y ago

I’m yet to see a signature matching system with defaults which are actually good. Don’t get me wrong, the ability to add regex filters are infinitely valuable for custom or emergency mitigations, but I think largely current systems blocking heuristics which look like attacks aren’t solving much. That being said, I’m interested to see anomaly detection engines that learn data patterns and flag potentially malicious traffic.

viraptor 4y ago
I don't believe reasonable defaults are possible for that use case. I mean, some anomalies are sure interesting. But I don't care about people trying to exploit MS exchange on my Linux boxes... Everyone runs something different, so unless a lot of profiles are provided that you can turn on/off, what would a reasonable default even be?
- hsbauauvhabzb 4y ago
  That’s my point. Though, maybe there is something to be said for blocking things like OWA vulnerabilities as compared to generic xss/sqli detection rules.

EricE 4y ago

General purpose CPUs seem to have peaked - purpose built hardware is definitely on the rise. It worked well for GPUs (FPUs used to be external too) so I guess it shouldn't be a surprise. Mainframe on the desktop!

defrost 4y ago

For completeness, this is a (very good) November 2020 (two years ago) write up on a paper [1] with a video presentation (19 minutes) [2]. [1] https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf [2] https://www.youtube.com/watch?v=KCUCUMNPMl0 https://www.youtube.com/watch?v=KCUCUMNPMl0

xwowsersx 4y ago

Direct link to the PDF https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf (not suggesting skipping the intro post as it had some good context/color)

dang 4y ago

Discussed at the time: Achieving 100Gbps intrusion prevention on a single server - https://news.ycombinator.com/item?id=25108392 https://news.ycombinator.com/item?id=25108392 - Nov 2020 (35 comments)