3 ms·

I really like this comment. I think you're saying that moving data is easy, while computing the data hasn't kept up? Then again, I could be reading into this t

by tdy721 10y ago

I really like this comment. I think you're saying that moving data is easy, while computing the data hasn't kept up? Then again, I could be reading into this too much, and the computing part has always been a bottleneck at backbone level.

alkjshdkfjasdf 10y ago
It seems to me that he's saying in order to move data at volume, you have to compute on it. Computation is tricky. Per-bit, if you can handle the network input, you're probably able to fire packets up to the OS layer. But when you need to run stats on the incoming data, e.g. an ML classifier of "bad/not bad" or "stop/passthrough", you might be O(n^2) or worse. Moore's can't hang.
- dharma1 10y ago
  Interesting. What ML algos are used to classify packets against DDoS?
  - violentvinyl 10y ago
    I think that's the secret sauce for these quys. I'd be surprised if you can find out a lot about current techniques without signing an NDA and leaving your mobile phone in a box at security.
  - windowsworkstoo 10y ago
    None, really. It's mostly filters against common types of attacks at L3/L4, then OODA. Variations from normal get looked at and custom filters applied as appropriate. And of course, there's lots of NOC to NOC back channel comms around this stuff constantly to stay relatively on top of things.
  - maksimum 10y ago
    Not sure if this applies to DDoS, but a baseline ML method for security is outlier detection. For example, (1) you get a dataset that is mostly "good" data, with some "bad" data (2) you cluster it using something fast like k-means (3) data points are labelled as outliers if they fall further than some threshold from a cluster center.
dredmorbius 10y ago
Linespeed is pretty fast, and there's a lot moving through. Routers are a bit like GNU grep -- they way to be fast is to not touch most of the data. The more you have to touch, and the deeper you have to touch it, the more expensive it gets. The real trick is figuring out what's good, what's evil, and downrating the latter whilst allowing the good. Given peering relations, BGP routing, and the sorry state of much of those protocols, tracing problems to their source, quickly, and getting a useful response, is difficult.
Jweb_Guru 10y ago
At line rate, with millions of small packets coming in every second, even counting the number of packets per flow, a prerequisite for some of the simplest mitigation strategies, is really hard (often requires either expensive hardware like ternary content-addressable memory, or exotic data structures like counter braids that are very expensive to decode). A lot of the time people try to get around this by probabilistic sampling, etc. Similarly, MICA's ability to handle key value lookups at line rate on commodity hardware was considered a big success in the database community: https://www.usenix.org/node/179748 https://www.usenix.org/node/179748. Hopefully that should be a good indicator of the challenges inherent in performing any sort of nontrivial computation at line rate, even really, really simple computation. (This is why most DDOS mitigation strategies involve getting peers to load balance their traffic when it's still manageable, rather than buying bigger and bigger pipes; it's also why ultimately a large part of the responsibility for handling DDOS attacks rests on the shoulders of ISPs).

by tdy721 10y ago

alkjshdkfjasdf 10y ago

It seems to me that he's saying in order to move data at volume, you have to compute on it. Computation is tricky. Per-bit, if you can handle the network input, you're probably able to fire packets up to the OS layer. But when you need to run stats on the incoming data, e.g. an ML classifier of "bad/not bad" or "stop/passthrough", you might be O(n^2) or worse. Moore's can't hang.

dharma1 10y ago
Interesting. What ML algos are used to classify packets against DDoS?
- violentvinyl 10y ago
  I think that's the secret sauce for these quys. I'd be surprised if you can find out a lot about current techniques without signing an NDA and leaving your mobile phone in a box at security.
- windowsworkstoo 10y ago
  None, really. It's mostly filters against common types of attacks at L3/L4, then OODA. Variations from normal get looked at and custom filters applied as appropriate. And of course, there's lots of NOC to NOC back channel comms around this stuff constantly to stay relatively on top of things.
- maksimum 10y ago
  Not sure if this applies to DDoS, but a baseline ML method for security is outlier detection. For example, (1) you get a dataset that is mostly "good" data, with some "bad" data (2) you cluster it using something fast like k-means (3) data points are labelled as outliers if they fall further than some threshold from a cluster center.

dredmorbius 10y ago

Linespeed is pretty fast, and there's a lot moving through. Routers are a bit like GNU grep -- they way to be fast is to not touch most of the data. The more you have to touch, and the deeper you have to touch it, the more expensive it gets. The real trick is figuring out what's good, what's evil, and downrating the latter whilst allowing the good. Given peering relations, BGP routing, and the sorry state of much of those protocols, tracing problems to their source, quickly, and getting a useful response, is difficult.

Jweb_Guru 10y ago

At line rate, with millions of small packets coming in every second, even counting the number of packets per flow, a prerequisite for some of the simplest mitigation strategies, is really hard (often requires either expensive hardware like ternary content-addressable memory, or exotic data structures like counter braids that are very expensive to decode). A lot of the time people try to get around this by probabilistic sampling, etc. Similarly, MICA's ability to handle key value lookups at line rate on commodity hardware was considered a big success in the database community: https://www.usenix.org/node/179748 https://www.usenix.org/node/179748. Hopefully that should be a good indicator of the challenges inherent in performing any sort of nontrivial computation at line rate, even really, really simple computation. (This is why most DDOS mitigation strategies involve getting peers to load balance their traffic when it's still manageable, rather than buying bigger and bigger pipes; it's also why ultimately a large part of the responsibility for handling DDOS attacks rests on the shoulders of ISPs).