3 ms·
Why not take samples and process that instead? No need to look at all of it.
by lima 9y ago
Why not take samples and process that instead? No need to look at all of it.
- teraflop 9y agoTo meet the given sensitivity and precision requirements, you could potentially have to sample a significant fraction of the stream. And then you're still faced with the problem of finding the heavy hitters in the sample.
- makmanalp 9y agoI think it depends on how big you think each thing you want to identify is. If a heavy hitter contains 10% or 50% of traffic, then sampling is absolutely a great approach. Since we're talking about DNS traffic here (very small packets), a "significant" increase in queries for one domain could be e.g. 0.01% or less, which could be much easier to miss with sampling.
- skate22 9y agoCount min sketch guarentees that elements that occur more than a certain number of times (determined by your parameters) will be returned,sampling makes no such promise to a 100% degree of certainty.