4 ms·
That Cloudflare article is a little frustrating. > While we could think of more sophisticated data structures like Cuckoo filter, maybe we can be simpler Yes,
by vlmutolo 4y ago
That Cloudflare article is a little frustrating.
> While we could think of more sophisticated data structures like Cuckoo filter, maybe we can be simpler
Yes, standard Bloom filters fail for large filter sizes and/or very small false-positive rates. But we've known this for decades, and tons of other probabilistic filters have come out since then to address the problem. Cuckoo filters in particular are incredible.
Was there really no easy way to bring in an open-source Cuckoo filter implementation? They're not that complicated. Maybe I'm just used to modern languages having good package managers.
Plus the author's solution is basically the idea behind Cuckoo filters: "what if we just use a hash table, but allow for collisions?" Cuckoo hashing is just clever about guaranteeing O(1) worst-case lookup.
And for absolute dead-simple filters, a blocked Bloom filter is basically just as easy as a Bloom filter. It's like two or three more lines of code. It's just changing the indexing operation to be modulo some large "block" size. That said, I don't think they'd work too well in this case, since the author has a pretty small false-positive rate (0.000019), and in my experience small Bloom filters (which is what comprise blocked Bloom filters) don't handle small FP rates well.
But guess what excel at small FP rates… cuckoo filters.
- deleted 4y ago[deleted]
- dundarious 4y agoA linear probing hash table is simpler. That’s the trade off they were going for at that time. I don’t think that’s the most efficient solution given the hardware they were using, and I don’t think the blog author would either, but it’s certainly easier to write such a hash table — and it’s well written, but still mostly interview level stuff. To me the blog post is not about cuckoo or bloom filters or hash tables at all. It’s about profiling and highlighting that a naive reading of the literature can easily lead you astray (worse performance and complexity). In school they don’t teach you that mov is the biggest cycle-eater of them all — at least it’s not the lesson people remember.
- xxs 4y agoCuckoo is terrible from constant const point of view, linear probe is where is it at, indeed. >"mov" is the biggest cycle-eater of them all. The R part in 'RAM' is so wrong nowadays.