5 ms·
This paper is needlessly hard to read (4 pages in before there's even an explanation!) so first a refresher: A traditional bloom filter sets k bits to 1, using
by colmmacc 12y ago
This paper is needlessly hard to read (4 pages in before there's even an explanation!) so first a refresher: A traditional bloom filter sets k bits to 1, using k hash functions to determine which bits. So for input 'i', we might get something like:
h1(i) = 0
h2(i) = 5
h3(i) = 7
and upon insertion, the bloom filter would resemble:
1 | 0 | 0 | 0 | 0 | 1 | 0 | 1
to check if an item is the bloom filter, you check the corresponding bits. After you add items to a bloom filter there's some probability that the corresponding bits for items that haven't been added are also set. So it can sometimes lie to you and say something is included, even when it's not.
IBLTs replace the bits with counts of items added, and a sum of the key and value. So where Cs is the count sum, Ks is the Key sum and Vs is the value sum, the filter might look like;
Cs=1 Ks=10 Vs=20 | Cs=0 Ks=0 Vs=0 | ...
the positions in the table are computed just as before; the count in the count increments for every item added, and the key and value "sums" are a union of the keys and values that have been added (this might be a simple rolling count of a small checksum, or an xor).
Items can be removed from the table individually, by subtracting them from the relevant counts (just like a counting bloom filter) and rolling sums. A "dump" of the table can be derived by iterating through all possible values (you need to know what they might be) and deleting them one by one, and it takes trickery to support duplicate entries.
This seems like a big downside; if the input set is constrained and there is so small a finite set of inputs that you can just iterate over them, then I think it is easier to use composite numbers as your invertible lookup table.
Here's how; for each potential input element, assign it a unique prime number, keep this assignment in a static lookup table. Start the table with the value "1". To store an element, multiply the current table value by the element's value. A composite number will be generated that is the product of all elements inserted into the table.
To check for element presence; just use mod. Duplicates work; if the value is still congruent, then the element is still there. To remove an element, just divide by its value. A surprisingly large number of elements can be represented this way in a 64k-sized composite number.
A good example of this this technique is to model Internet AS paths; give every AS number a unique prime number (there are only about 100k of them) and model AS path as the composite of its component ASes; using vectorized DIV/MOD instructions it is incredibly fast to large graph operations.
- pradn 12y agoExcellent explanation. Thank you! If academics want their work to be used, writing with the utmost clarity is necessary.
- justin66 12y agoIdeally, academics would push out their papers as they have been but they'd also make blog posts on their work. All the buttons get pushed that way. And of course ideally they'd release source code of their implementations, and release all of their data, and so on... and we'd all ride ponies...
- diegolo 12y agoUnfortunately, the business of the academics is writing papers, and not writing code/blog posts. Usually they also have to write project proposals, teach, go to conferences etc. If you write a paper with simple words, it will probably be rejected since "it's trivial" or "it's not well formalized" ;)
- justicezyx 12y agoside note: me a PhD. The reason to put many seemingly unrelated stuff is to clearly explain the prior arts and give a thorough overview and background. The goal is to make a person have no idea on the specific topic (but need to have general background in the broader scope, a person with background in general hashing but nothing in Bloom filter is considered such person), to understand the paper. I can attest that reviewers can be obnoxious, if they dont understand the paper, they generally will reject the paper. That does not mean that the paper has to be write in this tedious fashion to guarantee readability, but that does guarantee that if a reviewer say "cannot understand", the authors can argue "I have xxx in page xxx line xxx".
- Yen 12y agoMinor correction - the method the paper describes to 'dump' the table is a bit different. It sounds like you're proposing that they iterate through all possible keys, check if the key is in the table, and if it is, dump that key and delete it. This would take O(2^b) time, where b is the number of bits in the key. Obviously, for reasonably-sized b, this can get a bit out of hand. The paper takes a different approach. It looks for a cell with Cs = 1. Cs = 1 means Ks and Vs represent exactly a key-value pair. Dump that (Ks,Vs) pair, then delete (Ks,Vs) from the table. In the process of deleting, there will probably be some other cell that goes from Cs = 2 to Cs = 1. continue from there. Eventually, you should get all Cs = 0. If you ever get stuck, you're unable to dump the entire list. That method takes O(n), where n is the number of elements in the table, and the fastest you could expect to dump any kind of list.