3 ms·
That doesn't sound compact at all! Every monkey's banana count uses a fixed number n of bits, where n = max(amountOfBananasOwned). That's horribly inefficient,
by dTal 7y ago
That doesn't sound compact at all! Every monkey's banana count uses a fixed number n of bits, where n = max(amountOfBananasOwned). That's horribly inefficient, when an ordinary binary counter uses n = log2(amountOfBananasOwned).
Which is not a criticism of Pilosa - I'm sure it's doing something very clever - I just don't understand what.
- fnordsensei 7y agoRight! I meant in comparison to serializing data, which is what I (maybe incorrectly) assumed that the parent was referring to. E.g., turning something like a JSON string of the same data into bits and sticking it in Pilosa.
- ytklx 7y agoRoaring bitmaps is quite good at compressing bits. Just as an example: Say, most monkeys have the same number of bananas. Pilosa doesn't store it as MonkeyCount bits, but just a few bytes.
- jaffee 7y agoBit-sliced indexing is the clever magic here. This post goes very deep on it https://www.pilosa.com/blog/range-encoded-bitmaps/ https://www.pilosa.com/blog/range-encoded-bitmaps/ But really, you use one bitmap for each binary bit of an integer, and it turns out you can generate arbitrary range queries on your dataset by doing various combinations of boolean operations on those bitmaps.
- kazinator 7y agoThis page perpetrates an apparent contradiction. Starts off with: > Pilosa is built around the concept of representing everything as bitmaps. What this literally means is that not a single datum is stored that is not in the bitmap. But then the diagrammed examples look like this: manatee loris sea_horse Vertebrate 1 1 1 Invertebrate 0 0 0 Breathes Air 1 1 0 Pardon my ignorance, but I see here a bitmap of 9 bits plus six character strings that are obviously not inside that bitmap. If those strings are removed, the bitmap means jack squat. Are they understood to be in another bitmap?
- jaffee 7y agoOne does have to maintain some understanding of the how integer row and column ids are linked to what they actually represent. Sometimes this is a function which might map (for example) row 3 to the letter 'd', 4 to 'e', and so on. Sometimes it has to be a lookup table which can be kept within Pilosa, or externally. Sometimes the IDs map directly to what they represent (day-of-month, year, passenger count, etc.) So strictly speaking, not everything is a bitmap, but the bulk of the heavy lifting in terms of serving queries is computation on bitmaps.