4 ms·

Not sure who this document is aimed to. It's not technical enough to appeal to programmers that are working closely with Pilosa. And it's not written in a way t

Not sure who this document is aimed to. It's not technical enough to appeal to programmers that are working closely with Pilosa. And it's not written in a way to make it easy to understand for people that don't know anything about Pilosa (such as myself). I mean a subtitle called "Time Quantum" is enough to make me confused. Would appreciate a more generic "what is this" intro if possible.

kitd 7y ago
The link goes straight to the "Data Model" section. The Introduction section is a better starting point. There are other sections (Getting Started, Architecture and API Reference) for programmers who want to know the details.
dmos62 7y ago
I found the use cases section the most informative. You can click on a use case to get a write-up. Here's a few excerpts from transportation: > Pilosa is a distributed bitmap index that sits on top of a data store. The key to understanding and then using Pilosa is converting data such that it is represented in ones and zeros. This dramatically reduces the size as well as accelerates query times. > For example, timestamps are important information, but we tend to be interested in individual components of a timestamp, especially when analyzing data with cyclic trends. Timestamp components are stored as groups of bitmaps, known as “frames”. We create one frame for the day of the week, as illustrated in the following table. Along with similar frames for year, month, and time of day, this accelerates queries that ask questions about rides belonging to any logical combination of these time groups. > [...] > Because each data point includes pickup/dropoff times and total distance travelled, it’s easy to determine the average speed of the trip. As an example, we use this as a first order approximation of congestion. We created a frame representing average speed, with a spacing of 1 mph. > In order to answer questions about congestion, we needed to first determine what speeds constitute slow traffic. One of the basic queries in Pilosa is the TopN function, and we used that to get a list of all the different average speeds. By performing a count on each we built a histogram of how many rides fall into each speed bucket, and decided from there which buckets deviate enough from the norm to constitute congestion.
- dTal 7y ago
  >converting data such that it is represented in ones and zeros er, what? Isn't it all? >This dramatically reduces the size huh? There is no symbolic encoding less efficient for length than binary.
  - jaffee 7y ago
    Good catch... that sounds pretty silly. It should probably read more like "converting relationships to be represented by single bits" As a concrete example, we took the NYC taxi ride data set which is something like 300GB of CSV files and when it was indexed in Pilosa, the total size of all the bitmap files was closer to 40GB.
    - kazinator 7y ago
      It's quite clear to hopefully everyone that a matrix of bits can represent a DAG. That's called an adjacency matrix. What's not obvious is what we're associating with what in the NYC taxi ride data set. A bitmap can also represent a set: the bit positions denote enumerated element symbols, and the value indicates whether that element is present. So we rearrange the NYC taxi ride data into a data structure based on graphs and sets, and make large bitmaps?
  - fnordsensei 7y ago
    Well, you could decide that one axis is monkeyIndex, and the other is amountOfBananasOwned, and have a quite compact representation of which monkey owns what number of bananas. I.e., decide on a symbolic meaning for the axes rather than converting data wholesale.
    - ytklx 7y ago
      This is very similar to how Pilosa saves integers. https://www.pilosa.com/docs/latest/data-model/#bsi-range-encoding https://www.pilosa.com/docs/latest/data-model/#bsi-range-enc...
      - 7y ago
maycotte 7y ago
Great feedback. We could also do a better job of documenting the most frequent use cases which include high cardinality segmentation, personalization at scale, rapid data exploration/discovery, edge analytics, fast BI, threat detection, bioinformatics and more.

kitd 7y ago

The link goes straight to the "Data Model" section. The Introduction section is a better starting point. There are other sections (Getting Started, Architecture and API Reference) for programmers who want to know the details.

dmos62 7y ago

I found the use cases section the most informative. You can click on a use case to get a write-up. Here's a few excerpts from transportation: > Pilosa is a distributed bitmap index that sits on top of a data store. The key to understanding and then using Pilosa is converting data such that it is represented in ones and zeros. This dramatically reduces the size as well as accelerates query times. > For example, timestamps are important information, but we tend to be interested in individual components of a timestamp, especially when analyzing data with cyclic trends. Timestamp components are stored as groups of bitmaps, known as “frames”. We create one frame for the day of the week, as illustrated in the following table. Along with similar frames for year, month, and time of day, this accelerates queries that ask questions about rides belonging to any logical combination of these time groups. > [...] > Because each data point includes pickup/dropoff times and total distance travelled, it’s easy to determine the average speed of the trip. As an example, we use this as a first order approximation of congestion. We created a frame representing average speed, with a spacing of 1 mph. > In order to answer questions about congestion, we needed to first determine what speeds constitute slow traffic. One of the basic queries in Pilosa is the TopN function, and we used that to get a list of all the different average speeds. By performing a count on each we built a histogram of how many rides fall into each speed bucket, and decided from there which buckets deviate enough from the norm to constitute congestion.

dTal 7y ago
>converting data such that it is represented in ones and zeros er, what? Isn't it all? >This dramatically reduces the size huh? There is no symbolic encoding less efficient for length than binary.
- jaffee 7y ago
  Good catch... that sounds pretty silly. It should probably read more like "converting relationships to be represented by single bits" As a concrete example, we took the NYC taxi ride data set which is something like 300GB of CSV files and when it was indexed in Pilosa, the total size of all the bitmap files was closer to 40GB.
  - kazinator 7y ago
    It's quite clear to hopefully everyone that a matrix of bits can represent a DAG. That's called an adjacency matrix. What's not obvious is what we're associating with what in the NYC taxi ride data set. A bitmap can also represent a set: the bit positions denote enumerated element symbols, and the value indicates whether that element is present. So we rearrange the NYC taxi ride data into a data structure based on graphs and sets, and make large bitmaps?
- fnordsensei 7y ago
  Well, you could decide that one axis is monkeyIndex, and the other is amountOfBananasOwned, and have a quite compact representation of which monkey owns what number of bananas. I.e., decide on a symbolic meaning for the axes rather than converting data wholesale.
  - ytklx 7y ago
    This is very similar to how Pilosa saves integers. https://www.pilosa.com/docs/latest/data-model/#bsi-range-encoding https://www.pilosa.com/docs/latest/data-model/#bsi-range-enc...
    - 7y ago

maycotte 7y ago

Great feedback. We could also do a better job of documenting the most frequent use cases which include high cardinality segmentation, personalization at scale, rapid data exploration/discovery, edge analytics, fast BI, threat detection, bioinformatics and more.