4 ms·

Ok so it's an inmemory product, sharded no less. They speak about compressing the data before "storing" it. I don't have a lot of experience with inmemory any

Ok so it's an inmemory product, sharded no less. They speak about compressing the data before "storing" it. I don't have a lot of experience with inmemory anything, but are we talking about retaining the compressed format in server memory here? Ie, RAM is your datastore. Then, at some point, to serve requests/queries for the data don't you have to get it "out of" RAM and uncompress it, also an inmemory operation? Did I get this right?

storyinmemo 10y ago
The paper on the algorithm is here: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf http://www.vldb.org/pvldb/vol8/p1816-teller.pdf Somebody implemented the algorithm in go based on the paper here: https://github.com/dgryski/go-tsz https://github.com/dgryski/go-tsz A short answer that may work for your question: the bits that are set in RAM are xor values relative to previous values. To provide an answer as to what the value is, a series of read|xor operations are performed.
- jdonaldson 10y ago
  They refer to it as "delta of delta", which implies the encoding handles things like acceleration and momentum. I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.
  - Scaevolus 10y ago
    Not quite. Delta of deltas means that periodic timestamps turn into a long run of zeros (since the difference between each point is constant), which is very easy to compress. In the Gorilla paper, 96% of timestamps compress down to zero, which is encoded by a single zero bit. For series without periodic sampling, delta-of-deltas performs similarly to normal deltas.
  - rakoo 10y ago
    "delta of delta" is only used for timestamps, not for values. In any standard monitoring setup, the acceleration of timestamps will be 0 for a big portion of measurements, and very small for a big portion of the rest, so it makes compression extremely easy and efficient.
  - throwaway2016a 10y ago
    > I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series. I thought they were talking about healthcare data too when I first read it but seeing as this is Facebook I realized they are talking about server health not human health.
  - machineman44 10y ago
    Haha, I thought jdonaldson was making a joke :P

storyinmemo 10y ago

The paper on the algorithm is here: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf http://www.vldb.org/pvldb/vol8/p1816-teller.pdf Somebody implemented the algorithm in go based on the paper here: https://github.com/dgryski/go-tsz https://github.com/dgryski/go-tsz A short answer that may work for your question: the bits that are set in RAM are xor values relative to previous values. To provide an answer as to what the value is, a series of read|xor operations are performed.

jdonaldson 10y ago
They refer to it as "delta of delta", which implies the encoding handles things like acceleration and momentum. I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.
- Scaevolus 10y ago
  Not quite. Delta of deltas means that periodic timestamps turn into a long run of zeros (since the difference between each point is constant), which is very easy to compress. In the Gorilla paper, 96% of timestamps compress down to zero, which is encoded by a single zero bit. For series without periodic sampling, delta-of-deltas performs similarly to normal deltas.
- rakoo 10y ago
  "delta of delta" is only used for timestamps, not for values. In any standard monitoring setup, the acceleration of timestamps will be 0 for a big portion of measurements, and very small for a big portion of the rest, so it makes compression extremely easy and efficient.
- throwaway2016a 10y ago
  > I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series. I thought they were talking about healthcare data too when I first read it but seeing as this is Facebook I realized they are talking about server health not human health.
- machineman44 10y ago
  Haha, I thought jdonaldson was making a joke :P