4 ms·

For Wavefront I got this down to about 1.5 bytes per metric reported (on average across customers) where each metric was a tuple of: name, source, timestamp (s

For Wavefront I got this down to about 1.5 bytes per metric reported (on average across customers) where each metric was a tuple of: name, source, timestamp (s), value (int) and a set of name/value pairs Key to optimizing it was having a database that supported different compression strategies based on the data.

londons_explore 4y ago
That actually sounds pretty bad to me, when many metrics will be constant or very rarely changing, or perfectly correlated with another metric. Were you arithmetic coding stuff, or still doing bit-twiddling tricks?
- spullara 4y ago
  If you think that sounds bad, I would love to see what you think is good and show your work then they can just add it to one of the options. It does basically all the tricks in the article and then some. This is a mean, not a median. The median is likely much lower but no one cares about the median when calculating how much space you need. RE: Correlated with another metric. That would be extremely expensive one to calculate. You need an algorithm that can operate when ingesting millions of points per second 24/7.
lucb1e 4y ago
> Key to optimizing it was having a database that supported different compression strategies based on the data. I keep wondering why automatic detection of a compression strategy is hardly ever used. Different algorithms are just better at different things. I thought timsort was one of the few things that does this, but a quick look at Wikipedia actually indicates that I misremember. With that gone, off the top of my head I cannot think of any utility or library that does this sort of thing. It doesn't seem hard to just take a few kilobytes of data, try a handful of good algorithms, pick the best one, and repeat after a few megabytes. I imagine the gains should be double digit percentages for at least a fifth of the datasets out there, without necessarily losing out on (de)compression speed. Curious if anyone ever looked into this, though some uncertainty will always be about bias in the dataset selection. E.g. 9 out of 10 times xz is clearly better than bzip2 (both at max compression setting), but then sometimes bzip2 actually beats xz with some margin. I don't know when or why, but it has me trying both options whenever I care about the final compressed size. I've also had that a lower level of gzip actually compressed better (I think it was -8 vs -9, but it also have been between -1 and -3 or so).
- spullara 4y ago
  I've always liked this kind of thing. I've also done some experiments with automatically memorizing better starting dictionaries for a corpus eras where the data is small: https://github.com/spullara/corpuscompression https://github.com/spullara/corpuscompression Another interesting case is using a compressed stream to indicate anomalies in the data when the compression ratio spikes down like in log analysis.

londons_explore 4y ago

That actually sounds pretty bad to me, when many metrics will be constant or very rarely changing, or perfectly correlated with another metric. Were you arithmetic coding stuff, or still doing bit-twiddling tricks?

spullara 4y ago
If you think that sounds bad, I would love to see what you think is good and show your work then they can just add it to one of the options. It does basically all the tricks in the article and then some. This is a mean, not a median. The median is likely much lower but no one cares about the median when calculating how much space you need. RE: Correlated with another metric. That would be extremely expensive one to calculate. You need an algorithm that can operate when ingesting millions of points per second 24/7.

lucb1e 4y ago

> Key to optimizing it was having a database that supported different compression strategies based on the data. I keep wondering why automatic detection of a compression strategy is hardly ever used. Different algorithms are just better at different things. I thought timsort was one of the few things that does this, but a quick look at Wikipedia actually indicates that I misremember. With that gone, off the top of my head I cannot think of any utility or library that does this sort of thing. It doesn't seem hard to just take a few kilobytes of data, try a handful of good algorithms, pick the best one, and repeat after a few megabytes. I imagine the gains should be double digit percentages for at least a fifth of the datasets out there, without necessarily losing out on (de)compression speed. Curious if anyone ever looked into this, though some uncertainty will always be about bias in the dataset selection. E.g. 9 out of 10 times xz is clearly better than bzip2 (both at max compression setting), but then sometimes bzip2 actually beats xz with some margin. I don't know when or why, but it has me trying both options whenever I care about the final compressed size. I've also had that a lower level of gzip actually compressed better (I think it was -8 vs -9, but it also have been between -1 and -3 or so).

spullara 4y ago
I've always liked this kind of thing. I've also done some experiments with automatically memorizing better starting dictionaries for a corpus eras where the data is small: https://github.com/spullara/corpuscompression https://github.com/spullara/corpuscompression Another interesting case is using a compressed stream to indicate anomalies in the data when the compression ratio spikes down like in log analysis.