6 ms·
> For this case that might mean convert the data into deltas, then write those deltas in a byte-oriented variable length integer fashion. Calculating a trend l
by jhgb 5y ago
> For this case that might mean convert the data into deltas, then write those deltas in a byte-oriented variable length integer fashion.
Calculating a trend line and storing deltas off of that trend line might be mildly better for some series. Having an AR model and storing errors of the model might work even better, although the obvious (and common) problem is which AR model of the many possible ones you go for.
- ballenf 5y agoOr simply a delta off the last delta (or mean of last n deltas, I guess), so you can do the compression in a stream.
- jhgb 5y agoThat might also be equivalent to one of the ARMA models, although I'd have to do the math to prove that (and to derive which one corresponds to what you're describing). Edit: It seems that if you detrend the data, the "delta off the last delta", if I understand it correctly, is an MA(1) model. I think. Maybe. My math is wobbly.
- wallacoloo 5y agoIf I understand you & GP, these are both just particular instantiations of Linear Predictive Coding [1]. For both of these approaches, you’d probably get similar or better results just throwing your data into an off-the-shelf LPC, and with less effort. [1] https://en.m.wikipedia.org/wiki/Linear_predictive_coding https://en.m.wikipedia.org/wiki/Linear_predictive_coding
- NohatCoder 5y agoIf you know the specifics of your data, maybe. But try to err on the side of simpler models, every detail you add slows your code down, and the difference after compression is often small.