3 ms·
> It is the EMA for each digit, rather than each code. So you have one EMA for all the first digits of all the codes, one for all the second digits, and so on.
by cinquemb 5y ago
> It is the EMA for each digit, rather than each code. So you have one EMA for all the first digits of all the codes, one for all the second digits, and so on.
> You can then make it more intelligent by having even more different EMAs and picking between them in some more intelligent way than just by digit. By digit is just the easiest option.
Ok cool, so if I'm using uint8_t, ill have an ema for each of the 8 bits. And yeah, I'll start with the easiet just so I can nail the implementation then try playing around with different emas if i'm not satisfied with it (right now I can turn 29kb to 5kb with paq8fthis2 compared to 12kb with snappy, so hopefully using something like this will make it go lower).
> However, if the probabilities change throughout the file, the dynamic version again gains the upper hand since it can adapt to changing data. How quickly you let the EMA change also affects this.
Yeah, I suspect this to likely be true. I'll need to probably play around with the ema weights to see how that will change things.
Thanks for helping me understand this all! I started out trying to compress the data by representing as a matrix with fewer # row operations to get back to the original, then ended up having to deal with it traditionally… lol