3 ms·
I see, thank you. Another thing that I'm sure you explored, and I'd love to hear how it went, would be to rearrange the elements in the vectors such that perha
by dotancohen 1y ago
I see, thank you.
Another thing that I'm sure you explored, and I'd love to hear how it went, would be to rearrange the elements in the vectors such that perhaps the denser parts could be more contiguous, and the sparser parts could be more contiguous, on average. That sounds like something that would be easier to compress. Were the distributions such that a rearrangement like this might have been possible? Or were they very evenly distributed?
I.e. could you have rearranged a Gaussian-like distribution into a Poisson-like distribution?
- Scene_Cast2 1y agoWhat ended up launching is a fancy product quantization based on k-means. Some of the tricks were storing magnitude separately (i.e. removing the mean) and rearranging dimensions based on variance (and/or rotation based on PCA) for PQ to work better. I also remember trying to fit a distribution so that I can generate synthetic data (not for a lack of data, but more for understanding the problem space better). The synthetic data quantized pretty differently - my guess is that it's because of random areas of density and sparsity. I'm not quite following your exact rearranging idea though. Not sure if the above answers the question.
- dotancohen 1y agoI was just wondering, it seems like an interesting problem to explore.