4 ms·
OK, so we're talking about a use case where you're compressing data in main memory and trying to decompress it only within L1 cache. I guess there must be a lot
by kentonv 6y ago
OK, so we're talking about a use case where you're compressing data in main memory and trying to decompress it only within L1 cache. I guess there must be a lot of data sitting around in RAM that isn't accessed very often. Search index leaf nodes I suppose?
It doesn't seem to me like Protobuf is ideal for this use case, but sure, I see how the light compression afforded by Protobuf encoding could lead to a win vs. bare C structs.
I think a better answer here, though, would be to use an actual compression algorithm that has been tuned for this purpose.
Of course, then the uncompressed data needs to be position-independent, so no (native) pointers. You could use something hand-rolled here... but also, this is exactly the problem zero-copy serializations solve, so they might be a good fit here. Hmm!
I'd be pretty interested to compare layering a zero-copy serialization on top of compression vs. protobuf encoding in these high-performance computing scenarios. Is that something you tried doing?
- senderista 6y agoIf this is your thing, then this is your book: https://www.cambridge.org/core/books/compact-data-structures/68A5983E6F1176181291E235D0B7EB44 https://www.cambridge.org/core/books/compact-data-structures...