3 ms·
In-memory compression is another technique not to be forgotten. If you can efficiently compress your in-memory representations of data you would otherwise have
by barrkel 17y ago
In-memory compression is another technique not to be forgotten. If you can efficiently compress your in-memory representations of data you would otherwise have to hit disk or network for, you can dramatically increase the amount of data you can cache in memory, especially if you're aware of the structure and general patterns of the data.
Once I had to cache file paths, millions of them. I got a fairly easy 6x improvement by storing them in a Patricia trie, with each node's string stored as a range of UTF-8 encoded bytes, and indexed by arrays of bitfields - i.e. bit-packed arrays. Child nodes were similarly indexed using bitfield arrays. Turning what would be a 32-bit (or worse, 64-bit) pointer into (e.g.) a 28-bit integer really starts to add up when you have millions of them.