3 ms·
What on Earth required this? Is this part of some home grown compression scheme?
by hiptobecubic 5y ago
What on Earth required this? Is this part of some home grown compression scheme?
- marginalia_nu 5y agoIt is indeed, it's part of the compression scheme for the keyword lexicon for my search engine. It's a domain-specific compression that uses the fact that the keyword dictionary is a dictionary of keywords to compress N-grams into dense blobs using this function. Overall it achieves about 70% compression. The lexicon supports several billion entries entirely in memory, which requires a lot these eldritch coding tricks (since Java certainly wasn't built for that many objects, GC and header overhead extremely prohibitive so objects are entirely out of the question). Most of the data is kept in contiguous memory mapped byte arrays.