3 ms·
In a "hold my beer"-moment, I've ported GNU Trove's hash map to use memory mapped files instead of allocated memory. With good reason, granted, and it's not com
by marginalia_nu 5y ago
In a "hold my beer"-moment, I've ported GNU Trove's hash map to use memory mapped files instead of allocated memory. With good reason, granted, and it's not commonly used, but compared to almost anything else, it is fairly slow.
- spicybright 5y agoThat sounds pretty interesting (in a sick way ;) ) Are you able to elaborate a bit more what you were doing?
- marginalia_nu 5y agoI'm using it for the lexicon in my search engine. I basically need a mapping from arbitrary short strings to unique integer identifiers, with a cardinality in the billions. Too much to keep in memory, most db solutions keel over as well.
- rurban 5y agofor this I would use CHM, ordered minimal perfect hashes with external mmap'ed keys. still constant lookup, just the position needs to be stored in the value also.
- marginalia_nu 5y agoProblem is I don't know the items beforehand, instead I need to build the hash as I encounter new pages. I don't know any way of reconcile that with a minimal perfect hash.
- rurban 5y agook, with trillions keys creating a CHM MPH is risky, and needs a few secs. then your external hash is good enough.