3 ms·
I'm using it for the lexicon in my search engine. I basically need a mapping from arbitrary short strings to unique integer identifiers, with a cardinality in
by marginalia_nu 5y ago
I'm using it for the lexicon in my search engine.
I basically need a mapping from arbitrary short strings to unique integer identifiers, with a cardinality in the billions. Too much to keep in memory, most db solutions keel over as well.
- rurban 5y agofor this I would use CHM, ordered minimal perfect hashes with external mmap'ed keys. still constant lookup, just the position needs to be stored in the value also.
- marginalia_nu 5y agoProblem is I don't know the items beforehand, instead I need to build the hash as I encounter new pages. I don't know any way of reconcile that with a minimal perfect hash.
- rurban 5y agook, with trillions keys creating a CHM MPH is risky, and needs a few secs. then your external hash is good enough.