3 ms·
We are using a constant database for a similar use-case (MNP database), albeit much smaller, around 15 million phone numbers. It requires almost no RAM and you
by fasteo 11y ago
We are using a constant database for a similar use-case (MNP database), albeit much smaller, around 15 million phone numbers. It requires almost no RAM and you update the database as a whole with no downtime.
[1] https://en.wikipedia.org/wiki/Cdb_(software) https://en.wikipedia.org/wiki/Cdb_(software)
- vidarh 11y agoI'm sure that works great for 15m. (EDIT: To be clear, I love CDB, and I'm not being sarcastic - for 15m it's probably perfectly fine; it's just space inefficient for large datasets where the combined key+value lengths are very low) But even if we ignore the hash table pointer tables which can be made arbitrarily small at the cost of probing a higher number of entries on average before finding the right key (but in reality you'd want them to be fairly large), the minimum space used per entry is [1]: 8 bytes for the length of the key and length of the value + 5 bytes for the key + 2 bytes for the value, so 15 bytes per number, or 6GB (EDIT: fixed numbers to account for BCD encoding of number). Which means you'd need to switch to one of the (non-standard) 64-bit CDB-inspired formats, as CDB itself can only handle 4GB data files. [1] CDB format: http://cr.yp.to/cdb/cdb.txt http://cr.yp.to/cdb/cdb.txt
- waterside81 11y agoFor the gophers in the crowd, a CDB implementation in pure Go. Memory mapped, thread safe. https://github.com/repustate/go-cdb https://github.com/repustate/go-cdb