3 ms·
>so maybe it'll cost lower to build a search engine index compared to an internet preservation project? An index is just hashmap of words and list of urls. So
by DeathArrow 3y ago
>so maybe it'll cost lower to build a search engine index compared to an internet preservation project?
An index is just hashmap of words and list of urls. So you have to parse the page, and add the urls and word frequencies to the list.
In terms of storage is cheaper, in terms of computing power is more expensive.
- marginalia_nu 3y agoA hash table is not a good backing structure for a search engine. Hash tables almost guarantee worst case disk read patterns. You use something like a skip list or a b-tree, since that's makes much better use of the hardware, and on top of that allows you to do incredibly fast joins.
- DeathArrow 3y agoThat's true, but I was referring to a hash table as a mere mental model, not an actual implementation. A better name would be a dictionary instead of hash table.
- fomine3 3y agoPerl culture here.