3 ms·
Yep! That is one of the things we’ve worked the hardest on. Completely new indexing structure, using an append-only hash trie which scales really well. We’ve te
by mafintosh 6y ago
Yep! That is one of the things we’ve worked the hardest on. Completely new indexing structure, using an append-only hash trie which scales really well. We’ve tested it with many big datasets including importing all of Wikipedia as files in a single folder. Worked like a charm :)
- 0x006A 6y agothis one? https://dumps.wikimedia.org/other/static_html_dumps/current/en/ https://dumps.wikimedia.org/other/static_html_dumps/current/... how long does it take to import it?
- mafintosh 6y agoI think it was that one yes. Can’t remember the exact time it took, as we ran it over a couple of days due to some unrelated computer issues.