4 ms·

does it scale now? previously i was not able to import large folders (several TB) or many files (+100k files)

• mafintosh 6y ago
  Yep! That is one of the things we’ve worked the hardest on. Completely new indexing structure, using an append-only hash trie which scales really well. We’ve tested it with many big datasets including importing all of Wikipedia as files in a single folder. Worked like a charm :)
  • 0x006A 6y ago
    this one? https://dumps.wikimedia.org/other/static_html_dumps/current/en/ https://dumps.wikimedia.org/other/static_html_dumps/current/... how long does it take to import it?
    • mafintosh 6y ago
      I think it was that one yes. Can’t remember the exact time it took, as we ran it over a couple of days due to some unrelated computer issues.