3 ms·
I just tried splitting the Anna's Archive 250GB WorldCat JSON file out into separate files so I could random-access it. My app crashed at 7m files due to some c
by qingcharles 3y ago
I just tried splitting the Anna's Archive 250GB WorldCat JSON file out into separate files so I could random-access it. My app crashed at 7m files due to some corruption in the source file.
I can tell you, it took a real, real long time to delete seven million files on NTFS. It was not happy.
- Zardoz84 3y agoNTFS and Windows filesystem IO does poorly with many small files. it's one of the reasons that git for windows (and Linux subsystem version 1) is inevitably slow.
- Keyframe 3y ago> NTFS was angry that day, my friends! When it's not angry?
- pilimi_anna 3y agoSince the file is sorted you can binary-search it: https://annas-software.org/AnnaArchivist/annas-archive/-/blob/e2057592df482cf0d8d10159e3e068a76dca02aa/allthethings/utils.py#L1463 https://annas-software.org/AnnaArchivist/annas-archive/-/blo...
- qingcharles 3y agoLooks like sorted by ID, though, sadly? :(