4 ms·
what makes me jealous,is not the double performance ,but the fact that he can handle 13000 files in 2 seconds. i am at my ${corporate} job now, and have about 4
by dgan 2y ago
what makes me jealous,is not the double performance ,but the fact that he can handle 13000 files in 2 seconds. i am at my ${corporate} job now, and have about 40'000 json files stored locally on the windows laptop. when I need to load a them, it takes minutes. i am still not sure if i read files wrong or NTFS is a pile of trash when comes to reading many smallish files
- Hackbraten 2y agoThere might me some kind of malware scanner at play. Maybe it's time to rebuild those with mimalloc, too.
- nurettin 2y agoDoes ${corporate} prevent you from merging your jsons to a single file with a decent format like parquet?
- dgan 2y agothey don't but that defeats the purpose. i want to be able to inspect the insides, so binary formats are out, i have to easily open & read the content
- nurettin 2y agoI don't get how you can't put the content within the merged file and query it but good luck.
- dgan 2y agobecause that d be a several hundreds megabytes json :) Anyway with suggestion in another comment, i moved the root directory into an antivirus-excluded location, which gave me almost a 10 times performance win
- HdS84 2y agoTry excluding that directory from the virus scanner. Also a dev drive might come in handy for this workload
- dgan 2y agoyou were right. it didn't occur to me that antivirus would scan everything I open. Moved all files into an excluded directory, reading files is now almost 10 times faster. Thanks kind stranger !
- SkiFire13 2y agoNTFS is definitely very slow when it comes to reading many small files. Windows Defender or whatever antivirus you might be using can also further slow this down.
- v1ne 2y agoNTFS is slow, especially when you operate on a lot of tiny files (nobody in the Windows world would do that, you'd always put your tiny data blobs into a bigger container file, e.g. asset files in games), but from my corporate experience, it's mostly the _multiple_ "endpoint security" solutions that bog file system performance down. It's the reason I so far use a Mac at work, which has its own issues, and a lot of them.
- scoopr 2y agoCould be perhaps worth the trouble to try dev-drive/ReFS? [0] [0] https://learn.microsoft.com/en-us/windows/dev-drive/ https://learn.microsoft.com/en-us/windows/dev-drive/