3 ms·
As part of my undergrad research project I used magnetic simulation software (running on windows and therefore NTFS) The program output magnetization by timest
by mikrl 3y ago
As part of my undergrad research project I used magnetic simulation software (running on windows and therefore NTFS)
The program output magnetization by timestep so for the simple structure I was investigating, I ended up with about 100K files of a few KB each which were a massive pain to move to my Linux machine for analysis.
I looked into changing the batching but couldn’t figure it out, and this prompted me for the first time to consider how the structure of data on a device could affect processing throughout even if the absolute quantity of data was the same.
- naikrovek 3y agoall filesystems have made tradeoffs. ntfs made a tradeoff that resulted in file operations having a high overhead per file. you really, really feel this when you're working with 100s of thousands of tiny files in a single directory.