4 ms·
Thanks for the additional information but sometimes I wonder that 15k+ files count as 'many' nowadays. I mean, an off-the-shelf Canon DSLR produces 20k files pe
by arendtio 8y ago
Thanks for the additional information but sometimes I wonder that 15k+ files count as 'many' nowadays. I mean, an off-the-shelf Canon DSLR produces 20k files per directory (10k JPG + 10k raw). Nevertheless, I did some test your approach on my PC:
On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance. Without modification, it takes 0.5 seconds to list a directory (on a slow consumer grade HDD) with 14448 files and after setting
export LS_COLORS='ex=00:su=00:sg=00:ca=00:'
it takes 4.4 seconds.
However, listing a local (SSD) directory with 15k files takes just about 0.12 seconds and gets faster after settings the variable to 0.06 seconds.
For all tests I drop the caches before testing, e.g.:
echo 3 > /proc/sys/vm/drop_caches; time ls --color=always /mnt/nfs/many | wc -l
- mrb 8y ago"On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance" Run "strace -o logfile ls --color=always" and diff the two logfiles.
- arendtio 8y agoWell, this looks rather strange. As expected, settings the variable causes large changes as it removes a lot of lstat calls and also removes the color codes from the writes. So there seems to be no obvious explanation, why it becomes slower when doing less... # du -sh logfile_nfs* 1.7M logfile_nfs 668K logfile_nfs_exported However, the strace -c output looks kinda different: Without modifying the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEbtark/TMSDm9gT+jT0OK5ME8fE= https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb... With setting the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEbtark/TMSDm9gT+jT0OK5ME8fE= https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb...
- mrb 8y agoYou pasted the same link for both cases. Can't tell what's going on.
- arendtio 8y agoOh sorry, my fault. The second link should be https://bin.disroot.org/?56f3dcb618240df0#x3StRGZZgSzM0UJ0BK5V5MajQ8VradUoBlaZS0Da7i0= https://bin.disroot.org/?56f3dcb618240df0#x3StRGZZgSzM0UJ0BK...
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- gnufx 8y agoPlus/minus what? Note that caching on the server can be relevant, and you can't control that from the client. That's one thing that requires care for performance measurements on networked filesystems.
- arendtio 8y agoSounds as if you don't trust my statement. Sure there is some variance, but when one is measured in seconds and the other in the tenth of a second it should be clear that it is not a coincidence... But for completeness, I wrote a few loops which should answer your question: =NFS= Normal: https://bin.disroot.org/?8967a0f34b26e512#aTYmbyESfeuqAXS802NHmg4vTvDnsmRixODRGDfoOvg= https://bin.disroot.org/?8967a0f34b26e512#aTYmbyESfeuqAXS802... With exported Variable: https://bin.disroot.org/?c405d7aa74b50ee0#KverJYJEzNhct7CmdWXAELJV/bVJmQP70M6b0bKRu9k= https://bin.disroot.org/?c405d7aa74b50ee0#KverJYJEzNhct7CmdW... ----- =Local= Normal: https://bin.disroot.org/?b94e1d1f58e3edb7#tKszD/tjvwBwepJLunZwpYEmN6xNRPsl94zTc3XC4NU= https://bin.disroot.org/?b94e1d1f58e3edb7#tKszD/tjvwBwepJLun... With exported Variable: https://bin.disroot.org/?d3ac83f1dff9e767#WVODjGnvL1QbOJdWPpTaNGX23TIyVCyktx9iK87X2Wg= https://bin.disroot.org/?d3ac83f1dff9e767#WVODjGnvL1QbOJdWPp... I also tried dropping the cache on the NFS server but that didn't seem to have a major effect on the performance (probably because reading a 15k file index from a local disk doesn't take that long after all).