4 ms·

Having a billion files in a single directory versus a more typical tree has its own challenges. http://be-n.com/spw/you-can-list-a-million-files-in-a-directory

• jeffbee 3y ago
  Possibly, but at the scale discussed in the article, not so much. % time ls | wc 10000000 10000000 78888897 ls 1.90s user 0.73s system 99% cpu 2.637 total wc 0.28s user 0.02s system 11% cpu 2.597 total ETA: and most of that time is spent sorting the output. % time ls -U | wc 10000000 10000000 78888897 ls -U 0.53s user 0.29s system 99% cpu 0.823 total wc 0.26s user 0.02s system 33% cpu 0.822 total
  • sllabres 3y ago
    To my experience (and if only the filename is needed) the "-1" option helps saving some time too. Be aware that output from ls without options differ, if redirected. time (real user sys) in seconds with 1e6 files in a xfs filesystem. Directory used 27 MB space. ls -l > /dev/null : 49.12 23.90 13.47 ls -lU > /dev/null : 46.62 33.11 8.01 ls > /dev/null : 6.36 1.97 3.63 ls -1 > /dev/null : 2.49 1.83 0.36 ls -1U > /dev/null : 0.31 0.18 0.08 (find) : 0.77 0.49 0.19 and for 1e6 files in an ext4 filesystem. Directory used 23 MB space. (but as i had no ext4 fs with one million free inodes on a loop device) ls -l > /dev/null : 51.39 34.02 11.39 ls -lU > /dev/null : 47.39 30.72 11.09 ls > /dev/null : 4.83 3.56 0.65 ls -1 > /dev/null : 4.46 3.45 0.50 ls -1U > /dev/null : 0.51 0.18 0.26 (find) : 1.29 0.76 0.36
    • jeffbee 3y ago
      Good info. But isn't `-1` implicit when the output is a pipe?
      • sllabres 3y ago
        Good reply, you are right. I have even written ("Be aware that output from ls without options differ, if redirected") that it is so but was somewhat fooled by my experiment and the different timings for "ls" and "ls -1" when using XFS. Of course I was not able to reproduce this difference and assume that I did work on the system and it was not idle. I have repeated the the tests, on another mostly idle system. (Therefor the timing is not comparable to the first test). This time both filesystems live in a loop device and the benchmarks were run with hyperfine (3 warmup loops, 3 measurements each). XFS ls -l >/dev/null Time (mean ± σ): 5.890 s ± 0.394 s [User: 1.096 s, System: 3.491 s] ls -lU >/dev/null Time (mean ± σ): 2.254 s ± 0.386 s [User: 0.648 s, System: 1.602 s] ls >/dev/null Time (mean ± σ): 542.7 ms ± 4.4 ms [User: 443.8 ms, System: 98.7 ms] ls -1 >/dev/null Time (mean ± σ): 534.4 ms ± 2.3 ms [User: 450.1 ms, System: 84.1 ms] ls -1U >/dev/null Time (mean ± σ): 115.8 ms ± 1.8 ms [User: 80.9 ms, System: 34.6 ms] ls -U >/dev/null Time (mean ± σ): 121.0 ms ± 14.6 ms [User: 80.9 ms, System: 40.0 ms] find >/dev/null Time (mean ± σ): 225.0 ms ± 16.2 ms [User: 150.1 ms, System: 74.7 ms] EXT4 ls -l >/dev/null Time (mean ± σ): 10.493 s ± 0.392 s [User: 2.495 s, System: 5.032 s] ls -lU >/dev/null Time (mean ± σ): 8.395 s ± 0.157 s [User: 0.735 s, System: 5.049 s] ls >/dev/null Time (mean ± σ): 1.966 s ± 0.010 s [User: 1.760 s, System: 0.202 s] ls -1 >/dev/null Time (mean ± σ): 2.014 s ± 0.032 s [User: 1.798 s, System: 0.216 s] ls -1U >/dev/null Time (mean ± σ): 221.4 ms ± 1.3 ms [User: 83.7 ms, System: 137.6 ms] ls -U >/dev/null Time (mean ± σ): 222.5 ms ± 2.0 ms [User: 78.9 ms, System: 143.6 ms] find >/dev/null Time (mean ± σ): 531.0 ms ± 4.0 ms [User: 377.3 ms, System: 153.7 ms]
• JackSlateur 3y ago
  That is easily implemented with a double loop, where the first one is a directory (integer hexed with printf '%x') and the second is the files in it However, that was not the point of my reply