4 ms·

The syscall to list a directory is synchronous, but the follow-up stat calls to read file attributes (type, permission, attributes, size) are done one file at a

The syscall to list a directory is synchronous, but the follow-up stat calls to read file attributes (type, permission, attributes, size) are done one file at a time, so there are opportunities for parallelization. Incidentally, this is why ls without colors or type symbols (-l, -F, --color, and so forth) is a lot faster: it doesn't have to query every file to find out what it is in order to display the color/symbol/permissions/etc. Handy if you're on a slow networked file system and just need to see names.

edwintorok 5y ago
It would be better if it used fstatat instead of regular stat calls. On a deeply nested directory it can make a difference (e.g. a single-threaded 'find' easily beats a parallel 'fd' with warm caches on my machine, 'exa' could similarly benefit)
userbinator 5y ago
I don't see how parallelism could work with stat() in general, since the OS has to keep the FS structures consistent and ultimately there is only one disk to read from, unless you have RAID.
- vidarh 5y ago
  Depends how much of the overhead is context switches vs. the disk IO. Really something that needs benchmarking.
- mhio 5y ago
  Filling the disk queue usually helps with the IO throughout part, sequential stats not attempting to queue requests will never saturate the bandwidth of a modern device. There's also the page cache involved for reads, so it can be read while actual IO is queued.
R0b0t1 5y ago
> but the follow-up stat calls to read file attributes (type, permission, attributes, size) are done one file at a time, so there are opportunities for parallelization. You may expect so but this is usually not the case. Multiple threads submitting stat or other fs ioctl commands to the disk driver are going to be sequenced and may not be threaded. This is definitely how it works on Windows. Linux is better, but a lot of disk IO is behind the times. io_uring only has read/write so far.

edwintorok 5y ago

It would be better if it used fstatat instead of regular stat calls. On a deeply nested directory it can make a difference (e.g. a single-threaded 'find' easily beats a parallel 'fd' with warm caches on my machine, 'exa' could similarly benefit)

userbinator 5y ago

I don't see how parallelism could work with stat() in general, since the OS has to keep the FS structures consistent and ultimately there is only one disk to read from, unless you have RAID.

vidarh 5y ago
Depends how much of the overhead is context switches vs. the disk IO. Really something that needs benchmarking.
mhio 5y ago
Filling the disk queue usually helps with the IO throughout part, sequential stats not attempting to queue requests will never saturate the bandwidth of a modern device. There's also the page cache involved for reads, so it can be read while actual IO is queued.

R0b0t1 5y ago

> but the follow-up stat calls to read file attributes (type, permission, attributes, size) are done one file at a time, so there are opportunities for parallelization. You may expect so but this is usually not the case. Multiple threads submitting stat or other fs ioctl commands to the disk driver are going to be sequenced and may not be threaded. This is definitely how it works on Windows. Linux is better, but a lot of disk IO is behind the times. io_uring only has read/write so far.