4 ms·

"stat 30k files in 10s of milliseconds" - I'm not sure how is this possible, as GetFileAttributesEx() is roughly 65k/s - e.g. 64-65k ops (my computer, my drive,

by malkia 6y ago

"stat 30k files in 10s of milliseconds" - I'm not sure how is this possible, as GetFileAttributesEx() is roughly 65k/s - e.g. 64-65k ops (my computer, my drive, etc.) - all cached. It's possible that Ninja maybe using something else (directory lookup?)

JNRowe 6y ago
The paragraph mentions that it is talking about Linux. Further down in the article it digs in to the speed problems on Windows, and links to a quick run down of the speed comparisons¹. 1. https://github.com/ninja-build/ninja/wiki/Timing-Numbers https://github.com/ninja-build/ninja/wiki/Timing-Numbers
- malkia 6y ago
  Just saw it, if I'm interpreting the numbers there - it's even worse (as expected on Windows/NTFS), even from cache. Still though I love ninja, and just recently used it for generating .proto files to 4 different languages. Also love bazel (after using 2-3 years blaze), but for the use case above it would've been a real "killer" :)
acqq 6y ago
Have you measured the speed of FindFirstFile, if the goal is only to have many file times, maybe that could be faster for a hot cache, compared to GetFileAttributesEx? (For cold cache, everything is slow anyway.) In specific case of ninja build tool, it can be that they simply don't achieve the same speed on Linux and Windows, as JNRowe writes. It is known that NTFS and Windows are traditionally both visibly slower with many operations on many files, compared to Linux.
- malkia 6y ago
  You can FindFirstFile (not that, but directly reading the directory as a file) really fast (haven't measured that recently), but there is latency
  - acqq 6y ago
    > FindFirstFile (not that, but directly reading the directory as a file) I always believed FindFirstFile actually does read the directory as a file once (or in some convenient chunks), keeps the handle to that and returns an element of that each time FindNextFile is called. Moreover, NTFS has some stat-like file information in the directory file, exactly to be able to return it this way. And as always, then the problem with having the cached information is, what if it is stale. But sometimes, for some purposes, it can be good enough.
- evmar 6y ago
  The first attempt at Windows tried to do some memory-resident file watching thing that I eventually rejected as too complex. We circled a few more times on different approaches and the final Windows approach is pretty complex. Look at the WIN32-related ifdefs in https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc https://github.com/ninja-build/ninja/blob/master/src/disk_in... (I would link to something specifically but it's not that long of a file.)
  - malkia 6y ago
    I wonder how that works, and haven't it run into problems? (E.g. relying on FindFirstFile for info). Maybe it's all right when comes to compiling... just this https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=8813 https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=88...
    - acqq 6y ago
      > Maybe it's all right when comes to compiling I would not expect problems, once there are no bugs in the code. As far as I understand, a build tool doesn't need file sizes, only the timestamps. The code appears to do sensible things, e.g. it notes "FindExInfoBasic is 30% faster than FindExInfoStandard" and uses the former when possible. Looking "from 20000 ft", the only to me unexpected thing I see is handling ".." there, somehow I expected that code which uses that wouldn't even need the timestamp of "..".
- pianoben 6y ago
  As it turns out, this is more or less what ninja does: https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc#L104 https://github.com/ninja-build/ninja/blob/master/src/disk_in...

by malkia 6y ago

JNRowe 6y ago

The paragraph mentions that it is talking about Linux. Further down in the article it digs in to the speed problems on Windows, and links to a quick run down of the speed comparisons¹. 1. https://github.com/ninja-build/ninja/wiki/Timing-Numbers https://github.com/ninja-build/ninja/wiki/Timing-Numbers

malkia 6y ago
Just saw it, if I'm interpreting the numbers there - it's even worse (as expected on Windows/NTFS), even from cache. Still though I love ninja, and just recently used it for generating .proto files to 4 different languages. Also love bazel (after using 2-3 years blaze), but for the use case above it would've been a real "killer" :)

acqq 6y ago

Have you measured the speed of FindFirstFile, if the goal is only to have many file times, maybe that could be faster for a hot cache, compared to GetFileAttributesEx? (For cold cache, everything is slow anyway.) In specific case of ninja build tool, it can be that they simply don't achieve the same speed on Linux and Windows, as JNRowe writes. It is known that NTFS and Windows are traditionally both visibly slower with many operations on many files, compared to Linux.

malkia 6y ago
You can FindFirstFile (not that, but directly reading the directory as a file) really fast (haven't measured that recently), but there is latency
- acqq 6y ago
  > FindFirstFile (not that, but directly reading the directory as a file) I always believed FindFirstFile actually does read the directory as a file once (or in some convenient chunks), keeps the handle to that and returns an element of that each time FindNextFile is called. Moreover, NTFS has some stat-like file information in the directory file, exactly to be able to return it this way. And as always, then the problem with having the cached information is, what if it is stale. But sometimes, for some purposes, it can be good enough.
evmar 6y ago
The first attempt at Windows tried to do some memory-resident file watching thing that I eventually rejected as too complex. We circled a few more times on different approaches and the final Windows approach is pretty complex. Look at the WIN32-related ifdefs in https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc https://github.com/ninja-build/ninja/blob/master/src/disk_in... (I would link to something specifically but it's not that long of a file.)
- malkia 6y ago
  I wonder how that works, and haven't it run into problems? (E.g. relying on FindFirstFile for info). Maybe it's all right when comes to compiling... just this https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=8813 https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=88...
  - acqq 6y ago
    > Maybe it's all right when comes to compiling I would not expect problems, once there are no bugs in the code. As far as I understand, a build tool doesn't need file sizes, only the timestamps. The code appears to do sensible things, e.g. it notes "FindExInfoBasic is 30% faster than FindExInfoStandard" and uses the former when possible. Looking "from 20000 ft", the only to me unexpected thing I see is handling ".." there, somehow I expected that code which uses that wouldn't even need the timestamp of "..".
pianoben 6y ago
As it turns out, this is more or less what ninja does: https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc#L104 https://github.com/ninja-build/ninja/blob/master/src/disk_in...