7 ms·

Have you measured the speed of FindFirstFile, if the goal is only to have many file times, maybe that could be faster for a hot cache, compared to GetFileAttrib

by acqq 6y ago

Have you measured the speed of FindFirstFile, if the goal is only to have many file times, maybe that could be faster for a hot cache, compared to GetFileAttributesEx? (For cold cache, everything is slow anyway.) In specific case of ninja build tool, it can be that they simply don't achieve the same speed on Linux and Windows, as JNRowe writes. It is known that NTFS and Windows are traditionally both visibly slower with many operations on many files, compared to Linux.

malkia 6y ago
You can FindFirstFile (not that, but directly reading the directory as a file) really fast (haven't measured that recently), but there is latency
- acqq 6y ago
  > FindFirstFile (not that, but directly reading the directory as a file) I always believed FindFirstFile actually does read the directory as a file once (or in some convenient chunks), keeps the handle to that and returns an element of that each time FindNextFile is called. Moreover, NTFS has some stat-like file information in the directory file, exactly to be able to return it this way. And as always, then the problem with having the cached information is, what if it is stale. But sometimes, for some purposes, it can be good enough.
evmar 6y ago
The first attempt at Windows tried to do some memory-resident file watching thing that I eventually rejected as too complex. We circled a few more times on different approaches and the final Windows approach is pretty complex. Look at the WIN32-related ifdefs in https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc https://github.com/ninja-build/ninja/blob/master/src/disk_in... (I would link to something specifically but it's not that long of a file.)
- malkia 6y ago
  I wonder how that works, and haven't it run into problems? (E.g. relying on FindFirstFile for info). Maybe it's all right when comes to compiling... just this https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=8813 https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=88...
  - acqq 6y ago
    > Maybe it's all right when comes to compiling I would not expect problems, once there are no bugs in the code. As far as I understand, a build tool doesn't need file sizes, only the timestamps. The code appears to do sensible things, e.g. it notes "FindExInfoBasic is 30% faster than FindExInfoStandard" and uses the former when possible. Looking "from 20000 ft", the only to me unexpected thing I see is handling ".." there, somehow I expected that code which uses that wouldn't even need the timestamp of "..".
pianoben 6y ago
As it turns out, this is more or less what ninja does: https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc#L104 https://github.com/ninja-build/ninja/blob/master/src/disk_in...

by acqq 6y ago

malkia 6y ago

You can FindFirstFile (not that, but directly reading the directory as a file) really fast (haven't measured that recently), but there is latency

acqq 6y ago
> FindFirstFile (not that, but directly reading the directory as a file) I always believed FindFirstFile actually does read the directory as a file once (or in some convenient chunks), keeps the handle to that and returns an element of that each time FindNextFile is called. Moreover, NTFS has some stat-like file information in the directory file, exactly to be able to return it this way. And as always, then the problem with having the cached information is, what if it is stale. But sometimes, for some purposes, it can be good enough.

evmar 6y ago

The first attempt at Windows tried to do some memory-resident file watching thing that I eventually rejected as too complex. We circled a few more times on different approaches and the final Windows approach is pretty complex. Look at the WIN32-related ifdefs in https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc https://github.com/ninja-build/ninja/blob/master/src/disk_in... (I would link to something specifically but it's not that long of a file.)

malkia 6y ago
I wonder how that works, and haven't it run into problems? (E.g. relying on FindFirstFile for info). Maybe it's all right when comes to compiling... just this https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=8813 https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=88...
- acqq 6y ago
  > Maybe it's all right when comes to compiling I would not expect problems, once there are no bugs in the code. As far as I understand, a build tool doesn't need file sizes, only the timestamps. The code appears to do sensible things, e.g. it notes "FindExInfoBasic is 30% faster than FindExInfoStandard" and uses the former when possible. Looking "from 20000 ft", the only to me unexpected thing I see is handling ".." there, somehow I expected that code which uses that wouldn't even need the timestamp of "..".

pianoben 6y ago

As it turns out, this is more or less what ninja does: https://github.com/ninja-build/ninja/blob/master/src/disk_interface.cc#L104 https://github.com/ninja-build/ninja/blob/master/src/disk_in...