3 ms·

Windows is (or at least was) a dog when it comes to this even if the files are in the cache. Like barrkel said, the number I quoted was for the case when the OS

Windows is (or at least was) a dog when it comes to this even if the files are in the cache. Like barrkel said, the number I quoted was for the case when the OS doesn't hit the drive.

to3m 12y ago
Random rant time! You have to code for Windows just so. Dear POSIX people in general, there is more to cross platform coding than being able to build it with cygwin. Every grep I've used on Windows handles wildcards, so it must be doing the expansion itself. The right way to do this sort of thing on Windows is to call FindFirstFile/FindNextFile, which is a cross between readdir and stat in that each call returns not only the name of one directory entry that matches the pattern (as per readdir+fnmatch) but also its metadata (as per stat). So, if you were going to call readdir (to get the names) and then stat each name (to get the metadata), you should really be doing all of it in just this one loop so that each bit of data is retrieved just the once. But this is exactly what POSIX programmers would seemingly never, ever do. Probably because doing it that way would involve calling a Win32 function. Well, more fool them, because getting this data from FindFirstFile/FindNextFile seems pretty quick, and getting it any other way seems comparatively slow. I cobbled together a native Win32 port of the_silver_searcher a while ago and it was about twice as fast as the stock EXE thanks to retrieving all the file data in one pass. In this case the readdir wrapper just needed to fill in the file type in the dirent struct; luckily it seems that some POSIX-style systems do this, so there was already code to handle this and it just needed the right #define adding. (I have absolutely no idea why MingW32 doesn't do this already.) Prior to switching to the_silver_searcher, I used to use grep; the grep I usually used is the GNU-Win32 one (http://gnuwin32.sourceforge.net/packages/grep.htm http://gnuwin32.sourceforge.net/packages/grep.htm), and it looks to call readdir to get the names, and then stat to get each name's details. I checked that just now, and after all those years I can finally imagine why it's so slow. GNU-Win32 find is really slow, too.
- lucian1900 12y ago
  Perhaps it'd be less bad if Windows wasn't so gratuitously different. On a more serious note, there's a pretty large gulf between the Windows and Unix development worlds. Try getting a game developer to use curl ...
- herf 12y ago
  Yes don't call GetFileAttributes (which your POSIX layer does to emulate grep) - especially over a network (samba) connection. But the original point is OPEN is expensive, which it really is, especially over a network connection.
- shantanugoel 12y ago
  Aside to the discussion, can you point out a link to the changes you did? I'd like to get silver searcher faster as well on windows.
  - to3m 12y ago
    Sure. Here's a binary: https://github.com/tom-seddon/the_silver_searcher/tree/_vs2012/bin https://github.com/tom-seddon/the_silver_searcher/tree/_vs20... The source code is there too, but I wouldn't look too closely. "Cobbled together", like I said. Though I've been using it all the time, in conjunction with ag-mode - https://github.com/Wilfred/ag.el https://github.com/Wilfred/ag.el - and haven't noticed any obvious problems.

Windows is (or at least was) a dog when it comes to this even if the files are in the cache. Like barrkel said, the number I quoted was for the case when the OS doesn't hit the drive.

to3m 12y ago

Random rant time! You have to code for Windows just so. Dear POSIX people in general, there is more to cross platform coding than being able to build it with cygwin. Every grep I've used on Windows handles wildcards, so it must be doing the expansion itself. The right way to do this sort of thing on Windows is to call FindFirstFile/FindNextFile, which is a cross between readdir and stat in that each call returns not only the name of one directory entry that matches the pattern (as per readdir+fnmatch) but also its metadata (as per stat). So, if you were going to call readdir (to get the names) and then stat each name (to get the metadata), you should really be doing all of it in just this one loop so that each bit of data is retrieved just the once. But this is exactly what POSIX programmers would seemingly never, ever do. Probably because doing it that way would involve calling a Win32 function. Well, more fool them, because getting this data from FindFirstFile/FindNextFile seems pretty quick, and getting it any other way seems comparatively slow. I cobbled together a native Win32 port of the_silver_searcher a while ago and it was about twice as fast as the stock EXE thanks to retrieving all the file data in one pass. In this case the readdir wrapper just needed to fill in the file type in the dirent struct; luckily it seems that some POSIX-style systems do this, so there was already code to handle this and it just needed the right #define adding. (I have absolutely no idea why MingW32 doesn't do this already.) Prior to switching to the_silver_searcher, I used to use grep; the grep I usually used is the GNU-Win32 one (http://gnuwin32.sourceforge.net/packages/grep.htm http://gnuwin32.sourceforge.net/packages/grep.htm), and it looks to call readdir to get the names, and then stat to get each name's details. I checked that just now, and after all those years I can finally imagine why it's so slow. GNU-Win32 find is really slow, too.

lucian1900 12y ago
Perhaps it'd be less bad if Windows wasn't so gratuitously different. On a more serious note, there's a pretty large gulf between the Windows and Unix development worlds. Try getting a game developer to use curl ...
herf 12y ago
Yes don't call GetFileAttributes (which your POSIX layer does to emulate grep) - especially over a network (samba) connection. But the original point is OPEN is expensive, which it really is, especially over a network connection.
shantanugoel 12y ago
Aside to the discussion, can you point out a link to the changes you did? I'd like to get silver searcher faster as well on windows.
- to3m 12y ago
  Sure. Here's a binary: https://github.com/tom-seddon/the_silver_searcher/tree/_vs2012/bin https://github.com/tom-seddon/the_silver_searcher/tree/_vs20... The source code is there too, but I wouldn't look too closely. "Cobbled together", like I said. Though I've been using it all the time, in conjunction with ag-mode - https://github.com/Wilfred/ag.el https://github.com/Wilfred/ag.el - and haven't noticed any obvious problems.