4 ms·

Very interesting. I just benchmarked this (the python2.7 module only) with an internal application that walks over filesystems and found scandir.walk() to, inex

Very interesting. I just benchmarked this (the python2.7 module only) with an internal application that walks over filesystems and found scandir.walk() to, inexplicably, be slightly slower than the os.walk(). I think part of the issue (though I've not tested this yet) is that we're stat()-ing every single file anyway so with os cache considered, it really ends up not mattering anyway. I thought the additional cost of the extra system calls (even if they were entirely cached in memory) would add up, but, it seems like something the scandir module is doing is just less efficient in general. Devising some much simpler and more controllable tests (but still with our exact workload) and testing more though.

benhoyt 10y ago
That's very intriguing to me -- I've rarely/never seen it be slower, and usually at least several times faster. Are you using benchmark.py? If so, can you send a link to a Gist with the output? It may be that the C module is not compiling, and it's falling back to the much slower pure Python module.
- AdamJacobMuller 10y ago
  I wasn't running your specific benchmark test. I've been running some more controlled tests now that show it basically being on-par with listdir() which I still consider completely oddball. Also, not remotely hating on what you did here. I actually wrote/tested a very similar concept in python several years ago for the exact same reason. My C-skills (and mostly time resources) weren't really up to par to try to get it included in core. I mostly really want this to work as well as I think it should :) I'll run some tests with your benchmark.py (and compare that to my benchmark script) and post some results.
- AdamJacobMuller 10y ago
  https://gist.github.com/AdamJacobMuller/8ca79af8c0102285ff076247cdc6d8fe https://gist.github.com/AdamJacobMuller/8ca79af8c0102285ff07... Results are very similar/identical to what my tool said. Interestingly, If I look at strace call graphs the scandir version spends basically an identical amount of time in getdents as the os version, but, the os version spends a bunch of time in stat calls too. It just seems like on my workload, that time is considerably less than I expected.

benhoyt 10y ago

That's very intriguing to me -- I've rarely/never seen it be slower, and usually at least several times faster. Are you using benchmark.py? If so, can you send a link to a Gist with the output? It may be that the C module is not compiling, and it's falling back to the much slower pure Python module.

AdamJacobMuller 10y ago
I wasn't running your specific benchmark test. I've been running some more controlled tests now that show it basically being on-par with listdir() which I still consider completely oddball. Also, not remotely hating on what you did here. I actually wrote/tested a very similar concept in python several years ago for the exact same reason. My C-skills (and mostly time resources) weren't really up to par to try to get it included in core. I mostly really want this to work as well as I think it should :) I'll run some tests with your benchmark.py (and compare that to my benchmark script) and post some results.
AdamJacobMuller 10y ago
https://gist.github.com/AdamJacobMuller/8ca79af8c0102285ff076247cdc6d8fe https://gist.github.com/AdamJacobMuller/8ca79af8c0102285ff07... Results are very similar/identical to what my tool said. Interestingly, If I look at strace call graphs the scandir version spends basically an identical amount of time in getdents as the os version, but, the os version spends a bunch of time in stat calls too. It just seems like on my workload, that time is considerably less than I expected.