2 ms·
The problem here is that the code doesn't throw all the requests at once. In the previous version they where using a small number of threads which did provide a
by kevincox 6y ago
The problem here is that the code doesn't throw all the requests at once. In the previous version they where using a small number of threads which did provide a notable performance increase but issuing ~8 requests at a time doesn't allow for that much reordering.
It would be interesting to try a version that used async IO to try and read literally all of the files at the same time and let the kernel send you the data as the disk spins to it.
- pkolaczk 6y agoIt would be interesting to try, but probably that wouldn't work as a general-purpose solution. This would require opening all these files at the same time, risking hitting the open-file number limit. Although the limit could be technically raised, I don't think a user would like to do it only to be able to use the tool. Note we're talking about hundreds thousands or even millions of files here.
- kevincox 6y agoYeah. I would except something to fall over, but it would be interesting to try.
- londons_explore 6y agoBut if the disk is presented with a hundred thousand possible blocks to read next, I suspect it'll find pretty much the perfect option. I doubt a million options will be any better.
- pkolaczk 6y agoBut is the drive's reordering buffer so deep? NCQ in the system reorders only up to 32 requests. I doubt the drive's firmware is capable of reordering millions of requests. These things have typically small amount of memory, just enough for typical use-cases. Keeping a small window for sorting a subset of requests is much weaker than sorting all of them up front. Even if the window is 50% of all files, that would mean the average distance between requests increases 2 times (assuming blocks are distributed uniformly).