3 ms·

Ahh! Great catch, and thanks for taking the time to put that in writing. I did set gg to default to 4 threads, which seemed to be the optimal number on my mach

Ahh! Great catch, and thanks for taking the time to put that in writing. I did set gg to default to 4 threads, which seemed to be the optimal number on my machine for the typical repo sizes I navigate daily. Increasing the number of threads beyond that often results in unnecessary overhead for my personal use cases. I appreciate you pointing out the heuristic used in the ripgrep project. From what I understand, it also uses a fixed, machine-dependent number of threads, predetermined regardless of the task at hand (except for single-file tasks). This is something I was curious about while writing the code but couldn't fully answer due to my limited knowledge of the subject: could we potentially use a filesystem-specific heuristic to estimate the workload and dynamically adjust the number of threads accordingly? What I mean is a method, perhaps within the ignore crate, to estimate the amount of data to process—such as the number of files, file sizes, or number of lines—based on easily and cheaply accessible filesystem metadata.

burntsushi 2y ago
I'm not aware of one. Any tool that tells you disk space has to actually crawl the directory tree to report it. But that is precisely the thing we want to parallelize. The only other option I can think of is to dynamically adjust. Maybe after a certain amount of work has completed, spin up more threads. But I'm not sure it's worth doing.
- alexpasmantier 2y ago
  Looking at inode metadata—specifically the number of links for directory nodes—might iteratively provide a one-step-ahead view of what's left to crawl, allowing for preemptive thread adjustments during recursion. e.g. looking at the Links: 101 metadata on the `curl` codebase for src: $ stat -x src File: "src" Size: 3232 FileType: Directory Mode: (0755/drwxr-xr-x) Uid: ( 501/ alex) Gid: ( 20/ staff) Device: 1,22 Inode: 5857579 Links: 101 Access: Tue Aug 27 22:21:23 2024 Modify: Tue Aug 27 22:21:19 2024 Change: Tue Aug 27 22:21:19 2024 Birth: Tue Aug 27 22:21:19 2024 But then that still involves dynamically adjusting and might be kind of overkill for a relatively uncertain benefit...

burntsushi 2y ago

I'm not aware of one. Any tool that tells you disk space has to actually crawl the directory tree to report it. But that is precisely the thing we want to parallelize. The only other option I can think of is to dynamically adjust. Maybe after a certain amount of work has completed, spin up more threads. But I'm not sure it's worth doing.

alexpasmantier 2y ago
Looking at inode metadata—specifically the number of links for directory nodes—might iteratively provide a one-step-ahead view of what's left to crawl, allowing for preemptive thread adjustments during recursion. e.g. looking at the Links: 101 metadata on the `curl` codebase for src: $ stat -x src File: "src" Size: 3232 FileType: Directory Mode: (0755/drwxr-xr-x) Uid: ( 501/ alex) Gid: ( 20/ staff) Device: 1,22 Inode: 5857579 Links: 101 Access: Tue Aug 27 22:21:23 2024 Modify: Tue Aug 27 22:21:19 2024 Change: Tue Aug 27 22:21:19 2024 Birth: Tue Aug 27 22:21:19 2024 But then that still involves dynamically adjusting and might be kind of overkill for a relatively uncertain benefit...