You might be interested in how this problem was solved by our team at Meta, in EdenFS (https://github.com/facebook/sapling/blob/main/eden/fs/docs/Overview.md ht

You might be interested in how this problem was solved by our team at Meta, in EdenFS (https://github.com/facebook/sapling/blob/main/eden/fs/docs/Overview.md https://github.com/facebook/sapling/blob/main/eden/fs/docs/O...) and Watchman: https://github.com/facebook/watchman https://github.com/facebook/watchman. What you're describing is reasonably similar to EdenFS, except EdenFS runs in userspace. Watchman layers a consistent view of file metadata on top of inotify (etc), as well as providing stateless queries on top of EdenFS. It acts as a unified interface over regular filesystems as well as Eden that provides file lstat info and hashes over a Unix domain socket. Back in the day, Watchman sped up status queries by over 5x for a repo with hundreds of thousands of files: https://engineering.fb.com/2014/01/07/core-infra/scaling-mercurial-at-facebook/ https://engineering.fb.com/2014/01/07/core-infra/scaling-mer... I worked directly on this and co-wrote this blog post. In truth, getting these two components working to the standard expected by developers was a very difficult systems problem with a ton of event ordering and cache invalidation concerns. (With EdenFS, in particular, I believe there was machine learning involved to detect prefetch patterns.) For smaller repos, it is much simpler to do linear scans. Since it is really fast on modern hardware anyway, it is also the right thing to do, following the maxim of doing the simplest thing that works.