3 ms·

The article gives some examples, but here's another one: consider what happens when you run "git status" on a large repository. In order to determine the status

The article gives some examples, but here's another one: consider what happens when you run "git status" on a large repository. In order to determine the status of every file, git needs to check each subdirectory for a ".gitignore" file, even though the vast majority of the time there's only one at the root. All of those nonexistent files can become negative dentries. In theory, git could do its own caching, or it could be refactored so as to move the .gitignore scanning inline with the main directory traversal. In practice, it doesn't do so (at least as of 2.30.2, which is the version I just tested). And I don't think it's reasonable to expect every program to contort itself to minimize the number of times it looks for nonexistent files, when that's something that can reasonably be delegated to the OS.

Groxx 5y ago
The only way git could cache this is with a file-system watcher.... and tbh my experience with git fswatchers has been utterly abysmal. They miss things / have phantom changes routinely, on the order of a couple times per week, so I just disable them outright now. Without a truly reliable filesystem watching technique, git cannot cache this information. Any cached knowledge needs to be verified because it could have changed, which defeats the whole purpose of a cache here. It could change its strategy, to e.g. only check at repo root or require a list of "enabled" .gitignores, but it doesn't do that currently.
- teraflop 5y ago
  Oh, I just meant caching within the lifetime of a single git command's execution. As it stands, when you run "git status", the git executable goes through each directory in your working copy, lists all of its directory entries, and then immediately afterward calls open() to see if a .gitignore file exists in that directory -- even if there wasn't one in the list it just read. That's what I meant by saying that in principle, there's no reason git couldn't just cache the file's presence/absence on its own, just for those few microseconds (maybe "cache" was a poor choice of words). In practice, I can understand that the implementation complexity might not be worth it.
  - Groxx 5y ago
    ah, lol. yeah, that's a good point, that scenario could definitely be smarter - I wasn't thinking about it listing the directory first. plus the time between list and "look again for .gitignore" is small enough that any inconsistencies could very reasonably be labeled an acceptable resolution to a race between git and other file system changes.
- bsder 5y ago
  Does not caching this really help, though? What happens when the .gitignore above you changes while you are in the midst of scanning a subdirectory? Is that not the same problem? The problem is that git operations aren't transactional with respect to the filesystem, no? Sure, the window of uncertainty changes, but it's never zero.
  - beagle3 5y ago
    But that's an inherent problem with a non-transactional file system. If you remove .gitignore 1 microsecond before git open()s it, it's gone. If you remove .gitignore 1 microsecond after git open()s it (even before it has a chance to read it), unix file system semantics mean you still get the contents. There is always a race condition, you can play with the specifics, but cannot avoid it when multiple files referring to each other are edited independently.
bjourne 5y ago
Well, recursive configuration is a misfeature of git. No sane program should work that way. But regardless, if git scans every sub directory then the kernel should have already cached all "positive" dentries obviating the need for any negative ones. And that cache must be an order of magnitude larger than any negative cache.

Groxx 5y ago

The only way git could cache this is with a file-system watcher.... and tbh my experience with git fswatchers has been utterly abysmal. They miss things / have phantom changes routinely, on the order of a couple times per week, so I just disable them outright now. Without a truly reliable filesystem watching technique, git cannot cache this information. Any cached knowledge needs to be verified because it could have changed, which defeats the whole purpose of a cache here. It could change its strategy, to e.g. only check at repo root or require a list of "enabled" .gitignores, but it doesn't do that currently.

teraflop 5y ago
Oh, I just meant caching within the lifetime of a single git command's execution. As it stands, when you run "git status", the git executable goes through each directory in your working copy, lists all of its directory entries, and then immediately afterward calls open() to see if a .gitignore file exists in that directory -- even if there wasn't one in the list it just read. That's what I meant by saying that in principle, there's no reason git couldn't just cache the file's presence/absence on its own, just for those few microseconds (maybe "cache" was a poor choice of words). In practice, I can understand that the implementation complexity might not be worth it.
- Groxx 5y ago
  ah, lol. yeah, that's a good point, that scenario could definitely be smarter - I wasn't thinking about it listing the directory first. plus the time between list and "look again for .gitignore" is small enough that any inconsistencies could very reasonably be labeled an acceptable resolution to a race between git and other file system changes.
bsder 5y ago
Does not caching this really help, though? What happens when the .gitignore above you changes while you are in the midst of scanning a subdirectory? Is that not the same problem? The problem is that git operations aren't transactional with respect to the filesystem, no? Sure, the window of uncertainty changes, but it's never zero.
- beagle3 5y ago
  But that's an inherent problem with a non-transactional file system. If you remove .gitignore 1 microsecond before git open()s it, it's gone. If you remove .gitignore 1 microsecond after git open()s it (even before it has a chance to read it), unix file system semantics mean you still get the contents. There is always a race condition, you can play with the specifics, but cannot avoid it when multiple files referring to each other are edited independently.

bjourne 5y ago

Well, recursive configuration is a misfeature of git. No sane program should work that way. But regardless, if git scans every sub directory then the kernel should have already cached all "positive" dentries obviating the need for any negative ones. And that cache must be an order of magnitude larger than any negative cache.