4 ms·

How could a process accumulate hundreds of negative dentries during normal operation? If that happens something fishy is going on.

pdonis 5y ago
The third paragraph in the article addresses this. Most of the negative dentries a process generates are going to be invisible to the user.
- Groxx 5y ago
  Yeah, my immediate thought was of stuff like "ask nginx to give me this misspelled webpage" -> look up that file -> bam, dentry. Expose practically anything on the internet that leads to a piece of user input causing a filesystem operation (which is going to be extremely common and often unavoidable), and "hundreds" isn't the concern. Millions to billions is.
  - bjourne 5y ago
    That's what I meant with "fishy". If nginx fills up your negative dentry cache with millions to billions of misspelled urls you are being dos:ed.
    - pdonis 5y ago
      What if your C compiler fills up your negative dentry cache during, say, a kernel compile, because it's looking for a huge number of include files that don't exist, since the C preprocessor uses a very simplistic algorithm to decide where it needs to look for include files? (This is one of the examples in the paragraph in the article that I referred to.)
      - bjourne 5y ago
        Even in the worst case, that shouldn't amount to more than a few thousand files. Millions of missing include files is unrealistic.
teraflop 5y ago
The article gives some examples, but here's another one: consider what happens when you run "git status" on a large repository. In order to determine the status of every file, git needs to check each subdirectory for a ".gitignore" file, even though the vast majority of the time there's only one at the root. All of those nonexistent files can become negative dentries. In theory, git could do its own caching, or it could be refactored so as to move the .gitignore scanning inline with the main directory traversal. In practice, it doesn't do so (at least as of 2.30.2, which is the version I just tested). And I don't think it's reasonable to expect every program to contort itself to minimize the number of times it looks for nonexistent files, when that's something that can reasonably be delegated to the OS.
- Groxx 5y ago
  The only way git could cache this is with a file-system watcher.... and tbh my experience with git fswatchers has been utterly abysmal. They miss things / have phantom changes routinely, on the order of a couple times per week, so I just disable them outright now. Without a truly reliable filesystem watching technique, git cannot cache this information. Any cached knowledge needs to be verified because it could have changed, which defeats the whole purpose of a cache here. It could change its strategy, to e.g. only check at repo root or require a list of "enabled" .gitignores, but it doesn't do that currently.
  - teraflop 5y ago
    Oh, I just meant caching within the lifetime of a single git command's execution. As it stands, when you run "git status", the git executable goes through each directory in your working copy, lists all of its directory entries, and then immediately afterward calls open() to see if a .gitignore file exists in that directory -- even if there wasn't one in the list it just read. That's what I meant by saying that in principle, there's no reason git couldn't just cache the file's presence/absence on its own, just for those few microseconds (maybe "cache" was a poor choice of words). In practice, I can understand that the implementation complexity might not be worth it.
    - 5y ago

pdonis 5y ago

The third paragraph in the article addresses this. Most of the negative dentries a process generates are going to be invisible to the user.

Groxx 5y ago
Yeah, my immediate thought was of stuff like "ask nginx to give me this misspelled webpage" -> look up that file -> bam, dentry. Expose practically anything on the internet that leads to a piece of user input causing a filesystem operation (which is going to be extremely common and often unavoidable), and "hundreds" isn't the concern. Millions to billions is.
- bjourne 5y ago
  That's what I meant with "fishy". If nginx fills up your negative dentry cache with millions to billions of misspelled urls you are being dos:ed.
  - pdonis 5y ago
    What if your C compiler fills up your negative dentry cache during, say, a kernel compile, because it's looking for a huge number of include files that don't exist, since the C preprocessor uses a very simplistic algorithm to decide where it needs to look for include files? (This is one of the examples in the paragraph in the article that I referred to.)
    - bjourne 5y ago
      Even in the worst case, that shouldn't amount to more than a few thousand files. Millions of missing include files is unrealistic.

teraflop 5y ago

The article gives some examples, but here's another one: consider what happens when you run "git status" on a large repository. In order to determine the status of every file, git needs to check each subdirectory for a ".gitignore" file, even though the vast majority of the time there's only one at the root. All of those nonexistent files can become negative dentries. In theory, git could do its own caching, or it could be refactored so as to move the .gitignore scanning inline with the main directory traversal. In practice, it doesn't do so (at least as of 2.30.2, which is the version I just tested). And I don't think it's reasonable to expect every program to contort itself to minimize the number of times it looks for nonexistent files, when that's something that can reasonably be delegated to the OS.

Groxx 5y ago
The only way git could cache this is with a file-system watcher.... and tbh my experience with git fswatchers has been utterly abysmal. They miss things / have phantom changes routinely, on the order of a couple times per week, so I just disable them outright now. Without a truly reliable filesystem watching technique, git cannot cache this information. Any cached knowledge needs to be verified because it could have changed, which defeats the whole purpose of a cache here. It could change its strategy, to e.g. only check at repo root or require a list of "enabled" .gitignores, but it doesn't do that currently.
- teraflop 5y ago
  Oh, I just meant caching within the lifetime of a single git command's execution. As it stands, when you run "git status", the git executable goes through each directory in your working copy, lists all of its directory entries, and then immediately afterward calls open() to see if a .gitignore file exists in that directory -- even if there wasn't one in the list it just read. That's what I meant by saying that in principle, there's no reason git couldn't just cache the file's presence/absence on its own, just for those few microseconds (maybe "cache" was a poor choice of words). In practice, I can understand that the implementation complexity might not be worth it.
  - 5y ago