6 ms·

Can you explain more about this feature? If it is such an improvement, why would it not be enabled by default - are there any drawbacks to enabling it?

by nicky0 11y ago

bicolao 11y ago
Because it relies on filesystem changing directory's mtime every time a file/dir is added or deleted in that directory. Not every filesystem supports that.
- amelius 11y ago
  Can't git just check whether the filesystem does this? It would be a pity if an ignorant user copied a repository to another filesystem only to find out that certain operations suddenly break. Perhaps the option should include the filesystem UUID for which the option should apply (to prevent these kinds of surprises).
  - pilif 11y ago
    >Can't git just check whether the filesystem does this? it can by using `git update-index --test-untracked-cache` but it takes a long time to check this, so I guess they don't want to incur that penalty on every run. OTOH, >It would be a pity if an ignorant user copied a repository to another filesystem only to find out that certain operations suddenly break. yes. I agree. The way this is handled at the moment is dangerous to the point of this not being worth it at all.
    - amelius 11y ago
      > it can by using `git update-index --test-untracked-cache` but it takes a long time to check this How about just writing a file to a scratch directory, and see if the mtime of the directory was changed?
      - pilif 11y ago
        considering that they have actual test code which also takes quite a while to run, I would assume that there are systems misbehaving in many different ways, all of which might cause this feature to break, so all of them needs to be tested against. The check you're talking about is one of the many checks it does, but there was further breakage which all needs to be checked against.
      - dilap 11y ago
        or how about just turning it on automatically for known-supporting systems, and leaving it as a system-wide config option for unknown systems?
        Guvante 11y ago
        The underlying file system determines compatibility so there is no OS level switch to check.
        setpatchaddress 11y ago
        Seems like a flaw in POSIX -- there should be a bit in statfs (?) for something like this.
        telotortium 11y ago
        There is in struct statfs, which can be obtained for an open file descriptor using the fstatfs(2) system call, if you (a) want to make this autodetection OS-specific (statfs is Linux-specific, but I'm sure there's equivalents on other OSs) and (b) maintain your own list of compatible file system types.
      - Someone 11y ago
        I can't think of real world use cases that make sense to me, but I would guess they are worried about having mount points in the directory tree to check for modified files. If so, they have to check every directory in the tree (or, alternatively, figure out all mount points in a different way, but I don't know whether that can be done efficiently in a portable way)
  - bicolao 11y ago
    > It would be a pity if an ignorant user copied a repository to another filesystem only to find out that certain operations suddenly break. Once tested (or enabled without testing if you know what you're doing), the location of the repository and uname are recorded. If you copy the repository elsewhere, git should detect and disable the cache.
- ahel 11y ago
  Which one does that?
  - avar 11y ago
    On which one does it work? I haven't run across a Linux filesystem where it doesn't work, including mounts over NFS. But I suppose on some non-POSIX platforms you might have more issues, or with some esoteric mount options to screw with mtime or something. The reason it's disabled by default is first of all that it's relatively new, but secondly that if your FS stops behaving as expected (or you move your repo to one that doesn't behave) it'll degrade very badly, i.e. "git status" might now completely miss files that have been modified in your working tree. But if you're just running a system where you know you can trust the FS you can use the untracked cache and get a lot of "git status" speed-up, which'll matter more the larger your checkout size is in terms of checked out directories & files.
    - codys 11y ago
      Looks like parallel's shared filesystem (linux guest, osx host) fails the test: % ~/g/git/git-update-index --test-untracked-cache Testing mtime in '/media/psf/Home/Downloads/junk' . directory stat info does not change after adding a new file
TheSwordsman 11y ago
Here is the link to the documentation, it explains it better than I can without copying it verbatim: * https://git-scm.com/docs/git-update-index#_untracked_cache https://git-scm.com/docs/git-update-index#_untracked_cache The underlying filesystem needs to have certain features enabled (mtime of directories) to be able to support the untracked cache. There's no guarantee that all will, so it does not enable it. I suppose this could be enhanced to be an opportunistic feature. However, then there is some variability in what git is doing versus there being one default behavior.
VLM 11y ago
Non-technical detail reasons not to: 1) the gain is too small to be useful. Its still too slow for a synchronous poll in an IDE/editor or something so anyone concerned about latency will spawn a thread or process to asynchronously check the status. Now if we were going from hundreds of ms to tens of us... but we aren't. 2) the gain can be expressed in years of hardware evolution "just wait X years and your faster mass storage will naturally speed up enough...". vs the time it would take to implement bulletproof code. Better off just waiting for faster drives. Simple code on fast drives beats complex code on slow drives. 3) speaking of faster drives, $$$ can be turned into speed in a pretty smooth interchangeable market of professionals and exotic hardware. This is extremely well developed and widely understood. Pop in a SSD, parallel off the NAS, whatever. Practically nobody knows how to troubleshoot the git enhancement. Best of luck, as an overall lifetime system cost on a large scale its going to be an extremely expensive way to gain performance compared to slapping in a SSD. 4) The larger the code is the more complex it is and the fewer people can understand it. Industrial revolution dogma about specialization doesn't work in code. So outside the new feature, the rest of the code will suffer because the bar will raise that much. Simplicate and add lightness. Some of the technical reason workarounds are missing "fun" scenarios like some madman has a git repo spanning multiple filesystems so you can't just check the repo root directories capabilities you have to traverse the whole repo tree (ugh) or there's a backup-restore cycle where the filesystem type changes (maybe as part of a hardware and OS upgrade?) or there's layered filesystem problems (stored on ext3, but exported via an obscure userfs or networked file system). Also caching must be hilarious, doesn't NFS have mount option timers like acdirmin acdirmax and noac that mess with when mtime is updated, so two clients on the same NFS mounted dir could react differently based on wall time the command is run, LOL that one would be fun/hilarious to troubleshoot.
- avar 11y ago
  In reply to that: 1) I find that for the large repos I work on the difference goes from "noticeably slow" to "I don't notice it" which is somewhere on the magical ~200ms boundary. 2) This really doesn't help, the Git repo is likely in the FS cache anyway, you'll get exactly the same results / speedup if you do this in /dev/shm i.e. the in-memory filesystem. This has nothing to do with fast drives, it has to do with syscall overhead. Recursively stat-ing a huge directory is simply never going to be all that fast. 3) Irrelevant for the reasons noted above.

Can you explain more about this feature? If it is such an improvement, why would it not be enabled by default - are there any drawbacks to enabling it?

by nicky0 11y ago

Can you explain more about this feature? If it is such an improvement, why would it not be enabled by default - are there any drawbacks to enabling it?

bicolao 11y ago

Because it relies on filesystem changing directory's mtime every time a file/dir is added or deleted in that directory. Not every filesystem supports that.

amelius 11y ago
Can't git just check whether the filesystem does this? It would be a pity if an ignorant user copied a repository to another filesystem only to find out that certain operations suddenly break. Perhaps the option should include the filesystem UUID for which the option should apply (to prevent these kinds of surprises).
- pilif 11y ago
  >Can't git just check whether the filesystem does this? it can by using `git update-index --test-untracked-cache` but it takes a long time to check this, so I guess they don't want to incur that penalty on every run. OTOH, >It would be a pity if an ignorant user copied a repository to another filesystem only to find out that certain operations suddenly break. yes. I agree. The way this is handled at the moment is dangerous to the point of this not being worth it at all.
  - amelius 11y ago
    > it can by using `git update-index --test-untracked-cache` but it takes a long time to check this How about just writing a file to a scratch directory, and see if the mtime of the directory was changed?
    - pilif 11y ago
      considering that they have actual test code which also takes quite a while to run, I would assume that there are systems misbehaving in many different ways, all of which might cause this feature to break, so all of them needs to be tested against. The check you're talking about is one of the many checks it does, but there was further breakage which all needs to be checked against.
    - dilap 11y ago
      or how about just turning it on automatically for known-supporting systems, and leaving it as a system-wide config option for unknown systems?
      - Guvante 11y ago
        The underlying file system determines compatibility so there is no OS level switch to check.
        setpatchaddress 11y ago
        Seems like a flaw in POSIX -- there should be a bit in statfs (?) for something like this.
        telotortium 11y ago
        There is in struct statfs, which can be obtained for an open file descriptor using the fstatfs(2) system call, if you (a) want to make this autodetection OS-specific (statfs is Linux-specific, but I'm sure there's equivalents on other OSs) and (b) maintain your own list of compatible file system types.
    - Someone 11y ago
      I can't think of real world use cases that make sense to me, but I would guess they are worried about having mount points in the directory tree to check for modified files. If so, they have to check every directory in the tree (or, alternatively, figure out all mount points in a different way, but I don't know whether that can be done efficiently in a portable way)
- bicolao 11y ago
  > It would be a pity if an ignorant user copied a repository to another filesystem only to find out that certain operations suddenly break. Once tested (or enabled without testing if you know what you're doing), the location of the repository and uname are recorded. If you copy the repository elsewhere, git should detect and disable the cache.
ahel 11y ago
Which one does that?
- avar 11y ago
  On which one does it work? I haven't run across a Linux filesystem where it doesn't work, including mounts over NFS. But I suppose on some non-POSIX platforms you might have more issues, or with some esoteric mount options to screw with mtime or something. The reason it's disabled by default is first of all that it's relatively new, but secondly that if your FS stops behaving as expected (or you move your repo to one that doesn't behave) it'll degrade very badly, i.e. "git status" might now completely miss files that have been modified in your working tree. But if you're just running a system where you know you can trust the FS you can use the untracked cache and get a lot of "git status" speed-up, which'll matter more the larger your checkout size is in terms of checked out directories & files.
  - codys 11y ago
    Looks like parallel's shared filesystem (linux guest, osx host) fails the test: % ~/g/git/git-update-index --test-untracked-cache Testing mtime in '/media/psf/Home/Downloads/junk' . directory stat info does not change after adding a new file

TheSwordsman 11y ago

Here is the link to the documentation, it explains it better than I can without copying it verbatim: * https://git-scm.com/docs/git-update-index#_untracked_cache https://git-scm.com/docs/git-update-index#_untracked_cache The underlying filesystem needs to have certain features enabled (mtime of directories) to be able to support the untracked cache. There's no guarantee that all will, so it does not enable it. I suppose this could be enhanced to be an opportunistic feature. However, then there is some variability in what git is doing versus there being one default behavior.

VLM 11y ago

Non-technical detail reasons not to: 1) the gain is too small to be useful. Its still too slow for a synchronous poll in an IDE/editor or something so anyone concerned about latency will spawn a thread or process to asynchronously check the status. Now if we were going from hundreds of ms to tens of us... but we aren't. 2) the gain can be expressed in years of hardware evolution "just wait X years and your faster mass storage will naturally speed up enough...". vs the time it would take to implement bulletproof code. Better off just waiting for faster drives. Simple code on fast drives beats complex code on slow drives. 3) speaking of faster drives, $$$ can be turned into speed in a pretty smooth interchangeable market of professionals and exotic hardware. This is extremely well developed and widely understood. Pop in a SSD, parallel off the NAS, whatever. Practically nobody knows how to troubleshoot the git enhancement. Best of luck, as an overall lifetime system cost on a large scale its going to be an extremely expensive way to gain performance compared to slapping in a SSD. 4) The larger the code is the more complex it is and the fewer people can understand it. Industrial revolution dogma about specialization doesn't work in code. So outside the new feature, the rest of the code will suffer because the bar will raise that much. Simplicate and add lightness. Some of the technical reason workarounds are missing "fun" scenarios like some madman has a git repo spanning multiple filesystems so you can't just check the repo root directories capabilities you have to traverse the whole repo tree (ugh) or there's a backup-restore cycle where the filesystem type changes (maybe as part of a hardware and OS upgrade?) or there's layered filesystem problems (stored on ext3, but exported via an obscure userfs or networked file system). Also caching must be hilarious, doesn't NFS have mount option timers like acdirmin acdirmax and noac that mess with when mtime is updated, so two clients on the same NFS mounted dir could react differently based on wall time the command is run, LOL that one would be fun/hilarious to troubleshoot.

avar 11y ago
In reply to that: 1) I find that for the large repos I work on the difference goes from "noticeably slow" to "I don't notice it" which is somewhere on the magical ~200ms boundary. 2) This really doesn't help, the Git repo is likely in the FS cache anyway, you'll get exactly the same results / speedup if you do this in /dev/shm i.e. the in-memory filesystem. This has nothing to do with fast drives, it has to do with syscall overhead. Recursively stat-ing a huge directory is simply never going to be all that fast. 3) Irrelevant for the reasons noted above.