5 ms·

But why? Why create a tool that does `git status` 10x faster?

by thwd 7y ago

romka2 7y ago
The job of gitstatusd is to put Git status in your shell prompt. See screenshot at the top of https://github.com/romkatv/gitstatus https://github.com/romkatv/gitstatus. This is an amazingly convenient tool. When you are working on chromium, on every command you type gitstatusd needs to list the contents of 25,000 directories. Low level optimizations like the ones described here are what makes gitstatusd 10 times faster than `git status`, which in turn makes prompt responsive when otherwise it would be sluggish.
- Lorkki 7y ago
  Isn't inotify meant for exactly this kind of thing, though? You are already introducing a hard dependency on Linux syscalls in "v4" of the optimisations, so it would seem advantageous to make use of that to avoid the full directory traversal for most of the time.
  - romka2 7y ago
    This directory-listing code only executes on a "cold" run of gitstatusd, which happens when you `cd` into a repo for the first time. Users obviously can tolerate higher prompt latency in this case but faster is still better.
  - majewsky 7y ago
    inotify requires you to hold open fds for all the dirs and files you're watching iirc, so in repos that large, you'll be crushed by the file-descriptor-per-process limit.
    - avar 7y ago
      It doesn't, you might need to tweak its limits via /proc/sys/fs/inotify/max_* on such repositories, but those limits are not the same as the much lower open FD limits (as in ulimit -n ...).
      - romka2 7y ago
        I've posted some details explaining why gitstatusd doesn't use inotify in https://github.com/romkatv/gitstatus/commit/050aaaa04b652e15ab4c28b669f1ed753f9d0519 https://github.com/romkatv/gitstatus/commit/050aaaa04b652e15.... The short version is that the default max_user_watches limit is much too low to be useful for gitstatusd and I cannot ask or expect users to change their system settings.
- avar 7y ago
  Your documentation indicates that you've tried with core.untrackedCache, but have you tried core.fsmonitor with the watchman hook? Obviously your "gitstatus" is still generally useful, but if your main itch you're trying to scratch is to have a faster "status" in a particular repo then you should be able to get that down to a few milliseconds with "git status", if you're willing to have watchman sit in the background and monitor it.
  - romka2 7y ago
    > [...] have you tried core.fsmonitor with the watchman hook? I read the docs but it seems like it's not something I can enable on users' machines. Or maybe there is a way to take advantage of it even if it's not enabled? I really haven't looked much into it. To clarify, my main motivation isn't to make my own prompt latency low (I don't even work on large git projects) but to make Powerlevel10k as good a product as possible. Git latency is a pain point for the existing users, so I work on optimizing it. I also cannot rely on users to enable untracked cache even if their filesystem allows it. So gitstatusd performs tests similar to `git update-index --test-untracked-cache` in the background and builds its own untracked cache if tests pass. This makes for a good user experience with no setup or configuration fiddling.
    - univerio 7y ago
      You could use it if available, and fall back to existing behavior if not. (Not sure if it would be faster, but probably worth experimenting with.)
wereHamster 7y ago
Because `git status` takes a really long time if the repository is large. And no, the linux kernel is not considered a large repository, there are far larger ones around, mostly private at companies, such as Microsoft and others.
cgag 7y ago
Why be slow when you can be fast?
- kazinator 7y ago
  - So you can be fast at changing direction. - So you mitigate the damage if you crash.
_wmd 7y ago
Why create a computer when an abacus counts just as effectively :) Time, precious time! We're all born with a limited number of breaths, let's not waste them sighing while waiting for a simple task to complete. In this way faster tooling is a very real method otherwise pure-tech projects can prolong the lives of people. Fix a slow popular tool and the effect is multiplied
masklinn 7y ago
Because you might want to display an up-to-date status permanently in a UI of some sort (e.g. an IDE or editor status bar or a shell prompt — the latter seems to be the use case here), and on big repositories it can be noticeable. Takes 150ms on the rust repository on my machine, with warm cache, that's a serious stutter when it runs on every shell prompt. The tool's page specifically quotes Chromium which takes ~300ms to status on the author's system.

But why? Why create a tool that does `git status` 10x faster?

by thwd 7y ago

But why? Why create a tool that does `git status` 10x faster?

The job of gitstatusd is to put Git status in your shell prompt. See screenshot at the top of https://github.com/romkatv/gitstatus https://github.com/romkatv/gitstatus. This is an amazingly convenient tool. When you are working on chromium, on every command you type gitstatusd needs to list the contents of 25,000 directories. Low level optimizations like the ones described here are what makes gitstatusd 10 times faster than `git status`, which in turn makes prompt responsive when otherwise it would be sluggish.

Lorkki 7y ago
Isn't inotify meant for exactly this kind of thing, though? You are already introducing a hard dependency on Linux syscalls in "v4" of the optimisations, so it would seem advantageous to make use of that to avoid the full directory traversal for most of the time.
- romka2 7y ago
  This directory-listing code only executes on a "cold" run of gitstatusd, which happens when you `cd` into a repo for the first time. Users obviously can tolerate higher prompt latency in this case but faster is still better.
- majewsky 7y ago
  inotify requires you to hold open fds for all the dirs and files you're watching iirc, so in repos that large, you'll be crushed by the file-descriptor-per-process limit.
  - avar 7y ago
    It doesn't, you might need to tweak its limits via /proc/sys/fs/inotify/max_* on such repositories, but those limits are not the same as the much lower open FD limits (as in ulimit -n ...).
    - romka2 7y ago
      I've posted some details explaining why gitstatusd doesn't use inotify in https://github.com/romkatv/gitstatus/commit/050aaaa04b652e15ab4c28b669f1ed753f9d0519 https://github.com/romkatv/gitstatus/commit/050aaaa04b652e15.... The short version is that the default max_user_watches limit is much too low to be useful for gitstatusd and I cannot ask or expect users to change their system settings.
avar 7y ago
Your documentation indicates that you've tried with core.untrackedCache, but have you tried core.fsmonitor with the watchman hook? Obviously your "gitstatus" is still generally useful, but if your main itch you're trying to scratch is to have a faster "status" in a particular repo then you should be able to get that down to a few milliseconds with "git status", if you're willing to have watchman sit in the background and monitor it.
- romka2 7y ago
  > [...] have you tried core.fsmonitor with the watchman hook? I read the docs but it seems like it's not something I can enable on users' machines. Or maybe there is a way to take advantage of it even if it's not enabled? I really haven't looked much into it. To clarify, my main motivation isn't to make my own prompt latency low (I don't even work on large git projects) but to make Powerlevel10k as good a product as possible. Git latency is a pain point for the existing users, so I work on optimizing it. I also cannot rely on users to enable untracked cache even if their filesystem allows it. So gitstatusd performs tests similar to `git update-index --test-untracked-cache` in the background and builds its own untracked cache if tests pass. This makes for a good user experience with no setup or configuration fiddling.
  - univerio 7y ago
    You could use it if available, and fall back to existing behavior if not. (Not sure if it would be faster, but probably worth experimenting with.)

wereHamster 7y ago

Because `git status` takes a really long time if the repository is large. And no, the linux kernel is not considered a large repository, there are far larger ones around, mostly private at companies, such as Microsoft and others.

cgag 7y ago

Why be slow when you can be fast?

kazinator 7y ago
- So you can be fast at changing direction. - So you mitigate the damage if you crash.

_wmd 7y ago

Why create a computer when an abacus counts just as effectively :) Time, precious time! We're all born with a limited number of breaths, let's not waste them sighing while waiting for a simple task to complete. In this way faster tooling is a very real method otherwise pure-tech projects can prolong the lives of people. Fix a slow popular tool and the effect is multiplied

masklinn 7y ago

Because you might want to display an up-to-date status permanently in a UI of some sort (e.g. an IDE or editor status bar or a shell prompt — the latter seems to be the use case here), and on big repositories it can be noticeable. Takes 150ms on the rust repository on my machine, with warm cache, that's a serious stutter when it runs on every shell prompt. The tool's page specifically quotes Chromium which takes ~300ms to status on the author's system.