10 ms·

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that,

> Just use grep? First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core. But you don't NEED to do this do you? I'm ALREADY in a repository, I just don't want to check out, say all of WebKit, I just need to find where a specific reference is defined. Maybe, maybe on a really serious day do I need to search an entire organization. But hardly ever. I have never, in over a decade ever, wanted sophisticated symbolic searching from GitHub code search, I just need remote grep. Why is the code search not feature bisected into this 99% use case, and then the occasional global repository search, which can behave entirely differently?

100k 4y ago
(I work on this.) If you check out our prior blog post, "A brief history of code search at GitHub" (https://github.blog/2021-12-15-a-brief-history-of-code-search-at-github/ https://github.blog/2021-12-15-a-brief-history-of-code-searc...), you can learn a bit about the evolution of this feature. And, in fact, we used to use git grep to search repositories. This doesn't work well at GitHub's scale. We have 100M users and over 200M repositories in a multi-tenant environment. Your git grep is going to be competing for resources with other user's pushes and clones.
- nickysielicki 4y ago
  From a resource utilization perspective, is it easier on your servers if I just clone the repo and do the grep myself? Because when the search doesn’t show me what I’m looking for on the first page, this is exactly what I do. (And I can never remember if it’s —shallow or —depth=1, so they’re not shallow clones, either.)
  - plugin-baby 4y ago
    Same. In addition: > when the search doesn’t show me what I’m looking for on the first page, this is exactly what I do This is pretty much always.
- deleted 4y ago
  [deleted]
joshuamorton 4y ago
> I'm ALREADY in a repository, I just don't want to check out, say all of WebKit, I just need to find where a specific reference is defined. If you're in some repository that uses a webkit api, and you want to know how that api is defined, how do you do that without global cross references or a global lookup? Even for local lookups, indicies are useful (as any ctags user will tell you!), but for any kind of cross repo xrefs they're ncecessary.
tiagod 4y ago
I use github-wide searches all the time to see how people are using certain APIs, to find libraries used in some blob from the strings I find there, to find people working with the same data I'm about to attempt to work it, and the list goes on. What you use github search for doesn't require all this engineering, but what I use it for does. Why wouldn't they build something that satisfies both our necessities well?
- rtuin 4y ago
  Same here. I find doing a quick org-wide code search for a specific call a great starting point for impact analysis for API changes. Using it multiple times a week!
- saurik 4y ago
  Same. I'll go even further and say that the narrower the search the less useful it is, as if I want to search a single repository I can and will just download it as local grep is so much better than remote grep it is painful. They could make a really amazing remote search for a repository--with a find-as-you-type low-latency display that supported language-specific syntax tree queries--but it would require way too many resources and so they are never going to bother. The result: if I am searching on GitHub at all, it is probably a full-site query, and I absolutely do use full-organization searches to figure out what repository to download as so many people now do extremely fine-grained linked repositories (though, in my experience, GitHub's search for this purpose sucks... maybe it is better now).
IshKebab 4y ago
Actually I've been using https://grep.app https://grep.app for ages and while I agree on GitHub I basically only search the repo I'm in, that's mainly because Github's existing search sucks. On grep.app I regularly search all repos. It's very useful for finding out how to use APIs or where APIs from dependencies are defined. So I suspect you don't want it because subconsciously you know that Github's "search all" feature won't return you useful results. Hell they still don't provide a way to filter out test directories which makes the code search inside a single repo useless a lot of the time.
- Xeophon 4y ago
  Recently discovered grep.app and it is incredibly useful. Wish I’ve knew it earlier.
  - shurane 4y ago
    This is pretty neat. Feels like real time full text search. The user interface really reminds me of https://hn.algolia.com https://hn.algolia.com as well.
- andrewmcwatters 4y ago
  Thank you for sharing. This is basically what I want.
nonethewiser 4y ago
That's not 99% of use cases. That's just how you use it.
muchospandas 4y ago
If I want to search locally, I'll use some tool to search my checked-out code. I used to search my org's code on a daily basis and all of GitHub a few times a week at least, back when I used git for work. I think you may be overestimating how common your specific use-case is.
userbinator 4y ago
or about 0.6 GB/sec/core. That seems very slow by today's standards. There's a rather... eccentric guy who easily beat that almost 10 years ago with his implementations of string search: https://news.ycombinator.com/item?id=6954298 https://news.ycombinator.com/item?id=6954298
- ygra 4y ago
  The difference may be regex matching. This can often be optimized to an impressive degree, depending on the regex, but unless it's a simple substring search without any metacharacters, I'm not sure those approaches are comparable.
WorldMaker 4y ago
Have you tried using github.dev for your remote grep? Press . in a code repo (while logged in) and it loads a remote VS Code for you in that repo. I don't entirely understand the complex interplay between how github.dev uses both the GitHub search indexes and in-browser in-memory grep, but being as it tries its best reproduce the local VS Code ripgrep experience, it may still be more optimized for your use cases.

100k 4y ago

(I work on this.) If you check out our prior blog post, "A brief history of code search at GitHub" (https://github.blog/2021-12-15-a-brief-history-of-code-search-at-github/ https://github.blog/2021-12-15-a-brief-history-of-code-searc...), you can learn a bit about the evolution of this feature. And, in fact, we used to use git grep to search repositories. This doesn't work well at GitHub's scale. We have 100M users and over 200M repositories in a multi-tenant environment. Your git grep is going to be competing for resources with other user's pushes and clones.

nickysielicki 4y ago
From a resource utilization perspective, is it easier on your servers if I just clone the repo and do the grep myself? Because when the search doesn’t show me what I’m looking for on the first page, this is exactly what I do. (And I can never remember if it’s —shallow or —depth=1, so they’re not shallow clones, either.)
- plugin-baby 4y ago
  Same. In addition: > when the search doesn’t show me what I’m looking for on the first page, this is exactly what I do This is pretty much always.
deleted 4y ago
[deleted]

joshuamorton 4y ago

> I'm ALREADY in a repository, I just don't want to check out, say all of WebKit, I just need to find where a specific reference is defined. If you're in some repository that uses a webkit api, and you want to know how that api is defined, how do you do that without global cross references or a global lookup? Even for local lookups, indicies are useful (as any ctags user will tell you!), but for any kind of cross repo xrefs they're ncecessary.

tiagod 4y ago

I use github-wide searches all the time to see how people are using certain APIs, to find libraries used in some blob from the strings I find there, to find people working with the same data I'm about to attempt to work it, and the list goes on. What you use github search for doesn't require all this engineering, but what I use it for does. Why wouldn't they build something that satisfies both our necessities well?

rtuin 4y ago
Same here. I find doing a quick org-wide code search for a specific call a great starting point for impact analysis for API changes. Using it multiple times a week!
saurik 4y ago
Same. I'll go even further and say that the narrower the search the less useful it is, as if I want to search a single repository I can and will just download it as local grep is so much better than remote grep it is painful. They could make a really amazing remote search for a repository--with a find-as-you-type low-latency display that supported language-specific syntax tree queries--but it would require way too many resources and so they are never going to bother. The result: if I am searching on GitHub at all, it is probably a full-site query, and I absolutely do use full-organization searches to figure out what repository to download as so many people now do extremely fine-grained linked repositories (though, in my experience, GitHub's search for this purpose sucks... maybe it is better now).

IshKebab 4y ago

Actually I've been using https://grep.app https://grep.app for ages and while I agree on GitHub I basically only search the repo I'm in, that's mainly because Github's existing search sucks. On grep.app I regularly search all repos. It's very useful for finding out how to use APIs or where APIs from dependencies are defined. So I suspect you don't want it because subconsciously you know that Github's "search all" feature won't return you useful results. Hell they still don't provide a way to filter out test directories which makes the code search inside a single repo useless a lot of the time.

Xeophon 4y ago
Recently discovered grep.app and it is incredibly useful. Wish I’ve knew it earlier.
- shurane 4y ago
  This is pretty neat. Feels like real time full text search. The user interface really reminds me of https://hn.algolia.com https://hn.algolia.com as well.
andrewmcwatters 4y ago
Thank you for sharing. This is basically what I want.

nonethewiser 4y ago

That's not 99% of use cases. That's just how you use it.

muchospandas 4y ago

If I want to search locally, I'll use some tool to search my checked-out code. I used to search my org's code on a daily basis and all of GitHub a few times a week at least, back when I used git for work. I think you may be overestimating how common your specific use-case is.

userbinator 4y ago

or about 0.6 GB/sec/core. That seems very slow by today's standards. There's a rather... eccentric guy who easily beat that almost 10 years ago with his implementations of string search: https://news.ycombinator.com/item?id=6954298 https://news.ycombinator.com/item?id=6954298

ygra 4y ago
The difference may be regex matching. This can often be optimized to an impressive degree, depending on the regex, but unless it's a simple substring search without any metacharacters, I'm not sure those approaches are comparable.

WorldMaker 4y ago

Have you tried using github.dev for your remote grep? Press . in a code repo (while logged in) and it loads a remote VS Code for you in that repo. I don't entirely understand the complex interplay between how github.dev uses both the GitHub search indexes and in-browser in-memory grep, but being as it tries its best reproduce the local VS Code ripgrep experience, it may still be more optimized for your use cases.