5 ms·

GitHub code search – waiting list signup

99 points by mmq 4y ago

mcovalt 4y ago
Is this the product that tree-sitter-graph [0] is being used for? [0] https://github.com/tree-sitter/tree-sitter-graph https://github.com/tree-sitter/tree-sitter-graph
- 100k 4y ago
  Yes, Stack Graphs power code navigation for supported languages (currently, Python). We also use tree-sitter parsers for CTags-style symbol navigation.
juujian 4y ago
Honestly, I have caught myself looking for an advanced search option on my github page more than a couple of times. Just to remember that it does not exist. Long overdue!
- YetAnotherNick 4y ago
  Press . in any repo and then search :)
pamelafox 4y ago
I used this recently when I was searching for examples of CPython extensions implementing a particular feature, and it actually did help me find what I was looking for! I also used it when looking for examples of Github action workflows that did particular things. It's nice that you can use the path: option to specify the file extension and/or the path to the file.
semitones 4y ago
This looks very similar to the internal code search product at Google (called CodeSearch). If the quality of this is going to be anywhere near that of CodeSearch, I would use it every day
100k 4y ago
Hello from the GitHub code search team! I'm really proud of this release. It's the result of a year of hard work from multiple teams to bring the custom code search engine we built for cs.github.com to github.com. In the background, we've also spent a lot of time since the launch rebuilding the index format to let us scale up the system. My colleague Tim Clem will be speaking at GitHub Universe today about some of technical details: https://githubuniverse.com/events/detail/virtual-schedule/49426c3f-89ed-477d-b6e1-ab08ee201aca https://githubuniverse.com/events/detail/virtual-schedule/49...
- anonymoushn 4y ago
  Can you search for special characters, and are large files indexed?
  - 100k 4y ago
    All characters are indexed (UTF-8 only). Files larger than 350 KiB are currently excluded, though we welcome feedback on use cases that require larger files to be indexed.
- schroeding 4y ago
  A few weeks earlier and I could have not be forced to hack together a tool to search for specific repos in specific languages that contain certain code-snippets by scraping the existing search API for a week. :P Joking aside though, looks really great, thank you so much! :D
- sytelus 4y ago
  Last time when I heard this, the GitHub search was the shittiest search ever I came across. It didn't had even basic dedup. You search for something and there will be 10 pages of results with exact same code from thousand forks. The intern guy who worked on it claimed he had used "AI" however.
  - sytelus 4y ago
    Looks as non-promising as before: https://cs.github.com/?scopeName=All+repos&scope=&q=gpt2+training https://cs.github.com/?scopeName=All+repos&scope=&q=gpt2+tra...
thinkingemote 4y ago
Does it still only index the primary/default branch of a repository, or can it search within all branches? If it can now, can we filter the search for stuff within the branches of forks for a specific repository? So many interesting work can only be found manually, laboriously (but is so rewarding) within the explore network graph where people work on their own branches in their own forks. Their code is never found from the search function.
- 100k 4y ago
  Currently, we index the head of the default branch. However, the index format will support branch indexing, and we use that functionality to index forks, which was not possible in prior incarnations of GitHub code search. Indexing specific commits (e.g., associated with a PR or feature branch) is on our roadmap.
- mdaniel 4y ago
  https://docs.sourcegraph.com/getting-started/github-vs-sourcegraph#searching-repositories-branches-and-forks https://docs.sourcegraph.com/getting-started/github-vs-sourc... highlights just such a comparison, and they have both "fork:yes" and "fork:only" and their "rev:" keyword supports branch globs
bonney_io 4y ago
Inching closer and closer to each Github project page being one-click away from a complete, web-based IDE.
adamnemecek 4y ago
Does it dedup?
- 100k 4y ago
  Content is deduplicated, yes.
canyonero 4y ago
One feature for code search that I've wanted for some time that could unlock some really interesting results would be to allow search to reach into the git file history. I've found myself sometimes thinking that something existed in a particular codebase and I go to look for only to discover that I'm unable find it. Then, I realize that the code I was looking for has been removed. I can only find it by browsing back into the history figuring out the events that led to the deletion or change. Being able to search the history would be killer IMO.
- LifeIsBio 4y ago
  I’ve run into this exact situation multiple times. Searching the whole history would be revolutionary.
  - slimsag 4y ago
    FWIW you can do this with Sourcegraph today, searching over both diffs (code changes)[0] and commit messages[1]: [0] https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/golang/go%24+type:diff+TODO&patternType=standard https://sourcegraph.com/search?q=context:global+repo:%5Egith... [1] https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/golang/go%24+type:commit+TODO&patternType=standard https://sourcegraph.com/search?q=context:global+repo:%5Egith... (I work there, just commenting on my own though. we're all pretty happy to have competition, more awareness of code search, etc.)
- durazabu 4y ago
  Are you familiar with git pickaxe? For example: git log -pS 'deletedvar'
  - disqard 4y ago
    Thank you for sharing that! TIL.
- sdesol 4y ago
  This is something that I'm working on and I believe a couple of others as well. My solution won't be available until early next year though, as indexing public repos at scale is far from trivial and I'm expecting this will consume over 40TB of fast storage to be performant.
zbhoy 4y ago
Is this a Sourcegraph killer or does SG have other features that still keep it competitive?
- jdorfman 4y ago
  We (Sourcegraph) have other features to stay competitive. One thing we have been working on indexing other code hosts like Eclipse Foundation, Crates.io, etc. We put this together anticipating these questions: https://docs.sourcegraph.com/getting-started/github-vs-sourcegraph https://docs.sourcegraph.com/getting-started/github-vs-sourc... We also just launched Search Ranking and Relevance: https://about.sourcegraph.com/blog/new-search-ranking https://about.sourcegraph.com/blog/new-search-ranking More to come! And BTW we are happy to have competition, GitHub is a great brand and it will only make code search better for all developers.
  - zbhoy 4y ago
    Thanks for the response and for having a doc ready. Code search definitely seems to be on the rise as a needed tool
    - jdorfman 4y ago
      You're welcome. We agree, good times ahead =)
throwaway743 4y ago
Digging it. Fingers crossed github's app search options are updated as well
hbn 4y ago
I've been using this since January and it's become a near daily tool for me. Very good stuff
acedTrex 4y ago
I have been using cs.github.com religiously since it originally went into beta and it has increased productivity for me to insane levels. There are many features I still want but its usually *good enough
davea37 4y ago
A couple of years ago you stopped indexing repos that hadn't had activity for a year [0]. Will such repos be indexed by the new code search? [0]: https://github.blog/changelog/2020-12-17-changes-to-code-search-indexing/ https://github.blog/changelog/2020-12-17-changes-to-code-sea...
- 100k 4y ago
  Our ultimate goal is to surpass the current code search index. As of today, there are about 25 million repositories in the new index, but we have more work to do before we reach full scale.
bottlepalm 4y ago
Developers, developers developers! Very happy Microsoft bought Github.
jo_the_zeroton 4y ago
HUGE
andrewmcwatters 4y ago
> Way more than grep. Heck, it'd be nice if the current system right now at least did grep with users being able to toggle POSIX options. It doesn't even do that. Instead, I have to clone the entire repo to do a trivial lookup. To be fair, supporting this would require GitHub to do the compute themselves, but I have no idea what they could possibly be doing right now that's more effective.
eximius 4y ago
My only problem with the new code search is I've noticed some examples where exact string searches fail to locate some matches. I was looking for callers of a function and assumed no results meant no callers. But a simple 'search all' in VS Code found results missing from GitHub code search.
barbazoo 4y ago
This will also improve the github.com code search, right? I always found it frustrating that I couldn't really search for exact strings, even using quotes. I hope this fixes that too.
- 100k 4y ago
  This will become the new github.com code search. Last year, we launched a Technology Preview of our new code search as a standalone site: cs.github.com. Today, we're bringing that powerful code search, plus a redesigned code navigation UX, to github.com.
revskill 4y ago
Dark theme is the default theme now for modern web. I found it a good thing.
elforce002 4y ago
It's in beta. Will this service have a free version for individuals and organizations?
- 100k 4y ago
  This is addressed in the FAQ. It will be included with GitHub, just like the current code search.

99 points by mmq 4y ago

Is this the product that tree-sitter-graph [0] is being used for? [0] https://github.com/tree-sitter/tree-sitter-graph https://github.com/tree-sitter/tree-sitter-graph

100k 4y ago
Yes, Stack Graphs power code navigation for supported languages (currently, Python). We also use tree-sitter parsers for CTags-style symbol navigation.

juujian 4y ago

Honestly, I have caught myself looking for an advanced search option on my github page more than a couple of times. Just to remember that it does not exist. Long overdue!

YetAnotherNick 4y ago
Press . in any repo and then search :)

pamelafox 4y ago

I used this recently when I was searching for examples of CPython extensions implementing a particular feature, and it actually did help me find what I was looking for! I also used it when looking for examples of Github action workflows that did particular things. It's nice that you can use the path: option to specify the file extension and/or the path to the file.

semitones 4y ago

This looks very similar to the internal code search product at Google (called CodeSearch). If the quality of this is going to be anywhere near that of CodeSearch, I would use it every day

100k 4y ago

Hello from the GitHub code search team! I'm really proud of this release. It's the result of a year of hard work from multiple teams to bring the custom code search engine we built for cs.github.com to github.com. In the background, we've also spent a lot of time since the launch rebuilding the index format to let us scale up the system. My colleague Tim Clem will be speaking at GitHub Universe today about some of technical details: https://githubuniverse.com/events/detail/virtual-schedule/49426c3f-89ed-477d-b6e1-ab08ee201aca https://githubuniverse.com/events/detail/virtual-schedule/49...

anonymoushn 4y ago
Can you search for special characters, and are large files indexed?
- 100k 4y ago
  All characters are indexed (UTF-8 only). Files larger than 350 KiB are currently excluded, though we welcome feedback on use cases that require larger files to be indexed.
schroeding 4y ago
A few weeks earlier and I could have not be forced to hack together a tool to search for specific repos in specific languages that contain certain code-snippets by scraping the existing search API for a week. :P Joking aside though, looks really great, thank you so much! :D
sytelus 4y ago
Last time when I heard this, the GitHub search was the shittiest search ever I came across. It didn't had even basic dedup. You search for something and there will be 10 pages of results with exact same code from thousand forks. The intern guy who worked on it claimed he had used "AI" however.
- sytelus 4y ago
  Looks as non-promising as before: https://cs.github.com/?scopeName=All+repos&scope=&q=gpt2+training https://cs.github.com/?scopeName=All+repos&scope=&q=gpt2+tra...

thinkingemote 4y ago

Does it still only index the primary/default branch of a repository, or can it search within all branches? If it can now, can we filter the search for stuff within the branches of forks for a specific repository? So many interesting work can only be found manually, laboriously (but is so rewarding) within the explore network graph where people work on their own branches in their own forks. Their code is never found from the search function.

100k 4y ago
Currently, we index the head of the default branch. However, the index format will support branch indexing, and we use that functionality to index forks, which was not possible in prior incarnations of GitHub code search. Indexing specific commits (e.g., associated with a PR or feature branch) is on our roadmap.
mdaniel 4y ago
https://docs.sourcegraph.com/getting-started/github-vs-sourcegraph#searching-repositories-branches-and-forks https://docs.sourcegraph.com/getting-started/github-vs-sourc... highlights just such a comparison, and they have both "fork:yes" and "fork:only" and their "rev:" keyword supports branch globs

bonney_io 4y ago

Inching closer and closer to each Github project page being one-click away from a complete, web-based IDE.

adamnemecek 4y ago

Does it dedup?

100k 4y ago
Content is deduplicated, yes.

canyonero 4y ago

One feature for code search that I've wanted for some time that could unlock some really interesting results would be to allow search to reach into the git file history. I've found myself sometimes thinking that something existed in a particular codebase and I go to look for only to discover that I'm unable find it. Then, I realize that the code I was looking for has been removed. I can only find it by browsing back into the history figuring out the events that led to the deletion or change. Being able to search the history would be killer IMO.

LifeIsBio 4y ago
I’ve run into this exact situation multiple times. Searching the whole history would be revolutionary.
- slimsag 4y ago
  FWIW you can do this with Sourcegraph today, searching over both diffs (code changes)[0] and commit messages[1]: [0] https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/golang/go%24+type:diff+TODO&patternType=standard https://sourcegraph.com/search?q=context:global+repo:%5Egith... [1] https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/golang/go%24+type:commit+TODO&patternType=standard https://sourcegraph.com/search?q=context:global+repo:%5Egith... (I work there, just commenting on my own though. we're all pretty happy to have competition, more awareness of code search, etc.)
durazabu 4y ago
Are you familiar with git pickaxe? For example: git log -pS 'deletedvar'
- disqard 4y ago
  Thank you for sharing that! TIL.
sdesol 4y ago
This is something that I'm working on and I believe a couple of others as well. My solution won't be available until early next year though, as indexing public repos at scale is far from trivial and I'm expecting this will consume over 40TB of fast storage to be performant.

zbhoy 4y ago

Is this a Sourcegraph killer or does SG have other features that still keep it competitive?

jdorfman 4y ago
We (Sourcegraph) have other features to stay competitive. One thing we have been working on indexing other code hosts like Eclipse Foundation, Crates.io, etc. We put this together anticipating these questions: https://docs.sourcegraph.com/getting-started/github-vs-sourcegraph https://docs.sourcegraph.com/getting-started/github-vs-sourc... We also just launched Search Ranking and Relevance: https://about.sourcegraph.com/blog/new-search-ranking https://about.sourcegraph.com/blog/new-search-ranking More to come! And BTW we are happy to have competition, GitHub is a great brand and it will only make code search better for all developers.
- zbhoy 4y ago
  Thanks for the response and for having a doc ready. Code search definitely seems to be on the rise as a needed tool
  - jdorfman 4y ago
    You're welcome. We agree, good times ahead =)

throwaway743 4y ago

Digging it. Fingers crossed github's app search options are updated as well

hbn 4y ago

I've been using this since January and it's become a near daily tool for me. Very good stuff

acedTrex 4y ago

I have been using cs.github.com religiously since it originally went into beta and it has increased productivity for me to insane levels. There are many features I still want but its usually *good enough

davea37 4y ago

A couple of years ago you stopped indexing repos that hadn't had activity for a year [0]. Will such repos be indexed by the new code search? [0]: https://github.blog/changelog/2020-12-17-changes-to-code-search-indexing/ https://github.blog/changelog/2020-12-17-changes-to-code-sea...

100k 4y ago
Our ultimate goal is to surpass the current code search index. As of today, there are about 25 million repositories in the new index, but we have more work to do before we reach full scale.

bottlepalm 4y ago

Developers, developers developers! Very happy Microsoft bought Github.

jo_the_zeroton 4y ago

HUGE

andrewmcwatters 4y ago

> Way more than grep. Heck, it'd be nice if the current system right now at least did grep with users being able to toggle POSIX options. It doesn't even do that. Instead, I have to clone the entire repo to do a trivial lookup. To be fair, supporting this would require GitHub to do the compute themselves, but I have no idea what they could possibly be doing right now that's more effective.

eximius 4y ago

My only problem with the new code search is I've noticed some examples where exact string searches fail to locate some matches. I was looking for callers of a function and assumed no results meant no callers. But a simple 'search all' in VS Code found results missing from GitHub code search.

barbazoo 4y ago

This will also improve the github.com code search, right? I always found it frustrating that I couldn't really search for exact strings, even using quotes. I hope this fixes that too.

100k 4y ago
This will become the new github.com code search. Last year, we launched a Technology Preview of our new code search as a standalone site: cs.github.com. Today, we're bringing that powerful code search, plus a redesigned code navigation UX, to github.com.

revskill 4y ago

Dark theme is the default theme now for modern web. I found it a good thing.

elforce002 4y ago

It's in beta. Will this service have a free version for individuals and organizations?

100k 4y ago
This is addressed in the FAQ. It will be included with GitHub, just like the current code search.