5 ms·

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Somet

by ezekg 4y ago

I use their new code search a lot to grok how people use certain features, or implement certain things. But I do wish there was a way to filter out forks. Sometimes I search a string and just get a bunch of forks all with the same result. For example, searching a common class in a Rails app often just shows a bunch of rails/rails forks, which is a lot of noise to sift through when you're trying to see how devs commonly use a certain feature.

100k 4y ago
Thanks for the feedback! That's coming, we've been prioritizing scaling the index and ingest process and haven't had a chance got add that yet. There are a bunch of value-add features like this I am looking forward to knocking out soon.
- kowlo 4y ago
  Well done
- BaculumMeumEst 4y ago
  thanks so much, you guys are doing amazing work
- orf 4y ago
  Out of interest, if I have a repo with many millions of files that compress quite nicely down to about a 1.4gb packfile, is it better for the ingestion and/or indexer if I break this down into many smaller pushes or one large push? Because I pushed such a repo yesterday and it’s still not been indexed.
  - 100k 4y ago
    Sorry, there are size limits on the number of files in a repositories that we'll index. That is probably why your repository wasn't indexed. Out of curiosity, what are you storing in this repository?
    - orf 4y ago
      Sequences of code from Python package releases from pypi, as an experiment I’m working on. They compress quite nicely as the deltas between releases are fairly small. I thought it would be nice to be able to search through them, but I guess the file limit was reached. That’s a shame.
      - joshuawarner32 4y ago
        Oooh that sounds really useful, would you mind dropping a link?
        bspammer 4y ago
        Sorry for the cyber-stalking orf, but I'm guessing it's this: https://github.com/orf/pypi-data https://github.com/orf/pypi-data
        orf 4y ago
        It's not - it's something that builds upon that. It's not really ready for prime time yet, but it will be here: https://github.com/orf/pypi-code-repo https://github.com/orf/pypi-code-repo if it works, which I really hope it does, we should have a repo containing every file (sub-25mb) published to pypi. it's not really useful to clone this, but the git packfile + index seems to be pretty ideal for storing + querying this data, reducing 14tb of compressed packages down to like 100gb or so. this should enable large scale exploration of the contents of pypi from only your laptop, which is useful if you want to look at how the python language is evolving (how many packages use f-strings over time?). the rust tool i've built to do the heavy lifting is here: https://github.com/orf/pypi-import-test https://github.com/orf/pypi-import-test. i'm learning a lot about git internals, it's quite fun
- jpgvm 4y ago
  Is it possible for you to clarify the bigram weight function? I'm assuming it's inverse frequency but that would a be great tidbit to better understand why the covering ngrams work.
simonw 4y ago
As an open source library maintainer I've been using it for that too - it's fantastic for answering questions like "is anyone using this API method I wrote?" and "how much of a mess would it likely cause people if I deprecated this function?"
- xvilka 4y ago
  Would be nice to make a tool automating this and printing the "coverage" statistics.
qabqabaca 4y ago
Sourcegraph[1] does this better and has done for a couple of years now. I use it for this reason all the time. [1] https://sourcegraph.com https://sourcegraph.com
- Arnavion 4y ago
  Yes, just change the URL from https://github.com/foo/bar https://github.com/foo/bar to https://sourcegraph.com/github.com/foo/bar https://sourcegraph.com/github.com/foo/bar to be dropped in to a code search for that GH repo.
mnutt 4y ago
I run into this a lot where I'd like to see some real world use cases of a method in FooClass, but get a hundred pages of forks of the FooClass.h header file. In some cases I've been successful adding "-filename:FooClass.h" to filter out that header file. If the method is used a lot in the primary project it can be a game of whack-a-mole but it often eventually works.
ljm 4y ago
But also, you know what you’re looking for and you can filter the results. Someone less experienced than you may take any of those results as gospel. Gospel the same way StackOverflow could be, but this time by a source who will do it’s best to tell you what you want. my personal rule about AI is about the same as the rule about being a lawyer. Never ask a question you don’t know the answer to.
chatmasta 4y ago
Perhaps it could benefit from something like a "dissimilarity" filter, which ranks the current result set by returning the most unique hits first. You wouldn't always want this, because sometimes you're searching for how something is typically used, and with many duplicative results you can confirm that's the preferred pattern. But other times you're looking for more estoteric usage of a certain function, and it would be nice to filter the "standard usage" from the results (although you can already do this with carefully chosen negated keywords). Personally, I'm happy with the new code search so far. I stopped using Sourcegraph because I could never get the deeper results I wanted - it would just return the top five repositories including some common code snippet and I couldn't explore further than that. GitHub Code Search doesn't seem to have this problem to such a degree, since I can use negation more naturally, and since my query is not limited to some shallow subset of the corpus before refining it.
- sqs 4y ago
  Re: Sourcegraph, we're working on improving that, and sorry you couldn't get the results you wanted. We primarily build for the code within customers, where this particular problem is less common than across all open-source repositories. But we want it to work really well in every case. Our new ranking (https://about.sourcegraph.com/blog/new-search-ranking https://about.sourcegraph.com/blog/new-search-ranking) should help a lot here, and it's live on https://sourcegraph.com https://sourcegraph.com. Can you share some of the queries you tried so we can see how much ranking helps and how to handle them better?
  - chatmasta 4y ago
    Sure! I will try it next time I'm using code search. I can't remember the specific queries I've tried, but I only ever used it when looking for "how have people used this function," so I remember the limited depth of results being particularly annoying. Thanks for the link to your blog post. I didn't realize Steve Yegge had joined your team - congratulations, that's quite an endorsement! I've always liked Sourcegraph just because our company names are so similar (founder of Splitgraph here)... we might have even gotten a few candidates because of that initial confusion. :) How has GitHub Code Search impacted your product direction? Do you see it as an opportunity to focus more on the internal use case, or do you have plans for some other differentiation? It's always unfortunate when a big company introduces a product so similar to the core product of a startup, but I'm sure there is a silver lining there, especially when you have a talented team and a mature codebase (for example, Fly.io has been able to carve out a niche for itself despite Cloudflare moving to compete in the same areas). Either way, best of luck to you from a fellow S-grapher!
pohuing 4y ago
I think forks without any changes shouldn't really show up in search tbh.
damidekronik 4y ago
When trying out some C++ I was surprised by how many projects bundled the library source code, essentially finding any actual client code a pain.
- saurik 4y ago
  As a C++ developer, I maybe-ironically have this same complaint with JavaScript projects ;P. I particularly hate it when people embed minified versions of libraries, as line-based search and display then will experience a hit for essentially every query on the one giant line that is tens of kilocharacters long.
  - runlevel1 4y ago
    If your language lacks a de facto way to install dependencies, people will commit vendored code. See: C, C++. If your language's package manager puts your deps within the repo directory by default, people will commit the vendored code. See: Node, Go (since go.mod). As long as we can all agree that committing compiled code is a crime punishable by 24 hours in the shame cube.
- shaunsingh0207 4y ago
  I have this issue with GitHub code search very often. I'll search up a function to see some examples of how to use it, and the first 20 or so pages of results are just copies of the source where its defined.
nitwit005 4y ago
Similar to this, there are some generated files it would be nice to rank very low, or even completely ignore, as they probably weren't intended to be committed in the first place. Some search results are dominated by files in paths like ".npm/_cacache/...", ".apm/...", ".svn/...".
lumb63 4y ago
I’m glad to see some folks in the comments section here using this. My first thought on seeing the headline was, “why not just use grep?”, which the article addressed. My second was, “who in earth wants to search all of GitHub?” I’m happy someone does.

by ezekg 4y ago

100k 4y ago

Thanks for the feedback! That's coming, we've been prioritizing scaling the index and ingest process and haven't had a chance got add that yet. There are a bunch of value-add features like this I am looking forward to knocking out soon.

kowlo 4y ago
Well done
BaculumMeumEst 4y ago
thanks so much, you guys are doing amazing work
orf 4y ago
Out of interest, if I have a repo with many millions of files that compress quite nicely down to about a 1.4gb packfile, is it better for the ingestion and/or indexer if I break this down into many smaller pushes or one large push? Because I pushed such a repo yesterday and it’s still not been indexed.
- 100k 4y ago
  Sorry, there are size limits on the number of files in a repositories that we'll index. That is probably why your repository wasn't indexed. Out of curiosity, what are you storing in this repository?
  - orf 4y ago
    Sequences of code from Python package releases from pypi, as an experiment I’m working on. They compress quite nicely as the deltas between releases are fairly small. I thought it would be nice to be able to search through them, but I guess the file limit was reached. That’s a shame.
    - joshuawarner32 4y ago
      Oooh that sounds really useful, would you mind dropping a link?
      - bspammer 4y ago
        Sorry for the cyber-stalking orf, but I'm guessing it's this: https://github.com/orf/pypi-data https://github.com/orf/pypi-data
        orf 4y ago
        It's not - it's something that builds upon that. It's not really ready for prime time yet, but it will be here: https://github.com/orf/pypi-code-repo https://github.com/orf/pypi-code-repo if it works, which I really hope it does, we should have a repo containing every file (sub-25mb) published to pypi. it's not really useful to clone this, but the git packfile + index seems to be pretty ideal for storing + querying this data, reducing 14tb of compressed packages down to like 100gb or so. this should enable large scale exploration of the contents of pypi from only your laptop, which is useful if you want to look at how the python language is evolving (how many packages use f-strings over time?). the rust tool i've built to do the heavy lifting is here: https://github.com/orf/pypi-import-test https://github.com/orf/pypi-import-test. i'm learning a lot about git internals, it's quite fun
jpgvm 4y ago
Is it possible for you to clarify the bigram weight function? I'm assuming it's inverse frequency but that would a be great tidbit to better understand why the covering ngrams work.

simonw 4y ago

As an open source library maintainer I've been using it for that too - it's fantastic for answering questions like "is anyone using this API method I wrote?" and "how much of a mess would it likely cause people if I deprecated this function?"

xvilka 4y ago
Would be nice to make a tool automating this and printing the "coverage" statistics.

qabqabaca 4y ago

Sourcegraph[1] does this better and has done for a couple of years now. I use it for this reason all the time. [1] https://sourcegraph.com https://sourcegraph.com

Arnavion 4y ago
Yes, just change the URL from https://github.com/foo/bar https://github.com/foo/bar to https://sourcegraph.com/github.com/foo/bar https://sourcegraph.com/github.com/foo/bar to be dropped in to a code search for that GH repo.

mnutt 4y ago

I run into this a lot where I'd like to see some real world use cases of a method in FooClass, but get a hundred pages of forks of the FooClass.h header file. In some cases I've been successful adding "-filename:FooClass.h" to filter out that header file. If the method is used a lot in the primary project it can be a game of whack-a-mole but it often eventually works.

ljm 4y ago

But also, you know what you’re looking for and you can filter the results. Someone less experienced than you may take any of those results as gospel. Gospel the same way StackOverflow could be, but this time by a source who will do it’s best to tell you what you want. my personal rule about AI is about the same as the rule about being a lawyer. Never ask a question you don’t know the answer to.

chatmasta 4y ago

Perhaps it could benefit from something like a "dissimilarity" filter, which ranks the current result set by returning the most unique hits first. You wouldn't always want this, because sometimes you're searching for how something is typically used, and with many duplicative results you can confirm that's the preferred pattern. But other times you're looking for more estoteric usage of a certain function, and it would be nice to filter the "standard usage" from the results (although you can already do this with carefully chosen negated keywords). Personally, I'm happy with the new code search so far. I stopped using Sourcegraph because I could never get the deeper results I wanted - it would just return the top five repositories including some common code snippet and I couldn't explore further than that. GitHub Code Search doesn't seem to have this problem to such a degree, since I can use negation more naturally, and since my query is not limited to some shallow subset of the corpus before refining it.

sqs 4y ago
Re: Sourcegraph, we're working on improving that, and sorry you couldn't get the results you wanted. We primarily build for the code within customers, where this particular problem is less common than across all open-source repositories. But we want it to work really well in every case. Our new ranking (https://about.sourcegraph.com/blog/new-search-ranking https://about.sourcegraph.com/blog/new-search-ranking) should help a lot here, and it's live on https://sourcegraph.com https://sourcegraph.com. Can you share some of the queries you tried so we can see how much ranking helps and how to handle them better?
- chatmasta 4y ago
  Sure! I will try it next time I'm using code search. I can't remember the specific queries I've tried, but I only ever used it when looking for "how have people used this function," so I remember the limited depth of results being particularly annoying. Thanks for the link to your blog post. I didn't realize Steve Yegge had joined your team - congratulations, that's quite an endorsement! I've always liked Sourcegraph just because our company names are so similar (founder of Splitgraph here)... we might have even gotten a few candidates because of that initial confusion. :) How has GitHub Code Search impacted your product direction? Do you see it as an opportunity to focus more on the internal use case, or do you have plans for some other differentiation? It's always unfortunate when a big company introduces a product so similar to the core product of a startup, but I'm sure there is a silver lining there, especially when you have a talented team and a mature codebase (for example, Fly.io has been able to carve out a niche for itself despite Cloudflare moving to compete in the same areas). Either way, best of luck to you from a fellow S-grapher!

pohuing 4y ago

I think forks without any changes shouldn't really show up in search tbh.

damidekronik 4y ago

When trying out some C++ I was surprised by how many projects bundled the library source code, essentially finding any actual client code a pain.

saurik 4y ago
As a C++ developer, I maybe-ironically have this same complaint with JavaScript projects ;P. I particularly hate it when people embed minified versions of libraries, as line-based search and display then will experience a hit for essentially every query on the one giant line that is tens of kilocharacters long.
- runlevel1 4y ago
  If your language lacks a de facto way to install dependencies, people will commit vendored code. See: C, C++. If your language's package manager puts your deps within the repo directory by default, people will commit the vendored code. See: Node, Go (since go.mod). As long as we can all agree that committing compiled code is a crime punishable by 24 hours in the shame cube.
shaunsingh0207 4y ago
I have this issue with GitHub code search very often. I'll search up a function to see some examples of how to use it, and the first 20 or so pages of results are just copies of the source where its defined.

nitwit005 4y ago

Similar to this, there are some generated files it would be nice to rank very low, or even completely ignore, as they probably weren't intended to be committed in the first place. Some search results are dominated by files in paths like ".npm/_cacache/...", ".apm/...", ".svn/...".

lumb63 4y ago

I’m glad to see some folks in the comments section here using this. My first thought on seeing the headline was, “why not just use grep?”, which the article addressed. My second was, “who in earth wants to search all of GitHub?” I’m happy someone does.