5 ms·

Feedback: code search in a repository (or multiple repositories) should allow blacklisting and/or whitelisting files and directories. Every time I try to search

by oefrha 6y ago

Feedback: code search in a repository (or multiple repositories) should allow blacklisting and/or whitelisting files and directories. Every time I try to search for something on GitHub, I have to sift through tons and tons of useless results from tests. Edit: The discussion has taken a weird turn into support for ignore files. To be clear, this feedback is about dynamically filtering searches when using a website to do code search, most likely searching someone else’s repo on someone else’s website; you don’t have access to the filesystem, the search form is the only input. So all this “rg supports ignore files” (I’m aware, but thanks) is not really relevant.

the_mitsuhiko 6y ago
Ripgrep honors ignore files.
- oefrha 6y ago
  The has nothing to do with ignore files. Tests pushed to a GitHub repository obviously won't be matched by gitignore (unless you ignore them in gitignore then force commit them, which would be really rare), I'm just not interested in them in most of my searches. I'm talking about excluding files and directories during searches, through a directive like, say, "-exclude tests -exclude test_*.py". Sure, I guess technically you can add a bunch of additional patterns to gitignore before you use this web frontend, but that would be a really weird workflow, and it's impossible when the web frontend is run by someone else.
  - woadwarrior01 6y ago
    You could add them to .rgignore or .ignore (which also works with ag, IIRC) instead of adding them to .gitignore.
    - oefrha 6y ago
      Which is again impossible when you use someone else’s web frontend for code search. Plus if you’re editing .rgignore on the filesystem for every search, what’s the point of using a web frontend?
      - the_mitsuhiko 6y ago
        Not sure why ignore files are not the solution here. You can check on an ignore file into your repo then whoever uses ripgrep through whatever fronend gets that logic applied. I’m using this and i don’t quite see the downsides of that.
        oefrha 6y ago
        - I can’t control what someone else checks into their repo. I use web frontends for code search almost exclusively on other people’s repos, because my own repos are already on my machine, so what’s the point. - Most of the time I don’t want to see search results in tests, but occasionally I actually want to find things in tests and not anywhere else. Different people have different filtering needs at different times, you can’t have a ignore file that satisfies every need. Search is about filtering to begin with.
        the_mitsuhiko 6y ago
        Datasette is a tool you run against your stuff usually. I have control over my repos. Once you do cross repos search providing global filters on every query turns into an awful user experience. WRT web frontends: datasette runs locally.
        oefrha 6y ago
        Well, all this tool does is calling rg from a directory with a bunch of repos, so providing global filters on every query is as awful as rg providing a --glob flag... Which is arguably not that awful to a lot of users. > Datasette is a tool you run against your stuff usually. ... datasette runs locally. I don't think so? I could be wrong but I thought a major use case (or the major use case) is providing a frontend for your data to other people. Like on https://ripgrep.datasette.io/-/ripgrep https://ripgrep.datasette.io/-/ripgrep. To quote https://docs.datasette.io/en/stable/ https://docs.datasette.io/en/stable/, > Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world.
  - the_mitsuhiko 6y ago
    There are more ignore files that ripgrep honors then gitignore.
burntsushi 6y ago
I agree your replies are weird. Ignore files would be nice if you wanted repo specific filters or even user wide global filters. But for on-demand filters at search time, the frontend could add a feature for it and implement it with the `-g/--glob` flag.
- the_mitsuhiko 6y ago
  So I guess since I started this entire thread here is why it even came up: datasette (a tool I use) is something you run locally. In particular this plugin also is pointed at a local checkout of the repositories you're working with. So for instance in my case I have all the code I work with in ~/Development. Since I also use ripgrep for local development I typically include .ignore/.rgignore files to control what I'm searching around in my repos. Doing "excludes"/"includes" across a variety of repositories is hard at query time because they layouts of those repositories can be very different. For instance in some JS repos you do want to search in "dist" whereas in others you don't because "dist" gets generated out of "src" for instance and just ends up being a duplicate etc.
  - burntsushi 6y ago
    Ah I see. I didn't know what datasette was. The way I see doing includes/excludes at query time is this. You run a search without excludes for example. You see a bunch of stuff coming back all in one directory that you don't want for this particular search. So you add an exclude rule for that directory. And then just refine results that way. If you have a bunch of similarly named directories, then I think that just means that the exclude rules at query time need to be more specific. e.g., use the full path to the directory. I think the feature being requested here is an interactive feature of the user interface, where as ignore files are more like a static fact of a directory tree.
arcatek 6y ago
GitHub searches are particularly bad. They always give you test folders and examples before and after actual relevant source code. You'd think they would know that folders named "test" should be in a separate section...
- bostonvaulter2 6y ago
  The worst thing about github searches is that it only shows the first result for each file, often that's not the interesting result. I almost never use github's search and instead clone the repository locally and use ripgrep on it (or git grep)
simonw 6y ago
I really want this feature. Filed an issue here, thanks: https://github.com/simonw/datasette-ripgrep/issues/9 https://github.com/simonw/datasette-ripgrep/issues/9
- oefrha 6y ago
  Thanks!
derefr 6y ago
> To be clear, this feedback is about dynamically filtering searches when using a website to do code search, most likely searching someone else’s repo on someone else’s website; you don’t have access to the filesystem, the search form is the only input. I think people are imagining that it should be the repo-author's responsibility to create the equivalent of an `.rgignore` file ahead-of-time (since they know the codebase's structure, they know what's useful vs. useless to show up in a search of the code — pretty objectively, "code" should show up, and "not-code" shouldn't); and that code-search on websites like Github should respect the rules in such a file if it were discovered in the repo. If such a thing existed and was in wide use, you — a random passerby to the codebase, who doesn't yet understand it — wouldn't need to struggle to do any dynamic filtering, because static filtering would already have been done for you.
- oefrha 6y ago
  > If such a thing existed and was in wide use, you — a random passerby to the codebase, who doesn't yet understand it — wouldn't need to struggle to do any dynamic filtering, because static filtering would already have been done for you. That's the thing, it's impossible to create a generic ignore list that caters to everyone's interest. Could the_mitsuhiko, the creator of flask, provide such a list for flask? Nope. Some people want to understand how register_blueprint is implemented and don't want to see anything from test, docs and examples. Some people want to learn how to use register_blueprint, so they specifically want to limit themselves to examples. Some other people want to check if there are test cases for a certain API. You can't ignore any file that may be interesting to someone, which is basically all of them.
  - burntsushi 6y ago
    Right. Ignore files and search-time dynamic filters solve different problems. They aren't mutually exclusive. Otherwise, I wouldn't have added the -g/--glob flag to ripgrep in the first place.
  - derefr 6y ago
    > You can't ignore any file that may be interesting to someone, which is basically all of them. ripgrep already does ignore certain files! Specifically, it ignores binary and hidden files, and files in your .gitignore. This is part of what makes ripgrep's default output so much more "readable" than GNU grep's default output: its few default filters mean that it's not matching from e.g. the contents of your .git/ directory, whereas GNU grep does show those matches by default. > Could the_mitsuhiko, the creator of flask, provide such a list for flask? That wasn't what I was suggesting. I'm saying that each repo owner should be maintaining such a list for their own repo, custom-tailored to it. Just like every repo owner manages their repo's .gitignore, .gitattributes, .dockerignore, etc. And specifically, that list shouldn't be a set of things that are always filtered out of all searches, but rather a set of patterns that apply abstract purpose-type tags to files, ala .gitattributes, such that it makes them much easier (or even "default") to filter them in/out of your searches. Given a patterns file that would look like: *.test.js purpose=test tests/** purpose=test apps/*/tests/** purpose=test ...ripgrep and other search tools could then see those files as having that purpose-tag "test" attached; and you'd be able to filter files in/out by their purpose-tag, just like you can filter files in/out by their filetype with ripgrep's -t/-T switches. (You could go wild from there, if you like, and let this patterns-file match not just files but lines inside files. For example, doc-comments lines [=~ lines with syntax X in files of type Y] could be tagged as having purpose=docs.) And I would then propose, on top of that, some higher-level behavior — not necessarily implemented in rg(1) itself — that could look at a file of "filter-context specifications" like this: default: code code: +purpose=impl -purpose=test -purpose=examples -purpose=docs ...and then, when you do a search, it would use that `default` search-scope if not otherwise specified; or use another scope if you name it. So given the config above, by default you'd be searching only code; but you could instead search your docs scope, or your examples scope, or an implicit "all" scope. The big win with those purpose-tags and scopes, would be if they had conventional or formal (e.g. URNs in an RDF namespace) names, such that different repos could agree on their usage. Then tools that do code-search across many different repos, like Github, could layer UI on top of these purpose-tags and search-scopes discovered from the repo, to enable the repo to be searched using this explicitly-encoded abstract understanding of the purpose/function of the different parts of the repo. (Picture the Github search autocomplete for a search 'foo' giving several drop-down options like "Search 'foo' in [code] of this repo", "Search 'foo' in [docs] of this repo", etc. Then, after you get to the search page, a set of checkboxes to refine your search from the original scope, into a custom scope, by including/excluding arbitrary purpose-tags.)
zeckalpha 6y ago
Include the filename in the output and pipe to grep :)
IshKebab 6y ago
I agree. https://grep.app/ https://grep.app/ does this really well.
simonw 6y ago
I've added the ability to use globs (via ripgrep --glob) to include and exclude directories and file path patterns: https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncClient&glob=datasette%2F%2A%2A https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncCl...

by oefrha 6y ago

the_mitsuhiko 6y ago

Ripgrep honors ignore files.

oefrha 6y ago
The has nothing to do with ignore files. Tests pushed to a GitHub repository obviously won't be matched by gitignore (unless you ignore them in gitignore then force commit them, which would be really rare), I'm just not interested in them in most of my searches. I'm talking about excluding files and directories during searches, through a directive like, say, "-exclude tests -exclude test_*.py". Sure, I guess technically you can add a bunch of additional patterns to gitignore before you use this web frontend, but that would be a really weird workflow, and it's impossible when the web frontend is run by someone else.
- woadwarrior01 6y ago
  You could add them to .rgignore or .ignore (which also works with ag, IIRC) instead of adding them to .gitignore.
  - oefrha 6y ago
    Which is again impossible when you use someone else’s web frontend for code search. Plus if you’re editing .rgignore on the filesystem for every search, what’s the point of using a web frontend?
    - the_mitsuhiko 6y ago
      Not sure why ignore files are not the solution here. You can check on an ignore file into your repo then whoever uses ripgrep through whatever fronend gets that logic applied. I’m using this and i don’t quite see the downsides of that.
      - oefrha 6y ago
        - I can’t control what someone else checks into their repo. I use web frontends for code search almost exclusively on other people’s repos, because my own repos are already on my machine, so what’s the point. - Most of the time I don’t want to see search results in tests, but occasionally I actually want to find things in tests and not anywhere else. Different people have different filtering needs at different times, you can’t have a ignore file that satisfies every need. Search is about filtering to begin with.
        the_mitsuhiko 6y ago
        Datasette is a tool you run against your stuff usually. I have control over my repos. Once you do cross repos search providing global filters on every query turns into an awful user experience. WRT web frontends: datasette runs locally.
        oefrha 6y ago
        Well, all this tool does is calling rg from a directory with a bunch of repos, so providing global filters on every query is as awful as rg providing a --glob flag... Which is arguably not that awful to a lot of users. > Datasette is a tool you run against your stuff usually. ... datasette runs locally. I don't think so? I could be wrong but I thought a major use case (or the major use case) is providing a frontend for your data to other people. Like on https://ripgrep.datasette.io/-/ripgrep https://ripgrep.datasette.io/-/ripgrep. To quote https://docs.datasette.io/en/stable/ https://docs.datasette.io/en/stable/, > Datasette is aimed at data journalists, museum curators, archivists, local governments and anyone else who has data that they wish to share with the world.
- the_mitsuhiko 6y ago
  There are more ignore files that ripgrep honors then gitignore.

burntsushi 6y ago

I agree your replies are weird. Ignore files would be nice if you wanted repo specific filters or even user wide global filters. But for on-demand filters at search time, the frontend could add a feature for it and implement it with the `-g/--glob` flag.

the_mitsuhiko 6y ago
So I guess since I started this entire thread here is why it even came up: datasette (a tool I use) is something you run locally. In particular this plugin also is pointed at a local checkout of the repositories you're working with. So for instance in my case I have all the code I work with in ~/Development. Since I also use ripgrep for local development I typically include .ignore/.rgignore files to control what I'm searching around in my repos. Doing "excludes"/"includes" across a variety of repositories is hard at query time because they layouts of those repositories can be very different. For instance in some JS repos you do want to search in "dist" whereas in others you don't because "dist" gets generated out of "src" for instance and just ends up being a duplicate etc.
- burntsushi 6y ago
  Ah I see. I didn't know what datasette was. The way I see doing includes/excludes at query time is this. You run a search without excludes for example. You see a bunch of stuff coming back all in one directory that you don't want for this particular search. So you add an exclude rule for that directory. And then just refine results that way. If you have a bunch of similarly named directories, then I think that just means that the exclude rules at query time need to be more specific. e.g., use the full path to the directory. I think the feature being requested here is an interactive feature of the user interface, where as ignore files are more like a static fact of a directory tree.

arcatek 6y ago

GitHub searches are particularly bad. They always give you test folders and examples before and after actual relevant source code. You'd think they would know that folders named "test" should be in a separate section...

bostonvaulter2 6y ago
The worst thing about github searches is that it only shows the first result for each file, often that's not the interesting result. I almost never use github's search and instead clone the repository locally and use ripgrep on it (or git grep)

simonw 6y ago

I really want this feature. Filed an issue here, thanks: https://github.com/simonw/datasette-ripgrep/issues/9 https://github.com/simonw/datasette-ripgrep/issues/9

oefrha 6y ago
Thanks!

derefr 6y ago

> To be clear, this feedback is about dynamically filtering searches when using a website to do code search, most likely searching someone else’s repo on someone else’s website; you don’t have access to the filesystem, the search form is the only input. I think people are imagining that it should be the repo-author's responsibility to create the equivalent of an `.rgignore` file ahead-of-time (since they know the codebase's structure, they know what's useful vs. useless to show up in a search of the code — pretty objectively, "code" should show up, and "not-code" shouldn't); and that code-search on websites like Github should respect the rules in such a file if it were discovered in the repo. If such a thing existed and was in wide use, you — a random passerby to the codebase, who doesn't yet understand it — wouldn't need to struggle to do any dynamic filtering, because static filtering would already have been done for you.

oefrha 6y ago
> If such a thing existed and was in wide use, you — a random passerby to the codebase, who doesn't yet understand it — wouldn't need to struggle to do any dynamic filtering, because static filtering would already have been done for you. That's the thing, it's impossible to create a generic ignore list that caters to everyone's interest. Could the_mitsuhiko, the creator of flask, provide such a list for flask? Nope. Some people want to understand how register_blueprint is implemented and don't want to see anything from test, docs and examples. Some people want to learn how to use register_blueprint, so they specifically want to limit themselves to examples. Some other people want to check if there are test cases for a certain API. You can't ignore any file that may be interesting to someone, which is basically all of them.
- burntsushi 6y ago
  Right. Ignore files and search-time dynamic filters solve different problems. They aren't mutually exclusive. Otherwise, I wouldn't have added the -g/--glob flag to ripgrep in the first place.
- derefr 6y ago
  > You can't ignore any file that may be interesting to someone, which is basically all of them. ripgrep already does ignore certain files! Specifically, it ignores binary and hidden files, and files in your .gitignore. This is part of what makes ripgrep's default output so much more "readable" than GNU grep's default output: its few default filters mean that it's not matching from e.g. the contents of your .git/ directory, whereas GNU grep does show those matches by default. > Could the_mitsuhiko, the creator of flask, provide such a list for flask? That wasn't what I was suggesting. I'm saying that each repo owner should be maintaining such a list for their own repo, custom-tailored to it. Just like every repo owner manages their repo's .gitignore, .gitattributes, .dockerignore, etc. And specifically, that list shouldn't be a set of things that are always filtered out of all searches, but rather a set of patterns that apply abstract purpose-type tags to files, ala .gitattributes, such that it makes them much easier (or even "default") to filter them in/out of your searches. Given a patterns file that would look like: *.test.js purpose=test tests/** purpose=test apps/*/tests/** purpose=test ...ripgrep and other search tools could then see those files as having that purpose-tag "test" attached; and you'd be able to filter files in/out by their purpose-tag, just like you can filter files in/out by their filetype with ripgrep's -t/-T switches. (You could go wild from there, if you like, and let this patterns-file match not just files but lines inside files. For example, doc-comments lines [=~ lines with syntax X in files of type Y] could be tagged as having purpose=docs.) And I would then propose, on top of that, some higher-level behavior — not necessarily implemented in rg(1) itself — that could look at a file of "filter-context specifications" like this: default: code code: +purpose=impl -purpose=test -purpose=examples -purpose=docs ...and then, when you do a search, it would use that `default` search-scope if not otherwise specified; or use another scope if you name it. So given the config above, by default you'd be searching only code; but you could instead search your docs scope, or your examples scope, or an implicit "all" scope. The big win with those purpose-tags and scopes, would be if they had conventional or formal (e.g. URNs in an RDF namespace) names, such that different repos could agree on their usage. Then tools that do code-search across many different repos, like Github, could layer UI on top of these purpose-tags and search-scopes discovered from the repo, to enable the repo to be searched using this explicitly-encoded abstract understanding of the purpose/function of the different parts of the repo. (Picture the Github search autocomplete for a search 'foo' giving several drop-down options like "Search 'foo' in [code] of this repo", "Search 'foo' in [docs] of this repo", etc. Then, after you get to the search page, a set of checkboxes to refine your search from the original scope, into a custom scope, by including/excluding arbitrary purpose-tags.)

zeckalpha 6y ago

Include the filename in the output and pipe to grep :)

IshKebab 6y ago

I agree. https://grep.app/ https://grep.app/ does this really well.

simonw 6y ago

I've added the ability to use globs (via ripgrep --glob) to include and exclude directories and file path patterns: https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncClient&glob=datasette%2F%2A%2A https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncCl...