4 ms·

I'll share my similarly named tool `grep-ast` [0], which sort of does the opposite of the OP's `ast-grep`. The OP's tool lets you specify your search as a chunk

I'll share my similarly named tool `grep-ast` [0], which sort of does the opposite of the OP's `ast-grep`. The OP's tool lets you specify your search as a chunk of code/AST (and then do AST transforms on matches). My tool let's you grep a regex as usual, but shows you the matches in a helpful AST aware way. It works with most popular languages, thanks to tree-sitter. It uses the abstract syntax tree (AST) of the source code to show how the matching lines fit into the code structure. It shows relevant code from every layer of the AST, above and below the matches. It's useful when you're grepping to understand how functions, classes, variables etc are used within a non-trivial codebase. Here's a snippet that shows grep-ast searching the django repo. Notice that it finds `ROOT_URLCONF` and then shows you the method and class that contain the matching line, including a helpful part of the docstring. If you ran this in the terminal, it would also colorize the matches. django$ grep-ast ROOT_URLCONF middleware/locale.py: │from django.conf import settings │from django.conf.urls.i18n import is_language_prefix_patterns_used │from django.http import HttpResponseRedirect ⋮... │class LocaleMiddleware(MiddlewareMixin): │ """ │ Parse a request and decide what translation │ object to install in the current thread context. ⋮... │ def process_request(self, request): ▶ urlconf = getattr(request, "urlconf", settings.ROOT_URLCONF) [0] https://github.com/paul-gauthier/grep-ast https://github.com/paul-gauthier/grep-ast

herrington_d 3y ago
Hey paulg, ast-grep author here! This is something I also want to do in ast-grep! ast-grep prints the surrounding lines around matches but they are not aware of which function/scope the matches are in. May I ask how you do the scope detection in a general fashion? (say language agnostic) https://github.com/ast-grep/ast-grep/issues/155 https://github.com/ast-grep/ast-grep/issues/155
- anotherpaulg 3y ago
  Nice, thanks for checking out grep-ast. The command line tool is a thin wrapper around the `TreeContext` class, whose purpose is show you a set of "lines of interest" in the context of the entire AST. This all exists because my other project aider [0] uses TreeContext to display a repository map [1] so that GPT-4 can understand how the most important classes, methods, functions, etc fit into the entire code base of a git repository. But it was easy to make a CLI interface to grep lines of interest and display them with TreeContext, and it turned out to be quite useful. The TreeContext class is line-oriented, and is mainly interested in tracking language constructs whose scope spans multiple lines. Typically these are things like classes, methods, functions, loops, if/else constructs, etc. Given a line of interest, we look at all the multi-line scopes which contain it. For each such multi-line scope, we want to display some "header" lines to provide context. In this example, the match for "two" is contained in the multi-line scopes of a method and a class. So we print their headers. $ grep-ast two example.py ⋮... │class MyClass: │ "MyClass is great" ⋮... │ def print2(self): ▶ print("two") ⋮... The trick is how to determine the header for each multi-line scope? It's not ideal to just use the first line. For example, it's nice that the header for the class included the docstring as well as the bare `class MyClass:` line. For any multi-line scope, we look at all the other AST scopes which start on the same line. We take the smallest such co-occurring scope, and declare the header to be the lines that it spans. For a simple method like `def print2(self):`, that's all that gets picked up. But a complex method like `print1()` below picks up all the lines which are part of its full function signature: $ grep-ast one example.py ⋮... │class MyClass: │ "MyClass is great" ⋮... │ def print1( │ self, │ prefix, │ suffix, │ ): ⋮... ▶ print(f"{prefix} one {suffix}") ⋮... It's a heuristic, but it seems to work well in practice. [0] https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider [1] https://aider.chat/docs/repomap.html https://aider.chat/docs/repomap.html
mingodad 3y ago
There is also gram_grep[0] "Search text using a grammar, lexer, or straight regex. Chain searches for greater refinement." See also parsertl-playground[1] for online edit/test grammars. [0]https://github.com/BenHanson/gram_grep https://github.com/BenHanson/gram_grep [1]https://mingodad.github.io/parsertl-playground/playground/ https://mingodad.github.io/parsertl-playground/playground/
e40 3y ago
It would be great if your tool was available in Homebrew.

herrington_d 3y ago

Hey paulg, ast-grep author here! This is something I also want to do in ast-grep! ast-grep prints the surrounding lines around matches but they are not aware of which function/scope the matches are in. May I ask how you do the scope detection in a general fashion? (say language agnostic) https://github.com/ast-grep/ast-grep/issues/155 https://github.com/ast-grep/ast-grep/issues/155

anotherpaulg 3y ago
Nice, thanks for checking out grep-ast. The command line tool is a thin wrapper around the `TreeContext` class, whose purpose is show you a set of "lines of interest" in the context of the entire AST. This all exists because my other project aider [0] uses TreeContext to display a repository map [1] so that GPT-4 can understand how the most important classes, methods, functions, etc fit into the entire code base of a git repository. But it was easy to make a CLI interface to grep lines of interest and display them with TreeContext, and it turned out to be quite useful. The TreeContext class is line-oriented, and is mainly interested in tracking language constructs whose scope spans multiple lines. Typically these are things like classes, methods, functions, loops, if/else constructs, etc. Given a line of interest, we look at all the multi-line scopes which contain it. For each such multi-line scope, we want to display some "header" lines to provide context. In this example, the match for "two" is contained in the multi-line scopes of a method and a class. So we print their headers. $ grep-ast two example.py ⋮... │class MyClass: │ "MyClass is great" ⋮... │ def print2(self): ▶ print("two") ⋮... The trick is how to determine the header for each multi-line scope? It's not ideal to just use the first line. For example, it's nice that the header for the class included the docstring as well as the bare `class MyClass:` line. For any multi-line scope, we look at all the other AST scopes which start on the same line. We take the smallest such co-occurring scope, and declare the header to be the lines that it spans. For a simple method like `def print2(self):`, that's all that gets picked up. But a complex method like `print1()` below picks up all the lines which are part of its full function signature: $ grep-ast one example.py ⋮... │class MyClass: │ "MyClass is great" ⋮... │ def print1( │ self, │ prefix, │ suffix, │ ): ⋮... ▶ print(f"{prefix} one {suffix}") ⋮... It's a heuristic, but it seems to work well in practice. [0] https://github.com/paul-gauthier/aider https://github.com/paul-gauthier/aider [1] https://aider.chat/docs/repomap.html https://aider.chat/docs/repomap.html

mingodad 3y ago

There is also gram_grep[0] "Search text using a grammar, lexer, or straight regex. Chain searches for greater refinement." See also parsertl-playground[1] for online edit/test grammars. [0]https://github.com/BenHanson/gram_grep https://github.com/BenHanson/gram_grep [1]https://mingodad.github.io/parsertl-playground/playground/ https://mingodad.github.io/parsertl-playground/playground/

e40 3y ago

It would be great if your tool was available in Homebrew.