6 ms·
Show HN: GritQL, a Rust CLI for rewriting source code
Hi everyone!
I’m excited to open source GritQL, a Rust CLI for searching and transforming source code.
GritQL comes from my experiences with conducting large scale refactors and migrations.
Usually, I would start exploring a codebase with grep. This is easy to start with, but most migrations end up accumulating additional requirements like ensuring the right packages are imported and excluding cases which don’t have a viable migration path.
Eventually, to build a complex migration, I usually ended up having to write a full codemod program with a tool like jscodeshift. This comes with its own problems:
- Most of the exploratory work has to be abandoned as you figure out how to represent your original regex search as an AST.
- Reading/writing a codemod requires mentally translating from AST names back to what source code actually looks like.
- Performance is often an afterthought, so iterating on a large codemod can be painfully slow.
- Codemod frameworks are language-specific, so if you’re hopping between multiple languages—or trying to migrate a shared API—you have to learn different tools.
GritQL is an attempt to develop a powerful middle ground:
- Exploratory analysis is easy: just put a code snippet in backticks and use $metavariables for placeholders.
- Incrementally add complexity by introducing side conditions with where clauses.
- Reuse named patterns to avoid rebuilding queries, and use shared patterns from our standard library for common tasks like ensuring modules are imported.
- Iterate on large codebases quickly: we use Rust for maximum performance
GritQL has already been used on thousands of repositories for complex migrations[1] but we're excited to collaborate more with the open source community.
[1] Ex. https://github.com/openai/openai-python/discussions/742 https://github.com/openai/openai-python/discussions/742
- mlejva 3y agoCongrats to the launch! Managing tech debt is and will be an important issue in the world of AI software. I'm glad there's a company solving this. Additionally, this is something the current generation of LLMs are already helpful at solving. GritQL is very unique way of approaching this problem. OT: We've been working with Grit and Morgante for the past months and I have only positive things to say about them. If GritQL looks useful to you, you should reach out.
- psalv 3y ago++ congratulations to the team! We've tackled some pretty gnarly migrations with GritQL and it has saved us a lot of time. We have used ts-morph in the past for similar types of migrations but found the barrier to entry felt high, which prevented adoption. We were impressed by how complicated of migrations we could use GritQL for—we have been using it to migrate from mobx to react hooks.
- morgante 3y agoThanks Paul! It's been great getting your feedback as we've evolved Grit over time.
- jxnlco 3y agothis definitly helped a tonne to migrate from openai <1.0 to 1.0
- ranman 3y agoI'd love to see one of these for migrating from Claude's prompt API to their messaging API
- rattray 3y agoVery happy Grit customer here. Wish I'd had this instead of jscodeshift et al back when I was doing oodles of migrations at Stripe, but glad I have it now, and very excited to see them go open-source!
- anotherpaulg 3y agoThis looks great, thanks for building and sharing it. Interested folks may also want to check out ast-grep: https://github.com/ast-grep/ast-grep https://github.com/ast-grep/ast-grep
- morgante 3y agoast-grep is great too, we've learned a lot from it. If you like yaml, you should definitely try ast-grep. The main area GritQL shines is in taking simpler transformation patterns and composing them into complex migrations. For example, the OpenAI migration was built by incrementally handling the different edge cases we've seen in the wild: https://github.com/getgrit/stdlib/blob/main/.grit/patterns/python/openai.md https://github.com/getgrit/stdlib/blob/main/.grit/patterns/p...
- ryanjshaw 3y agoConsider linking this and other complex examples in your readme, with before/after blocks to demonstrate the power of your software. I've spent the last 6 months building a static analyzer that mostly uses ASTs and couldn't understand what you were offering after skimming the readme until I saw this link. Now I'm thinking of how I could integrate something like this into my analyzer to simplify the implementation of some of the simpler rules.
- morgante 3y agoThanks for the feedback, we're going to link to more complex examples. Let me know if I can be helpful with integrating GritQL.
- herrington_d 3y agothanks for mentioning ast-grep!
- practicalrs 3y ago"a Rust CLI" - "C 96.5% Rust 1.4%" Not that I'm complaining ;) Is it a new way to market a C tool?
- morgante 3y agoThe C is all generated parsers, all our actual coding is done in Rust.
- mdaniel 3y agothis audience will also enjoy(?) the tree-sitter discussion going on here: https://news.ycombinator.com/item?id=39762495 https://news.ycombinator.com/item?id=39762495 there are very similar complaints about "yikes that thing just vomits a massive amount of loosey goosey C" so I hope you've not experienced the crashes they're talking about
- mdaniel 3y agoapologies if this should be a discussion/issue/whatever but: Do you envision going up against CodeQL and/or <https://www.jetbrains.com/help/qodana/about-qodana.html https://www.jetbrains.com/help/qodana/about-qodana.html> by making semantic information available to the ast nodes? OT1H, I can imagine it could be an overwhelming increase in project scope, but OTOH it could also truly lead to some stunning transformation patterns e.g. https://github.com/github/codeql/blob/v1.27.0/java/ql/examples/snippets/unusedlocalvar.ql https://github.com/github/codeql/blob/v1.27.0/java/ql/exampl... or even more "textual" semantics such as var foo = "hello".substring(1); // knowing "foo" is a String Since you already have GitLab on your docs page, if I could pay for CodeQL over on GL that would be a game changer IMHO
- morgante 3y agoOne of the specific design goals we had was to make sure that the query engine isn't tightly coupled to AST nodes. We started with the AST, since that's ultimately the foundation for code but the goal is to add more graphs on top. So you could do a query like this to find all cases where a function called `unsafe` is queried with an unknown value: `unsafe($val)` where $val <: type `unknown` Is your interest in CodeQL primarily for security scans?
- mdaniel 3y agosecurity is up there, but from reading the examples in CodeQL it just seemed like it would be possible to express some truly great versions of "don't do that" rules in it. I am a total JetBrains fanboi, and their introspections are world-class, but getting Qodana to run to completion before the heat death of the universe has proven to require more glucose than I have to offer it. Thus, I'm always interested in alternate implementations, even though I am acutely aware of the computational complexity of what I'm asking I recalled another link I wish I had included in my question from the SourceGraph folks https://github.com/sourcegraph/scip#scip-code-intelligence-protocol https://github.com/sourcegraph/scip#scip-code-intelligence-p... which started out life as "Language Server Indexing Protocol" and seems to solve some similar project-wide introspection questions but TBH since their rug pull I've been a lot less willing to hitch my wagon to their train
- herrington_d 3y agoCongrats from ast-grep team! Nice job Morgante! GritQL shares a lot of common ground with ast-grep, which is also a code rewriter based on tree-sitter. Interested folks may wonder about their difference. The key difference is that their surface APIs are quite different. GritQL is more like a DSL (or SQL). while ast-grep is more like pattern language + embedded configuration in YAML. Check it out here if you are interested https://github.com/ast-grep/ast-grep https://github.com/ast-grep/ast-grep
- snthpy 3y agoThis is great. Thanks for the link. I'm just starting to learn about tree-sitter but my understanding is that it can also handle nested syntax, for example code in documentation html pages. Does sg support this a well? What I would like to do is use sg to update my project's code samples. Is that possible? What part of the docs should I look at for this?
- herrington_d 3y agoHi! This is Grit's release news so please allow me to only briefly answer your question. > nested syntax sg does not support it natively now and you need API to extract relevant text out. For update code, I recommend first read the quick start, pattern and rule essentials https://ast-grep.github.io/guide/quick-start.html https://ast-grep.github.io/guide/quick-start.html https://ast-grep.github.io/guide/pattern-syntax.html https://ast-grep.github.io/guide/pattern-syntax.html https://ast-grep.github.io/guide/rule-config.html https://ast-grep.github.io/guide/rule-config.html
- Princess_98 3y ago[flagged]
- setr 3y agoAre either of these exposed conveniently to build on? One scenario I dealt with recently was replacing a pattern that required rudimentary type resolution to apply properly. Specifically, While($conn.hasrows()){ … $conn.commit() } Where $conn needed to be of a specific connection class to be a match. I ended up taking a Java parser, adding simple type resolution myself, and then the rest of the find-replace logic from there. I wouldn’t expect a tree-sitter based tool to deal with types, but if there’s a hook to add metadata to the parse tree and replace utilizing it, then this might have saved me some trouble
- MichaelMug 3y agoCan I use this to translate one language to another?
- morgante 3y agoIn theory, you could write a full transpiler in GritQL. In practice, it's not really the ideal tool for translating between languages. It does work well for cases where the differences are minor (ex. converting from JavaScript to TypeScript, or between different SQL dialects).
- ayewo 3y agoBut Grit should be able to handle translating a PHP dialect like Hacklang to a modern version of PHP? In your opinion, what tools would be ideal for general purpose transpilation between languages?
- morgante 3y ago> But Grit should be able to handle translating a PHP dialect like Hacklang to a modern version of PHP? Yeah, that should be doable (though GritQL doesn't have PHP support yet). > In your opinion, what tools would be ideal for general purpose transpilation between languages? Writing a transpiler is quite complicated and requires a lot of development to get right. I don't know of any high-quality generalized transpilers—you're best looking for tools specific to the language pair you're targeting.
- snthpy 3y agoI have the same question. I'm particularly interested in the SQL dialects case. Does the rewrite code also have to be syntactically correct or is that a simple text replacement because that's what I see as the main impediment; if I'm using the sqlite grammar to parse and produce postgresql output, the output won't conform to the grammar. Also how do I dynamically load custom tree-sitter grammars like in ast-grep?
- morgante 3y ago
- jcranmer 3y agoSounds like a cool tool. The ability to write rewrites as regular code fragments with metavariables feels like some ideas I've had in the past. However, it seems to me that your tool lacks the ability to do something like (in C/C++/etc. terms) "replace every use of S::method($arg1) with S::method2(arg1, nullptr, $arg1)", where you have to do some amount of name resolution to know that x->method is a call to S::method, which seems like it is somewhat less useful for several kinds of rewrites I'm interested in. (One of the main things I keep playing around with in my head is converting a C codebase into C++ by converting its ad-hoc vtable implementation into regular C++ virtual methods.)
- morgante 3y agoWe don't have semantic information yet (it's on the roadmap), but in many cases it's sufficient to get that information from the AST by looking where the variable was originally defined. For example: https://app.grit.io/studio?key=TteyWLNGZGWFIC9EOiCRi https://app.grit.io/studio?key=TteyWLNGZGWFIC9EOiCRi `$x.old($arg)` => `$x.new("method", _, $arg)` where { $program <: contains `$x = new Thing($_)` } --- Long term we want to incorporate more semantic analysis, but for quick rewrites this has been pretty effective for us.
- primitivesuave 3y agoThis is a great example of an open-core business model that I hope others will follow. Make something useful for everyone, and sell additional features in a hosted version (grit.io). I actually didn't even know there was a hosted version until I clicked the outgoing links from the documentation. I've experienced how much time these migrations take (and how error prone they are), especially once you have double-digit team sizes, so I definitely see the value proposition.
- morgante 3y agoThank you! We definitely want to give useful value to everyone. To expand more on our business model, we see GritQL as a powerful but also relatively low-level tool. It can speed up your migration, but writing the query and generating/testing the PRs still takes considerable time. If you'd prefer to have migrations handled entirely for you, you can pay us to tackle that work. In additional to providing the migration code, we generate pull requests and hook into your CI pipeline to fix downstream test failures. [0] [0] https://docs.grit.io/workflows/healing https://docs.grit.io/workflows/healing
- bilekas 3y agoThis looks really interesting, especially the GritQL side of things, I remember playing around with doing something simular on a much smaller and tailored use-case so I have an idea how much effort is involved here, but seems like a really great idea to introduce a query language to move things smoother. Great work, can't wait to play around.
- jkrubin 3y agoThis really and truly looks fantastic.
- lightandlight 3y agoThe first tool like this I saw was [comby](https://comby.dev/ https://comby.dev/). I remember it being a bit annoying to use for small-to-medium refactors. Next time I'll try `grit` and see how it does.
- snthpy 3y agoThis looks great! Thank you for building it. I'm looking forward to learning to use it. I'm just starting to learn about tree-sitter but my understanding is that it can also handle nested syntax, for example code in documentation html pages. Does grit support this a well? What I would like to do is use grit to update my project's code samples with new releases or syntax changes. Is that possible? What part of the docs should I look at for this?
- morgante 3y agoYes, Grit does support embedded languages. Right now we support Vue syntax: https://github.com/getgrit/gritql/blob/4ff989e8d665cc9ffa2cbb8a9829e8c7e1072f6d/crates/language/src/javascript.rs#L173-L184 https://github.com/getgrit/gritql/blob/4ff989e8d665cc9ffa2cb... We don't have support for HTML yet, but if you open an issue we're happy to explore it.
- CJefferson 3y agoNormally I wouldn't suggest this, but have you considered integrating with AIs? One tool I've been looking for (and I wonder if this is such a tool?) is something that can do things like rename variables in functions to "more sensible names". Now, it's not hard to get ChatGPT to read a function and suggest new names for variables, but if you ask it to also do the substitutions, there is an annoyingly high chance it break the function in the process.
- worldsayshi 3y agoI suspect that ChatGPT would be quite challenged trying to write code in any language that wasn't part of its training data. So a DSL that was just released would be a challenge. Worth trying out though. Perhaps rule creation can be turned into function calls so it wouldn't need to understand new syntax.
- tempaccount420 3y agoYou could fine tune a Mistral model, it should be good enough for this use case.
- morgante 3y agoStay tuned, we're working on some deep AI integrations.
- Walden98 3y ago[flagged]
- Princess_98 3y ago[flagged]
- codesnik 3y agovery powerful approach. While I'm still grepping for stuff, and for big refactorings and renames I sometime write perl onelineners, I'll give your tool a try in a brief moment before we all do refactors instructing some LLM.
- ThouYS 3y agogrep truly is the king of tools
- _f9cu 3y agoIn my words, build a tool that can trace the source of a variable's data, for every data variables in the source code. Lay them down in a timeline. Easy if it is single threaded language. Flatten the branches. Applications 1. minification. 2. cross programming language code generation. 3. Source tracing complements control flow tracing. Bringing in AST based solution will complicate the problem. What do you think ?
- jackbridger 3y agoLooks great, thanks for sharing.
- joshxyz 3y agowild how this one also uses tree-sitter.