dcreager

10 ms·

61.
by dcreager 5y ago
The generated parsers are implemented in C, and so currently the only way to use them is via FFI. (Unless you're writing your analysis tool itself in C!) The `tree-sitter generate` command also produces a full description of the gramma
62.
by dcreager 5y ago
It's not quite what you're asking, but it you want to limit your searches to a particular repo, you can just type that into the URL bar (and then bookmark that): https://cs.github.com/$OWNER/$REPO , just like
63.
by dcreager 5y ago
Unfortunately not yet — we have working stack rules for Python, but they use an older internal version of the graph DSL. We’re actively working on porting it to the open source tree-sitter-graph DSL and adding it to the public tree-sitter-p
64.
by dcreager 5y ago
Right now code nav on GitHub only works within a repository, and so every link you follow keeps you within the commit that you’re already viewing. As we move to cross-repo code nav, you’re right that it will be difficult to determine the ri
65.
by dcreager 5y ago
They're currently in a private repo, only because they use an old crusty implementation of the graph DSL. We're porting them over to the open-source graph DSL (tree-sitter/tree-sitter-graph), and we'll add them to the t
66.
by dcreager 5y ago
As mentioned elsewhere on this thread, stack graphs and Semantic were built by the same team (which I manage). Semantic is not abandoned, we've just been focusing on a different layer of our tech stack for the past year or so. http
67.
by dcreager 5y ago
You touch on an important difference between languages. A lot of existing tooling in this space focuses on static languages, exactly because there are existing tools that you can build on to get something implemented more quickly. But, ve
68.
by dcreager 5y ago
I talk a bit in my FOSDEM talk from last year how the “local editor” and “hosted service” versions of code navigation have enough real differences that it's not obviously best to reuse local editor solutions in a hosted service like Gi
69.
by dcreager 5y ago
Your `f(x)` example is similar to one of the harder examples I mention (but don't dive into) at the end of the blog post. You need a way to pass along information about `x` (such as its type) so that whatever you've constructed t
70.
by dcreager 5y ago
This early (and rough around the edges) design doc goes into more detail about scope stacks, and works through a couple of examples that rely on them: https://github.github.io/stack-graph-docs/
71.
by dcreager 5y ago
This relies on “scope stacks”, which are another piece that I didn’t really have a chance to discuss in either the blog post or Strange Loop talk. In brief, they allow you to “package up” context from one part of a file and “send it over” t
72.
by dcreager 5y ago
Haha also I like food :-)
73.
by dcreager 5y ago
Yes! Those are exactly the kinds of examples that mess up this kind of analysis. Anything where the structure of your program depends on arbitrary computation: https://twitter.com/dcreager/status/146765425251658
74.
by dcreager 5y ago
The most straightforward option would probably be to write an LSP wrapper around the stack graphs code. One of the people on my team wrote a very “contrib directory” version of that for internal testing, which lets us test our stack graph
75.
by dcreager 5y ago
Hard agree, this is very much a “standing on the shoulders of giants” situation. Eelco's group has done amazing work on scope graphs, and we would not have been able to make this without that work as a foundation.
76.
by dcreager 5y ago
Thanks very much, I appreciate the kind words!
77.
by dcreager 5y ago
Ooh that's a good one. Right now, our lookups are only within the single repository. And so if you had two files that _could_ provide the same (fully qualified) symbol, we aren't doing any PYTHONPATH analysis to determine which
78.
by dcreager 5y ago
I go into some amount of detail in a talk I gave at last year's FOSDEM: https://dcreager.net/talks/2020-fosdem/ For LSP, the short version is that running separate sidecar services in production for every lan
79.
by dcreager 5y ago
It's not “easy”, but we've found that because it's based on a declarative DSL it's less effort than you might expect. And we're finding that there are common patterns that you use in your graph construction rules,
80.
by dcreager 5y ago
Great questions! The framework is based on some great existing academic work from Eelco Visser's group at TU Delft. Their framework is called “scope graphs”: https://pl.ewi.tudelft.nl/research/projects/scope
81.
by dcreager 5y ago
> And how to prune and simplify CST to AST at scale (supporting many languages)? We're not doing any pruning or CST→AST translation, we just operate directly on the CST. With the new graph DSL you should be able to implement someth
82.
by dcreager 5y ago
> And, probably not closely related to stack graph, but about using tree-sitter for c/c++ understanding, how to handle the preprocessor? Ha yeah that's a good question. Some uses of the preprocessor won't be problematic
83.
by dcreager 5y ago
> how can it guarantee the python shadowing rule? it doesn't seem to encode any order preference. does the code traverse the source file in the reverse order basically? That snippet of graph DSL does not show the precedences being a
84.
by dcreager 5y ago
It's from the same team (which I am the manager of), but it's not using that same codebase. In Semantic, you would have to write Haskell code to add support for a new language, and we've found that the declarative DSLs that
85.
by dcreager 5y ago
> For production, is there a good database system that can index this graph structure? For awhile, we were storing this in a (very large) MySQL database, sharded with Vitess. The sharding behavior worked great (since repo ID gives you a
86.
by dcreager 5y ago
OP author here. I also gave a talk about this at Strange Loop back in October if folks want to watch/listen instead of read: https://dcreager.net/talks/2021-strange-loop/
87.
by dcreager 5y ago
We’ve been working on a framework called “stack graphs” that lets us extract exactly this kind of information without having to build anything. More details in my Strange Loop talk from October: https://dcreager.net/talks&#x
88.
by dcreager 5y ago
And also note that if you often need to add this kind of qualifier to many searches, you can create a "custom scope" that includes it for you transparently.
89.
by dcreager 5y ago
It’s all Docker images under the covers, so our devs also have the option of downloading and running that image locally instead of in a Codespaces VM. That was always possible before, of course, but Codespaces gave us the impetus to invest
90.
by dcreager 5y ago
Yes, I use neovim myself! Codespaces looks for a dotfiles repo under your user account, which you can use to install any personal customizations. Mine, which installs a bunch of neovim plugins and configs, is here: https://github

More ›