4 ms·

Anyone knows, why github code navigation, which is based on tree-sitter [1], supports only subset of languages for which tree-sitter has parsers [2]? For exampl

Anyone knows, why github code navigation, which is based on tree-sitter [1], supports only subset of languages for which tree-sitter has parsers [2]? For example, Common Lisp is not supported. I asked github [3], but they are silent. 1 - the "search based" navigation is based on tree sitter https://docs.github.com/en/repositories/working-with-files/using-files/navigating-code-on-github https://docs.github.com/en/repositories/working-with-files/u... 2 - the list of parsers in the official docs: https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/ 3 - https://github.com/orgs/community/discussions/55704 https://github.com/orgs/community/discussions/55704

kfir 3y ago
I believe this paper has an answer to your question - https://queue.acm.org/detail.cfm?id=3487022 https://queue.acm.org/detail.cfm?id=3487022
- avodonosov 3y ago
  Thank you very much for the link. I don't see an answer in there, though. They use tagging (as described also here: https://tree-sitter.github.io/tree-sitter/code-navigation-systems https://tree-sitter.github.io/tree-sitter/code-navigation-sy...). From both docs I assume `tree-sitter tags` works out of box for any language that has a parser. (Since neither doc instructs to save a custom config for tag extraction query, I assume every language plugin provides tagging queries).
awayto 3y ago
I worked with tree sitter a bit for a small project. It seems to be the case that, while it covers a large set of languages, the actual implementation of each subset doesn't follow the exact same underlying API. And it follows that there is no universal AST parser as far as I can tell. Languages have many types of atomic parts to such an extent that unifying them is a tremendously complex task.
- avodonosov 3y ago
  Have you used the `tree-sitter tags` command?
  - awayto 3y ago
    Unfortunately not. My stint of use was focused around dynamically loading in any given tree-sitter-langaugeName package via docker and then parsing any given file type in a runtime environment, in order to coalesce differing languages' parts into a unified metadata structure. So, I wasn't trying to identify named things, like the tags command might seem to do, but more generic parts of the langauges like function, var, import/export statements, etc. I ultimately made my own walk functionality, but pulling out those atomic parts per language is still a very heavy task (basically need a mini lexer for each lang), even though TS seems to provide all the info to do the task.
euiq 3y ago
While we're talking about this, I've been quite disappointed by the new GitHub file browser: the new features look neat, but the core file browsing experience is significantly worse for me because the new UI is much more complicated and every action seems to do way more work. I couldn't immediately find a way to opt out; I wouldn't mind the new file browser so much if I could drop into it on demand instead of having it forced down my throat.
- throwaway290 3y ago
  It was an optional opt-in labs feature that I tried for an hour then turned off, but now they force it. The new UI is more busy for no good reason. Sidebar makes sense in a local IDE, but over web every time you expand a subtree it takes ages anyway. And because it merely exists it is distracting and makes pages load slower...
- gurjeet 3y ago
  Click on your profile in the top-right, and then in the resulting menu, click on 'Feature preview'. You might be able to disable it from there.
  - euiq 3y ago
    It’s not there anymore, presumably because it graduated from beta earlier this month: <https://github.blog/changelog/2023-05-08-the-new-code-search-and-code-view-is-now-generally-available/ https://github.blog/changelog/2023-05-08-the-new-code-search...>
- eyelidlessness 3y ago
  I’ve had few (some, but minimal) problems with the new file tree browser, but the actual individual file view has become a minefield. It hijacks find in page, presumably to enhance it with semantic relations, but it breaks frequently when navigating. Previous searches (cmd-G) will fall back to the browser’s native search. If you’re lucky, you searched the same term in both, and if you’re luckier still the search term is in the top N lines of code (because the code itself is lazy loaded I guess?). Most of the time I’m not lucky and I have to start over each time I view a different file. Even though it feels more like a “single page app” than it previously did, I lose all of that state every time I do anything. I get what they’re trying to do, and in some ways it’s clearly beta growing pains, but in other ways it’s clear they’re fighting the browser environment in ways that don’t and probably shouldn’t work.
cstrahan 3y ago
I believe they use semantic (Haskell program that uses tree-sitter) for navigation: https://github.com/github/semantic https://github.com/github/semantic So the answer may be that semantic does not yet have support for the language in question.
- ghuntley 3y ago
  correct. adding support for new language is a matter of opening up a pull-request with the appropriate gluecode. https://github.com/github/semantic/blob/793a876ae45d38a6bd1744e137db71d148081bf1/semantic-rust/src/Language/Rust/AST.hs#L30 https://github.com/github/semantic/blob/793a876ae45d38a6bd17...
  - avodonosov 3y ago
    It would be more realistic to think of pull requests if github allowed per-repo configs with custom parsing scripts, similar to gitLab LSIF support - https://docs.gitlab.com/ee/user/project/code_intelligence.html https://docs.gitlab.com/ee/user/project/code_intelligence.ht.... Otherwise, how can a contrioutor be sure that his pull request really produces convenient code navigation in github UI. There is no way to test it.
  - avodonosov 3y ago
    The Semantic seems not used anymore: > During this time, the team migrated from a Semantic-based tagging service to one that operated entirely with (newly developed) Tree-sitter queries. Though Semantic performed well, using the Tree-sitter query language allowed faster iteration and avoided the operational overhead of a program-analysis framework. https://queue.acm.org/detail.cfm?id=3487022 https://queue.acm.org/detail.cfm?id=3487022
avodonosov 3y ago
Update: https://github.com/orgs/community/discussions/55704#discussioncomment-6025454 https://github.com/orgs/community/discussions/55704#discussi...
ghuntley 3y ago
ah, easy. it's because support has not been added into https://github.com/github/semantic https://github.com/github/semantic which is the tech that powers the GitHub UI. Adding support is pretty easy/mainly glue code [1] that imports the tree sitter API. [1] https://github.com/github/semantic/blob/793a876ae45d38a6bd1744e137db71d148081bf1/semantic-rust/src/Language/Rust/AST.hs#L30 https://github.com/github/semantic/blob/793a876ae45d38a6bd17...

kfir 3y ago

I believe this paper has an answer to your question - https://queue.acm.org/detail.cfm?id=3487022 https://queue.acm.org/detail.cfm?id=3487022

avodonosov 3y ago
Thank you very much for the link. I don't see an answer in there, though. They use tagging (as described also here: https://tree-sitter.github.io/tree-sitter/code-navigation-systems https://tree-sitter.github.io/tree-sitter/code-navigation-sy...). From both docs I assume `tree-sitter tags` works out of box for any language that has a parser. (Since neither doc instructs to save a custom config for tag extraction query, I assume every language plugin provides tagging queries).

awayto 3y ago

I worked with tree sitter a bit for a small project. It seems to be the case that, while it covers a large set of languages, the actual implementation of each subset doesn't follow the exact same underlying API. And it follows that there is no universal AST parser as far as I can tell. Languages have many types of atomic parts to such an extent that unifying them is a tremendously complex task.

avodonosov 3y ago
Have you used the `tree-sitter tags` command?
- awayto 3y ago
  Unfortunately not. My stint of use was focused around dynamically loading in any given tree-sitter-langaugeName package via docker and then parsing any given file type in a runtime environment, in order to coalesce differing languages' parts into a unified metadata structure. So, I wasn't trying to identify named things, like the tags command might seem to do, but more generic parts of the langauges like function, var, import/export statements, etc. I ultimately made my own walk functionality, but pulling out those atomic parts per language is still a very heavy task (basically need a mini lexer for each lang), even though TS seems to provide all the info to do the task.

euiq 3y ago

While we're talking about this, I've been quite disappointed by the new GitHub file browser: the new features look neat, but the core file browsing experience is significantly worse for me because the new UI is much more complicated and every action seems to do way more work. I couldn't immediately find a way to opt out; I wouldn't mind the new file browser so much if I could drop into it on demand instead of having it forced down my throat.

throwaway290 3y ago
It was an optional opt-in labs feature that I tried for an hour then turned off, but now they force it. The new UI is more busy for no good reason. Sidebar makes sense in a local IDE, but over web every time you expand a subtree it takes ages anyway. And because it merely exists it is distracting and makes pages load slower...
gurjeet 3y ago
Click on your profile in the top-right, and then in the resulting menu, click on 'Feature preview'. You might be able to disable it from there.
- euiq 3y ago
  It’s not there anymore, presumably because it graduated from beta earlier this month: <https://github.blog/changelog/2023-05-08-the-new-code-search-and-code-view-is-now-generally-available/ https://github.blog/changelog/2023-05-08-the-new-code-search...>
eyelidlessness 3y ago
I’ve had few (some, but minimal) problems with the new file tree browser, but the actual individual file view has become a minefield. It hijacks find in page, presumably to enhance it with semantic relations, but it breaks frequently when navigating. Previous searches (cmd-G) will fall back to the browser’s native search. If you’re lucky, you searched the same term in both, and if you’re luckier still the search term is in the top N lines of code (because the code itself is lazy loaded I guess?). Most of the time I’m not lucky and I have to start over each time I view a different file. Even though it feels more like a “single page app” than it previously did, I lose all of that state every time I do anything. I get what they’re trying to do, and in some ways it’s clearly beta growing pains, but in other ways it’s clear they’re fighting the browser environment in ways that don’t and probably shouldn’t work.

cstrahan 3y ago

I believe they use semantic (Haskell program that uses tree-sitter) for navigation: https://github.com/github/semantic https://github.com/github/semantic So the answer may be that semantic does not yet have support for the language in question.

ghuntley 3y ago
correct. adding support for new language is a matter of opening up a pull-request with the appropriate gluecode. https://github.com/github/semantic/blob/793a876ae45d38a6bd1744e137db71d148081bf1/semantic-rust/src/Language/Rust/AST.hs#L30 https://github.com/github/semantic/blob/793a876ae45d38a6bd17...
- avodonosov 3y ago
  It would be more realistic to think of pull requests if github allowed per-repo configs with custom parsing scripts, similar to gitLab LSIF support - https://docs.gitlab.com/ee/user/project/code_intelligence.html https://docs.gitlab.com/ee/user/project/code_intelligence.ht.... Otherwise, how can a contrioutor be sure that his pull request really produces convenient code navigation in github UI. There is no way to test it.
- avodonosov 3y ago
  The Semantic seems not used anymore: > During this time, the team migrated from a Semantic-based tagging service to one that operated entirely with (newly developed) Tree-sitter queries. Though Semantic performed well, using the Tree-sitter query language allowed faster iteration and avoided the operational overhead of a program-analysis framework. https://queue.acm.org/detail.cfm?id=3487022 https://queue.acm.org/detail.cfm?id=3487022

avodonosov 3y ago

Update: https://github.com/orgs/community/discussions/55704#discussioncomment-6025454 https://github.com/orgs/community/discussions/55704#discussi...

ghuntley 3y ago

ah, easy. it's because support has not been added into https://github.com/github/semantic https://github.com/github/semantic which is the tech that powers the GitHub UI. Adding support is pretty easy/mainly glue code [1] that imports the tree sitter API. [1] https://github.com/github/semantic/blob/793a876ae45d38a6bd1744e137db71d148081bf1/semantic-rust/src/Language/Rust/AST.hs#L30 https://github.com/github/semantic/blob/793a876ae45d38a6bd17...