3 ms·

The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tok

by duped 5y ago

The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tokens" API. Language servers can choose to implement this however they want, and the API allows for incremental additions. I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specific grammars) to add syntax highlighting. That way it can be applied to any editor that supports LSP integration. Really the world would be a better place for IDEs and text editors if we could just get LSP more standardized, and VS Code's dominance is a decent place to start with it. I'm tired of relying on editor and IDE authors for language support.

da39a3ee 5y ago
I believe I've read in various places that the difficulty of doing syntax highlighting fast enough is precisely why it isn't done via LSP. Am I right in thinking that the Semantic Tokens part of the spec (currently) falls short of saying "this is for all your syntax-highlighting needs, please go ahead and implement full-blown syntax highlighters using it" I definitely agree with you that the more we can share between different editors/IDEs the better.
fcurts 5y ago
> The tokenization speed issue is already addressed by the language server protocol [...] It's not really addressed. The semantic tokens API is intended for semantic highlighting: > Semantic tokenization allows language servers to provide additional token information based on the language server's knowledge on how to resolve symbols in the context of a project. Abusing the semantic tokens API for syntactic highlighting is slow, unnecessarily complex (why do I need to implement a language server just to do syntactic highlighting?), and only a partial solution (still need a TM grammar, still don't get correct code folding, etc.).
tsar9x 5y ago
> I think this is a much better approach that baking tree sitter into VS Code they're implementing both, with tree sitter being 'dumb' version of LSP syntax highlighting: https://github.com/microsoft/vscode-anycode https://github.com/microsoft/vscode-anycode
- da39a3ee 5y ago
  Oh thanks for that link, that's really interesting. Can someone explain the paragraph below -- I thought "this is an invocation of a function named bar" was what we mean by semantic information: > All features are based on parse trees and there is no semantic information - that means there is no guarantee for correctness. Parse trees allow to identify declarations and usages, like "these lines define a function named foo" or "this is an invocation of a function named bar"
  - fcurts 5y ago
    It uses heuristics to derive semantic information from the parse tree without doing a full semantic analysis.
    - da39a3ee 5y ago
      Is "semantic analysis" in the sense you used it similar to "type checking"?
      - fcurts 5y ago
        Semantic analysis is a broader term that (for languages with a type checker) includes type checking. https://cs.lmu.edu/~ray/notes/semanticanalysis/ https://cs.lmu.edu/~ray/notes/semanticanalysis/
- fcurts 5y ago
  What's "dumb" about Anycode is its semantic features such as "go to definition", code completion, etc., not its syntax highlighting. Alas, Anycode is a separate project.
- eyelidlessness 5y ago
  I was curious about the implementation, and was surprised to find: - they’re importing query files, with a .scm extension, in TypeScript - they’re using esbuild to handle those imports It’s surprising to me that the Microsoft, who created and maintains TypeScript, is using an alternative TypeScript compiler… in part to query semantic information from TypeScript.

by duped 5y ago

da39a3ee 5y ago

I believe I've read in various places that the difficulty of doing syntax highlighting fast enough is precisely why it isn't done via LSP. Am I right in thinking that the Semantic Tokens part of the spec (currently) falls short of saying "this is for all your syntax-highlighting needs, please go ahead and implement full-blown syntax highlighters using it" I definitely agree with you that the more we can share between different editors/IDEs the better.

fcurts 5y ago

> The tokenization speed issue is already addressed by the language server protocol [...] It's not really addressed. The semantic tokens API is intended for semantic highlighting: > Semantic tokenization allows language servers to provide additional token information based on the language server's knowledge on how to resolve symbols in the context of a project. Abusing the semantic tokens API for syntactic highlighting is slow, unnecessarily complex (why do I need to implement a language server just to do syntactic highlighting?), and only a partial solution (still need a TM grammar, still don't get correct code folding, etc.).

tsar9x 5y ago

> I think this is a much better approach that baking tree sitter into VS Code they're implementing both, with tree sitter being 'dumb' version of LSP syntax highlighting: https://github.com/microsoft/vscode-anycode https://github.com/microsoft/vscode-anycode

da39a3ee 5y ago
Oh thanks for that link, that's really interesting. Can someone explain the paragraph below -- I thought "this is an invocation of a function named bar" was what we mean by semantic information: > All features are based on parse trees and there is no semantic information - that means there is no guarantee for correctness. Parse trees allow to identify declarations and usages, like "these lines define a function named foo" or "this is an invocation of a function named bar"
- fcurts 5y ago
  It uses heuristics to derive semantic information from the parse tree without doing a full semantic analysis.
  - da39a3ee 5y ago
    Is "semantic analysis" in the sense you used it similar to "type checking"?
    - fcurts 5y ago
      Semantic analysis is a broader term that (for languages with a type checker) includes type checking. https://cs.lmu.edu/~ray/notes/semanticanalysis/ https://cs.lmu.edu/~ray/notes/semanticanalysis/
fcurts 5y ago
What's "dumb" about Anycode is its semantic features such as "go to definition", code completion, etc., not its syntax highlighting. Alas, Anycode is a separate project.
eyelidlessness 5y ago
I was curious about the implementation, and was surprised to find: - they’re importing query files, with a .scm extension, in TypeScript - they’re using esbuild to handle those imports It’s surprising to me that the Microsoft, who created and maintains TypeScript, is using an alternative TypeScript compiler… in part to query semantic information from TypeScript.