5 ms·

There is at least one compelling argument for making new languages easy for machines to parse: it makes it easier to write good supporting tools. We don’t just

There is at least one compelling argument for making new languages easy for machines to parse: it makes it easier to write good supporting tools. We don’t just use a compiler or interpreter any more. We also use editors and refactoring tools and debuggers and profilers and style checkers and source formatters and static safety analysers and diff tools and… If the developers of these tools don’t have to worry so much about the mechanics of parsing the source, that leaves more resources to spend on making each tool useful. To some extent you could achieve the same benefit by making a library available to parse the source and return some sort of annotated AST or similar data structure. However, unless you can provide an easy way to call that library from every language someone might want to use to write a tool, avoiding unnecessarily complicated parsing still seems advantageous.

wenc 6y ago
As you alluded to in your 3rd paragraph, that is exactly the problem that "language servers" [1] aim to solve. The API is JSON-RPC based, which is callable from any tool. If every modern language shipped with a canonical language server, machine parseability becomes a non-issue since the parser would be the same as that of the compiler's. For instance, VS Code already uses language servers to support autocomplete/refactoring for languages like Python and they're pretty zippy (I used to have concerns about speed, but in practice it's a non-issue). Those same language servers are also supported in Vim, emacs, etc. The editor itself doesn't need to know anything about the underlying language. And it looks like a Rust language server exists: https://rls.booyaa.wtf/ https://rls.booyaa.wtf/ In fact, to take the idea further, the goal of the Roslyn [2] project is to expose APIs to the compiler itself in order to provide services to external dev tooling. Imagine a third-party generic debugger being able to tap into compiler or runtime internals and provide services around them it without explicitly knowing anything about the underlying language. [1] https://en.wikipedia.org/wiki/Language_Server_Protocol https://en.wikipedia.org/wiki/Language_Server_Protocol [2] https://en.wikipedia.org/wiki/Roslyn_(compiler)#Architecture https://en.wikipedia.org/wiki/Roslyn_(compiler)#Architecture
- steveklabnik 6y ago
  > And it looks like a Rust language server exists: There are two, actually. That one, and https://github.com/rust-analyzer/rust-analyzer https://github.com/rust-analyzer/rust-analyzer which is quickly supplanting it. The intention is that the compiler will eventually be a language server. Someday.
- Chris_Newton 6y ago
  It probably won’t surprise you to learn that I had LSP in mind when writing my previous comment. :-) I think there is a wider issue here than the (still very useful) scope of LSP, though. If you are building a tool that depends heavily on the semantics of the source language, beyond common operations like “go to definition”, you might need more than a standardised language server can provide, and then you’re back to the position I described before.
  - wenc 6y ago
    I had the sense it was on the tip of your tongue since the way you described it matched to a T, but I decided to flesh it out for folks who hadn't heard of LSPs. :) You're right, at this moment, the LSP is circumscribed in that it doesn't expose the AST (even though it could) which means one is limited to the capabilities it does expose. The claim is that the goal is parity across languages and exposing the ASTs is contrary to this [1]. I'm hoping this philosophy is challenged. In the mean time one might have to rely on languages themselves exposing their ASTs, eg. Python is able to expose its ast (via the ast module). So yes, right now one wouldn't be able to write say an IntelliJ IDEA type IDE based off a language server alone -- one wouldd have to be able to create the AST oneself. [1] https://github.com/Microsoft/language-server-protocol/issues/258 https://github.com/Microsoft/language-server-protocol/issues...

wenc 6y ago

As you alluded to in your 3rd paragraph, that is exactly the problem that "language servers" [1] aim to solve. The API is JSON-RPC based, which is callable from any tool. If every modern language shipped with a canonical language server, machine parseability becomes a non-issue since the parser would be the same as that of the compiler's. For instance, VS Code already uses language servers to support autocomplete/refactoring for languages like Python and they're pretty zippy (I used to have concerns about speed, but in practice it's a non-issue). Those same language servers are also supported in Vim, emacs, etc. The editor itself doesn't need to know anything about the underlying language. And it looks like a Rust language server exists: https://rls.booyaa.wtf/ https://rls.booyaa.wtf/ In fact, to take the idea further, the goal of the Roslyn [2] project is to expose APIs to the compiler itself in order to provide services to external dev tooling. Imagine a third-party generic debugger being able to tap into compiler or runtime internals and provide services around them it without explicitly knowing anything about the underlying language. [1] https://en.wikipedia.org/wiki/Language_Server_Protocol https://en.wikipedia.org/wiki/Language_Server_Protocol [2] https://en.wikipedia.org/wiki/Roslyn_(compiler)#Architecture https://en.wikipedia.org/wiki/Roslyn_(compiler)#Architecture

steveklabnik 6y ago
> And it looks like a Rust language server exists: There are two, actually. That one, and https://github.com/rust-analyzer/rust-analyzer https://github.com/rust-analyzer/rust-analyzer which is quickly supplanting it. The intention is that the compiler will eventually be a language server. Someday.
Chris_Newton 6y ago
It probably won’t surprise you to learn that I had LSP in mind when writing my previous comment. :-) I think there is a wider issue here than the (still very useful) scope of LSP, though. If you are building a tool that depends heavily on the semantics of the source language, beyond common operations like “go to definition”, you might need more than a standardised language server can provide, and then you’re back to the position I described before.
- wenc 6y ago
  I had the sense it was on the tip of your tongue since the way you described it matched to a T, but I decided to flesh it out for folks who hadn't heard of LSPs. :) You're right, at this moment, the LSP is circumscribed in that it doesn't expose the AST (even though it could) which means one is limited to the capabilities it does expose. The claim is that the goal is parity across languages and exposing the ASTs is contrary to this [1]. I'm hoping this philosophy is challenged. In the mean time one might have to rely on languages themselves exposing their ASTs, eg. Python is able to expose its ast (via the ast module). So yes, right now one wouldn't be able to write say an IntelliJ IDEA type IDE based off a language server alone -- one wouldd have to be able to create the AST oneself. [1] https://github.com/Microsoft/language-server-protocol/issues/258 https://github.com/Microsoft/language-server-protocol/issues...