7 ms·

Building a language server for Muon

101 points by nickmqb 7y ago

jimws 7y ago
Lisp had this kind of interactive programming support for a long time using SLIME and Emacs. Why isn't this a more common feature in the more recent languages?
- susam 7y ago
  > Lisp had this kind of interactive programming support for a long time using SLIME and Emacs. For those like me who have Vim burnt into their fingers and brain, there is Slimv[1] for Vim. Another alternative is Vlime[2] for Vim. Both these plugins are based on the same client-server architecture that SLIME is based on. In fact, these plugins rely on Swank TCP server (the same thing that SLIME also relies on). Swank receives SLIME commands from Slimv or Vlime and executes them. Of these two Vim plugins, I prefer Slimv personally because apart from supporting Common Lisp, it supports MIT Scheme and Clojure too. Vlime supports Common Lisp only at this time. [1]: https://github.com/kovisoft/slimv https://github.com/kovisoft/slimv [2]: https://github.com/l04m33/vlime https://github.com/l04m33/vlime
  - dunefox 7y ago
    Or Spacemacs with Slime.
- giancarlostoro 7y ago
  For that matter, VB, C#, Java and even Delphi all had a RAD approach to developing UIs. Why don't more modern languages work on even a basic UI as part of their standard libraries. What kind of system doesn't have a UI these days? Outside of servers obviously. The closest Go and Rust (modern languages...) are getting to a UI seems to be WebAssembly, which for now points towards an Electron UI solution. I'm not super familiar with SLIME, but that's the goals of LangServ is to consolidate everything into a known protocol. There's also a spec for debugging. Oddly enough the Wikipedia page for SLIME doesn't seem to mention Racket[0], but it does mention Clojure. [0]: https://en.wikipedia.org/wiki/SLIME https://en.wikipedia.org/wiki/SLIME
  - gmueckl 7y ago
    A flexible UI library that is up to modern expectations is complex and hard. It is even harder if it is supposed to be cross platform. So the only realistic option is to bind to a good existing library. However, these are all written in C++ and hard to bind to anything else. Gtk isn't sufficiently cross-platform to be attractive. In the end, this is a project that is at least as much effort a creating a programming language.
    - oaiey 7y ago
      And here are UWP Apps. They are based on a vNext edition of COM and use language projections to enable a OO-style usage in C#, JavaScript, C++. Actually works. In the end you work in the languages with these objects like any other object in that language. It is possible, just require major engineering smartness (like writing and improving compilers, code generation, etc).
      - giancarlostoro 7y ago
        Hell, it doesn't even have to be ultra advanced. Something close enough to Tk (not GTK, but Tcl/Tk which gets used by Python and a few other languages) would be fine in my book. It is simple, and a start, and allows the community to derive from a working solution to form amazing alternatives. In fact... Racket has their own UI approach, but I'm not sure if it's just GTK under the hood or what, but it works cross platform.
      - gmueckl 7y ago
        You're pretty much making my point for me, aren't you? UWP is a single platform interface only and the development of the bindings has taken a tremendous amount of work, as you say yourself. Projects like sip and PySide show how much effort it takes to make Qt bindings form Python. And all of this discussion just the bindings, not the actual widget libraries themselves and the effort that goes into those.
        oaiey 7y ago
- oaiey 7y ago
  There are two aspects here: Interactive Development is incredible difficult. There is a talk of Anders Heijlsberg about that compiler building is no longer the same than 10 years ago. When you code, you need sub-100-ms reactions, the parser needs to be happy parsing incomplete, typically errorous code and then you need a code analysis engine dealing with that half-baked AST. Also traditional compilers are a strict pipeline and not a loop etc. Microsoft had to rewrite the C# compiler for that. The PHP LSP had to create an own parser engine because no suitable was available. It is essentially a lecture for future language creator: If you write a compiler, do not only built it for binary emitting but also for editors and analyzers. And yes, the smart-asses of the 70/80s (aka LISP and Smalltalk) thought about that long time before anyone else. But hey, today's landscape is a bit more complex (think about CSS styles in a HTML embedded in a TypeScript enriched JavaScript JSX file). The second aspect: LSP is great because it was a cross-vendor, cross-language initiative which had in its first year already two dozen languages on board in addition to half a dozen editors (VS Code, Emacs, Atom, VIM, ...). Emacs plugins could never deliver that because they were bound to Emacs.
  - TeMPOraL 7y ago
    RE the second aspect: that's why I'm a fan of LSP and the related debugging server protocol. There's no way Emacs can have native IDE-like support for all the languages out there, and the small developer base makes it hard to compete with IDEs for popular languages, that have commercial interests backing them. LSP promises to level the playing field - it only takes one good language server to make IDE features suddenly available not just for IDEs, but also Emacs and Vim and others.
- laumars 7y ago
  It's actually really common but the problem was that, until recently, it was an IDE specific feature and some of the best IDEs weren't free. What this article is about is the new(ish) open protocol originally developed for Visual Studio Code that is being adopted by quite a few other editors. The language server protocol (LSP) enables developers to write the kind of tools you're describing and have them support any editor (which has LSP support) on any OS. It's pretty cool. However it's also a fairly involved process writing one so many of the smaller languages haven't yet had the bandwidth to writing their own language servers.
- barrkel 7y ago
  The problem is much easier (at least 10x easier) for a dynamic language with a REPL or playground environment (scratch in Emacs) than it is with a static language, or providing stateless insight based on cursor location rather than the set of forms you've evaluated. With a REPL or playground, you've got all the libraries loaded, all the function definitions in a map in your interpreter, and if you have a partial identifier you can simply do a prefix search in the interpreter. The interpreter is live; define a new function, and it's added to the map, so it's available for subsequent completions. If the function definition is wrong, hey, it's a REPL, next line try again please. In a stateless situation, or for a static language, you need to build that map. Ideally you use the compiler's front end, but then the front end needs to be hardened to recover properly from errors. It needs to be told about the cursor location, and when it hits the cursor location in the middle of an incomplete symbol, it needs to figure out what's appropriate to complete at that point based on the stack of scopes available, and jump back out to the editor (or use coroutines, or pause its thread, or whatever) with what it found. It's simply more work. And if you can't use the front end, because the front end wasn't built for it, and you can't modify it, well you're going to have to build what is effectively a new front end to solve the problem.
UglyToad 7y ago
An interesting article. I was minded to write an Ask HN related to this. Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support? I get that all languages are just text at the end of the day but are people really holding the whole API surface and type arguments of standard libraries and 3rd party libraries in their head? Do they remember variable names across files and directories. I feel like my memory must be absolutely terrible because there's no way I could work in a language without decent autocomplete.
- codetrotter 7y ago
  > Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support? Probably because developing all of that stuff takes a lot of time still, even though it is probably easier now than it was before. Everything gets done eventually if it’s important, but it may take a while until something gets done even though it is important because there is so much else to do as well.
- oaiey 7y ago
  Because you either invest in language features or editor features. I agree but most languages are built by single persons or very small teams. They have to select their battles. The last languages developed with that in mind were in order: TypeScript (essentially only for editors), Kotlin (developed by an IDE company) and C# (developed in a HUGE team including a huge IDE team next office door).
  - Quenty 7y ago
    Notably both C# and TypeScript are developed in the same Microsoft division as Visual Studio and VSCode. :p
    - seanmcdirmid 7y ago
      Not only that, but many of the people are the same.
ktm5j 7y ago
I thought this article was about "building a language server" like the title says, but this is really just a grammar set for an existing language server.. unless I'm missing something? I realize that this is being picky about wording, but it's a little misleading.
- _bxg1 7y ago
  My impression was that it's glue code that wraps around an existing compiler and turns it into a language server. There are piles of resources out there already on writing compilers
- laumars 7y ago
  Is there any good resources on writing language servers? I've tried writing my own one but while LSP is well documented, it wasn't really beginner friendly (ie where do I start when writing one from scratch?).
  - nickmqb 7y ago
    I didn't look for any resources beyond the LSP spec and VS Code documentation, so I can't offer any recommendations here I'm afraid. That said, looking back I'm pretty happy with the choice of using VS Code as the initial editor to get things up and running. While not perfect (e.g. VS Code will silently ignore your JSON message if you miss a closing brace), the docs are pretty good and it offers a reasonable level of debugging functionality (such as an output window where you can see messages sent to the language server). The Muon VS Code extension [1] is pretty minimal in itself, and it's MIT licensed so feel free fork and use it as a starting point. [1] https://github.com/nickmqb/vscode-muon https://github.com/nickmqb/vscode-muon
    - laumars 7y ago
      Thank you. I may well do just that. (this is what I love about open source -- when it works well everyone is giving each other a helping leg up).
  - ahaferburg 7y ago
duckqlz 7y ago
Moun’s for loops make me angry.
- all2 7y ago
  Can you expand on this?
  - saagarjha 7y ago
    I’m also confused; Muon’s loop syntax and semantics looks nothing out of the ordinary except that it allows omitting expressions…
  - lioeters 7y ago
    From "Muon by example" [0], the for loops look pretty natural/intuitive subjectively. for i := 1; i < n { temp := a a += b b = temp } > `for` works mostly like in other imperative languages. For convenience, the 3rd term can be omitted; in that case, the index variable is incremented by 1 at the end of each iteration of the loop. The index variable is scoped to the loop body. The variable can be omitted if you want to use an existing index variable, e.g.: for ; i < n; i += 1 { ... } With arrays: for x in arr { sum += x } for x, i in arr { sum += x } for arr { sum += it } > A secondary loop variable may be specified which holds the index of the current element. If no loop variable is specified, the name `it` is used. [0] https://github.com/nickmqb/muon/blob/master/docs/muon_by_example.md https://github.com/nickmqb/muon/blob/master/docs/muon_by_exa...
ngcc_hk 7y ago
Good article. In fact the muon is very doc. Still, one of the curse when coping segment of python code is to handle those tab and white space. The muon help to have a feature (first line say tab or space). Sigh muon opt for that turn me off.
- nickmqb 7y ago
  I agree that Muon's current handling of whitespace is not ideal. I have plans to allow users to customize it. For now, you can use the //tab_size=N directive as documented here: https://github.com/nickmqb/muon/blob/master/docs/muon_by_example.md#significant-whitespace https://github.com/nickmqb/muon/blob/master/docs/muon_by_exa...

101 points by nickmqb 7y ago

Lisp had this kind of interactive programming support for a long time using SLIME and Emacs. Why isn't this a more common feature in the more recent languages?

susam 7y ago
> Lisp had this kind of interactive programming support for a long time using SLIME and Emacs. For those like me who have Vim burnt into their fingers and brain, there is Slimv[1] for Vim. Another alternative is Vlime[2] for Vim. Both these plugins are based on the same client-server architecture that SLIME is based on. In fact, these plugins rely on Swank TCP server (the same thing that SLIME also relies on). Swank receives SLIME commands from Slimv or Vlime and executes them. Of these two Vim plugins, I prefer Slimv personally because apart from supporting Common Lisp, it supports MIT Scheme and Clojure too. Vlime supports Common Lisp only at this time. [1]: https://github.com/kovisoft/slimv https://github.com/kovisoft/slimv [2]: https://github.com/l04m33/vlime https://github.com/l04m33/vlime
- dunefox 7y ago
  Or Spacemacs with Slime.
giancarlostoro 7y ago
For that matter, VB, C#, Java and even Delphi all had a RAD approach to developing UIs. Why don't more modern languages work on even a basic UI as part of their standard libraries. What kind of system doesn't have a UI these days? Outside of servers obviously. The closest Go and Rust (modern languages...) are getting to a UI seems to be WebAssembly, which for now points towards an Electron UI solution. I'm not super familiar with SLIME, but that's the goals of LangServ is to consolidate everything into a known protocol. There's also a spec for debugging. Oddly enough the Wikipedia page for SLIME doesn't seem to mention Racket[0], but it does mention Clojure. [0]: https://en.wikipedia.org/wiki/SLIME https://en.wikipedia.org/wiki/SLIME
- gmueckl 7y ago
  A flexible UI library that is up to modern expectations is complex and hard. It is even harder if it is supposed to be cross platform. So the only realistic option is to bind to a good existing library. However, these are all written in C++ and hard to bind to anything else. Gtk isn't sufficiently cross-platform to be attractive. In the end, this is a project that is at least as much effort a creating a programming language.
  - oaiey 7y ago
    And here are UWP Apps. They are based on a vNext edition of COM and use language projections to enable a OO-style usage in C#, JavaScript, C++. Actually works. In the end you work in the languages with these objects like any other object in that language. It is possible, just require major engineering smartness (like writing and improving compilers, code generation, etc).
    - giancarlostoro 7y ago
      Hell, it doesn't even have to be ultra advanced. Something close enough to Tk (not GTK, but Tcl/Tk which gets used by Python and a few other languages) would be fine in my book. It is simple, and a start, and allows the community to derive from a working solution to form amazing alternatives. In fact... Racket has their own UI approach, but I'm not sure if it's just GTK under the hood or what, but it works cross platform.
    - gmueckl 7y ago
      You're pretty much making my point for me, aren't you? UWP is a single platform interface only and the development of the bindings has taken a tremendous amount of work, as you say yourself. Projects like sip and PySide show how much effort it takes to make Qt bindings form Python. And all of this discussion just the bindings, not the actual widget libraries themselves and the effort that goes into those.
      - oaiey 7y ago
oaiey 7y ago
There are two aspects here: Interactive Development is incredible difficult. There is a talk of Anders Heijlsberg about that compiler building is no longer the same than 10 years ago. When you code, you need sub-100-ms reactions, the parser needs to be happy parsing incomplete, typically errorous code and then you need a code analysis engine dealing with that half-baked AST. Also traditional compilers are a strict pipeline and not a loop etc. Microsoft had to rewrite the C# compiler for that. The PHP LSP had to create an own parser engine because no suitable was available. It is essentially a lecture for future language creator: If you write a compiler, do not only built it for binary emitting but also for editors and analyzers. And yes, the smart-asses of the 70/80s (aka LISP and Smalltalk) thought about that long time before anyone else. But hey, today's landscape is a bit more complex (think about CSS styles in a HTML embedded in a TypeScript enriched JavaScript JSX file). The second aspect: LSP is great because it was a cross-vendor, cross-language initiative which had in its first year already two dozen languages on board in addition to half a dozen editors (VS Code, Emacs, Atom, VIM, ...). Emacs plugins could never deliver that because they were bound to Emacs.
- TeMPOraL 7y ago
  RE the second aspect: that's why I'm a fan of LSP and the related debugging server protocol. There's no way Emacs can have native IDE-like support for all the languages out there, and the small developer base makes it hard to compete with IDEs for popular languages, that have commercial interests backing them. LSP promises to level the playing field - it only takes one good language server to make IDE features suddenly available not just for IDEs, but also Emacs and Vim and others.
laumars 7y ago
It's actually really common but the problem was that, until recently, it was an IDE specific feature and some of the best IDEs weren't free. What this article is about is the new(ish) open protocol originally developed for Visual Studio Code that is being adopted by quite a few other editors. The language server protocol (LSP) enables developers to write the kind of tools you're describing and have them support any editor (which has LSP support) on any OS. It's pretty cool. However it's also a fairly involved process writing one so many of the smaller languages haven't yet had the bandwidth to writing their own language servers.
barrkel 7y ago
The problem is much easier (at least 10x easier) for a dynamic language with a REPL or playground environment (scratch in Emacs) than it is with a static language, or providing stateless insight based on cursor location rather than the set of forms you've evaluated. With a REPL or playground, you've got all the libraries loaded, all the function definitions in a map in your interpreter, and if you have a partial identifier you can simply do a prefix search in the interpreter. The interpreter is live; define a new function, and it's added to the map, so it's available for subsequent completions. If the function definition is wrong, hey, it's a REPL, next line try again please. In a stateless situation, or for a static language, you need to build that map. Ideally you use the compiler's front end, but then the front end needs to be hardened to recover properly from errors. It needs to be told about the cursor location, and when it hits the cursor location in the middle of an incomplete symbol, it needs to figure out what's appropriate to complete at that point based on the stack of scopes available, and jump back out to the editor (or use coroutines, or pause its thread, or whatever) with what it found. It's simply more work. And if you can't use the front end, because the front end wasn't built for it, and you can't modify it, well you're going to have to build what is effectively a new front end to solve the problem.

UglyToad 7y ago

An interesting article. I was minded to write an Ask HN related to this. Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support? I get that all languages are just text at the end of the day but are people really holding the whole API surface and type arguments of standard libraries and 3rd party libraries in their head? Do they remember variable names across files and directories. I feel like my memory must be absolutely terrible because there's no way I could work in a language without decent autocomplete.

codetrotter 7y ago
> Why do so many new languages launch without what I'd consider to be essential functionality for any language; autocomplete, good IDE integration, intellisense, auto refactoring and navigation support? Probably because developing all of that stuff takes a lot of time still, even though it is probably easier now than it was before. Everything gets done eventually if it’s important, but it may take a while until something gets done even though it is important because there is so much else to do as well.
oaiey 7y ago
Because you either invest in language features or editor features. I agree but most languages are built by single persons or very small teams. They have to select their battles. The last languages developed with that in mind were in order: TypeScript (essentially only for editors), Kotlin (developed by an IDE company) and C# (developed in a HUGE team including a huge IDE team next office door).
- Quenty 7y ago
  Notably both C# and TypeScript are developed in the same Microsoft division as Visual Studio and VSCode. :p
  - seanmcdirmid 7y ago
    Not only that, but many of the people are the same.

ktm5j 7y ago

I thought this article was about "building a language server" like the title says, but this is really just a grammar set for an existing language server.. unless I'm missing something? I realize that this is being picky about wording, but it's a little misleading.

_bxg1 7y ago
My impression was that it's glue code that wraps around an existing compiler and turns it into a language server. There are piles of resources out there already on writing compilers
laumars 7y ago
Is there any good resources on writing language servers? I've tried writing my own one but while LSP is well documented, it wasn't really beginner friendly (ie where do I start when writing one from scratch?).
- nickmqb 7y ago
  I didn't look for any resources beyond the LSP spec and VS Code documentation, so I can't offer any recommendations here I'm afraid. That said, looking back I'm pretty happy with the choice of using VS Code as the initial editor to get things up and running. While not perfect (e.g. VS Code will silently ignore your JSON message if you miss a closing brace), the docs are pretty good and it offers a reasonable level of debugging functionality (such as an output window where you can see messages sent to the language server). The Muon VS Code extension [1] is pretty minimal in itself, and it's MIT licensed so feel free fork and use it as a starting point. [1] https://github.com/nickmqb/vscode-muon https://github.com/nickmqb/vscode-muon
  - laumars 7y ago
    Thank you. I may well do just that. (this is what I love about open source -- when it works well everyone is giving each other a helping leg up).
- ahaferburg 7y ago

duckqlz 7y ago

Moun’s for loops make me angry.

all2 7y ago
Can you expand on this?
- saagarjha 7y ago
  I’m also confused; Muon’s loop syntax and semantics looks nothing out of the ordinary except that it allows omitting expressions…
- lioeters 7y ago
  From "Muon by example" [0], the for loops look pretty natural/intuitive subjectively. for i := 1; i < n { temp := a a += b b = temp } > `for` works mostly like in other imperative languages. For convenience, the 3rd term can be omitted; in that case, the index variable is incremented by 1 at the end of each iteration of the loop. The index variable is scoped to the loop body. The variable can be omitted if you want to use an existing index variable, e.g.: for ; i < n; i += 1 { ... } With arrays: for x in arr { sum += x } for x, i in arr { sum += x } for arr { sum += it } > A secondary loop variable may be specified which holds the index of the current element. If no loop variable is specified, the name `it` is used. [0] https://github.com/nickmqb/muon/blob/master/docs/muon_by_example.md https://github.com/nickmqb/muon/blob/master/docs/muon_by_exa...

ngcc_hk 7y ago

Good article. In fact the muon is very doc. Still, one of the curse when coping segment of python code is to handle those tab and white space. The muon help to have a feature (first line say tab or space). Sigh muon opt for that turn me off.

nickmqb 7y ago
I agree that Muon's current handling of whitespace is not ideal. I have plans to allow users to customize it. For now, you can use the //tab_size=N directive as documented here: https://github.com/nickmqb/muon/blob/master/docs/muon_by_example.md#significant-whitespace https://github.com/nickmqb/muon/blob/master/docs/muon_by_exa...