5 ms·

The way a language server should work, imo, is it holds all the code and serves projections of the code on the fly to the editor. The user of the editor makes c

The way a language server should work, imo, is it holds all the code and serves projections of the code on the fly to the editor. The user of the editor makes changes to the code and commits it back, at which point the language server parses the code and integrates it into the codebase, pretty-printing it to files as necessary for source control. But, the file layout should be an implementation detail that the editor knows nothing about.

slimsag 3y ago
That's exactly how LSP is designed. The editor e.g. requests 'the user is hovering over code <here>, what should I show?' and the server responds with some Markdown text. Or 'the user wants to navigate to the thing under their cursor <here>, where should I go?' and the server responds with a file/line. Once you try to use LSP for anything not in that form.. it's not so fun.
- fiddlerwoaroof 3y ago
  No, this is the opposite of how LSP is designed: LSP assumes the editor is looking at a file and this results in a complicated sync dance between editor state and file state. I want the editor to look at temporary buffers served up from the language server and have the code be “written” by sending the temporary buffer back to the language server which handles writing it out to files itself. As far as I know, the only languages with something even vaguely like this are Pharo Smalltalk (with its git integration) and unison
  - fiddlerwoaroof 3y ago
    The advantage here is this enables some interesting workflows, like “show me this method and every definition that overrides it in a single view” or “show me this method and all the methods it calls directly”. So you can project the interesting part of your code base into a temporary editor buffer and then edit it and the language server takes care of persistence. Also, it ends the formatting wars because the in-repository format is disconnected from the user’s preferred format.
  - zarzavat 3y ago
    > I want the editor to look at temporary buffers served up from the language server and have the code be “written” by sending the temporary buffer back to the language server which handles writing it out to files itself. I don’t think any editor would want to accept this workflow. The biggest issue is that writing is reputation-critical, if the editor writes to the wrong place, or it fails to write when it should have written, then users lose their work which makes them very unhappy. So the editor has to take responsibility for doing the writing, and that is the user’s mental model when they save their work.
    - fiddlerwoaroof 3y ago
      I want a system more like Bank Python[1] or Unison[2] or Pharo Smalltalk[3] that gets us beyond the idea of "code in files". And I have plans to build this on top of Common Lisp + SLIME [1]: https://calpaterson.com/bank-python.html https://calpaterson.com/bank-python.html [2]: https://www.unison-lang.org https://www.unison-lang.org [3]: https://pharo.org https://pharo.org
      - pawelmurias 3y ago
        Loads of people absolutely hate having to use a weird custom IDE just to try a language. That's why all the smalltalk like things are doomed for failure unless they are enforced by a platform.
        fiddlerwoaroof 3y ago
        Yeah, this is the problem my proposal solves.
  - arp242 3y ago
    I don't use LSP for all the real-time stuff, but I do use it for other things like rename symbol, lookup documentation, jump to, etc. and just "the file you have open" isn't really enough for that in most cases. You need to know about the "file layout" or "complete project" for that sort of thing.
jenadine 3y ago
Imho it should be the other way around: the editor knows about file layout and everything, and the language server queries the editor when it needs contents for a specific file. Unfortunately the LSP don't have calls to query contents. The rationale is that only the editor knows what files are open and modified. Also we can imagine scenario where the editor and the language server are on different remote (eg GitHub codespace or the browser version of vscode. The language server could be a wasm build running where the editor is running, but the editor may access files in a remote server, or vice versa)
- fiddlerwoaroof 3y ago
  Well, a lot of this is that I think files are a bad way to represent code. You really want something more shaped like a database that allows for multiple views into the same code. Git would for serialization, but the source of truth should be inside the language server and either the files or the repository.
  - Reisen 3y ago
    This is something I have been wanting for nearly a decade. A lot of writing software isn't just implementing your logic and abstractions but actively thinking about how to organize code to the constraints of the filesystem. Having to actively model your modules around file paths, Rust for example tightly binding the use of `mod` to your layout. Refactoring is the same, a non-trivial amount of time on large projects when re-factoring is realising you need to re-organise some module hierarchy and that involves modifying the file system too. I really dislike this, instead of a fuzzy file finder I want a fuzzy function finder, where all functions are just kept in a database that I can pull into buffers at will. Where hierarchy is only based on the logical structure of your program and the filesystem ceases exist. "New Function" over "New File". You can get the "Fuzzy Function" finder part somewhat with LSP Symbols, but it doesn't get rid of the having to think about files. Unfortunately I don't think you can get this without first-class support by the language itself, and new languages getting critical adoption isn't a regular thing.
jesperlang 3y ago
Exactly! There is such a shift in paradigm that needs to happen here and the only project I know of that is moving in this direction is Unison. I don't want to edit a "file", I want to edit these two functions that exist in some module(s), why can't I just see those two? I constantly jump between many different languages and the cognitive load is noticeable: was it "!=", "/=" or "~="? Why am I writing/viewing ascii art when I'm coding? If I am most comfortable/fluent viewing python, why can't I view a javascript source as python? I think the remaining challenge is what sort of projections are the best/most useful? How do we manipulate these projections? I have played around with these ideas and made an AST viewer for the browser where you could configure exactly how a node was represented (using CSS) and navigation was done in block mode (node traversal) but I found it really hard to build an editing experience that felt smooth..
- rayuela 3y ago
  This browser AST viewer sounds pretty neat. Mind sharing a link to your repo of it?
- Hendrikto 3y ago
  > If I am most comfortable/fluent viewing python, why can't I view a javascript source as python? Because they are not isomorphic. At all. Even if you just consider the languages themselves, and ignore their ecosystems, which, in practice, you cannot.
  - spunker540 3y ago
    Some percent of python code I’ve written code be rewritten as JavaScript code at the function-level.
    - Conscat 3y ago
      Meta's "Transcoders" or whatever they changed the name to demonstrate that. However, if we want perfectly semantically equivalent functions, as soon as you add two numbers, then Python -> JS is impossible. The best we can do is approximately translate the behavior.
oaiey 3y ago
Last time I read the LSP specs/repos, there were 2 nuances regarding your statement: (1) the editor read the files, display then, and then sends changes to the LSP which then mirrors the file to compile, analyze, etc it. The file is never persisted in that stage to the file system (needs to because otherwise no syntax highlighting while editing. (2) there are conversations about that in both the LSP and Editor space about virtual file systems. Google "language server virtual file system". The core author of the LSP spec has written one issue very related to it.
- fiddlerwoaroof 3y ago
  (2) is interesting, it would be cool if LSP evolved a sort of “file system server” ability that could be integrated into things like TRAMP in emacs
layer8 3y ago
That sounds like it would make for bad latency while editing. And what about other files in the project, other than the files of the specific programming language? The IDE needs to understand their file layout anyway, and often there are dependencies to the layout and naming of the programming-language source files. And you want to do stuff like textual search across all project files. Effectively your LSP server would have to become a full-scale remote IDE.
NoGravitas 3y ago
That's basically a subset of image-based development, as seen in Smalltalk and Lisp since forever.
- fiddlerwoaroof 3y ago
  Yes, and I write a lot of lisp

slimsag 3y ago

That's exactly how LSP is designed. The editor e.g. requests 'the user is hovering over code <here>, what should I show?' and the server responds with some Markdown text. Or 'the user wants to navigate to the thing under their cursor <here>, where should I go?' and the server responds with a file/line. Once you try to use LSP for anything not in that form.. it's not so fun.

fiddlerwoaroof 3y ago
No, this is the opposite of how LSP is designed: LSP assumes the editor is looking at a file and this results in a complicated sync dance between editor state and file state. I want the editor to look at temporary buffers served up from the language server and have the code be “written” by sending the temporary buffer back to the language server which handles writing it out to files itself. As far as I know, the only languages with something even vaguely like this are Pharo Smalltalk (with its git integration) and unison
- fiddlerwoaroof 3y ago
  The advantage here is this enables some interesting workflows, like “show me this method and every definition that overrides it in a single view” or “show me this method and all the methods it calls directly”. So you can project the interesting part of your code base into a temporary editor buffer and then edit it and the language server takes care of persistence. Also, it ends the formatting wars because the in-repository format is disconnected from the user’s preferred format.
- zarzavat 3y ago
  > I want the editor to look at temporary buffers served up from the language server and have the code be “written” by sending the temporary buffer back to the language server which handles writing it out to files itself. I don’t think any editor would want to accept this workflow. The biggest issue is that writing is reputation-critical, if the editor writes to the wrong place, or it fails to write when it should have written, then users lose their work which makes them very unhappy. So the editor has to take responsibility for doing the writing, and that is the user’s mental model when they save their work.
  - fiddlerwoaroof 3y ago
    I want a system more like Bank Python[1] or Unison[2] or Pharo Smalltalk[3] that gets us beyond the idea of "code in files". And I have plans to build this on top of Common Lisp + SLIME [1]: https://calpaterson.com/bank-python.html https://calpaterson.com/bank-python.html [2]: https://www.unison-lang.org https://www.unison-lang.org [3]: https://pharo.org https://pharo.org
    - pawelmurias 3y ago
      Loads of people absolutely hate having to use a weird custom IDE just to try a language. That's why all the smalltalk like things are doomed for failure unless they are enforced by a platform.
      - fiddlerwoaroof 3y ago
        Yeah, this is the problem my proposal solves.
- arp242 3y ago
  I don't use LSP for all the real-time stuff, but I do use it for other things like rename symbol, lookup documentation, jump to, etc. and just "the file you have open" isn't really enough for that in most cases. You need to know about the "file layout" or "complete project" for that sort of thing.

jenadine 3y ago

Imho it should be the other way around: the editor knows about file layout and everything, and the language server queries the editor when it needs contents for a specific file. Unfortunately the LSP don't have calls to query contents. The rationale is that only the editor knows what files are open and modified. Also we can imagine scenario where the editor and the language server are on different remote (eg GitHub codespace or the browser version of vscode. The language server could be a wasm build running where the editor is running, but the editor may access files in a remote server, or vice versa)

fiddlerwoaroof 3y ago
Well, a lot of this is that I think files are a bad way to represent code. You really want something more shaped like a database that allows for multiple views into the same code. Git would for serialization, but the source of truth should be inside the language server and either the files or the repository.
- Reisen 3y ago
  This is something I have been wanting for nearly a decade. A lot of writing software isn't just implementing your logic and abstractions but actively thinking about how to organize code to the constraints of the filesystem. Having to actively model your modules around file paths, Rust for example tightly binding the use of `mod` to your layout. Refactoring is the same, a non-trivial amount of time on large projects when re-factoring is realising you need to re-organise some module hierarchy and that involves modifying the file system too. I really dislike this, instead of a fuzzy file finder I want a fuzzy function finder, where all functions are just kept in a database that I can pull into buffers at will. Where hierarchy is only based on the logical structure of your program and the filesystem ceases exist. "New Function" over "New File". You can get the "Fuzzy Function" finder part somewhat with LSP Symbols, but it doesn't get rid of the having to think about files. Unfortunately I don't think you can get this without first-class support by the language itself, and new languages getting critical adoption isn't a regular thing.

jesperlang 3y ago

Exactly! There is such a shift in paradigm that needs to happen here and the only project I know of that is moving in this direction is Unison. I don't want to edit a "file", I want to edit these two functions that exist in some module(s), why can't I just see those two? I constantly jump between many different languages and the cognitive load is noticeable: was it "!=", "/=" or "~="? Why am I writing/viewing ascii art when I'm coding? If I am most comfortable/fluent viewing python, why can't I view a javascript source as python? I think the remaining challenge is what sort of projections are the best/most useful? How do we manipulate these projections? I have played around with these ideas and made an AST viewer for the browser where you could configure exactly how a node was represented (using CSS) and navigation was done in block mode (node traversal) but I found it really hard to build an editing experience that felt smooth..

rayuela 3y ago
This browser AST viewer sounds pretty neat. Mind sharing a link to your repo of it?
Hendrikto 3y ago
> If I am most comfortable/fluent viewing python, why can't I view a javascript source as python? Because they are not isomorphic. At all. Even if you just consider the languages themselves, and ignore their ecosystems, which, in practice, you cannot.
- spunker540 3y ago
  Some percent of python code I’ve written code be rewritten as JavaScript code at the function-level.
  - Conscat 3y ago
    Meta's "Transcoders" or whatever they changed the name to demonstrate that. However, if we want perfectly semantically equivalent functions, as soon as you add two numbers, then Python -> JS is impossible. The best we can do is approximately translate the behavior.

oaiey 3y ago

Last time I read the LSP specs/repos, there were 2 nuances regarding your statement: (1) the editor read the files, display then, and then sends changes to the LSP which then mirrors the file to compile, analyze, etc it. The file is never persisted in that stage to the file system (needs to because otherwise no syntax highlighting while editing. (2) there are conversations about that in both the LSP and Editor space about virtual file systems. Google "language server virtual file system". The core author of the LSP spec has written one issue very related to it.

fiddlerwoaroof 3y ago
(2) is interesting, it would be cool if LSP evolved a sort of “file system server” ability that could be integrated into things like TRAMP in emacs

layer8 3y ago

That sounds like it would make for bad latency while editing. And what about other files in the project, other than the files of the specific programming language? The IDE needs to understand their file layout anyway, and often there are dependencies to the layout and naming of the programming-language source files. And you want to do stuff like textual search across all project files. Effectively your LSP server would have to become a full-scale remote IDE.

NoGravitas 3y ago

That's basically a subset of image-based development, as seen in Smalltalk and Lisp since forever.

fiddlerwoaroof 3y ago
Yes, and I write a lot of lisp