3 ms·

Enter hell: LSP assumes that it's the source of truth, but you still need to access the filesystem yourself, and do it in a synchronized way LSP is an example

Enter hell: LSP assumes that it's the source of truth, but you still need to access the filesystem yourself, and do it in a synchronized way LSP is an example of utterly horrid technical design. Stop letting Microsoft design protocols and APIs. They are so. bad. at. it.

duttish 7d ago
I've never looked into LSP, how would you design it?
- packetlost 7d ago
  Well for one I wouldn't design it so both the LSP and the editor need a synchronized view of the underlying file.
- mitxela 7d ago
  Start with synchronous function calls instead of JSON. Microsoft knows how to do that - they invented COM and OLE. Function calls enable whatever data sharing is necessary to maintain a coherent view. Imagine trying to do OLE with JSON - just wouldn't work. (Does OLE still exist?)
  - cyberax 7d ago
    Stupid idea. So the LSP crashes and/or goes into a runaway memory consumption loop. And your main application dies with it. Or what if you want, you know, to be able to use the same LSP from TWO different applications at the same time? Never mind issues with other managed runtimes not expecting to deal with something else in their address space.
  - cyberax 7d ago
    Moreover, it's not just stupid, it also does not actually solve _anything_. A synchronous function can also have obsolete indexing information if it races with the code updates.
    - mitxela 7d ago
      But it doesn't have to, because you call AboutToUpdateCode and the LSP doesn't return until it's safe, and then you update code and call DoneUpdatingCode.
      - cyberax 7d ago
        Yes, it does because the physical code update is asynchronous. Or you'd have to do a roundtrip for every little edit, resulting in unusably slow experience.
        mitxela 6d ago
        I don't know what you're talking about. What is "physical code"? A function call roundtrip costs nothing, that's the reason we are saying it should be function calls.
        cyberax 6d ago
        TCP roundtrip overhead over localhost costs approximately nothing. But running the LSP itself is slow. It's not instant, and you can't run it in a blocking way for each keypress.
        mitxela 6d ago
        A function call costs nanoseconds. You can sprinkle them literally everywhere. A JSON-formatted TCP round trip to localhost costs, I'd guess conservatively, at least ten microseconds, 1000x slower.
        cyberax 6d ago
        OK. Here's my simple implementation of an LSP: > func doLspStuff() { sleep(60000); } Now sprinkle them after each keypress. Do you see the issue now?
  - rspeele 7d ago
    I feel like MS actually learned their lesson with synchronously integrating the language intelligence into the IDE. Old versions of VS would hang or crash based on bugs in the language tooling trying to provide intellisense. You'd restart and it'd work fine till you hit some other weird edge case. Generally this settled to a level of rare-bugginess where you were happy enough with the advantages not to go back to Emacs/VIM, but still annoyed at the occasional restart needed. In no way does this mean LSP is a perfect solution, but anything synchronous would be a step backwards.
  - jcranmer 7d ago
    That doesn't really fix anything. The filesystem is fundamentally a racy shared data structure. If you made the entrypoint API synchronous, any half-way decent editor would shove the LSP queries to the synchronous API into a different thread, because "Do Not Block the UI Thread" is a fundamental principle of good UI programming. Once you get past that, JSON-over-TCP is just another kind of asynchronous RPC mechanism, one that has the advantage that you can build it in just about any language with out-of-the-box tools. Trying to make a plugin system or a COM or CORBA or OLE based system really cuts out the ability to build language servers in most languages, because you have to be able to build the code in just the right way.
    - mitxela 7d ago
      Big assumption that the editor, which is required to make calls synchronously, would make them on a different thread. Isn't correctness more important than avoiding plugin calls in the UI thread? The good thing about having an actual function interface is that you can tailor each one to its actual needs, instead of insisting high-overhead async is okay for everything. It's like GDI vs X11.
      - ablob 7d ago
        I don't know about you, but I do not want to block on every keystroke. Program analysis can take time, asynchronous feedback is a natural fit. None of the features I use from LSP should be synchronous. I can almost guarantee you that a synchronous API would yield a much more complicated design just because many operations can not be expected to reliably work within a few milliseconds. You want to rename something across multiple files? Well now you add file system overhead and blow straight past reliable frame timings. Good luck waiting for that operation to finish. Of course you could make the API beginRename, and queryRename, or whatever you fancy to see if the operation was successful, but now you're back to what you wanted to avoid: an asynchronous API. Do mind that the example is actually one of the better cases, as many things you might want to do with a codebase are actually more expensive. You will feel the hiccups from waiting in the UI thread and you will loathe the program for it.
  - pjmlp 7d ago
    Actually, COM is inspired by DCE/RPC and the initial versions had some similarities. Parallel to that, IBM had SOM on OS/2, which was even better allowing for metaclasses and proper class inheritance, it was the key mechanism between Smalltalk and C++ on OS/2, where Smalltalk enjoyed a role similar to .NET on Windows nowadays. OLE naturally still exists when using Office natively on Windows, other vendors seem to have forgotten about it. COM's role on Windows has grown since Vista, and the Windows team redid many of the Longhorn ideas originally implemented in .NET into COM/C++, with WinRT being an evolution of COM.
    - mamp 7d ago
      I've never heard of anyone calling SOM 'better' at anything this century. I came across it when it was the foundation for OpenDoc at Apple, your comment brought back many bad memories
      - pjmlp 7d ago
        Compared with COM's design, it was much better.
        mitxela 7d ago
        Classes and metaclasses sound like architecture astronomy here. COM is fundamentally just a standard ABI and a way to look up DLL paths in the registry. There are more layers on top of that, but that's the basics and all you need for a plugin architecture.
        pjmlp 7d ago
        That applies to SOM just as it does to COM. Interesting to complain about one and then ignore exactly the same boilerplate for the other. Leaving aside the fact that Microsoft's tooling for COM has already had multiple reboots between VB OCX, MFC, .NET Framework RCW/CCW, .NET ComWrappers, ATL, WRL, WIL, WinRT, each one with its share of astronomy. No one uses the bare bones vtbl and nothing else.
  - throwaway7356 7d ago
    > Start with synchronous function calls instead of JSON. LSP is a function call protocol. So that is already done?
- lenkite 7d ago
  Wouldn't a normalized protocol based on AST vocabulary and tree operations make more sense ?
  - looperhacks 6d ago
    That would require the editor to be able to build an AST - defeating the point of LSP, because now the editor needs to know every possible language again
    - lenkite 6d ago
      I meant the language server builds the normalized, lossless AST and gives it to the editor. Changes in the AST can be provided as events/diffs to the editor. The normalized, source-mapped AST is effectively the stable language protocol. (Possibly given in a columnar, database form via Apache Arrow, but thats impl specific). Was discussing alternative architectures with a collague since even the best language servers really suck on large projects.
- kuschku 7d ago
  One of the largest issues of LSP is that each language implementation needs to do everything separately, and that means each language will work differently. Imagine if all the editing tools in microsoft word were specific to the language you used, and if you mixed German and English, each had different tooling. Now if you mix Rust, HTML, JS, and CSS, they'll all have separate tooling, seperate "go to definition", and separate refactoring. Worst of all, none of them can see the definitions of the other ones. So what you'd actually want to do is parse each language into your AST, with proper annotations as to what is what, and have all the "go to definition", the UI rendering, highlighting, refactoring, etc all done generically by the IDE ontop. Which is much much closer to how Jetbrains IntelliJ does it, and why their tooling can handle "find usages" on an HTML element to find matching querySelector in .js files and matching selectors in .css And if you subscribe to the AI stuff, you'd also want your AI to operate on this AST so it can learn skills that generalize across all languages.
tredre3 7d ago
> Stop letting Microsoft design protocols and APIs. Unfortunately nobody else stepped up to do it. I'm glad that the code editors out there didn't wait for your theoretical better designed protocol and decided to adopt LSP. Otherwise we'd still have editor that only support one language properly, and the rest is treated like text. If the price to pay is that it sucks for the handful of people who have to work with it, so be it. For every LSP developer that suffers there are tens of thousands of downstream users who benefit from better language support in their favorite editor!
- octoberfranklin 7d ago
  we'd still have editor that only support one language properly Emacs supported zillions of languages before LSP. Zoom out.
  - pjmlp 7d ago
    As someone that got introduced to it in 1996, and was a fan of XEmacs variant, still remembers enough keybindings and Elisp, supported beyond plain syntax highlighting was very much hit and miss, even nowadays.
  - shakow 7d ago
    I have been an emacs user for over a decade now, and if you think that language “support” was as good before LSP, I have a bridge to sell you.
  - throwaway7356 7d ago
    If the old emacs approach is so much better, why is emacs switching to using LSP?
  - rcxdude 7d ago
    But the effort to do so was then entirely duplicated for other editors. LSP turns an N*M effort into an N+M effort, the upshot of which being far more complete coverage for both editors and languages.
ivanjermakov 7d ago
LSP is not that bad. They tried to decouple IDE features from language features. Resolving file paths is not trivial because it's language specific. Syncing source changes on the LSP server side is necessary to keep packet side small and patching is trivial (span, new text). (I made an LSP-powered text editor and implemented LSP client from scratch)

duttish 7d ago

I've never looked into LSP, how would you design it?

packetlost 7d ago
Well for one I wouldn't design it so both the LSP and the editor need a synchronized view of the underlying file.
mitxela 7d ago
Start with synchronous function calls instead of JSON. Microsoft knows how to do that - they invented COM and OLE. Function calls enable whatever data sharing is necessary to maintain a coherent view. Imagine trying to do OLE with JSON - just wouldn't work. (Does OLE still exist?)
- cyberax 7d ago
  Stupid idea. So the LSP crashes and/or goes into a runaway memory consumption loop. And your main application dies with it. Or what if you want, you know, to be able to use the same LSP from TWO different applications at the same time? Never mind issues with other managed runtimes not expecting to deal with something else in their address space.
- cyberax 7d ago
  Moreover, it's not just stupid, it also does not actually solve _anything_. A synchronous function can also have obsolete indexing information if it races with the code updates.
  - mitxela 7d ago
    But it doesn't have to, because you call AboutToUpdateCode and the LSP doesn't return until it's safe, and then you update code and call DoneUpdatingCode.
    - cyberax 7d ago
      Yes, it does because the physical code update is asynchronous. Or you'd have to do a roundtrip for every little edit, resulting in unusably slow experience.
      - mitxela 6d ago
        I don't know what you're talking about. What is "physical code"? A function call roundtrip costs nothing, that's the reason we are saying it should be function calls.
        cyberax 6d ago
        TCP roundtrip overhead over localhost costs approximately nothing. But running the LSP itself is slow. It's not instant, and you can't run it in a blocking way for each keypress.
        mitxela 6d ago
        A function call costs nanoseconds. You can sprinkle them literally everywhere. A JSON-formatted TCP round trip to localhost costs, I'd guess conservatively, at least ten microseconds, 1000x slower.
        cyberax 6d ago
        OK. Here's my simple implementation of an LSP: > func doLspStuff() { sleep(60000); } Now sprinkle them after each keypress. Do you see the issue now?
- rspeele 7d ago
  I feel like MS actually learned their lesson with synchronously integrating the language intelligence into the IDE. Old versions of VS would hang or crash based on bugs in the language tooling trying to provide intellisense. You'd restart and it'd work fine till you hit some other weird edge case. Generally this settled to a level of rare-bugginess where you were happy enough with the advantages not to go back to Emacs/VIM, but still annoyed at the occasional restart needed. In no way does this mean LSP is a perfect solution, but anything synchronous would be a step backwards.
- jcranmer 7d ago
  That doesn't really fix anything. The filesystem is fundamentally a racy shared data structure. If you made the entrypoint API synchronous, any half-way decent editor would shove the LSP queries to the synchronous API into a different thread, because "Do Not Block the UI Thread" is a fundamental principle of good UI programming. Once you get past that, JSON-over-TCP is just another kind of asynchronous RPC mechanism, one that has the advantage that you can build it in just about any language with out-of-the-box tools. Trying to make a plugin system or a COM or CORBA or OLE based system really cuts out the ability to build language servers in most languages, because you have to be able to build the code in just the right way.
  - mitxela 7d ago
    Big assumption that the editor, which is required to make calls synchronously, would make them on a different thread. Isn't correctness more important than avoiding plugin calls in the UI thread? The good thing about having an actual function interface is that you can tailor each one to its actual needs, instead of insisting high-overhead async is okay for everything. It's like GDI vs X11.
    - ablob 7d ago
      I don't know about you, but I do not want to block on every keystroke. Program analysis can take time, asynchronous feedback is a natural fit. None of the features I use from LSP should be synchronous. I can almost guarantee you that a synchronous API would yield a much more complicated design just because many operations can not be expected to reliably work within a few milliseconds. You want to rename something across multiple files? Well now you add file system overhead and blow straight past reliable frame timings. Good luck waiting for that operation to finish. Of course you could make the API beginRename, and queryRename, or whatever you fancy to see if the operation was successful, but now you're back to what you wanted to avoid: an asynchronous API. Do mind that the example is actually one of the better cases, as many things you might want to do with a codebase are actually more expensive. You will feel the hiccups from waiting in the UI thread and you will loathe the program for it.
- pjmlp 7d ago
  Actually, COM is inspired by DCE/RPC and the initial versions had some similarities. Parallel to that, IBM had SOM on OS/2, which was even better allowing for metaclasses and proper class inheritance, it was the key mechanism between Smalltalk and C++ on OS/2, where Smalltalk enjoyed a role similar to .NET on Windows nowadays. OLE naturally still exists when using Office natively on Windows, other vendors seem to have forgotten about it. COM's role on Windows has grown since Vista, and the Windows team redid many of the Longhorn ideas originally implemented in .NET into COM/C++, with WinRT being an evolution of COM.
  - mamp 7d ago
    I've never heard of anyone calling SOM 'better' at anything this century. I came across it when it was the foundation for OpenDoc at Apple, your comment brought back many bad memories
    - pjmlp 7d ago
      Compared with COM's design, it was much better.
      - mitxela 7d ago
        Classes and metaclasses sound like architecture astronomy here. COM is fundamentally just a standard ABI and a way to look up DLL paths in the registry. There are more layers on top of that, but that's the basics and all you need for a plugin architecture.
        pjmlp 7d ago
        That applies to SOM just as it does to COM. Interesting to complain about one and then ignore exactly the same boilerplate for the other. Leaving aside the fact that Microsoft's tooling for COM has already had multiple reboots between VB OCX, MFC, .NET Framework RCW/CCW, .NET ComWrappers, ATL, WRL, WIL, WinRT, each one with its share of astronomy. No one uses the bare bones vtbl and nothing else.
- throwaway7356 7d ago
  > Start with synchronous function calls instead of JSON. LSP is a function call protocol. So that is already done?
lenkite 7d ago
Wouldn't a normalized protocol based on AST vocabulary and tree operations make more sense ?
- looperhacks 6d ago
  That would require the editor to be able to build an AST - defeating the point of LSP, because now the editor needs to know every possible language again
  - lenkite 6d ago
    I meant the language server builds the normalized, lossless AST and gives it to the editor. Changes in the AST can be provided as events/diffs to the editor. The normalized, source-mapped AST is effectively the stable language protocol. (Possibly given in a columnar, database form via Apache Arrow, but thats impl specific). Was discussing alternative architectures with a collague since even the best language servers really suck on large projects.
kuschku 7d ago
One of the largest issues of LSP is that each language implementation needs to do everything separately, and that means each language will work differently. Imagine if all the editing tools in microsoft word were specific to the language you used, and if you mixed German and English, each had different tooling. Now if you mix Rust, HTML, JS, and CSS, they'll all have separate tooling, seperate "go to definition", and separate refactoring. Worst of all, none of them can see the definitions of the other ones. So what you'd actually want to do is parse each language into your AST, with proper annotations as to what is what, and have all the "go to definition", the UI rendering, highlighting, refactoring, etc all done generically by the IDE ontop. Which is much much closer to how Jetbrains IntelliJ does it, and why their tooling can handle "find usages" on an HTML element to find matching querySelector in .js files and matching selectors in .css And if you subscribe to the AI stuff, you'd also want your AI to operate on this AST so it can learn skills that generalize across all languages.

tredre3 7d ago

> Stop letting Microsoft design protocols and APIs. Unfortunately nobody else stepped up to do it. I'm glad that the code editors out there didn't wait for your theoretical better designed protocol and decided to adopt LSP. Otherwise we'd still have editor that only support one language properly, and the rest is treated like text. If the price to pay is that it sucks for the handful of people who have to work with it, so be it. For every LSP developer that suffers there are tens of thousands of downstream users who benefit from better language support in their favorite editor!

octoberfranklin 7d ago
we'd still have editor that only support one language properly Emacs supported zillions of languages before LSP. Zoom out.
- pjmlp 7d ago
  As someone that got introduced to it in 1996, and was a fan of XEmacs variant, still remembers enough keybindings and Elisp, supported beyond plain syntax highlighting was very much hit and miss, even nowadays.
- shakow 7d ago
  I have been an emacs user for over a decade now, and if you think that language “support” was as good before LSP, I have a bridge to sell you.
- throwaway7356 7d ago
  If the old emacs approach is so much better, why is emacs switching to using LSP?
- rcxdude 7d ago
  But the effort to do so was then entirely duplicated for other editors. LSP turns an N*M effort into an N+M effort, the upshot of which being far more complete coverage for both editors and languages.

ivanjermakov 7d ago

LSP is not that bad. They tried to decouple IDE features from language features. Resolving file paths is not trivial because it's language specific. Syncing source changes on the LSP server side is necessary to keep packet side small and patching is trivial (span, new text). (I made an LSP-powered text editor and implemented LSP client from scratch)