16 ms·

Apple's investment into Clangd and refactoring tooling

289 points by artisdom 8y ago

pjmlp 8y ago
Looking forward to it. It is really nice to finally start having C++ environments close to the promise of Energize C++ and Visual Age C++ 4.
- codetrotter 8y ago
  https://www.reddit.com/r/programming/comments/25r6pw/a_demo_of_lucids_1993_graphical_cc_programming/ https://www.reddit.com/r/programming/comments/25r6pw/a_demo_... They mentioned incremental compilation in the first few seconds of the video. Didn’t watch the rest of the video yet. I’ve always wished for incremental compilation. Imagine incremental compilation that was so fine grained it would recompile only the functions you changed. Of course this makes full-program optimization impossible but I think most of us don’t compile at the highest optimization levels during development anyway because doing so slows down builds without any benefit for development builds. For production builds we use the highest optimization levels of course. Additionally I would like to have hot patching. I know some people have this, personally I’ve never had that. If anyone knows about any systems for hot patching Rust code please let me know :)
  - pjmlp 8y ago
    While not perfect, Visual C++ makes it a nice experience. https://blogs.msdn.microsoft.com/vcblog/2016/10/05/faster-c-build-cycle-in-vs-15-with-debugfastlink/ https://blogs.msdn.microsoft.com/vcblog/2016/10/05/faster-c-... https://msdn.microsoft.com/en-us/library/4khtbfyf.aspx https://msdn.microsoft.com/en-us/library/4khtbfyf.aspx
- barrkel 8y ago
  The future is a lot easier to build when you're vertically integrated. The risk with vertical integration is that keeping all parts of the integrated stack at the cutting edge is too costly and you start to fall behind. A marketplace approach with broad consensus around interfaces is ideal, but it's hard to bootstrap. LSP via VSCode seems to have done the job.
vhbit 8y ago
I hope that it also means there will be Apple-provided LSP server for Swift soon.
- solarexplorer 8y ago
  There is one already: SourceKit http://www.jpsim.com/uncovering-sourcekit/ http://www.jpsim.com/uncovering-sourcekit/
saagarjha 8y ago
I'm glad to see that Apple is putting work into making their tooling better and easier to work with, but I'm still not clear on the benefits of this change. Would someone care to enlighten me? Currently, the benefits of libclang that I see are that I can easily drop it into a project without requiring a lot of dependencies, and I don't need to figure out how to get permission to run different processes and manage interprocess communication between them–if anyone doubts that this will be an issue, on iOS neither of these is really supported at all. In addition, I don't require a daemon to constantly be running to analyze my code. (Why is this bad? Take a look at Swift's sourcekitd.)
- vhbit 8y ago
  > if anyone doubts that this will be an issue, on iOS neither of these is really supported at all What for you might need it on iOS? Server is meant to run on a development machine.
  - JDevlieghere 8y ago
    > What for you might need it on iOS? Server is meant to run on a development machine. While I totally agree, I can see the author's point. Think for example about Swift Playgrounds.
  - saagarjha 8y ago
    > Server is meant to run on a development machine There's no reason to discount iOS as a development machine–it's perfectly fine for coding, as Swift Playgrounds shows. (Actually, there are some issues with executing code in third party apps, but I'm working on a way to bypass the restrictions around this. Check back again soon!)
- pjmlp 8y ago
  To be able use it from any editor that is able to understand LSP, instead of having everyone writing their own bindings to libclang, specially if the editors are written in type safe languages.
  - 8y ago
raverbashing 8y ago
Could someone explain what's the difference between both approaches and what are the advantages of Clangd?
- barrkel 8y ago
  In process vs out of process; and language-specific binding vs JSON-RPC. Moving stuff out of process means, increased reliability and potentially ability to share data across multiple clients. Moving everything to LSP with two supported transports (JSON-RPC and something more native) means that editors like Emacs, vim, VSCode can benefit from tooling without needing to link against a library and work with C bindings.
- ninkendo 8y ago
  Moving language parsing/refactoring/etc into a separate daemon is a step towards unifying tooling around specific protocols for interacting with compilers. Microsoft's effort towards a single language server protocol is worth mentioning: https://github.com/Microsoft/language-server-protocol https://github.com/Microsoft/language-server-protocol I'd love to see a future when new languages/compilers can implement this protocol for their external compiling daemons and just get support in all the various text editors/IDEs automatically.
- dwheeler 8y ago
  Clangd implements the "Language Server Protocol" (LSP). More info about LSP here: https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/ https://langserver.org/ https://langserver.org/ A quick cut-and-paste from the first URL explains why LSP was created: "Adding features like auto complete, go to definition, or documentation on hover for a programming language takes significant effort. Traditionally this work had to be repeated for each development tool, as each tool provides different APIs for implementing the same feature. A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication. The idea behind the Language Server Protocol (LSP) is to standardize the protocol for how such servers and development tools communicate. This way, a single Language Server can be re-used in multiple development tools, which in turn can support multiple languages with minimal effort."
fowl2 8y ago
I'm assuming this is the "LSP" discussed? https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/ Mapping json-rpc onto xpc... Sure why not
- icholy 8y ago
  He said that's an intermediate step so that they can get to testing faster.
- fokinsean 8y ago
  Woah I had never seen this before. That's really cool. So in theory you can make your own editor and provide a ton of IDE-like features by communicating with this server? Are there any other implementations of LSP by other orgs?
  - evmar 8y ago
    https://langserver.org/ https://langserver.org/
CodeArtisan 8y ago
There is an article from 2010 titled "Emacs is dead" that is being reposted on HN from time to time. The author args that the greatness of Emacs is to rely on external, editor agnostic tools where text act as an universal interface/medium but that the practice is fading out, and thus the dead of Emacs. Now, ten years later, Apple announces this. Emacs, the undead? https://tkf.github.io/2013/06/04/Emacs-is-dead.html https://tkf.github.io/2013/06/04/Emacs-is-dead.html
- oblio 8y ago
  I think you might be right, but the greater context is highly ironic. The company resurrecting the Emacs concepts is Microsoft. They're the ones that created OmniSharp, bringing the idea of language servers to the programming mainstream. They then created the Language Server Protocol for what now is Emacs's biggest long term rival in the flexible editor arena: Visual Studio Code. Visual Studio Code is for the moment on a somewhat bloated and shaky foundation because of Electron. But otherwise its design is quite solid. And even that foundation will probably become a lot stronger in the next few years as WebAssembly gains wide adoption. Visual Studio Code itself is written in Typescript and it's not hard to imagine Microsoft adding a WASM backend to it.
  - pjmlp 8y ago
    Here is how Lucid implemented their C++ knowledge repository for Energize C++. "Foundation for a C++ Programming Environment", chapter 7, Protocols https://pdfs.semanticscholar.org/84a4/8824fc7dd872414efa0ad6f238399f480561.pdf https://pdfs.semanticscholar.org/84a4/8824fc7dd872414efa0ad6... And how it looked like with their Emacs customization in 1993 on UNIX. https://www.youtube.com/watch?v=pQQTScuApWk https://www.youtube.com/watch?v=pQQTScuApWk
    - oblio 8y ago
      Lucid went bankrupt as far as I remember :)
      - 8y ago
Game_Ender 8y ago
Does anyone have any details about Apples cross language indexer they mention? A good indexer is the key to and efficient language server and I think clangd’s is a little weak right now.
flamedoge 8y ago
I tried building Clangd in trunk but failed to build.
- DannyBee 8y ago
  We build and use it every day, so ... report it on the list?
DannyBee 8y ago
So, just because there seems to be confusion: Clangd already exists. It already supports LSP. It is already part of the llvm project. It was written by !Apple, and used by a wide number of people. (I want to make sure these people still get credit, i already see people in this thread thanking apple for supporting LSP, which is not actually what happened). What apple has proposed contributing is: Making clangd work with xcode, and extensions to LSP. This is great (as long as the second ends up standardized over time). But the benefits people listed here so far already exist in clangd. The main benefit to work is "xcode support" and "better refactoring" which is awesome :)
- falanjito 8y ago
  Would it make sense to rewrite everything in Python to gain some speed?
  - tbodt 8y ago
    No.
- oflannabhra 8y ago
  The proposal is to split the LSP implementation into a transport layer and a logical layer. The current LSP implementation is directly tied to JSON-RPC. Apple's end goal is, as you say, to get clangd to work with Xcode. However, having a replaceable transport layer for LSP could benefit lots of people other than Apple.
  - DannyBee 8y ago
    Sure, it's nice, but again, being pragmatic, nobody else has desired such a thing that much. Ie so far the "lots of people" have not materialized, even outside clangd. I can certainly think of use cases for such a thing, but most people seem happy enough with json-rpc at the moment. Part of that is likely that trying to speak LSP over a non local connection may be a fools errand anyway. It's a chatty protocol no matter what the transport.
fwfwfwf 8y ago
foop!!
woolvalley 8y ago
This is great news! More parts of xcode are being exposed as open source and clangd is getting more engineering resources.
jupp0r 8y ago
I have mixed feelings about this. On one hand it's really nice to see Apple jump on the LSP train and use standard tooling for their IDE to interact with language tools. On the other hand, instead of having a consistent strategy of using proper LSP (which by specification is JSONRPC), they are shoehorning LSP on top of their in-house XPC transport. To do that they're planning to introduce additional complexity to clangd (a transport abstraction), while actually defeating one of the main purposes of LSP, which is reducing the m-times-n complexity problem of matching compilers with tools to an m-plus-n complexity problem. No other tools (unless they support LSP-over-XPC as well) will be able to talk to XCode. XCode won't be able to talk to other tools. I hope they rethink that decision, keep clangd simple and instead adopt proper (JSONRPC) LSP in all of their other tools for Swift, etc instead. That way they'd not only open up XCode to clangd, but also all other editors to their refactoring tools for Swift, etc.

289 points by artisdom 8y ago

Looking forward to it. It is really nice to finally start having C++ environments close to the promise of Energize C++ and Visual Age C++ 4.

codetrotter 8y ago
https://www.reddit.com/r/programming/comments/25r6pw/a_demo_of_lucids_1993_graphical_cc_programming/ https://www.reddit.com/r/programming/comments/25r6pw/a_demo_... They mentioned incremental compilation in the first few seconds of the video. Didn’t watch the rest of the video yet. I’ve always wished for incremental compilation. Imagine incremental compilation that was so fine grained it would recompile only the functions you changed. Of course this makes full-program optimization impossible but I think most of us don’t compile at the highest optimization levels during development anyway because doing so slows down builds without any benefit for development builds. For production builds we use the highest optimization levels of course. Additionally I would like to have hot patching. I know some people have this, personally I’ve never had that. If anyone knows about any systems for hot patching Rust code please let me know :)
- pjmlp 8y ago
  While not perfect, Visual C++ makes it a nice experience. https://blogs.msdn.microsoft.com/vcblog/2016/10/05/faster-c-build-cycle-in-vs-15-with-debugfastlink/ https://blogs.msdn.microsoft.com/vcblog/2016/10/05/faster-c-... https://msdn.microsoft.com/en-us/library/4khtbfyf.aspx https://msdn.microsoft.com/en-us/library/4khtbfyf.aspx
barrkel 8y ago
The future is a lot easier to build when you're vertically integrated. The risk with vertical integration is that keeping all parts of the integrated stack at the cutting edge is too costly and you start to fall behind. A marketplace approach with broad consensus around interfaces is ideal, but it's hard to bootstrap. LSP via VSCode seems to have done the job.

vhbit 8y ago

I hope that it also means there will be Apple-provided LSP server for Swift soon.

solarexplorer 8y ago
There is one already: SourceKit http://www.jpsim.com/uncovering-sourcekit/ http://www.jpsim.com/uncovering-sourcekit/

saagarjha 8y ago

I'm glad to see that Apple is putting work into making their tooling better and easier to work with, but I'm still not clear on the benefits of this change. Would someone care to enlighten me? Currently, the benefits of libclang that I see are that I can easily drop it into a project without requiring a lot of dependencies, and I don't need to figure out how to get permission to run different processes and manage interprocess communication between them–if anyone doubts that this will be an issue, on iOS neither of these is really supported at all. In addition, I don't require a daemon to constantly be running to analyze my code. (Why is this bad? Take a look at Swift's sourcekitd.)

vhbit 8y ago
> if anyone doubts that this will be an issue, on iOS neither of these is really supported at all What for you might need it on iOS? Server is meant to run on a development machine.
- JDevlieghere 8y ago
  > What for you might need it on iOS? Server is meant to run on a development machine. While I totally agree, I can see the author's point. Think for example about Swift Playgrounds.
- saagarjha 8y ago
  > Server is meant to run on a development machine There's no reason to discount iOS as a development machine–it's perfectly fine for coding, as Swift Playgrounds shows. (Actually, there are some issues with executing code in third party apps, but I'm working on a way to bypass the restrictions around this. Check back again soon!)
pjmlp 8y ago
To be able use it from any editor that is able to understand LSP, instead of having everyone writing their own bindings to libclang, specially if the editors are written in type safe languages.
- 8y ago

raverbashing 8y ago

Could someone explain what's the difference between both approaches and what are the advantages of Clangd?

barrkel 8y ago
In process vs out of process; and language-specific binding vs JSON-RPC. Moving stuff out of process means, increased reliability and potentially ability to share data across multiple clients. Moving everything to LSP with two supported transports (JSON-RPC and something more native) means that editors like Emacs, vim, VSCode can benefit from tooling without needing to link against a library and work with C bindings.
ninkendo 8y ago
Moving language parsing/refactoring/etc into a separate daemon is a step towards unifying tooling around specific protocols for interacting with compilers. Microsoft's effort towards a single language server protocol is worth mentioning: https://github.com/Microsoft/language-server-protocol https://github.com/Microsoft/language-server-protocol I'd love to see a future when new languages/compilers can implement this protocol for their external compiling daemons and just get support in all the various text editors/IDEs automatically.
dwheeler 8y ago
Clangd implements the "Language Server Protocol" (LSP). More info about LSP here: https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/ https://langserver.org/ https://langserver.org/ A quick cut-and-paste from the first URL explains why LSP was created: "Adding features like auto complete, go to definition, or documentation on hover for a programming language takes significant effort. Traditionally this work had to be repeated for each development tool, as each tool provides different APIs for implementing the same feature. A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication. The idea behind the Language Server Protocol (LSP) is to standardize the protocol for how such servers and development tools communicate. This way, a single Language Server can be re-used in multiple development tools, which in turn can support multiple languages with minimal effort."

fowl2 8y ago

I'm assuming this is the "LSP" discussed? https://microsoft.github.io/language-server-protocol/ https://microsoft.github.io/language-server-protocol/ Mapping json-rpc onto xpc... Sure why not

icholy 8y ago
He said that's an intermediate step so that they can get to testing faster.
fokinsean 8y ago
Woah I had never seen this before. That's really cool. So in theory you can make your own editor and provide a ton of IDE-like features by communicating with this server? Are there any other implementations of LSP by other orgs?
- evmar 8y ago
  https://langserver.org/ https://langserver.org/

CodeArtisan 8y ago

There is an article from 2010 titled "Emacs is dead" that is being reposted on HN from time to time. The author args that the greatness of Emacs is to rely on external, editor agnostic tools where text act as an universal interface/medium but that the practice is fading out, and thus the dead of Emacs. Now, ten years later, Apple announces this. Emacs, the undead? https://tkf.github.io/2013/06/04/Emacs-is-dead.html https://tkf.github.io/2013/06/04/Emacs-is-dead.html

oblio 8y ago
I think you might be right, but the greater context is highly ironic. The company resurrecting the Emacs concepts is Microsoft. They're the ones that created OmniSharp, bringing the idea of language servers to the programming mainstream. They then created the Language Server Protocol for what now is Emacs's biggest long term rival in the flexible editor arena: Visual Studio Code. Visual Studio Code is for the moment on a somewhat bloated and shaky foundation because of Electron. But otherwise its design is quite solid. And even that foundation will probably become a lot stronger in the next few years as WebAssembly gains wide adoption. Visual Studio Code itself is written in Typescript and it's not hard to imagine Microsoft adding a WASM backend to it.
- pjmlp 8y ago
  Here is how Lucid implemented their C++ knowledge repository for Energize C++. "Foundation for a C++ Programming Environment", chapter 7, Protocols https://pdfs.semanticscholar.org/84a4/8824fc7dd872414efa0ad6f238399f480561.pdf https://pdfs.semanticscholar.org/84a4/8824fc7dd872414efa0ad6... And how it looked like with their Emacs customization in 1993 on UNIX. https://www.youtube.com/watch?v=pQQTScuApWk https://www.youtube.com/watch?v=pQQTScuApWk
  - oblio 8y ago
    Lucid went bankrupt as far as I remember :)
    - 8y ago

Game_Ender 8y ago

Does anyone have any details about Apples cross language indexer they mention? A good indexer is the key to and efficient language server and I think clangd’s is a little weak right now.

flamedoge 8y ago

I tried building Clangd in trunk but failed to build.

DannyBee 8y ago
We build and use it every day, so ... report it on the list?

DannyBee 8y ago

So, just because there seems to be confusion: Clangd already exists. It already supports LSP. It is already part of the llvm project. It was written by !Apple, and used by a wide number of people. (I want to make sure these people still get credit, i already see people in this thread thanking apple for supporting LSP, which is not actually what happened). What apple has proposed contributing is: Making clangd work with xcode, and extensions to LSP. This is great (as long as the second ends up standardized over time). But the benefits people listed here so far already exist in clangd. The main benefit to work is "xcode support" and "better refactoring" which is awesome :)

falanjito 8y ago
Would it make sense to rewrite everything in Python to gain some speed?
- tbodt 8y ago
  No.
oflannabhra 8y ago
The proposal is to split the LSP implementation into a transport layer and a logical layer. The current LSP implementation is directly tied to JSON-RPC. Apple's end goal is, as you say, to get clangd to work with Xcode. However, having a replaceable transport layer for LSP could benefit lots of people other than Apple.
- DannyBee 8y ago
  Sure, it's nice, but again, being pragmatic, nobody else has desired such a thing that much. Ie so far the "lots of people" have not materialized, even outside clangd. I can certainly think of use cases for such a thing, but most people seem happy enough with json-rpc at the moment. Part of that is likely that trying to speak LSP over a non local connection may be a fools errand anyway. It's a chatty protocol no matter what the transport.

fwfwfwf 8y ago

foop!!

woolvalley 8y ago

This is great news! More parts of xcode are being exposed as open source and clangd is getting more engineering resources.

jupp0r 8y ago

I have mixed feelings about this. On one hand it's really nice to see Apple jump on the LSP train and use standard tooling for their IDE to interact with language tools. On the other hand, instead of having a consistent strategy of using proper LSP (which by specification is JSONRPC), they are shoehorning LSP on top of their in-house XPC transport. To do that they're planning to introduce additional complexity to clangd (a transport abstraction), while actually defeating one of the main purposes of LSP, which is reducing the m-times-n complexity problem of matching compilers with tools to an m-plus-n complexity problem. No other tools (unless they support LSP-over-XPC as well) will be able to talk to XCode. XCode won't be able to talk to other tools. I hope they rethink that decision, keep clangd simple and instead adopt proper (JSONRPC) LSP in all of their other tools for Swift, etc instead. That way they'd not only open up XCode to clangd, but also all other editors to their refactoring tools for Swift, etc.