6 ms·

This seems convoluted compared to say moving the Lisp implementation from Emacs Lisp to Common Lisp, of which several native code compiling implementations exis

This seems convoluted compared to say moving the Lisp implementation from Emacs Lisp to Common Lisp, of which several native code compiling implementations exists.

wtetzner 6y ago
You don't think it would be difficult to keep compatibility with all of the existing Emacs Lisp packages that exist now?
widdershins 6y ago
Rewriting thousands of packages (some of which have had man-decades of work poured into them) would also be rather convoluted.
- junke 6y ago
  You don't need to rewrite packages if you have a layer where Emacs Lisp is compiled/interpreted as Common Lisp. You can easily hack the readtable in Lisp, and rewrite Emacs Lisp sexps as Common Lisp sexps. Some interesting things to consider are file-local variables, buffer-local variables, dynamic-scope/lexical-scope, how to handle floating point (-0.0e+NaN in Emacs Lisp, custom floating point rouding modes/traps in SBCL for example), but this looks like a reasonable approach overall. For example (just a draft, this might be incorrect): USER> (defun make-local-variable (symbol) (eval `(define-symbol-macro ,symbol (bvar (quote ,symbol))))) MAKE-LOCAL-VARIABLE USER> (make-local-variable 'foo) FOO USER> (macroexpand-all '(list foo)) (LIST (BVAR 'FOO)) T T The BVAR forms would then be able to access a buffer variable, with the current buffer. This works with SETF too (but Emacs Lisp SETQ should be replaced by SETF during the transform). BVAR could expand into code that calls FFI functions, for compatibility. Using an FFI approach would allow to progressively rewrite some parts of the runtime into CL. Just to clarify, I know this is a huge work to undertake, but this comes from the assumption that we want to keep all existing Elisp files running identically.
  - kazinator 6y ago
    Emacs contains a quarter million lines of C. What do you do with all the elisp that calls into it? Maybe that could be turned into a "libemacs" and used from FFI.
    - junke 6y ago
      Primitive Emacs Lisp functions are defined in C, yes, using helper C macros. There might be a way to extract the symbol names from those macros, and maybe enough information to generate the FFI bindings. Any kind of rewrite of Emacs is going to be a lot of work, I am not denying that. But if we assume there is a layer where everything works as it currently does (the C primitives), the Emacs Lisp defined on top of that needs not be defined in C, as far as I know. Also, the last time I checked, there were a ton of legacy stuff in the configure script for different platforms.
    - wtetzner 6y ago
      Or maybe you rewrite it in Common Lisp. If you're going to take the Common Lisp route, I think it might be worth going all in and just re-implementing Emacs in Common Lisp.
      - mikelevins 6y ago
        That's already been done. More than once, in fact. The problem is that, although there are perfectly serviceable Emacsen written in Common Lisp, none of them are GNU Emacs. For example, there's Hemlock from CMUCL, and its descendants built into Lispworks and Clozure Common Lisp. Those implementations aren't likely to work well as a substitute for GNU Emacs. For one thing, there's a substantial ecosystem of software that depends on specific APIs and other characteristics of GNU Emacs. Writing code to bridge GNU Emacs APIs with those available in the Hemlock descendants would be a lot of work. There are some other obstacles as well. CCL's implementation of Hemlock uses the Cocoa text architecture, so it's not portable to platforms other than macOS. The Lispworks implementation is portable across Windows and numerous UNIXEN, but the Lispworks license will not allow delivery of a proper substitute for GNU Emacs (it forbids building an application that can be construed as a Lisp development system, and when I pressed them about exactly what limits that policy implies, they explicitly used Emacs as an example of an application that would be forbidden). The original CMUCL Hemlock and its portable version are designed to work with CLX. It could probably be made to work in a modern X environment, but making it fit well into modern GUI environments and porting it to all the platforms GNU Emacs works on would be a huge amount of work. I'm probably overlooking some other Emacsen, but I don't know of any off the top of my head for which the situation is any easier.
- slightwinder 6y ago
  I wonder whether a modern modularized architecture might be the better way to handle this. Similar to neovim where you have a backend which communicates with a detatched frontend and other backends over some rpc-mechanism. Refactor the existing emacs to work headless and become the legacy-core, while also establishing channels to attatch any other core and frontend.
  - gpderetta 6y ago
    An important optimization discussed in the video is to optimize through call into C code (i.e elisp->C->elisp). adding RPC at the boundaries would go in the opposite direction.
    - WhatIsDukkha 6y ago
      Well this is not to my taste but I take it as a proof of existence that an rpc at the boundary could make for some amazing things - https://github.com/rhysd/NyaoVim https://github.com/rhysd/NyaoVim
    - slightwinder 6y ago
      It's optimization for a single process vs multiple processes. Additionally, when you free up your architecture and loose up the internal bindings between different areas, you sometimes can open chances for significant improvments which prior where not possible, thus still be faster with a "worse" solution. Considering how old GNU Emacs is, how much historical problems it has and how often you hear that this and that can't be optimized for this and that reason, this seems like such a situation where loosing up would open faster roads long term.
  - Myrmornis 6y ago
    https://github.com/xi-editor https://github.com/xi-editor is an example of the architecture you're referring to I believe. Xi's README makes it clear that very high performance is one of the main goals: in particular it mentions the latency between keypresses and screen painting. To what extent does this RPC-based architecture limit performance compared to a single-process architecture? I know that the original author of Xi very much knows what he is doing, so I guess the answer is "it doesn't" -- I was just hoping to be educated on this point.
gumby 6y ago
I assume you mean writing an Emacs lisp interpreter in Common Lisp — which could be done with a small lexer, a lot of macros, and some library support. That would probably be a win.
- wtetzner 6y ago
  Depends on how difficult it is to integrate Common Lisp into Emacs. Or would you just be rewriting Emacs in Common Lisp?
  - gumby 6y ago
    The issue is the importance of all the existing packages — invalidate them and nobody will use your implementation. There are small syntactic differences between Emacs lisp and Common Lisp so a lexer that handled those would suffice. There are some small semantic differences but they could be handled by porting code. And the Emacs c code could be rewritten in lisp or ported into the Common Lisp implementation, depending on your taste.
    - wtetzner 6y ago
      The existing package ecosystem is one of the main value propositions of Emacs. Otherwise a re-implementation with a different extension language could have replaced it a long time ago. > There are some small semantic differences but they could be handled by porting code. I suspect there are more than just small differences. And someone would have to do the work to port the code. I suspect that if packages that people use don't work, they will just avoid using the new version of Emacs, as opposed to abandoning the package. It's not an impossible project, but it's not clear to me it's easier/better than the proposed solution.
      - gumby 6y ago
        The point of my suggestion was to not have to rewrite any existing .el files.
kazinator 6y ago
It brings GCC into the address space via libgccjit. GCC is not robust enough to be integrated into applications that stay running, and be repeatedly invoked.
- tgbugs 6y ago
  I don't think that is how libgccjit is being used in this project. Emacs is not being linked with libgccjit. libgccjit is only being use for AOT compilation.

This seems convoluted compared to say moving the Lisp implementation from Emacs Lisp to Common Lisp, of which several native code compiling implementations exists.

wtetzner 6y ago

You don't think it would be difficult to keep compatibility with all of the existing Emacs Lisp packages that exist now?

widdershins 6y ago

Rewriting thousands of packages (some of which have had man-decades of work poured into them) would also be rather convoluted.

junke 6y ago
You don't need to rewrite packages if you have a layer where Emacs Lisp is compiled/interpreted as Common Lisp. You can easily hack the readtable in Lisp, and rewrite Emacs Lisp sexps as Common Lisp sexps. Some interesting things to consider are file-local variables, buffer-local variables, dynamic-scope/lexical-scope, how to handle floating point (-0.0e+NaN in Emacs Lisp, custom floating point rouding modes/traps in SBCL for example), but this looks like a reasonable approach overall. For example (just a draft, this might be incorrect): USER> (defun make-local-variable (symbol) (eval `(define-symbol-macro ,symbol (bvar (quote ,symbol))))) MAKE-LOCAL-VARIABLE USER> (make-local-variable 'foo) FOO USER> (macroexpand-all '(list foo)) (LIST (BVAR 'FOO)) T T The BVAR forms would then be able to access a buffer variable, with the current buffer. This works with SETF too (but Emacs Lisp SETQ should be replaced by SETF during the transform). BVAR could expand into code that calls FFI functions, for compatibility. Using an FFI approach would allow to progressively rewrite some parts of the runtime into CL. Just to clarify, I know this is a huge work to undertake, but this comes from the assumption that we want to keep all existing Elisp files running identically.
- kazinator 6y ago
  Emacs contains a quarter million lines of C. What do you do with all the elisp that calls into it? Maybe that could be turned into a "libemacs" and used from FFI.
  - junke 6y ago
    Primitive Emacs Lisp functions are defined in C, yes, using helper C macros. There might be a way to extract the symbol names from those macros, and maybe enough information to generate the FFI bindings. Any kind of rewrite of Emacs is going to be a lot of work, I am not denying that. But if we assume there is a layer where everything works as it currently does (the C primitives), the Emacs Lisp defined on top of that needs not be defined in C, as far as I know. Also, the last time I checked, there were a ton of legacy stuff in the configure script for different platforms.
  - wtetzner 6y ago
    Or maybe you rewrite it in Common Lisp. If you're going to take the Common Lisp route, I think it might be worth going all in and just re-implementing Emacs in Common Lisp.
    - mikelevins 6y ago
      That's already been done. More than once, in fact. The problem is that, although there are perfectly serviceable Emacsen written in Common Lisp, none of them are GNU Emacs. For example, there's Hemlock from CMUCL, and its descendants built into Lispworks and Clozure Common Lisp. Those implementations aren't likely to work well as a substitute for GNU Emacs. For one thing, there's a substantial ecosystem of software that depends on specific APIs and other characteristics of GNU Emacs. Writing code to bridge GNU Emacs APIs with those available in the Hemlock descendants would be a lot of work. There are some other obstacles as well. CCL's implementation of Hemlock uses the Cocoa text architecture, so it's not portable to platforms other than macOS. The Lispworks implementation is portable across Windows and numerous UNIXEN, but the Lispworks license will not allow delivery of a proper substitute for GNU Emacs (it forbids building an application that can be construed as a Lisp development system, and when I pressed them about exactly what limits that policy implies, they explicitly used Emacs as an example of an application that would be forbidden). The original CMUCL Hemlock and its portable version are designed to work with CLX. It could probably be made to work in a modern X environment, but making it fit well into modern GUI environments and porting it to all the platforms GNU Emacs works on would be a huge amount of work. I'm probably overlooking some other Emacsen, but I don't know of any off the top of my head for which the situation is any easier.
slightwinder 6y ago
I wonder whether a modern modularized architecture might be the better way to handle this. Similar to neovim where you have a backend which communicates with a detatched frontend and other backends over some rpc-mechanism. Refactor the existing emacs to work headless and become the legacy-core, while also establishing channels to attatch any other core and frontend.
- gpderetta 6y ago
  An important optimization discussed in the video is to optimize through call into C code (i.e elisp->C->elisp). adding RPC at the boundaries would go in the opposite direction.
  - WhatIsDukkha 6y ago
    Well this is not to my taste but I take it as a proof of existence that an rpc at the boundary could make for some amazing things - https://github.com/rhysd/NyaoVim https://github.com/rhysd/NyaoVim
  - slightwinder 6y ago
    It's optimization for a single process vs multiple processes. Additionally, when you free up your architecture and loose up the internal bindings between different areas, you sometimes can open chances for significant improvments which prior where not possible, thus still be faster with a "worse" solution. Considering how old GNU Emacs is, how much historical problems it has and how often you hear that this and that can't be optimized for this and that reason, this seems like such a situation where loosing up would open faster roads long term.
- Myrmornis 6y ago
  https://github.com/xi-editor https://github.com/xi-editor is an example of the architecture you're referring to I believe. Xi's README makes it clear that very high performance is one of the main goals: in particular it mentions the latency between keypresses and screen painting. To what extent does this RPC-based architecture limit performance compared to a single-process architecture? I know that the original author of Xi very much knows what he is doing, so I guess the answer is "it doesn't" -- I was just hoping to be educated on this point.

gumby 6y ago

I assume you mean writing an Emacs lisp interpreter in Common Lisp — which could be done with a small lexer, a lot of macros, and some library support. That would probably be a win.

wtetzner 6y ago
Depends on how difficult it is to integrate Common Lisp into Emacs. Or would you just be rewriting Emacs in Common Lisp?
- gumby 6y ago
  The issue is the importance of all the existing packages — invalidate them and nobody will use your implementation. There are small syntactic differences between Emacs lisp and Common Lisp so a lexer that handled those would suffice. There are some small semantic differences but they could be handled by porting code. And the Emacs c code could be rewritten in lisp or ported into the Common Lisp implementation, depending on your taste.
  - wtetzner 6y ago
    The existing package ecosystem is one of the main value propositions of Emacs. Otherwise a re-implementation with a different extension language could have replaced it a long time ago. > There are some small semantic differences but they could be handled by porting code. I suspect there are more than just small differences. And someone would have to do the work to port the code. I suspect that if packages that people use don't work, they will just avoid using the new version of Emacs, as opposed to abandoning the package. It's not an impossible project, but it's not clear to me it's easier/better than the proposed solution.
    - gumby 6y ago
      The point of my suggestion was to not have to rewrite any existing .el files.

kazinator 6y ago

It brings GCC into the address space via libgccjit. GCC is not robust enough to be integrated into applications that stay running, and be repeatedly invoked.

tgbugs 6y ago
I don't think that is how libgccjit is being used in this project. Emacs is not being linked with libgccjit. libgccjit is only being use for AOT compilation.