6 ms·

You don't need to rewrite packages if you have a layer where Emacs Lisp is compiled/interpreted as Common Lisp. You can easily hack the readtable in Lisp, and

by junke 6y ago

You don't need to rewrite packages if you have a layer where Emacs Lisp is compiled/interpreted as Common Lisp. You can easily hack the readtable in Lisp, and rewrite Emacs Lisp sexps as Common Lisp sexps. Some interesting things to consider are file-local variables, buffer-local variables, dynamic-scope/lexical-scope, how to handle floating point (-0.0e+NaN in Emacs Lisp, custom floating point rouding modes/traps in SBCL for example), but this looks like a reasonable approach overall. For example (just a draft, this might be incorrect): USER> (defun make-local-variable (symbol) (eval `(define-symbol-macro ,symbol (bvar (quote ,symbol))))) MAKE-LOCAL-VARIABLE USER> (make-local-variable 'foo) FOO USER> (macroexpand-all '(list foo)) (LIST (BVAR 'FOO)) T T The BVAR forms would then be able to access a buffer variable, with the current buffer. This works with SETF too (but Emacs Lisp SETQ should be replaced by SETF during the transform). BVAR could expand into code that calls FFI functions, for compatibility. Using an FFI approach would allow to progressively rewrite some parts of the runtime into CL. Just to clarify, I know this is a huge work to undertake, but this comes from the assumption that we want to keep all existing Elisp files running identically.

kazinator 6y ago
Emacs contains a quarter million lines of C. What do you do with all the elisp that calls into it? Maybe that could be turned into a "libemacs" and used from FFI.
- junke 6y ago
  Primitive Emacs Lisp functions are defined in C, yes, using helper C macros. There might be a way to extract the symbol names from those macros, and maybe enough information to generate the FFI bindings. Any kind of rewrite of Emacs is going to be a lot of work, I am not denying that. But if we assume there is a layer where everything works as it currently does (the C primitives), the Emacs Lisp defined on top of that needs not be defined in C, as far as I know. Also, the last time I checked, there were a ton of legacy stuff in the configure script for different platforms.
- wtetzner 6y ago
  Or maybe you rewrite it in Common Lisp. If you're going to take the Common Lisp route, I think it might be worth going all in and just re-implementing Emacs in Common Lisp.
  - mikelevins 6y ago
    That's already been done. More than once, in fact. The problem is that, although there are perfectly serviceable Emacsen written in Common Lisp, none of them are GNU Emacs. For example, there's Hemlock from CMUCL, and its descendants built into Lispworks and Clozure Common Lisp. Those implementations aren't likely to work well as a substitute for GNU Emacs. For one thing, there's a substantial ecosystem of software that depends on specific APIs and other characteristics of GNU Emacs. Writing code to bridge GNU Emacs APIs with those available in the Hemlock descendants would be a lot of work. There are some other obstacles as well. CCL's implementation of Hemlock uses the Cocoa text architecture, so it's not portable to platforms other than macOS. The Lispworks implementation is portable across Windows and numerous UNIXEN, but the Lispworks license will not allow delivery of a proper substitute for GNU Emacs (it forbids building an application that can be construed as a Lisp development system, and when I pressed them about exactly what limits that policy implies, they explicitly used Emacs as an example of an application that would be forbidden). The original CMUCL Hemlock and its portable version are designed to work with CLX. It could probably be made to work in a modern X environment, but making it fit well into modern GUI environments and porting it to all the platforms GNU Emacs works on would be a huge amount of work. I'm probably overlooking some other Emacsen, but I don't know of any off the top of my head for which the situation is any easier.
    - metroholografix 6y ago
      It hasn't already been done, no. The Emacs variants you mentioned (that IMV should not be referred to as "Emacs") are from-scratch new implementations of just the concept behind Emacs, but not GNU Emacs itself. They are not compatible with GNU Emacs, they don't even support a small fraction of GNU Emacs features, and are thus doomed to obsolescence / non-existent market share. So far, nobody -that I know of- has tried to move GNU Emacs to Common Lisp. GNU Emacs is a well-known, well-used platform and moving to Common Lisp could very well be worth the upfront costs. Wishful thinking aside, the GNU Emacs development community has hunkered down behind Emacs Lisp, so improving it will bring immediate and tangible benefits to every GNU Emacs user. I love Common Lisp and do use SBCL for a lot of personal projects, but I also love GNU Emacs and I will take any improvements there that I can get. Watching Emacs Lisp become a more viable general purpose language (improved performance means an expanded set of problems that it can now address) is a great development. Stefan Monnier on emacs-devel: The main benefit of such a compiler is not to run existing Elisp code faster (99% of existing Elisp code runs fast enough that the user won't notice if it runs faster) but to make it practical to write other Elisp code which would otherwise be too slow. But for that to work well, you want the new compiler to be available "everywhere", rather than just on some platforms. So it will only start being useful when it works on GNU/Linux, macOS, Windows, ARM, RISC-V, x86, amd64, MIPS, younameit (AFAIK lbgccjit's CPU coverage is already good enough for that, so the main barrier here is the OS support), and when it's not just an option at compile-time but when it's included in all builds.
- koral 6y ago
  The real trouble of using Emacs C core from another language is that you have to use all the internal datastructures used by C core, and this exactly equivalent to all elisp datastructures. This implies you have either to convert back and forward everything or you just can't use the native datastructures of the new programming language (making the whole operation often quite pointless). I must confess that most of the comments in this thread seems to start from the assumption that people working on this in the last two+ decades are probably dumb, and this is sad.
ufo 6y ago
I would expect that an ELisp interpreter written in Common Lisp would likely be slower than the existing ELisp interpreter written in C. As for compilation to Common Lisp, I don't know how feasible it would be. Apparently they tried something similar with Guile and it ran into problems.
- bjoli 6y ago
  Guile (the VM) runs elisp just fine. There has been zero optimization work done though so it is quite slow. The reason they did not go with guile was more political than anything else. It was also understandable from their point of view (and I say this as a guile weenie)
  - rurban 6y ago
    No, the reason is still slow string buffers. Politically everybody wants emacs to switch to guile.
    - bjoli 6y ago
      Iirc Guile-emacs relies on Emacs for all string related functionality. The overhead comes for dynamic bindings that are currently slower in guiles elisp implementation. I haven't looked into it since talking to Robin about it in 2016. You maybe have some more inside info, but back in 2016 I remember a lot of complaints and that guile was a liability for the much bigger Emacs project. Maybe even people threatening to retire as maintainers
  - hvis 6y ago
    It was somewhat political, but it was more of an issue of manpower. Also, Guile is basically maintained by one developer (as far as I've heard). It's pretty dangerous for Emacs to rely on a project like that.
toohotatopic 6y ago
>You can easily hack the readtable in Lisp The funny thing about lisp is that all things are easy, but yet, other eco systems offer more finished libraries.
- reikonomusha 6y ago
  Other ecosystems also become obsolete or dead within years counted on one hand. That’s perfect for today’s culture of disposable code though. Lisp isn’t good for that.
- TeMPOraL 6y ago
  That's the curse right here. Anyone can easily make a half-baked solution to their problem, so those solutions end up as half-baked libraries, if they get released at all.

by junke 6y ago

kazinator 6y ago

Emacs contains a quarter million lines of C. What do you do with all the elisp that calls into it? Maybe that could be turned into a "libemacs" and used from FFI.

junke 6y ago
Primitive Emacs Lisp functions are defined in C, yes, using helper C macros. There might be a way to extract the symbol names from those macros, and maybe enough information to generate the FFI bindings. Any kind of rewrite of Emacs is going to be a lot of work, I am not denying that. But if we assume there is a layer where everything works as it currently does (the C primitives), the Emacs Lisp defined on top of that needs not be defined in C, as far as I know. Also, the last time I checked, there were a ton of legacy stuff in the configure script for different platforms.
wtetzner 6y ago
Or maybe you rewrite it in Common Lisp. If you're going to take the Common Lisp route, I think it might be worth going all in and just re-implementing Emacs in Common Lisp.
- mikelevins 6y ago
  That's already been done. More than once, in fact. The problem is that, although there are perfectly serviceable Emacsen written in Common Lisp, none of them are GNU Emacs. For example, there's Hemlock from CMUCL, and its descendants built into Lispworks and Clozure Common Lisp. Those implementations aren't likely to work well as a substitute for GNU Emacs. For one thing, there's a substantial ecosystem of software that depends on specific APIs and other characteristics of GNU Emacs. Writing code to bridge GNU Emacs APIs with those available in the Hemlock descendants would be a lot of work. There are some other obstacles as well. CCL's implementation of Hemlock uses the Cocoa text architecture, so it's not portable to platforms other than macOS. The Lispworks implementation is portable across Windows and numerous UNIXEN, but the Lispworks license will not allow delivery of a proper substitute for GNU Emacs (it forbids building an application that can be construed as a Lisp development system, and when I pressed them about exactly what limits that policy implies, they explicitly used Emacs as an example of an application that would be forbidden). The original CMUCL Hemlock and its portable version are designed to work with CLX. It could probably be made to work in a modern X environment, but making it fit well into modern GUI environments and porting it to all the platforms GNU Emacs works on would be a huge amount of work. I'm probably overlooking some other Emacsen, but I don't know of any off the top of my head for which the situation is any easier.
  - metroholografix 6y ago
    It hasn't already been done, no. The Emacs variants you mentioned (that IMV should not be referred to as "Emacs") are from-scratch new implementations of just the concept behind Emacs, but not GNU Emacs itself. They are not compatible with GNU Emacs, they don't even support a small fraction of GNU Emacs features, and are thus doomed to obsolescence / non-existent market share. So far, nobody -that I know of- has tried to move GNU Emacs to Common Lisp. GNU Emacs is a well-known, well-used platform and moving to Common Lisp could very well be worth the upfront costs. Wishful thinking aside, the GNU Emacs development community has hunkered down behind Emacs Lisp, so improving it will bring immediate and tangible benefits to every GNU Emacs user. I love Common Lisp and do use SBCL for a lot of personal projects, but I also love GNU Emacs and I will take any improvements there that I can get. Watching Emacs Lisp become a more viable general purpose language (improved performance means an expanded set of problems that it can now address) is a great development. Stefan Monnier on emacs-devel: The main benefit of such a compiler is not to run existing Elisp code faster (99% of existing Elisp code runs fast enough that the user won't notice if it runs faster) but to make it practical to write other Elisp code which would otherwise be too slow. But for that to work well, you want the new compiler to be available "everywhere", rather than just on some platforms. So it will only start being useful when it works on GNU/Linux, macOS, Windows, ARM, RISC-V, x86, amd64, MIPS, younameit (AFAIK lbgccjit's CPU coverage is already good enough for that, so the main barrier here is the OS support), and when it's not just an option at compile-time but when it's included in all builds.
koral 6y ago
The real trouble of using Emacs C core from another language is that you have to use all the internal datastructures used by C core, and this exactly equivalent to all elisp datastructures. This implies you have either to convert back and forward everything or you just can't use the native datastructures of the new programming language (making the whole operation often quite pointless). I must confess that most of the comments in this thread seems to start from the assumption that people working on this in the last two+ decades are probably dumb, and this is sad.

ufo 6y ago

I would expect that an ELisp interpreter written in Common Lisp would likely be slower than the existing ELisp interpreter written in C. As for compilation to Common Lisp, I don't know how feasible it would be. Apparently they tried something similar with Guile and it ran into problems.

bjoli 6y ago
Guile (the VM) runs elisp just fine. There has been zero optimization work done though so it is quite slow. The reason they did not go with guile was more political than anything else. It was also understandable from their point of view (and I say this as a guile weenie)
- rurban 6y ago
  No, the reason is still slow string buffers. Politically everybody wants emacs to switch to guile.
  - bjoli 6y ago
    Iirc Guile-emacs relies on Emacs for all string related functionality. The overhead comes for dynamic bindings that are currently slower in guiles elisp implementation. I haven't looked into it since talking to Robin about it in 2016. You maybe have some more inside info, but back in 2016 I remember a lot of complaints and that guile was a liability for the much bigger Emacs project. Maybe even people threatening to retire as maintainers
- hvis 6y ago
  It was somewhat political, but it was more of an issue of manpower. Also, Guile is basically maintained by one developer (as far as I've heard). It's pretty dangerous for Emacs to rely on a project like that.

toohotatopic 6y ago

>You can easily hack the readtable in Lisp The funny thing about lisp is that all things are easy, but yet, other eco systems offer more finished libraries.

reikonomusha 6y ago
Other ecosystems also become obsolete or dead within years counted on one hand. That’s perfect for today’s culture of disposable code though. Lisp isn’t good for that.
TeMPOraL 6y ago
That's the curse right here. Anyone can easily make a half-baked solution to their problem, so those solutions end up as half-baked libraries, if they get released at all.