10 ms·

Bringing GNU Emacs to native code [video]

223 points by zeveb 6y ago

fmakunbound 6y ago
This seems convoluted compared to say moving the Lisp implementation from Emacs Lisp to Common Lisp, of which several native code compiling implementations exists.
- wtetzner 6y ago
  You don't think it would be difficult to keep compatibility with all of the existing Emacs Lisp packages that exist now?
- widdershins 6y ago
  Rewriting thousands of packages (some of which have had man-decades of work poured into them) would also be rather convoluted.
  - junke 6y ago
    You don't need to rewrite packages if you have a layer where Emacs Lisp is compiled/interpreted as Common Lisp. You can easily hack the readtable in Lisp, and rewrite Emacs Lisp sexps as Common Lisp sexps. Some interesting things to consider are file-local variables, buffer-local variables, dynamic-scope/lexical-scope, how to handle floating point (-0.0e+NaN in Emacs Lisp, custom floating point rouding modes/traps in SBCL for example), but this looks like a reasonable approach overall. For example (just a draft, this might be incorrect): USER> (defun make-local-variable (symbol) (eval `(define-symbol-macro ,symbol (bvar (quote ,symbol))))) MAKE-LOCAL-VARIABLE USER> (make-local-variable 'foo) FOO USER> (macroexpand-all '(list foo)) (LIST (BVAR 'FOO)) T T The BVAR forms would then be able to access a buffer variable, with the current buffer. This works with SETF too (but Emacs Lisp SETQ should be replaced by SETF during the transform). BVAR could expand into code that calls FFI functions, for compatibility. Using an FFI approach would allow to progressively rewrite some parts of the runtime into CL. Just to clarify, I know this is a huge work to undertake, but this comes from the assumption that we want to keep all existing Elisp files running identically.
    - kazinator 6y ago
      Emacs contains a quarter million lines of C. What do you do with all the elisp that calls into it? Maybe that could be turned into a "libemacs" and used from FFI.
      - junke 6y ago
        Primitive Emacs Lisp functions are defined in C, yes, using helper C macros. There might be a way to extract the symbol names from those macros, and maybe enough information to generate the FFI bindings. Any kind of rewrite of Emacs is going to be a lot of work, I am not denying that. But if we assume there is a layer where everything works as it currently does (the C primitives), the Emacs Lisp defined on top of that needs not be defined in C, as far as I know. Also, the last time I checked, there were a ton of legacy stuff in the configure script for different platforms.
      - wtetzner 6y ago
        Or maybe you rewrite it in Common Lisp. If you're going to take the Common Lisp route, I think it might be worth going all in and just re-implementing Emacs in Common Lisp.
        mikelevins 6y ago
        That's already been done. More than once, in fact. The problem is that, although there are perfectly serviceable Emacsen written in Common Lisp, none of them are GNU Emacs. For example, there's Hemlock from CMUCL, and its descendants built into Lispworks and Clozure Common Lisp. Those implementations aren't likely to work well as a substitute for GNU Emacs. For one thing, there's a substantial ecosystem of software that depends on specific APIs and other characteristics of GNU Emacs. Writing code to bridge GNU Emacs APIs with those available in the Hemlock descendants would be a lot of work. There are some other obstacles as well. CCL's implementation of Hemlock uses the Cocoa text architecture, so it's not portable to platforms other than macOS. The Lispworks implementation is portable across Windows and numerous UNIXEN, but the Lispworks license will not allow delivery of a proper substitute for GNU Emacs (it forbids building an application that can be construed as a Lisp development system, and when I pressed them about exactly what limits that policy implies, they explicitly used Emacs as an example of an application that would be forbidden). The original CMUCL Hemlock and its portable version are designed to work with CLX. It could probably be made to work in a modern X environment, but making it fit well into modern GUI environments and porting it to all the platforms GNU Emacs works on would be a huge amount of work. I'm probably overlooking some other Emacsen, but I don't know of any off the top of my head for which the situation is any easier.
  - slightwinder 6y ago
    I wonder whether a modern modularized architecture might be the better way to handle this. Similar to neovim where you have a backend which communicates with a detatched frontend and other backends over some rpc-mechanism. Refactor the existing emacs to work headless and become the legacy-core, while also establishing channels to attatch any other core and frontend.
    - gpderetta 6y ago
      An important optimization discussed in the video is to optimize through call into C code (i.e elisp->C->elisp). adding RPC at the boundaries would go in the opposite direction.
      - WhatIsDukkha 6y ago
        Well this is not to my taste but I take it as a proof of existence that an rpc at the boundary could make for some amazing things - https://github.com/rhysd/NyaoVim https://github.com/rhysd/NyaoVim
      - slightwinder 6y ago
        It's optimization for a single process vs multiple processes. Additionally, when you free up your architecture and loose up the internal bindings between different areas, you sometimes can open chances for significant improvments which prior where not possible, thus still be faster with a "worse" solution. Considering how old GNU Emacs is, how much historical problems it has and how often you hear that this and that can't be optimized for this and that reason, this seems like such a situation where loosing up would open faster roads long term.
    - Myrmornis 6y ago
      https://github.com/xi-editor https://github.com/xi-editor is an example of the architecture you're referring to I believe. Xi's README makes it clear that very high performance is one of the main goals: in particular it mentions the latency between keypresses and screen painting. To what extent does this RPC-based architecture limit performance compared to a single-process architecture? I know that the original author of Xi very much knows what he is doing, so I guess the answer is "it doesn't" -- I was just hoping to be educated on this point.
- gumby 6y ago
  I assume you mean writing an Emacs lisp interpreter in Common Lisp — which could be done with a small lexer, a lot of macros, and some library support. That would probably be a win.
  - wtetzner 6y ago
    Depends on how difficult it is to integrate Common Lisp into Emacs. Or would you just be rewriting Emacs in Common Lisp?
    - gumby 6y ago
      The issue is the importance of all the existing packages — invalidate them and nobody will use your implementation. There are small syntactic differences between Emacs lisp and Common Lisp so a lexer that handled those would suffice. There are some small semantic differences but they could be handled by porting code. And the Emacs c code could be rewritten in lisp or ported into the Common Lisp implementation, depending on your taste.
      - wtetzner 6y ago
        The existing package ecosystem is one of the main value propositions of Emacs. Otherwise a re-implementation with a different extension language could have replaced it a long time ago. > There are some small semantic differences but they could be handled by porting code. I suspect there are more than just small differences. And someone would have to do the work to port the code. I suspect that if packages that people use don't work, they will just avoid using the new version of Emacs, as opposed to abandoning the package. It's not an impossible project, but it's not clear to me it's easier/better than the proposed solution.
        gumby 6y ago
        The point of my suggestion was to not have to rewrite any existing .el files.
- kazinator 6y ago
  It brings GCC into the address space via libgccjit. GCC is not robust enough to be integrated into applications that stay running, and be repeatedly invoked.
  - tgbugs 6y ago
    I don't think that is how libgccjit is being used in this project. Emacs is not being linked with libgccjit. libgccjit is only being use for AOT compilation.
gumby 6y ago
Note: this is a video. Here are the slides: http://akrl.sdf.org/gccemacs_els2020.pdf http://akrl.sdf.org/gccemacs_els2020.pdf
- pmiller2 6y ago
  I'd recognize a LaTeX beamer presentation anywhere. Here is the corresponding paper: https://arxiv.org/pdf/2004.02504.pdf https://arxiv.org/pdf/2004.02504.pdf
  - disgruntledphd2 6y ago
    From org-mode too, as it has the pointless Outline page that org always inserts into beamer presentations.
    - cion 6y ago
      It's not pointless, but the outline only shows up if the generated tex file is compiled twice. The default org to latex export will only compile it once. Alternatively, it is easy to suppress the outline slide altogether (add "toc:nil" to "#+OPTIONS:").
      - disgruntledphd2 6y ago
        Oh I know. I have toc:nil as my default because it annoys me. It's just a bit of a giveaway as to how the file was made.
poidos 6y ago
I tried this out on my system last night. Compile time was quite large (I would say about an hour and a half on my Ryzen 3600X.) I use the DOOM Emacs config, and was surprised to find most things working out of the box with the native compilation. I noticed no difference in startup time. The speed boost was surprisingly noticeable, however, when e.g. opening a buffer that causes a language server to start. Opening CCLS was... instant. It's usually quite quick, but this was noticeably faster. Great job all around!
- tincholio 6y ago
  Similar story for me (though unfortunately on a few years old i7, so the compile time was well over 2h). I haven't had any stability issues with it, either, it just works.
- s-km 6y ago
  does it do anything to change/fix Emacs shitting itself on large buffers? last time i gave it a try opening a large file would completely kill performance, and iirc in particular really long lines (ex. 1000+ chars) would make the thing chug even if the actual file wasn't super big or anything.
  - xxpor 6y ago
    Are you opening it in fundamental mode? Without actually doing profiling, I'd guess it's syntax highlighting that's the real issue there. I've opened 100+ MB files without an issue, but that was on a box with an absolute ton of RAM.
    - throwaway_pdp09 6y ago
      Nah. IME it really is just line length.
  - BeetleB 6y ago
    Large buffers are fine - long lines are not. If you don't have long lines but you are experiencing slowdowns on large buffers, your mode is doing something. As an example, large XML files bring my Emacs to a crawl, but if I switch to a text mode it's fine. For long lines, I believe the latest Emacs has a fix: https://www.reddit.com/r/emacs/comments/ccoksw/solong_mitigating_slowness_due_to_extremely_long/ https://www.reddit.com/r/emacs/comments/ccoksw/solong_mitiga...
harrygeez 6y ago
Just when I thought I know enough compsci to understand everything on a sufficiently high level, this guy totally lost me starting from LIMPLE
- philsnow 6y ago
  Compilers are "just" a series of transformations / translations from higher-level code to lower-level code. The top is code like C, python, elisp, whatever, and the bottom is machine code for amd64, arm7, whatever. All the in-between code is in some intermediate representation (IR). Each successive step takes care of different optimizations, modifying the code as it goes down. At the last step, he converts LIMPLE to an IR (intermediate representation) that libgccjit understands, and hands it off to gcc for native compilation. Could you just start with elisp and emit amd64 machine code in one step? Absolutely, but it would be hell to maintain, and then you lose out on all the pluggability of modern compilers. If you (consume and/or) emit standard(-ish) IRs, you get to participate in a pretty amazing ecosystem.
  - __s 6y ago
    Also even if you do "one step" you'll want to have an intermediate data format which is SSA for the more efficient optimizations
gpderetta 6y ago
As a exclusive emacs user for the last 20 years, I'm quite excited; my main complain about emacs is its slow down with some more sophisticated packages. I wonder if it improves magit performance with large codebases. Pigs fly just fine with with enough thrust.
- metroholografix 6y ago
  One reason magit is slow (that last I checked still hasn't been fixed) is that it spawns a large number of processes calling out to git for each operation. Most of these are redundant and could/should go away with either a redesign or a working caching scheme. This issue is more than obvious on platforms (e.g. some macOS versions) where fork/vfork are not as fast as one would expect. I like the paradigm behind magit, but the implementation leaves a lot to be desired. So the pig in this case is definitely not Emacs. Example: # Doing a simple magit refresh results in the following processes being spawned (tracked with dtrace) # Notice how many calls are completely redundant 2020 May 4 12:46:22 11819 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11820 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11821 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11822 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11823 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11824 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11825 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11826 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11827 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11828 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11829 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11830 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11831 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11832 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11833 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11834 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11835 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11836 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11837 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11838 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11839 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11840 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11841 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11842 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11843 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11844 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11845 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11846 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>
tgbugs 6y ago
I have a Gentoo ebuild [0] almost working for this (I think I'm missing the step where the eln files are loaded before dumping the base image). The compile times are ... substantial. However they don't affect the development process for new code since the interpreter is always there, and I am excited to see what performance gains we will see. From an engineering perspective this is a excellent example of a direct path from interpreted to compiled code. The trade-offs are clear (heck, they are number 0-3), and while there is complexity, all the engineering time has been effectively concentrated inside a single project, rather than forced upon tens of thousands of maintainers and users. Bravo. I wonder what other bytecode interpreters could benefit from this toolchain. Compile times from qlop. On a Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz Without comp `2020-05-03T15:33:27 >>> app-editors/emacs: 6′40″` With compile `2020-05-03T18:35:43 >>> app-editors/emacs: 2:11:45` 0. https://github.com/tgbugs/tgbugs-overlay/blob/master/app-editors/emacs/emacs-28.0.9999-r1.ebuild https://github.com/tgbugs/tgbugs-overlay/blob/master/app-edi...
- aoeuhtns 6y ago
  nice, thanks for the ebuild!
register 6y ago
I posted the same content 5 days ago here: https://news.ycombinator.com/item?id=23021574 https://news.ycombinator.com/item?id=23021574 No reaction at all. I am really curios to understand what I did wrong at the time.
- saagarjha 6y ago
  Nothing at all. You just lost the Hacker News lottery.
- guenthert 6y ago
  And to add insult to injury, your comment is being down-voted. It's a cruel web.
saagarjha 6y ago
Looks really cool! One thing I noticed was the generated code seemed to have fairly poor register allocation, looking more like it was just pulling things straight out from locals into registers and immediately storing them back. From the talk, it looks like that was what was being provided to libgccjit, but surely it could optimize that further?
nemoniac 6y ago
Very keen to try this out but that branch won't compile for me. What's the trick?
schiffl 6y ago
Did anyone successfully compile it for x86 32-bit target? The docker image is 64-bit and the compilation seems to be hitting the gcc-i386 limit of 3GiB.
ken 6y ago
"Arguably the most deployed Lisp today?" That's an interesting question. My other guess would have been Gimp, and based on a quick glance at Debian's popcon, I'd say Gimp might be slightly in the lead. Then again, like Open Firmware deployed Forth on millions of computers, right under our noses, it would not surprise me at all if there were a simple Lisp implementation hidden on every computer in the world. Is there anything more recent than DSSSL?
gongyiliao 6y ago
Here some rough, non-scientific testing/benchmark statistics. It took me 124.98 mins to build the native-comp branch with a 4-cores/8-threads i7-4790k, 32GB RAM on a LXC instance while the master branch took me 244 seconds. Both branches are obtained from latest available git snapshot as 2020-5-4 20:20 CDT. The following function is passed to (benchmark-run-compiled 10 ...) for each run. ;; -*- lexical-binding: t -*- (require 'cl-lib) (defun bf-1 nil (/ (apply '+ (cl-loop repeat 300000 collect (cl-random 1.0))) 300000.0)) gc-cons-threshold is set as 268435456 (~ 256MB) Before each run, (garbage-collect) function is called. With Emacs's built-in core lisp functions, cl-lib functions and the above benchmark function are all native-compiled, it took 0.5823 second to complete. With Emacs built-ins and cl-lib are native-compiled but the benchmark function is byte-compiled, it takes 0.6411 second to complete. With byte-compiled Emacs built-in/cl-lib/benchmark functions, it takes 1.3574 second to complete. With byte-compiled Emacs built-in/cl-lib functions and interpreted benchmark function, it takes 78.054 seconds with 1 GC taking 75.094 second, which implies the execution roughly takes 2.96 seconds. I also ran same benchmark on a 4-core A10-6800k. and observed similar ratios on the builds from 2020-5-3.

223 points by zeveb 6y ago

This seems convoluted compared to say moving the Lisp implementation from Emacs Lisp to Common Lisp, of which several native code compiling implementations exists.

wtetzner 6y ago
You don't think it would be difficult to keep compatibility with all of the existing Emacs Lisp packages that exist now?
widdershins 6y ago
Rewriting thousands of packages (some of which have had man-decades of work poured into them) would also be rather convoluted.
- junke 6y ago
  You don't need to rewrite packages if you have a layer where Emacs Lisp is compiled/interpreted as Common Lisp. You can easily hack the readtable in Lisp, and rewrite Emacs Lisp sexps as Common Lisp sexps. Some interesting things to consider are file-local variables, buffer-local variables, dynamic-scope/lexical-scope, how to handle floating point (-0.0e+NaN in Emacs Lisp, custom floating point rouding modes/traps in SBCL for example), but this looks like a reasonable approach overall. For example (just a draft, this might be incorrect): USER> (defun make-local-variable (symbol) (eval `(define-symbol-macro ,symbol (bvar (quote ,symbol))))) MAKE-LOCAL-VARIABLE USER> (make-local-variable 'foo) FOO USER> (macroexpand-all '(list foo)) (LIST (BVAR 'FOO)) T T The BVAR forms would then be able to access a buffer variable, with the current buffer. This works with SETF too (but Emacs Lisp SETQ should be replaced by SETF during the transform). BVAR could expand into code that calls FFI functions, for compatibility. Using an FFI approach would allow to progressively rewrite some parts of the runtime into CL. Just to clarify, I know this is a huge work to undertake, but this comes from the assumption that we want to keep all existing Elisp files running identically.
  - kazinator 6y ago
    Emacs contains a quarter million lines of C. What do you do with all the elisp that calls into it? Maybe that could be turned into a "libemacs" and used from FFI.
    - junke 6y ago
      Primitive Emacs Lisp functions are defined in C, yes, using helper C macros. There might be a way to extract the symbol names from those macros, and maybe enough information to generate the FFI bindings. Any kind of rewrite of Emacs is going to be a lot of work, I am not denying that. But if we assume there is a layer where everything works as it currently does (the C primitives), the Emacs Lisp defined on top of that needs not be defined in C, as far as I know. Also, the last time I checked, there were a ton of legacy stuff in the configure script for different platforms.
    - wtetzner 6y ago
      Or maybe you rewrite it in Common Lisp. If you're going to take the Common Lisp route, I think it might be worth going all in and just re-implementing Emacs in Common Lisp.
      - mikelevins 6y ago
        That's already been done. More than once, in fact. The problem is that, although there are perfectly serviceable Emacsen written in Common Lisp, none of them are GNU Emacs. For example, there's Hemlock from CMUCL, and its descendants built into Lispworks and Clozure Common Lisp. Those implementations aren't likely to work well as a substitute for GNU Emacs. For one thing, there's a substantial ecosystem of software that depends on specific APIs and other characteristics of GNU Emacs. Writing code to bridge GNU Emacs APIs with those available in the Hemlock descendants would be a lot of work. There are some other obstacles as well. CCL's implementation of Hemlock uses the Cocoa text architecture, so it's not portable to platforms other than macOS. The Lispworks implementation is portable across Windows and numerous UNIXEN, but the Lispworks license will not allow delivery of a proper substitute for GNU Emacs (it forbids building an application that can be construed as a Lisp development system, and when I pressed them about exactly what limits that policy implies, they explicitly used Emacs as an example of an application that would be forbidden). The original CMUCL Hemlock and its portable version are designed to work with CLX. It could probably be made to work in a modern X environment, but making it fit well into modern GUI environments and porting it to all the platforms GNU Emacs works on would be a huge amount of work. I'm probably overlooking some other Emacsen, but I don't know of any off the top of my head for which the situation is any easier.
- slightwinder 6y ago
  I wonder whether a modern modularized architecture might be the better way to handle this. Similar to neovim where you have a backend which communicates with a detatched frontend and other backends over some rpc-mechanism. Refactor the existing emacs to work headless and become the legacy-core, while also establishing channels to attatch any other core and frontend.
  - gpderetta 6y ago
    An important optimization discussed in the video is to optimize through call into C code (i.e elisp->C->elisp). adding RPC at the boundaries would go in the opposite direction.
    - WhatIsDukkha 6y ago
      Well this is not to my taste but I take it as a proof of existence that an rpc at the boundary could make for some amazing things - https://github.com/rhysd/NyaoVim https://github.com/rhysd/NyaoVim
    - slightwinder 6y ago
      It's optimization for a single process vs multiple processes. Additionally, when you free up your architecture and loose up the internal bindings between different areas, you sometimes can open chances for significant improvments which prior where not possible, thus still be faster with a "worse" solution. Considering how old GNU Emacs is, how much historical problems it has and how often you hear that this and that can't be optimized for this and that reason, this seems like such a situation where loosing up would open faster roads long term.
  - Myrmornis 6y ago
    https://github.com/xi-editor https://github.com/xi-editor is an example of the architecture you're referring to I believe. Xi's README makes it clear that very high performance is one of the main goals: in particular it mentions the latency between keypresses and screen painting. To what extent does this RPC-based architecture limit performance compared to a single-process architecture? I know that the original author of Xi very much knows what he is doing, so I guess the answer is "it doesn't" -- I was just hoping to be educated on this point.
gumby 6y ago
I assume you mean writing an Emacs lisp interpreter in Common Lisp — which could be done with a small lexer, a lot of macros, and some library support. That would probably be a win.
- wtetzner 6y ago
  Depends on how difficult it is to integrate Common Lisp into Emacs. Or would you just be rewriting Emacs in Common Lisp?
  - gumby 6y ago
    The issue is the importance of all the existing packages — invalidate them and nobody will use your implementation. There are small syntactic differences between Emacs lisp and Common Lisp so a lexer that handled those would suffice. There are some small semantic differences but they could be handled by porting code. And the Emacs c code could be rewritten in lisp or ported into the Common Lisp implementation, depending on your taste.
    - wtetzner 6y ago
      The existing package ecosystem is one of the main value propositions of Emacs. Otherwise a re-implementation with a different extension language could have replaced it a long time ago. > There are some small semantic differences but they could be handled by porting code. I suspect there are more than just small differences. And someone would have to do the work to port the code. I suspect that if packages that people use don't work, they will just avoid using the new version of Emacs, as opposed to abandoning the package. It's not an impossible project, but it's not clear to me it's easier/better than the proposed solution.
      - gumby 6y ago
        The point of my suggestion was to not have to rewrite any existing .el files.
kazinator 6y ago
It brings GCC into the address space via libgccjit. GCC is not robust enough to be integrated into applications that stay running, and be repeatedly invoked.
- tgbugs 6y ago
  I don't think that is how libgccjit is being used in this project. Emacs is not being linked with libgccjit. libgccjit is only being use for AOT compilation.

gumby 6y ago

Note: this is a video. Here are the slides: http://akrl.sdf.org/gccemacs_els2020.pdf http://akrl.sdf.org/gccemacs_els2020.pdf

pmiller2 6y ago
I'd recognize a LaTeX beamer presentation anywhere. Here is the corresponding paper: https://arxiv.org/pdf/2004.02504.pdf https://arxiv.org/pdf/2004.02504.pdf
- disgruntledphd2 6y ago
  From org-mode too, as it has the pointless Outline page that org always inserts into beamer presentations.
  - cion 6y ago
    It's not pointless, but the outline only shows up if the generated tex file is compiled twice. The default org to latex export will only compile it once. Alternatively, it is easy to suppress the outline slide altogether (add "toc:nil" to "#+OPTIONS:").
    - disgruntledphd2 6y ago
      Oh I know. I have toc:nil as my default because it annoys me. It's just a bit of a giveaway as to how the file was made.

poidos 6y ago

I tried this out on my system last night. Compile time was quite large (I would say about an hour and a half on my Ryzen 3600X.) I use the DOOM Emacs config, and was surprised to find most things working out of the box with the native compilation. I noticed no difference in startup time. The speed boost was surprisingly noticeable, however, when e.g. opening a buffer that causes a language server to start. Opening CCLS was... instant. It's usually quite quick, but this was noticeably faster. Great job all around!

tincholio 6y ago
Similar story for me (though unfortunately on a few years old i7, so the compile time was well over 2h). I haven't had any stability issues with it, either, it just works.
s-km 6y ago
does it do anything to change/fix Emacs shitting itself on large buffers? last time i gave it a try opening a large file would completely kill performance, and iirc in particular really long lines (ex. 1000+ chars) would make the thing chug even if the actual file wasn't super big or anything.
- xxpor 6y ago
  Are you opening it in fundamental mode? Without actually doing profiling, I'd guess it's syntax highlighting that's the real issue there. I've opened 100+ MB files without an issue, but that was on a box with an absolute ton of RAM.
  - throwaway_pdp09 6y ago
    Nah. IME it really is just line length.
- BeetleB 6y ago
  Large buffers are fine - long lines are not. If you don't have long lines but you are experiencing slowdowns on large buffers, your mode is doing something. As an example, large XML files bring my Emacs to a crawl, but if I switch to a text mode it's fine. For long lines, I believe the latest Emacs has a fix: https://www.reddit.com/r/emacs/comments/ccoksw/solong_mitigating_slowness_due_to_extremely_long/ https://www.reddit.com/r/emacs/comments/ccoksw/solong_mitiga...

harrygeez 6y ago

Just when I thought I know enough compsci to understand everything on a sufficiently high level, this guy totally lost me starting from LIMPLE

philsnow 6y ago
Compilers are "just" a series of transformations / translations from higher-level code to lower-level code. The top is code like C, python, elisp, whatever, and the bottom is machine code for amd64, arm7, whatever. All the in-between code is in some intermediate representation (IR). Each successive step takes care of different optimizations, modifying the code as it goes down. At the last step, he converts LIMPLE to an IR (intermediate representation) that libgccjit understands, and hands it off to gcc for native compilation. Could you just start with elisp and emit amd64 machine code in one step? Absolutely, but it would be hell to maintain, and then you lose out on all the pluggability of modern compilers. If you (consume and/or) emit standard(-ish) IRs, you get to participate in a pretty amazing ecosystem.
- __s 6y ago
  Also even if you do "one step" you'll want to have an intermediate data format which is SSA for the more efficient optimizations

gpderetta 6y ago

As a exclusive emacs user for the last 20 years, I'm quite excited; my main complain about emacs is its slow down with some more sophisticated packages. I wonder if it improves magit performance with large codebases. Pigs fly just fine with with enough thrust.

metroholografix 6y ago
One reason magit is slow (that last I checked still hasn't been fixed) is that it spawns a large number of processes calling out to git for each operation. Most of these are redundant and could/should go away with either a redesign or a working caching scheme. This issue is more than obvious on platforms (e.g. some macOS versions) where fork/vfork are not as fast as one would expect. I like the paradigm behind magit, but the implementation leaves a lot to be desired. So the pig in this case is definitely not Emacs. Example: # Doing a simple magit refresh results in the following processes being spawned (tracked with dtrace) # Notice how many calls are completely redundant 2020 May 4 12:46:22 11819 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11820 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11821 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11822 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11823 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11824 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11825 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11826 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11827 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11828 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11829 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11830 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11831 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11832 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11833 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11834 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11835 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11836 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11837 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11838 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11839 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11840 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...> 2020 May 4 12:46:22 11841 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11842 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11843 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11844 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11845 <67471> 64b /opt/local/bin/git --no-pager -c core.preloadindex=true -c log.showSignature=false <...> 2020 May 4 12:46:22 11846 <67471> 64b /opt/local/bin/git --no-pager --literal-pathspecs -c core.preloadindex=true -c <...>

tgbugs 6y ago

I have a Gentoo ebuild [0] almost working for this (I think I'm missing the step where the eln files are loaded before dumping the base image). The compile times are ... substantial. However they don't affect the development process for new code since the interpreter is always there, and I am excited to see what performance gains we will see. From an engineering perspective this is a excellent example of a direct path from interpreted to compiled code. The trade-offs are clear (heck, they are number 0-3), and while there is complexity, all the engineering time has been effectively concentrated inside a single project, rather than forced upon tens of thousands of maintainers and users. Bravo. I wonder what other bytecode interpreters could benefit from this toolchain. Compile times from qlop. On a Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz Without comp `2020-05-03T15:33:27 >>> app-editors/emacs: 6′40″` With compile `2020-05-03T18:35:43 >>> app-editors/emacs: 2:11:45` 0. https://github.com/tgbugs/tgbugs-overlay/blob/master/app-editors/emacs/emacs-28.0.9999-r1.ebuild https://github.com/tgbugs/tgbugs-overlay/blob/master/app-edi...

aoeuhtns 6y ago
nice, thanks for the ebuild!

I posted the same content 5 days ago here: https://news.ycombinator.com/item?id=23021574 https://news.ycombinator.com/item?id=23021574 No reaction at all. I am really curios to understand what I did wrong at the time.

saagarjha 6y ago
Nothing at all. You just lost the Hacker News lottery.
guenthert 6y ago
And to add insult to injury, your comment is being down-voted. It's a cruel web.

saagarjha 6y ago

Looks really cool! One thing I noticed was the generated code seemed to have fairly poor register allocation, looking more like it was just pulling things straight out from locals into registers and immediately storing them back. From the talk, it looks like that was what was being provided to libgccjit, but surely it could optimize that further?

nemoniac 6y ago

Very keen to try this out but that branch won't compile for me. What's the trick?

schiffl 6y ago

Did anyone successfully compile it for x86 32-bit target? The docker image is 64-bit and the compilation seems to be hitting the gcc-i386 limit of 3GiB.

ken 6y ago

"Arguably the most deployed Lisp today?" That's an interesting question. My other guess would have been Gimp, and based on a quick glance at Debian's popcon, I'd say Gimp might be slightly in the lead. Then again, like Open Firmware deployed Forth on millions of computers, right under our noses, it would not surprise me at all if there were a simple Lisp implementation hidden on every computer in the world. Is there anything more recent than DSSSL?

gongyiliao 6y ago

Here some rough, non-scientific testing/benchmark statistics. It took me 124.98 mins to build the native-comp branch with a 4-cores/8-threads i7-4790k, 32GB RAM on a LXC instance while the master branch took me 244 seconds. Both branches are obtained from latest available git snapshot as 2020-5-4 20:20 CDT. The following function is passed to (benchmark-run-compiled 10 ...) for each run. ;; -*- lexical-binding: t -*- (require 'cl-lib) (defun bf-1 nil (/ (apply '+ (cl-loop repeat 300000 collect (cl-random 1.0))) 300000.0)) gc-cons-threshold is set as 268435456 (~ 256MB) Before each run, (garbage-collect) function is called. With Emacs's built-in core lisp functions, cl-lib functions and the above benchmark function are all native-compiled, it took 0.5823 second to complete. With Emacs built-ins and cl-lib are native-compiled but the benchmark function is byte-compiled, it takes 0.6411 second to complete. With byte-compiled Emacs built-in/cl-lib/benchmark functions, it takes 1.3574 second to complete. With byte-compiled Emacs built-in/cl-lib functions and interpreted benchmark function, it takes 78.054 seconds with 1 GC taking 75.094 second, which implies the execution roughly takes 2.96 seconds. I also ran same benchmark on a 4-core A10-6800k. and observed similar ratios on the builds from 2020-5-3.