16 ms·

The Emacs dumper dispute

262 points by jordigh 10y ago

jmount 10y ago
Emacs is my primary editor, however Emacs dumper has always been a dumpster fire. Basically they coded up so much ill-concieved and inefficient Emacs Lisp that the editor would never start up in an acceptable time. Instead of engineering around this (lazy loading services, fixing things, not doing things nobody needs) they had the great idea they could start the editor once and then core dump the in-memory state of a running editor. Then on later editor starts they would map-in the core dump and instantly be in a (somewhat) good state. Fails all kinds of smell tests and really speaks to bad taste having unbounded consequences. It is an idea that should not work and it is only happenstance that it ever worked (and it gets harder and harder as we have things like address layout randomization, file handles, and so on). [edited "file" -> "fire", sorry! And yes I know lispers always dumped, but they are dumping the C memory environment here- not just their precious Lisp state. They should have had some appreciation for how the C environment actually worked since they decided to use it.]
- melling 10y ago
  Yes, I remember building Emacs in the late 1980's and watching the final dump. I guess 30 years later hardware hasn't improved enough to avoid this? People are building editors in JavaScript today. Perhaps the Emacs Lisp engine needs to be tuned a little more? That Guile Emacs has been decades in the making...
- fatbird 10y ago
  Sounds like it needs the neovim treatment: neomacs.
  - sooheon 10y ago
    God a neomacs written in a faster, more modern lisp would be a dream.
    - bryanlarsen 10y ago
      It's called guile emacs. https://lwn.net/Articles/615220/ https://lwn.net/Articles/615220/ https://www.emacswiki.org/emacs/GuileEmacs https://www.emacswiki.org/emacs/GuileEmacs
      - fatbird 10y ago
        Interesting. Any idea on why it's not got more traction? Or is it actually on track to become the canonical emacs? Reading that LWN article, it looks like Guile wasn't a great choice from a community perspective, but more than that, by working internally to the emacs project it's subjected itself to internal standards of interoperability that it'll never really be able to hit (where a pseudo-hostile fork like xEmacs was might have sufficient momentum to force some accommodation).
        qwertyuiop924 10y ago
        It sorta works, but it's hard to keep it working, so people don't really do much with it. Shockingly, the compatability was pretty good, last I checked. If you really want an emacs written in a sane Lisp, and don't mind losing gnuemacs compatability, check out Hemlock. or Edwin.
        draven 10y ago
        > check out Hemlock. or Edwin. Or Climacs https://www.common-lisp.net/project/climacs/ https://www.common-lisp.net/project/climacs/ It seems there's a rewrite as well: https://github.com/robert-strandh/Second-Climacs https://github.com/robert-strandh/Second-Climacs
        qwertyuiop924 10y ago
        Oh. I forgot Climacs. Well, there you go. Another CL editor.
        10y ago
  - pjmlp 10y ago
    It was called XEmacs.
  - klocksib 10y ago
    Once upon a time this basically happened: xemacs
  - hyperhopper 10y ago
    As for the more higher level stuff, spacemacs does a great job of that. It really makes configuration files, separation of concerns, vim bindings, and many other things sensible and user friendly. It doesn't touch the lower level code though, as it is just an elisp layer on top of emacs itself.
- pm215 10y ago
  I agree that dumping an image is not a great idea in the modern world, but the concept(1) dates right back to when emacs was still based on TECO on the PDP-10: http://pdp-10.trailing-edge.com/mit_emacs_170_teco_1220/01/info/mkdump.info.html http://pdp-10.trailing-edge.com/mit_emacs_170_teco_1220/01/i... (and I suspect the available CPU horsepower back then was such that you pretty much needed to do tricks like this to get quick startup even if you were careful about how much lisp code you ran at startup). (1) as a part of emacs, that is -- I'd guess that "language runtime takes an image of its pre-initialized state and then just loads that" was used in non-emacs contexts before that.
  - lispm 10y ago
    The first Lisp from the early 60s did already dump images.
    - int_19h 10y ago
      Did it dump Lisp images, or the image of the entire process (including all the non-Lisp stuff)? It appears that all the complexity and instability here comes from the latter part. Lisp itself was always designed around this image dumping concept. But C and its standard library were not.
      - qwertyuiop924 10y ago
        Actually, many early unixes included undump, which let you build an executable from a coredump.
        int_19h 10y ago
        And what were the stability guarantees for such things? Could you expect to take such a dump, bring it to a different system (potentially with different hardware, different minor OS revision etc), and expect it to work?
        jrochkind1 10y ago
- btilly 10y ago
  It is an idea that should not work and it is only happenstance that it ever worked In the 1980s it was a surprisingly common approach. For example most Unix variants had an undump command to turn a core dump into an executable. See https://www.ctan.org/tex-archive/obsolete/support/undump https://www.ctan.org/tex-archive/obsolete/support/undump for example. It was also how Microsoft Office file formats were implemented until about a decade ago. With the hindsight of experience, it is easy to say that it is a bad way to go and won't age well. But back when Emacs went this way, the problems were not as widely understood as they are now.
  - sooheon 10y ago
    Now that we do have the hindsight of experience, and it is time to make that decision again, why are the maintainers going with the clearly smellier approach?
    - jeremiep 10y ago
      Because even with experience you don't refactor a 40+ years old piece of software easily without breaking half the ecosystem. I'm sure there's a whole world of tradeoffs we're not seeing that the maintainers have to deal with.
    - MBCook 10y ago
      They have an immediate problem (things are slow/broken on newer glibc) that needs fixing. Someone has posted sane code to fix the issue in a similar fashion to what they were doing before but without needing to reach into the memory manager. The maintainer who is mad wants it fixed right (replace the whole thing and make the elisp load fast) but so far neither they nor anyone else seems to have posted code to do that. Bird in the hand. I would hope they'd just remove the need for the mechanism totally in the future, but for now.... bird in the hand.
      - btilly 10y ago
        What would fixing it right look like? Make elisp faster. Easier said than done. Any easy wins already happened. Introduce various lazy loading mechanisms and switch to that by default. That doesn't help any third party plugins that people might use. In fact any such work could create ways of breaking the expectations of said plugins! Identify/eliminate work spent on dead code paths in initial load. Again, easier said than done. And there are bound to be a lot of, "I don't see what that is useful for, but what third party plugins might need it?" In the meantime, end users faced with the choice of a slow new release versus a fast old one, will resist upgrading. And if there is a patch that simply makes things fast, that is generally acceptable. The current moment provides pressure to do things right. If the cheap fix is taken, there are no guarantees on when there will be another opportunity like this one. But pressures exist to take the easy out right now. And if taken, it won't be easier next time either.
- youdontknowtho 10y ago
  did you mean dumpster file or dumpster fire? I Kid! I KID!!!! EDIT: Seriously? Humor's not allowed? What if you read my comment with Triumph The Insult Comic Dog's voice? See? It's funny! EDIT EDIT: I'm actually an EMACS fan. It's weird that you have to propitiate yourself to a text editor over a joke.
  - youdontknowtho 10y ago
    Dude. That was comedy gold! Downvotes? Really? Come on! That was funny. Down vote all you like, that was funny.
    - deleted 10y ago
      [deleted]
- tumba 10y ago
  This may seem strange to you, but this is actually the way many lisp systems (and other image-based development systems, like Smalltalk) work. I build software using the commercial develop environment Allegro Common Lisp and an image dump is precisely the way I deliver software. The ultimate deliverable is an executable that loads the image and launches the main execution thread.
  - weavie 10y ago
    This works really well for us. To load up our web server takes a while due to having to load up caches etc.. so we tend to do all that once, dump the image and deploy it - cache already primed. It had an even bigger advantage for me today. Due to some git shenanigans I somehow managed to wipe out a few days of work. No idea how... Almost resigning myself to having to recode the whole lot I just remembered that I had built an image to test just prior to attempting to check in. Clozure CL saves the source code of each function in the image. I was able to probe the image and pull out all the source code that I had just lost, saving myself a few days of tedious recoding! Not a recommended use of images - but it sure saved my butt today!
    - gkop 10y ago
      > Due to some git shenanigans I somehow managed to wipe out a few days of work. What happened??
      - weavie 10y ago
        I actually have no idea. I had a lot of unchecked work - about a weeks worth. (I know, I should not have let it go that far.. it was just one of those tasks that just seemed to get bigger and bigger.) It was time to check in, so I went through file by file staging it all (using Magit). I then decided to do some more testing. Built a lisp image. Then I became distracted, started thinking about home time etc... so my recollection of what I did is a little hazy. I think I saw I had some changes that I wasn't wanting to commit and test. So I think I stashed them. Then probably surfed hacker news or something.. When I got back to things I suddenly noticed that none of my changes were there any more. Nothing was staged, nothing was stashed. It was all gone. I spent a few hours looking around for it but all to no avail. It is the second time I have lost work by misusing git. I think the other time I was messing with rebasing and branching and all my work disappeared. I do really need to learn how to use git properly. But in the meantime I am not taking any more chances. I have set up a zone on our internal SmartOS server which uses ZFS. An hourly cron job on my machine rsyncs my source directory to the zone and then creates a ZFS snapshot. So, hopefully no matter how badly I use git now I will never be able to lose more than about an hours worth of work.
- kazinator 10y ago
  They didn't have this great idea; it came from the Lisp culture. Lisp implementations have the ability to save images. It's not described in ANSI CL, but all the "industrial strength" Lisp have this, and it is one of the means for application delivery. If you're working on an interpreted Lisp which is getting slow to load due to its growing library, image dumping and restarting is the obvious "off the shelf" approach you already know about from being a Lisp programmer. > Instead of engineering around this (lazy loading services I'd be surprised if Emacs didn't have lazy loading too. Lazy loading only goes so far. Sometimes lazy loading brings in a lot of dependencies, so it is slow for those users who use whatever is being lazily loaded. Lazy loading is a hack to work around the lack of a fast image restore. The thing that is hacky about Emacs image saving is that it saves the C state. It's not exactly a Lisp image saving mechanism. This is why I was abel to port this dump/undump very easily to GNU Make, which doesn't share any data structures with Emacs, let alone any Lisp stuff. Dumping the image worked just fine for the C data structures on GNU Make's heap. It's because it's such a low-level mechanism that they are having difficulties with its portability. I don't hear about, for instance, the CLISP project having issues with the EXT:SAVEINITMEM function due to some GLibc support changing; it doesn't rely on that.
  - AceJohnny2 10y ago
    > I'd be surprised if Emacs didn't have lazy loading too. Not sure if this is exactly what you're referring to, but Emacs has the autoload [1] feature, which lets you declare some functions and in which elisp file to find them. Emacs won't load that file until the function is actually called. This is usually for user-called functions. Most packages use it. [1] https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload.html https://www.gnu.org/software/emacs/manual/html_node/elisp/Au...
- sedachv 10y ago
  > Instead of engineering around this (lazy loading services, fixing things, not doing things nobody needs) they had the great idea they could start the editor once and then core dump the in-memory state of a running editor. Then on later editor starts they would map-in the core dump and instantly be in a (somewhat) good state. Fails all kinds of smell tests and really speaks to bad taste having unbounded consequences. How else would you compile a dynamic programming language program into a binary executable?
  - sedachv 10y ago
    BTW, something that just came to mind: this "bad taste having unbounded consequences" of copying the entire process memory is exactly how fork(2) works.
    - firethief 10y ago
      Doing something that relies on OS internals is bad taste when done in userspace. Interface stability aside, what fork does is simpler: a copied process can rely on its environment matching its parent's enough for a large subset of the contents of the duplicated address space to retain their meaning. Serializing a process's address space's contents and then loading it into an "unrelated" process later makes it more complicated to know what memory contents can still be relied on (e.g. ASLR, file handles, etc.).
      - armitron 10y ago
        Nothing of what Emacs or Common Lisp systems or Smalltalk systems do when they dump images relies on OS internals. Emacs unexec relied on glibc internals but the new portable dumper does away with that and only relies on gasp Emacs internals. Where do you people come up with that stuff?
      - sedachv 10y ago
        > Interface stability aside, what fork does is simpler: a copied process can rely on its environment matching its parent's enough for a large subset of the contents of the duplicated address space to retain their meaning. Serializing a process's address space's contents and then loading it into an "unrelated" process later makes it more complicated to know what memory contents can still be relied on (e.g. ASLR, file handles, etc.). Everything you mentioned except for ASLR is a problem for forked processes, and it boils down to kernel state (file descriptors and signals being two that I can think of right now).
- gkafkg8y8 10y ago
  > Emacs dumper has always been a dumpster fire. This made me laugh. Thanks!
kazinator 10y ago
I ported undump from Emacs to GNU Make once! I was working in an organization that developed a big network switch, with a large C++ application running on it whose non-recursive Makefile took 30 seconds just to load and parse all of the include makefiles throughout the tree, before actually building anything. Half a minute of waiting just waiting for that second that it then takes to recompile a single .cpp to a single .o and link everything. I got tired and added a "make --dump" option which used the GNU Emacs undump code to dump an image of make with all the rules loaded from the Makefile. Then "make --restart" would instantly fire off the incremental rebuilds. (Of course, any changes to the makefiles or generated dependency makefiles required a new --dump to be taken to have an accurate rule tree.) Another idea would be just to add a darn REPL to make, so you can keep it running and just re-evaluate the rule tree.
- sqeaky 10y ago
  Isn't the ninja build system just a faster replacement for make. I use CMake primarily it lets me create makefiles, ninja files or vs projects. I noticed that in the general case ninja is 5% to 10% faster than make on builds that are more than few seconds long. Then I found that if I did serious file manipulation at build time, like copy trees of files dependent on other thing in the build, I could have tens of thousands of targets, one per file usually. Ninja might hiccup for a fraction of a second on these shenanigans but make often sits and spins for 20 or minutes. Unless you want to write the makefile yourself why not use ninja?
  - kazinator 10y ago
    Because then I have to ask users to install ninja before they can build my program. In the projects I'm working on, I don't have any of the issues that ninja solves.
    - viraptor 10y ago
      You can generate both. That gives you ninja for development (which had a problem you described) while the official builds still can use the makefiles that should result in the same output.
AdmiralAsshat 10y ago
Why is the core of Emacs still written in C? Is it just fear of trying to rewrite it from the ground up in Lisp, or is there something fundamental to the architecture that makes Lisp unsuitable?
- Jtsummers 10y ago
  History, it's been in continuous development for over three decades now. And, being open source, a port of the C portions to a new language would require developer effort that doesn't always appear on demand. Also portability. GNU Emacs supports a fair number of platforms and, consequently, has a lot of things included in it for different platforms, different presentation layers (GUI, CLI). It's not a small task to support the variety of environments that it presently does with a novel implementation. And then there's the bootstrap issue. Emacs lisp isn't compiled to a standard binary executable format, they use their own bytecode. So even if you did port more portions that are in C to elisp, you still need an environment to run that elisp in, and the problem that the dumper file solves is actually made worse, more elisp needs to be loaded and executed before you get into a functional editor. There is work to port elisp to run on type of Guile, but it's not done yet. Lacking developer resources and other things. Like the issue of losing support for various environments, it also doesn't 100% support the existing base of elisp packages out there, which is a deal breaker for many people.
- dima55 10y ago
  Because C compiles to machine code and computers only understand machine code.
- AceJohnny2 10y ago
  Because that's the part that needs to interact with the OS at a low-level. Think of it as the "OS bindings" if you will. C is the lowest-common-denominator to do that. Edit: I'm wrong, see avar's response below. It looks like it's for performance reasons.
  - avar 10y ago
    Relatively speaking this really couldn't be further from the truth. There's plenty of codebases that implement some programming language and only use C bindings for truly low-level primitives, such as external library calls, memory allocation etc, leaving any substantial logic that's built on top of those OS primitives to a higher level language. Emacs is not such a codebase, most of the C code by volume is things that could perfectly well be implemented in Emacs Lisp itself, but aren't because Emacs Lisp is relatively slow. So while it's fine for "scripting" Emacs itself, things like regexes, anything that has to do with low-level character handling, most of the GUI layout of Emacs itself (i.e. the buffer logic etc, not actually calling ncurses or X) etc. is written in C. Just browse through the C files in src/, most of this clearly has nothing to do with interacting with low-level OS primitives: http://git.savannah.gnu.org/cgit/emacs.git/tree/src http://git.savannah.gnu.org/cgit/emacs.git/tree/src
branchless 10y ago
I'm a little confused by this. I can understand why someone might want to preserve a repl env across invocations however the main reason given is startup time. Given emacs daemon, which allows connecting with a thin client, how much startup time are we talking here? Can they not start once and connect like the rest of us?
- cmiles74 10y ago
  My understanding is that this dumping of state is done during the Emacs build process. You build Emacs, initialize it and then dump out it's state. Every time you launch Emacs, your instance is starting from the dumped state.
  - branchless 10y ago
    But why? We have an emacs init process that is fast enough, why not put your init script in version control and bootstrap emacs this way?
    - Jtsummers 10y ago
      Emacs has two init stages, you only see the latter. The first is what gets it to the dump file. The second is where your personal .emacs file gets loaded and executed. The first is the one that takes too long and motivated them to create the dumper system to begin with, and needs some resolution (elimination and making that first init process faster, switching to a serialization of the objects rather than the C program memory, or something else).
      - branchless 10y ago
        So am I using this dumper system every time I launch emacs? I start emacs "normally" in that I simply launch it from the cmd line with the daemon flag.
        vilhelm_s 10y ago
cmiles74 10y ago
I'm not sure I understand, but this seems like a lot of work for little reward. Newer versions of Emacs function with the C library that's missing these dumper hooks and if you have a newer C library, you can update to a new version of Emacs. This seems like a lot of work for people who want to stick with an older release.
- massysett 10y ago
  Perhaps a related question: how essential is this functionality? I use Emacs on Mac OS X, which does not even have glibc. Yet the article suggests that this is only present on glibc. So should I be appalled at how long it takes my Emacs to load? Seems to me it's just as fast as it was on GNU/Linux.
  - to3m 10y ago
    It is supported on OS X as well: https://github.com/emacs-mirror/emacs/blob/master/src/unexmacosx.c https://github.com/emacs-mirror/emacs/blob/master/src/unexma... (And Windows: https://github.com/emacs-mirror/emacs/blob/master/src/unexw32.c https://github.com/emacs-mirror/emacs/blob/master/src/unexw3..., https://github.com/emacs-mirror/emacs/blob/master/src/w32heap.c#L27 https://github.com/emacs-mirror/emacs/blob/master/src/w32hea...)
Philipp__ 10y ago
Emacs and Vim are great tools. Amazing pieces of software. But in some areas you really see the prospect of time and aging. I used and learned both in some point in time, liked something from both, sticked to Vim, but I kinda felt the best text editor would be hybrid of these two. (Now don't run and tell me to install EVIL in Emacs, I tried that, but modal editing is not the only thing that gives Vim edge over Emacs) What got my attention recently is Xi (developed by Raph Levien). It is written in Rust, looks fairly interesting, can't wait for it to be in advanced state. I really wouldn't mind some nice, modern, terminal text editor. (I use NeoVim at the moment, and I think it is closest to that, but VimL :cringe:)
- eridius 10y ago
  Huh, apparently Xi is actually a Google project? I didn't realize anyone at Google was even touching Rust. https://github.com/google/xi-editor https://github.com/google/xi-editor Edit: Well, not actually an official Google project, but it's still on their GitHub account. The last line of the README says > This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.
  - sooheon 10y ago
    > The editor should never, ever block and prevent the user from getting their work done. A thousand times this. Emacs and vim are both guilty of failing this most basic principle. It frustrates me to no end that a text editor will hang while font locking text, or printing repl results.
    - qwertyuiop924 10y ago
      ...I have never had that problem. Well, okay. I did once. But still, it's uncommon.
      - sooheon 10y ago
        Maybe eval'ing to the repl is almost a tic of mine. But as a contrived example, (range 100000) is instantaneous in a clojure repl in the terminal, and interminably slow in cider (emacs package for clojure). There's also SO questions like this[1], and I've personally come across it a lot as well. Anytime lines get long emacs grinds to a halt. [1]: http://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow http://emacs.stackexchange.com/questions/598/how-do-i-preven...
chriswarbo 10y ago
So according to the article there are three approaches: - The existing implementation, which misbehaves with newer glibc - The partially-implemented patches, which replace the existing core dump hacks with a big pile of C - Some unspecified, potential future optimisations to the elisp loader To me, that looks more like a roadmap than a dispute.
- korethr 10y ago
  To me, the dispute is coming from what appears to be Zaretskii wanting nothing less than step 3. I can sympathize; I don't think I would want to receive a big pile of C were I a maintainer. It seems to me that temporary hacks tend to become permanent. However, I think he's mistaken. As you said, that list makes a decent roadmap. IMO, the best approach to hand is to take the currently offered patch, and continue to work towards specifying and implementing optimizations to the elisp loader.
- rurban 10y ago
  you forgot the forth: - use your own malloc, favorably ptmalloc3. This way you are independent on glibc changes and platform quirks. It is faster, has arena support, and should have been the default glibc malloc anyway.
zeveb 10y ago
From the comments to the article: > If Emacs adopted one of the proposals to use a standard Lisp dialect That should be 'if Emacs adopted the standard Lisp dialect.' There's only one standard Lisp, and that's Common Lisp[1][2]. I do indeed think that it would be wonderful if the elisp engine were reimplemented in Lisp, but it's a tremendous amount of work, with a lot of potential incompatibilities in the short term, with very little to show until the long term. [1] That doesn't mean that other Lisp-like languages aren't awesome. Racket, in particular, leaps to mind as something which is massively cool. Clojure isn't to my taste, but I can understand why some folks like it. Scheme has its virtues too. But none of them is Lisp, unlike elisp, which really is a Lisp. [2] There are also EuLisp & ISLISP, but they're effectively dead.
- AceJohnny2 10y ago
  I've been using Emacs for over a decade now. Every now and then (say, once a week), I feel the need to fiddle with some ELisp code. I read Stevey's Emergency Elisp guide: http://steve-yegge.blogspot.com/2008/01/emergency-elisp.html http://steve-yegge.blogspot.com/2008/01/emergency-elisp.html I reached the part about the then clause of if/then/else needing a (progn ...) if it was to be a multi-statement clause, but the else clause doesn't. But you can avoid the progn if you don't have an else clause by using when instead of if! I facepalmed so hard I needed reconstructive surgery. I haven't fully recovered from that. (in reality it's because, though ELisp itself is bad enough, Emacs' API is daunting)
  - qwertyuiop924 10y ago
    That's common in Lisps. Just use cond and when if it bothers you.
    - AceJohnny2 10y ago
      Why do Lisps separate these related semantics? Are there implementation reasons, or is it mostly "the way it's always been"? Edit: thanks folks! You've helped me get over the facepalm from years ago :)
AceJohnny2 10y ago
In a related discussion, it's interesting to read this overview of work required to get Emacs to support double-buffering on X11. Interestingly, it's by the same guy who proposed the new dumper patch, Daniel Colascione (who lurks around here) https://www.facebook.com/notes/daniel-colascione/buttery-smooth-emacs/10155313440066102 https://www.facebook.com/notes/daniel-colascione/buttery-smo... (Notes are Facebook's newish blog format) HN discussion: https://news.ycombinator.com/item?id=12830206 https://news.ycombinator.com/item?id=12830206
BipolarElsa 10y ago
I have a question: what's the benefit of using a text editor when an IDE can perform compiling/interpreting for you at the ready? Is a Emacs/Vim for coders who don't have to worry about minor errors? I'm relatively new to programming and I'm curious.
- swolchok 10y ago
  A text editor will always be with you, no matter what you're working on. IDEs tend to be for a specific host platform, language, and sometimes target platform. For example, Xcode only runs on a Mac, understands a limited set of languages, and is best used for building Mac and iOS apps. MS Visual Studio only runs on Windows, understands more but different languages, and is best used for building for Microsoft platforms. Android Studio is best used for Android things. IntelliJ and Eclipse are best used for Java. If anyone has heard of your programming language, there is probably an Emacs mode for it.
  - throwanem 10y ago
    More to the point, Xcode, Visual Studio, Android Studio, IntelliJ, Eclipse, and so forth all have generally different UIs, which means that unless you spend all your development time in a single environment, you have to do a lot of really expensive mental context switching. An editor like Emacs or Vim provides a unified user interface for all the different languages and development environments with which you use it, which eliminates almost all of that mental overhead - and can also invoke your compiler, (usually) integrate with your debugger, et cetera. IDEs still win on tight integration, but for any language popular enough that someone's invested the time to build a well-integrated IDE for it, Emacs or Vim will generally cover at least the 90% cases quite well, too.
  - CJefferson 10y ago
    Although often, is not very good. The Emacs support for C++11 was awful for years. May be better now, I left Emacs because it.
    - kevbin 10y ago
oelmekki 10y ago
> it's also the sort of thing that could give vi a definitive advantage in the interminable editor wars Is editors war still a thing? I was under the impression that emacs, vim and more recent editors (sublime and atom, to name two) each found their core audience and were quite distinct. Or maybe it's just that I've been using vim for long enough. I can change my main language every few years, but I can't see myself changing my editor. Maybe the "editor war" is more about hesitating between editors at the beginning.
- deleted 10y ago
  [deleted]
- caf 10y ago
  I think you'll find that is just Corbet's very dry sense of humour.
- flukus 10y ago
  It's still a thing, we're just temporarily united against some common enemies...
e40 10y ago
Rather than try to capture the state of the C library's memory-allocation subsystem, it simply marshals and saves the set of Elisp objects known to the editor. This is what Allegro CL (and I presume all the other Common Lisps) have done for 30+ years. I'm surprised they didn't move to the marshalling idea before, for fear of the Glibc hacks going away. EDIT: Actually, unexec() was used in the early days, so the 30+ years is wrong. It's been more like 20+ years.
rurban 10y ago
Interestingly Emacs is not the only project affected by this glibc/dumper dispute. I added the stone-old patches for perl to my cperl fork, to be able to dump/compile perl scripts to native binaries the fast way. Improved dumpers are here: https://github.com/perl11/cperl/commits/feature/gh176-unexec https://github.com/perl11/cperl/commits/feature/gh176-unexec Mostly unified error handling and a few darwin segment instabilities. It is very fragile to use with a static library, but ok as dynamic library. Emacs uses the dumper in the main exe, not in a library. Solaris is the easiest to use. So I know a little bit of the troubles they are talking about here. Dan's portable dumper would be nice to have, XEmacs had this decades ago, but it never made it over to Stallman emacs. Wonder why :) So looking at the new pdump, it really is horrible. I don't think I want to do that. I'll rather add a proper static malloc to cperl, such ptmalloc3, which is better than glibc malloc, i.e. ptmalloc2, anyway. They never switched to the better version, because it had more memory overhead. And I really can make use of the arena support there. Emacs should try the same. Much easier and much faster. Good bye glibc.
yason 10y ago
Never even heard about the dumper in Emacs. It sounds like a crazy idea that is prone to break at the slightest change. Any shelving-unshelving implementation for processes should come from the kernel which owns the virtual memory mappings: kernel can basically already shelve a running process by just swapping it out to disk completely. And even that scheme is prone to break as soon as anything will do I/O. FYI, as for Emacs: I just fire up "emacs -nw" inside tmux and let it run for months. I call make-frame-on-display to add a window on my X session but I can close that or restart X without having to kill my Emacs inside tmux, along with a few other long-running processes such as mail reader and IRC clients.
jblow 10y ago
Zaretskii's stance is weird. If you are going to run out of people who can work on the core of the editor's source code, then the editor will die. So the lack of ability to work on the code is the real problem. This is probably because it has accreted way too much complexity at this point, and way too many hacks. Shedding some of those hacks is a very good idea. If you wanted to keep it the old way, and depend on the nuances of how an allocator stores memory, then ship your own allocator. Video game people do this as a matter of course; it's not a big deal.
dzdt 10y ago
I am actually surprised the dumper paradigm doesn't get more love. Startup time is an issue for most large programs. The dumper route is a generally applicable way to drastically improve startup time. Think of it as splitting your code into two parts: setup phase which is run at compile time (i.e. pre-dump) and run phase which runs at run time. Undumping substitutes a simple load of a file for the setup phase. What is not to love?
- tener 10y ago
  > What is not to love? The complexity of that solution. By nature it is very fragile and leads to nasty bugs. But I can see certainly see the benefit of this approach - would be interesting to see this applied to some big frameworks/VMs, such as Java or .NET.
  - mobilerisotto 10y ago
    I've read something about dumping JVM state to decrease Clojure startup time (don't recall the details and I'm on mobile right now to check), so if I'm not wrong it was tried before...
- barrkel 10y ago
  At one point Borland C++ precompiled header files used this approach: dump the in-memory object graph for the header file whole. Makes more sense than for a whole executable.
dpc_pw 10y ago
I don't get it. In 2016, on i7 with 32GB of RAM, 2 stripped SSDs, emacs (spacemacs actually) still takes around 1-2s to start which forces me to use emacsclient, and that is already the second stage of it's initialization started from the snapshot? I'm impressed.
drfuchs 10y ago
OK, if you promise to stay off my lawn, I'll explain the history behind undump. Back in the 70's, the big CS departments typically had DEC 36-bit mainframes (PDP-10, PDP-20) running the Tops10/Tops20/Tenex/Waits/Sail family of operating systems. These are what Knuth used to do all of TeX, McCarthy LISP, and Stallman and Steele EMACS. Not Unix; and Linus hadn't touched a computer yet. Executable program files were not much more than memory images; to run a program, the OS pretty much just mapped the executable image into your address space and jumped to the start. But when the program stopped, your entire state was still there, sitting in your address space. If the program had stopped due to a crash of some sort, or if it had been in an infinite loop and you had hit control-C to interrupt it, the program was still sitting there, even though you were staring at the command prompt. And the OS had a basic debugging capability built-in, so you could simply start snooping around at the memory state of the halted program. You could continue a suspended program, or you could even restart it without the OS having to reload it from disk. It was kind of a work-space model. Translating into Linux-ish, it's as if you always used control-Z instead of control-C, and the exit() system call also behaved like control-Z; and gdb was a builtin function of the shell that you could invoke no matter how your program happened to have been paused, and it worked on the current paused process rather than a core file (which didn't exist). The OS also had a built-in command to allow you to SAVE the current memory image back into a new executable file. There wasn't much to this command, either, since executables weren't much more than a memory image to begin with. So, the equivalent of dump/undump was really just built into the OS, and wasn't considered any big deal or super-special feature. Of course, all language runtimes knew all about this, so they were always written to understand as a matter of course that they had to be able to deal with it properly. It pretty much came naturally if you were used to that environment, and wasn't a burden. Thus, when TeX (and I presume the various Lisp and Emacs and etc. that were birthed on these machines) were designed, it was completely expected that they'd work this way. Cycles were expensive, as was IO; so in TeX's case, for example, it took many seconds to read in the basic macro package and standard set of font metric files and to preprocess the hyphenation patterns into their data structure. By doing a SAVE of the resulting preloaded executable once during installation, everyone then saved these many seconds each time they ran TeX. But when TeX was ported over to Unix (and then Linux), it came as a bit of a surprise that the model was different, and that there was no convenient, predefined way to get this functionality, and that the runtimes weren't typically set up to make it easy to do. The undump stuff was created to deal with it, but it was never pretty, since it was bolted on. And many of use from those days wonder why there's still no good solution in the *nix world when there are still plenty of programs that take too damn long to start up.
gumby 10y ago
The dumper approach in Emacs predates GNU Emacs; it was how the original TECO Emacs worked. The real fix would be to make dynamically linkable compiled elisp files (i.e. .so files) and let the system linker make Emacs start quickly just like any other program.
- sedachv 10y ago
  > The real fix would be to make dynamically linkable compiled elisp files (i.e. .so files) and let the system linker make Emacs start quickly just like any other program. Dynamic linking is not quick! https://en.wikipedia.org/wiki/Prelink https://en.wikipedia.org/wiki/Prelink
  - gumby 10y ago
    That's just the cacheing that makes it fast. Obviously it's slower at some level than a static binary like a dump, but no big deal.
gkafkg8y8 10y ago
I've loved and used Emacs for ~20 years, but if Emacs were to become slow, then if I were to have a replacement editor that could do the following (w/no X or window manager) without additional config in Linux, I'd use it instead: arrow keys to move add and delete text anywhere paste from terminal buffer ctrl-s -> search (and continue to find next match) ctrl-v -> down ctrl-esc -> up ctrl-k -> kill line ctrl-x ctrl-s -> save ctrl-x ctrl-c -> quit ctrl-a -> goto beginning of line ctrl-e -> goto end of line I don't even use selection anymore, because I can just use the terminal window copy/paste.
- stevekemp 10y ago
  Antirez, of redis fame, wrote a minimal editor in C which got a lot of love here on hacker news a while back: https://news.ycombinator.com/item?id=12065217 https://news.ycombinator.com/item?id=12065217 I took that release and added Lua scripting support to it, which also seemed reasonably popular: https://news.ycombinator.com/item?id=12137698 https://news.ycombinator.com/item?id=12137698 It pretty much meets your requirements, and was a fun project. But it did make me realize that writing editors is sometimes harder than you'd think.
  - gkafkg8y8 10y ago
    Kilo's supported commands are[1]: CTRL-S: Save CTRL-Q: Quit CTRL-F: Find string in file (ESC to exit search, arrows to navigate) It's available in a lot of well-used distros: https://pkgs.org/search/kilo https://pkgs.org/search/kilo but doesn't look like it's in Arch, etc. Kilua looks cool also as it has more similar keybindings to Emacs[2]: Ctrl-x Ctrl-o Open a new file in the current buffer. Ctrl-x Ctrl-s Save the current file. Ctrl-x Ctrl-c Quit. Ctrl-x c Create a new buffer Ctrl-x n Move to the next buffer. Ctrl-x p Move to the previous buffer. Ctrl-x b Select buffer from a list M-x Evaluate lua at the prompt. Ctrl-r: Regular expression search. but the goal would be to have that available in a package manager in a default install, so that after logging into any server where I'm a sudoer, I could: sudo apt-get install some_package sudo yum -y install some_package pacman -S some_package ... [1]: https://github.com/antirez/kilo https://github.com/antirez/kilo [2]: https://github.com/skx/kilua https://github.com/skx/kilua
gonzo 10y ago
I ported emacs undump to the Convex machines 29 years ago. http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/1.0-RELEASE/ports/emacs/src/unexconvex.c http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-relea... Emacs is old and moldy. Let it die.
gomijacogeo 10y ago
Why not use LLVM and create a proper .so or binary?

262 points by jordigh 10y ago

Emacs is my primary editor, however Emacs dumper has always been a dumpster fire. Basically they coded up so much ill-concieved and inefficient Emacs Lisp that the editor would never start up in an acceptable time. Instead of engineering around this (lazy loading services, fixing things, not doing things nobody needs) they had the great idea they could start the editor once and then core dump the in-memory state of a running editor. Then on later editor starts they would map-in the core dump and instantly be in a (somewhat) good state. Fails all kinds of smell tests and really speaks to bad taste having unbounded consequences. It is an idea that should not work and it is only happenstance that it ever worked (and it gets harder and harder as we have things like address layout randomization, file handles, and so on). [edited "file" -> "fire", sorry! And yes I know lispers always dumped, but they are dumping the C memory environment here- not just their precious Lisp state. They should have had some appreciation for how the C environment actually worked since they decided to use it.]

melling 10y ago
Yes, I remember building Emacs in the late 1980's and watching the final dump. I guess 30 years later hardware hasn't improved enough to avoid this? People are building editors in JavaScript today. Perhaps the Emacs Lisp engine needs to be tuned a little more? That Guile Emacs has been decades in the making...
fatbird 10y ago
Sounds like it needs the neovim treatment: neomacs.
- sooheon 10y ago
  God a neomacs written in a faster, more modern lisp would be a dream.
  - bryanlarsen 10y ago
    It's called guile emacs. https://lwn.net/Articles/615220/ https://lwn.net/Articles/615220/ https://www.emacswiki.org/emacs/GuileEmacs https://www.emacswiki.org/emacs/GuileEmacs
    - fatbird 10y ago
      Interesting. Any idea on why it's not got more traction? Or is it actually on track to become the canonical emacs? Reading that LWN article, it looks like Guile wasn't a great choice from a community perspective, but more than that, by working internally to the emacs project it's subjected itself to internal standards of interoperability that it'll never really be able to hit (where a pseudo-hostile fork like xEmacs was might have sufficient momentum to force some accommodation).
      - qwertyuiop924 10y ago
        It sorta works, but it's hard to keep it working, so people don't really do much with it. Shockingly, the compatability was pretty good, last I checked. If you really want an emacs written in a sane Lisp, and don't mind losing gnuemacs compatability, check out Hemlock. or Edwin.
        draven 10y ago
        > check out Hemlock. or Edwin. Or Climacs https://www.common-lisp.net/project/climacs/ https://www.common-lisp.net/project/climacs/ It seems there's a rewrite as well: https://github.com/robert-strandh/Second-Climacs https://github.com/robert-strandh/Second-Climacs
        qwertyuiop924 10y ago
        Oh. I forgot Climacs. Well, there you go. Another CL editor.
        10y ago
- pjmlp 10y ago
  It was called XEmacs.
- klocksib 10y ago
  Once upon a time this basically happened: xemacs
- hyperhopper 10y ago
  As for the more higher level stuff, spacemacs does a great job of that. It really makes configuration files, separation of concerns, vim bindings, and many other things sensible and user friendly. It doesn't touch the lower level code though, as it is just an elisp layer on top of emacs itself.
pm215 10y ago
I agree that dumping an image is not a great idea in the modern world, but the concept(1) dates right back to when emacs was still based on TECO on the PDP-10: http://pdp-10.trailing-edge.com/mit_emacs_170_teco_1220/01/info/mkdump.info.html http://pdp-10.trailing-edge.com/mit_emacs_170_teco_1220/01/i... (and I suspect the available CPU horsepower back then was such that you pretty much needed to do tricks like this to get quick startup even if you were careful about how much lisp code you ran at startup). (1) as a part of emacs, that is -- I'd guess that "language runtime takes an image of its pre-initialized state and then just loads that" was used in non-emacs contexts before that.
- lispm 10y ago
  The first Lisp from the early 60s did already dump images.
  - int_19h 10y ago
    Did it dump Lisp images, or the image of the entire process (including all the non-Lisp stuff)? It appears that all the complexity and instability here comes from the latter part. Lisp itself was always designed around this image dumping concept. But C and its standard library were not.
    - qwertyuiop924 10y ago
      Actually, many early unixes included undump, which let you build an executable from a coredump.
      - int_19h 10y ago
        And what were the stability guarantees for such things? Could you expect to take such a dump, bring it to a different system (potentially with different hardware, different minor OS revision etc), and expect it to work?
        jrochkind1 10y ago
btilly 10y ago
It is an idea that should not work and it is only happenstance that it ever worked In the 1980s it was a surprisingly common approach. For example most Unix variants had an undump command to turn a core dump into an executable. See https://www.ctan.org/tex-archive/obsolete/support/undump https://www.ctan.org/tex-archive/obsolete/support/undump for example. It was also how Microsoft Office file formats were implemented until about a decade ago. With the hindsight of experience, it is easy to say that it is a bad way to go and won't age well. But back when Emacs went this way, the problems were not as widely understood as they are now.
- sooheon 10y ago
  Now that we do have the hindsight of experience, and it is time to make that decision again, why are the maintainers going with the clearly smellier approach?
  - jeremiep 10y ago
    Because even with experience you don't refactor a 40+ years old piece of software easily without breaking half the ecosystem. I'm sure there's a whole world of tradeoffs we're not seeing that the maintainers have to deal with.
  - MBCook 10y ago
    They have an immediate problem (things are slow/broken on newer glibc) that needs fixing. Someone has posted sane code to fix the issue in a similar fashion to what they were doing before but without needing to reach into the memory manager. The maintainer who is mad wants it fixed right (replace the whole thing and make the elisp load fast) but so far neither they nor anyone else seems to have posted code to do that. Bird in the hand. I would hope they'd just remove the need for the mechanism totally in the future, but for now.... bird in the hand.
    - btilly 10y ago
      What would fixing it right look like? Make elisp faster. Easier said than done. Any easy wins already happened. Introduce various lazy loading mechanisms and switch to that by default. That doesn't help any third party plugins that people might use. In fact any such work could create ways of breaking the expectations of said plugins! Identify/eliminate work spent on dead code paths in initial load. Again, easier said than done. And there are bound to be a lot of, "I don't see what that is useful for, but what third party plugins might need it?" In the meantime, end users faced with the choice of a slow new release versus a fast old one, will resist upgrading. And if there is a patch that simply makes things fast, that is generally acceptable. The current moment provides pressure to do things right. If the cheap fix is taken, there are no guarantees on when there will be another opportunity like this one. But pressures exist to take the easy out right now. And if taken, it won't be easier next time either.
youdontknowtho 10y ago
did you mean dumpster file or dumpster fire? I Kid! I KID!!!! EDIT: Seriously? Humor's not allowed? What if you read my comment with Triumph The Insult Comic Dog's voice? See? It's funny! EDIT EDIT: I'm actually an EMACS fan. It's weird that you have to propitiate yourself to a text editor over a joke.
- youdontknowtho 10y ago
  Dude. That was comedy gold! Downvotes? Really? Come on! That was funny. Down vote all you like, that was funny.
  - deleted 10y ago
    [deleted]
tumba 10y ago
This may seem strange to you, but this is actually the way many lisp systems (and other image-based development systems, like Smalltalk) work. I build software using the commercial develop environment Allegro Common Lisp and an image dump is precisely the way I deliver software. The ultimate deliverable is an executable that loads the image and launches the main execution thread.
- weavie 10y ago
  This works really well for us. To load up our web server takes a while due to having to load up caches etc.. so we tend to do all that once, dump the image and deploy it - cache already primed. It had an even bigger advantage for me today. Due to some git shenanigans I somehow managed to wipe out a few days of work. No idea how... Almost resigning myself to having to recode the whole lot I just remembered that I had built an image to test just prior to attempting to check in. Clozure CL saves the source code of each function in the image. I was able to probe the image and pull out all the source code that I had just lost, saving myself a few days of tedious recoding! Not a recommended use of images - but it sure saved my butt today!
  - gkop 10y ago
    > Due to some git shenanigans I somehow managed to wipe out a few days of work. What happened??
    - weavie 10y ago
      I actually have no idea. I had a lot of unchecked work - about a weeks worth. (I know, I should not have let it go that far.. it was just one of those tasks that just seemed to get bigger and bigger.) It was time to check in, so I went through file by file staging it all (using Magit). I then decided to do some more testing. Built a lisp image. Then I became distracted, started thinking about home time etc... so my recollection of what I did is a little hazy. I think I saw I had some changes that I wasn't wanting to commit and test. So I think I stashed them. Then probably surfed hacker news or something.. When I got back to things I suddenly noticed that none of my changes were there any more. Nothing was staged, nothing was stashed. It was all gone. I spent a few hours looking around for it but all to no avail. It is the second time I have lost work by misusing git. I think the other time I was messing with rebasing and branching and all my work disappeared. I do really need to learn how to use git properly. But in the meantime I am not taking any more chances. I have set up a zone on our internal SmartOS server which uses ZFS. An hourly cron job on my machine rsyncs my source directory to the zone and then creates a ZFS snapshot. So, hopefully no matter how badly I use git now I will never be able to lose more than about an hours worth of work.
kazinator 10y ago
They didn't have this great idea; it came from the Lisp culture. Lisp implementations have the ability to save images. It's not described in ANSI CL, but all the "industrial strength" Lisp have this, and it is one of the means for application delivery. If you're working on an interpreted Lisp which is getting slow to load due to its growing library, image dumping and restarting is the obvious "off the shelf" approach you already know about from being a Lisp programmer. > Instead of engineering around this (lazy loading services I'd be surprised if Emacs didn't have lazy loading too. Lazy loading only goes so far. Sometimes lazy loading brings in a lot of dependencies, so it is slow for those users who use whatever is being lazily loaded. Lazy loading is a hack to work around the lack of a fast image restore. The thing that is hacky about Emacs image saving is that it saves the C state. It's not exactly a Lisp image saving mechanism. This is why I was abel to port this dump/undump very easily to GNU Make, which doesn't share any data structures with Emacs, let alone any Lisp stuff. Dumping the image worked just fine for the C data structures on GNU Make's heap. It's because it's such a low-level mechanism that they are having difficulties with its portability. I don't hear about, for instance, the CLISP project having issues with the EXT:SAVEINITMEM function due to some GLibc support changing; it doesn't rely on that.
- AceJohnny2 10y ago
  > I'd be surprised if Emacs didn't have lazy loading too. Not sure if this is exactly what you're referring to, but Emacs has the autoload [1] feature, which lets you declare some functions and in which elisp file to find them. Emacs won't load that file until the function is actually called. This is usually for user-called functions. Most packages use it. [1] https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload.html https://www.gnu.org/software/emacs/manual/html_node/elisp/Au...
sedachv 10y ago
> Instead of engineering around this (lazy loading services, fixing things, not doing things nobody needs) they had the great idea they could start the editor once and then core dump the in-memory state of a running editor. Then on later editor starts they would map-in the core dump and instantly be in a (somewhat) good state. Fails all kinds of smell tests and really speaks to bad taste having unbounded consequences. How else would you compile a dynamic programming language program into a binary executable?
- sedachv 10y ago
  BTW, something that just came to mind: this "bad taste having unbounded consequences" of copying the entire process memory is exactly how fork(2) works.
  - firethief 10y ago
    Doing something that relies on OS internals is bad taste when done in userspace. Interface stability aside, what fork does is simpler: a copied process can rely on its environment matching its parent's enough for a large subset of the contents of the duplicated address space to retain their meaning. Serializing a process's address space's contents and then loading it into an "unrelated" process later makes it more complicated to know what memory contents can still be relied on (e.g. ASLR, file handles, etc.).
    - armitron 10y ago
      Nothing of what Emacs or Common Lisp systems or Smalltalk systems do when they dump images relies on OS internals. Emacs unexec relied on glibc internals but the new portable dumper does away with that and only relies on gasp Emacs internals. Where do you people come up with that stuff?
    - sedachv 10y ago
      > Interface stability aside, what fork does is simpler: a copied process can rely on its environment matching its parent's enough for a large subset of the contents of the duplicated address space to retain their meaning. Serializing a process's address space's contents and then loading it into an "unrelated" process later makes it more complicated to know what memory contents can still be relied on (e.g. ASLR, file handles, etc.). Everything you mentioned except for ASLR is a problem for forked processes, and it boils down to kernel state (file descriptors and signals being two that I can think of right now).
gkafkg8y8 10y ago
> Emacs dumper has always been a dumpster fire. This made me laugh. Thanks!

kazinator 10y ago

I ported undump from Emacs to GNU Make once! I was working in an organization that developed a big network switch, with a large C++ application running on it whose non-recursive Makefile took 30 seconds just to load and parse all of the include makefiles throughout the tree, before actually building anything. Half a minute of waiting just waiting for that second that it then takes to recompile a single .cpp to a single .o and link everything. I got tired and added a "make --dump" option which used the GNU Emacs undump code to dump an image of make with all the rules loaded from the Makefile. Then "make --restart" would instantly fire off the incremental rebuilds. (Of course, any changes to the makefiles or generated dependency makefiles required a new --dump to be taken to have an accurate rule tree.) Another idea would be just to add a darn REPL to make, so you can keep it running and just re-evaluate the rule tree.

sqeaky 10y ago
Isn't the ninja build system just a faster replacement for make. I use CMake primarily it lets me create makefiles, ninja files or vs projects. I noticed that in the general case ninja is 5% to 10% faster than make on builds that are more than few seconds long. Then I found that if I did serious file manipulation at build time, like copy trees of files dependent on other thing in the build, I could have tens of thousands of targets, one per file usually. Ninja might hiccup for a fraction of a second on these shenanigans but make often sits and spins for 20 or minutes. Unless you want to write the makefile yourself why not use ninja?
- kazinator 10y ago
  Because then I have to ask users to install ninja before they can build my program. In the projects I'm working on, I don't have any of the issues that ninja solves.
  - viraptor 10y ago
    You can generate both. That gives you ninja for development (which had a problem you described) while the official builds still can use the makefiles that should result in the same output.

AdmiralAsshat 10y ago

Why is the core of Emacs still written in C? Is it just fear of trying to rewrite it from the ground up in Lisp, or is there something fundamental to the architecture that makes Lisp unsuitable?

Jtsummers 10y ago
History, it's been in continuous development for over three decades now. And, being open source, a port of the C portions to a new language would require developer effort that doesn't always appear on demand. Also portability. GNU Emacs supports a fair number of platforms and, consequently, has a lot of things included in it for different platforms, different presentation layers (GUI, CLI). It's not a small task to support the variety of environments that it presently does with a novel implementation. And then there's the bootstrap issue. Emacs lisp isn't compiled to a standard binary executable format, they use their own bytecode. So even if you did port more portions that are in C to elisp, you still need an environment to run that elisp in, and the problem that the dumper file solves is actually made worse, more elisp needs to be loaded and executed before you get into a functional editor. There is work to port elisp to run on type of Guile, but it's not done yet. Lacking developer resources and other things. Like the issue of losing support for various environments, it also doesn't 100% support the existing base of elisp packages out there, which is a deal breaker for many people.
dima55 10y ago
Because C compiles to machine code and computers only understand machine code.
AceJohnny2 10y ago
Because that's the part that needs to interact with the OS at a low-level. Think of it as the "OS bindings" if you will. C is the lowest-common-denominator to do that. Edit: I'm wrong, see avar's response below. It looks like it's for performance reasons.
- avar 10y ago
  Relatively speaking this really couldn't be further from the truth. There's plenty of codebases that implement some programming language and only use C bindings for truly low-level primitives, such as external library calls, memory allocation etc, leaving any substantial logic that's built on top of those OS primitives to a higher level language. Emacs is not such a codebase, most of the C code by volume is things that could perfectly well be implemented in Emacs Lisp itself, but aren't because Emacs Lisp is relatively slow. So while it's fine for "scripting" Emacs itself, things like regexes, anything that has to do with low-level character handling, most of the GUI layout of Emacs itself (i.e. the buffer logic etc, not actually calling ncurses or X) etc. is written in C. Just browse through the C files in src/, most of this clearly has nothing to do with interacting with low-level OS primitives: http://git.savannah.gnu.org/cgit/emacs.git/tree/src http://git.savannah.gnu.org/cgit/emacs.git/tree/src

branchless 10y ago

I'm a little confused by this. I can understand why someone might want to preserve a repl env across invocations however the main reason given is startup time. Given emacs daemon, which allows connecting with a thin client, how much startup time are we talking here? Can they not start once and connect like the rest of us?

cmiles74 10y ago
My understanding is that this dumping of state is done during the Emacs build process. You build Emacs, initialize it and then dump out it's state. Every time you launch Emacs, your instance is starting from the dumped state.
- branchless 10y ago
  But why? We have an emacs init process that is fast enough, why not put your init script in version control and bootstrap emacs this way?
  - Jtsummers 10y ago
    Emacs has two init stages, you only see the latter. The first is what gets it to the dump file. The second is where your personal .emacs file gets loaded and executed. The first is the one that takes too long and motivated them to create the dumper system to begin with, and needs some resolution (elimination and making that first init process faster, switching to a serialization of the objects rather than the C program memory, or something else).
    - branchless 10y ago
      So am I using this dumper system every time I launch emacs? I start emacs "normally" in that I simply launch it from the cmd line with the daemon flag.
      - vilhelm_s 10y ago

cmiles74 10y ago

I'm not sure I understand, but this seems like a lot of work for little reward. Newer versions of Emacs function with the C library that's missing these dumper hooks and if you have a newer C library, you can update to a new version of Emacs. This seems like a lot of work for people who want to stick with an older release.

massysett 10y ago
Perhaps a related question: how essential is this functionality? I use Emacs on Mac OS X, which does not even have glibc. Yet the article suggests that this is only present on glibc. So should I be appalled at how long it takes my Emacs to load? Seems to me it's just as fast as it was on GNU/Linux.
- to3m 10y ago
  It is supported on OS X as well: https://github.com/emacs-mirror/emacs/blob/master/src/unexmacosx.c https://github.com/emacs-mirror/emacs/blob/master/src/unexma... (And Windows: https://github.com/emacs-mirror/emacs/blob/master/src/unexw32.c https://github.com/emacs-mirror/emacs/blob/master/src/unexw3..., https://github.com/emacs-mirror/emacs/blob/master/src/w32heap.c#L27 https://github.com/emacs-mirror/emacs/blob/master/src/w32hea...)

Philipp__ 10y ago

Emacs and Vim are great tools. Amazing pieces of software. But in some areas you really see the prospect of time and aging. I used and learned both in some point in time, liked something from both, sticked to Vim, but I kinda felt the best text editor would be hybrid of these two. (Now don't run and tell me to install EVIL in Emacs, I tried that, but modal editing is not the only thing that gives Vim edge over Emacs) What got my attention recently is Xi (developed by Raph Levien). It is written in Rust, looks fairly interesting, can't wait for it to be in advanced state. I really wouldn't mind some nice, modern, terminal text editor. (I use NeoVim at the moment, and I think it is closest to that, but VimL :cringe:)

eridius 10y ago
Huh, apparently Xi is actually a Google project? I didn't realize anyone at Google was even touching Rust. https://github.com/google/xi-editor https://github.com/google/xi-editor Edit: Well, not actually an official Google project, but it's still on their GitHub account. The last line of the README says > This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.
- sooheon 10y ago
  > The editor should never, ever block and prevent the user from getting their work done. A thousand times this. Emacs and vim are both guilty of failing this most basic principle. It frustrates me to no end that a text editor will hang while font locking text, or printing repl results.
  - qwertyuiop924 10y ago
    ...I have never had that problem. Well, okay. I did once. But still, it's uncommon.
    - sooheon 10y ago
      Maybe eval'ing to the repl is almost a tic of mine. But as a contrived example, (range 100000) is instantaneous in a clojure repl in the terminal, and interminably slow in cider (emacs package for clojure). There's also SO questions like this[1], and I've personally come across it a lot as well. Anytime lines get long emacs grinds to a halt. [1]: http://emacs.stackexchange.com/questions/598/how-do-i-prevent-extremely-long-lines-making-emacs-slow http://emacs.stackexchange.com/questions/598/how-do-i-preven...

chriswarbo 10y ago

So according to the article there are three approaches: - The existing implementation, which misbehaves with newer glibc - The partially-implemented patches, which replace the existing core dump hacks with a big pile of C - Some unspecified, potential future optimisations to the elisp loader To me, that looks more like a roadmap than a dispute.

korethr 10y ago
To me, the dispute is coming from what appears to be Zaretskii wanting nothing less than step 3. I can sympathize; I don't think I would want to receive a big pile of C were I a maintainer. It seems to me that temporary hacks tend to become permanent. However, I think he's mistaken. As you said, that list makes a decent roadmap. IMO, the best approach to hand is to take the currently offered patch, and continue to work towards specifying and implementing optimizations to the elisp loader.
rurban 10y ago
you forgot the forth: - use your own malloc, favorably ptmalloc3. This way you are independent on glibc changes and platform quirks. It is faster, has arena support, and should have been the default glibc malloc anyway.

zeveb 10y ago

From the comments to the article: > If Emacs adopted one of the proposals to use a standard Lisp dialect That should be 'if Emacs adopted the standard Lisp dialect.' There's only one standard Lisp, and that's Common Lisp[1][2]. I do indeed think that it would be wonderful if the elisp engine were reimplemented in Lisp, but it's a tremendous amount of work, with a lot of potential incompatibilities in the short term, with very little to show until the long term. [1] That doesn't mean that other Lisp-like languages aren't awesome. Racket, in particular, leaps to mind as something which is massively cool. Clojure isn't to my taste, but I can understand why some folks like it. Scheme has its virtues too. But none of them is Lisp, unlike elisp, which really is a Lisp. [2] There are also EuLisp & ISLISP, but they're effectively dead.

AceJohnny2 10y ago
I've been using Emacs for over a decade now. Every now and then (say, once a week), I feel the need to fiddle with some ELisp code. I read Stevey's Emergency Elisp guide: http://steve-yegge.blogspot.com/2008/01/emergency-elisp.html http://steve-yegge.blogspot.com/2008/01/emergency-elisp.html I reached the part about the then clause of if/then/else needing a (progn ...) if it was to be a multi-statement clause, but the else clause doesn't. But you can avoid the progn if you don't have an else clause by using when instead of if! I facepalmed so hard I needed reconstructive surgery. I haven't fully recovered from that. (in reality it's because, though ELisp itself is bad enough, Emacs' API is daunting)
- qwertyuiop924 10y ago
  That's common in Lisps. Just use cond and when if it bothers you.
  - AceJohnny2 10y ago
    Why do Lisps separate these related semantics? Are there implementation reasons, or is it mostly "the way it's always been"? Edit: thanks folks! You've helped me get over the facepalm from years ago :)

AceJohnny2 10y ago

In a related discussion, it's interesting to read this overview of work required to get Emacs to support double-buffering on X11. Interestingly, it's by the same guy who proposed the new dumper patch, Daniel Colascione (who lurks around here) https://www.facebook.com/notes/daniel-colascione/buttery-smooth-emacs/10155313440066102 https://www.facebook.com/notes/daniel-colascione/buttery-smo... (Notes are Facebook's newish blog format) HN discussion: https://news.ycombinator.com/item?id=12830206 https://news.ycombinator.com/item?id=12830206

BipolarElsa 10y ago

I have a question: what's the benefit of using a text editor when an IDE can perform compiling/interpreting for you at the ready? Is a Emacs/Vim for coders who don't have to worry about minor errors? I'm relatively new to programming and I'm curious.

swolchok 10y ago
A text editor will always be with you, no matter what you're working on. IDEs tend to be for a specific host platform, language, and sometimes target platform. For example, Xcode only runs on a Mac, understands a limited set of languages, and is best used for building Mac and iOS apps. MS Visual Studio only runs on Windows, understands more but different languages, and is best used for building for Microsoft platforms. Android Studio is best used for Android things. IntelliJ and Eclipse are best used for Java. If anyone has heard of your programming language, there is probably an Emacs mode for it.
- throwanem 10y ago
  More to the point, Xcode, Visual Studio, Android Studio, IntelliJ, Eclipse, and so forth all have generally different UIs, which means that unless you spend all your development time in a single environment, you have to do a lot of really expensive mental context switching. An editor like Emacs or Vim provides a unified user interface for all the different languages and development environments with which you use it, which eliminates almost all of that mental overhead - and can also invoke your compiler, (usually) integrate with your debugger, et cetera. IDEs still win on tight integration, but for any language popular enough that someone's invested the time to build a well-integrated IDE for it, Emacs or Vim will generally cover at least the 90% cases quite well, too.
- CJefferson 10y ago
  Although often, is not very good. The Emacs support for C++11 was awful for years. May be better now, I left Emacs because it.
  - kevbin 10y ago

oelmekki 10y ago

> it's also the sort of thing that could give vi a definitive advantage in the interminable editor wars Is editors war still a thing? I was under the impression that emacs, vim and more recent editors (sublime and atom, to name two) each found their core audience and were quite distinct. Or maybe it's just that I've been using vim for long enough. I can change my main language every few years, but I can't see myself changing my editor. Maybe the "editor war" is more about hesitating between editors at the beginning.

deleted 10y ago
[deleted]
caf 10y ago
I think you'll find that is just Corbet's very dry sense of humour.
flukus 10y ago
It's still a thing, we're just temporarily united against some common enemies...

e40 10y ago

Rather than try to capture the state of the C library's memory-allocation subsystem, it simply marshals and saves the set of Elisp objects known to the editor. This is what Allegro CL (and I presume all the other Common Lisps) have done for 30+ years. I'm surprised they didn't move to the marshalling idea before, for fear of the Glibc hacks going away. EDIT: Actually, unexec() was used in the early days, so the 30+ years is wrong. It's been more like 20+ years.

rurban 10y ago

Interestingly Emacs is not the only project affected by this glibc/dumper dispute. I added the stone-old patches for perl to my cperl fork, to be able to dump/compile perl scripts to native binaries the fast way. Improved dumpers are here: https://github.com/perl11/cperl/commits/feature/gh176-unexec https://github.com/perl11/cperl/commits/feature/gh176-unexec Mostly unified error handling and a few darwin segment instabilities. It is very fragile to use with a static library, but ok as dynamic library. Emacs uses the dumper in the main exe, not in a library. Solaris is the easiest to use. So I know a little bit of the troubles they are talking about here. Dan's portable dumper would be nice to have, XEmacs had this decades ago, but it never made it over to Stallman emacs. Wonder why :) So looking at the new pdump, it really is horrible. I don't think I want to do that. I'll rather add a proper static malloc to cperl, such ptmalloc3, which is better than glibc malloc, i.e. ptmalloc2, anyway. They never switched to the better version, because it had more memory overhead. And I really can make use of the arena support there. Emacs should try the same. Much easier and much faster. Good bye glibc.

yason 10y ago

Never even heard about the dumper in Emacs. It sounds like a crazy idea that is prone to break at the slightest change. Any shelving-unshelving implementation for processes should come from the kernel which owns the virtual memory mappings: kernel can basically already shelve a running process by just swapping it out to disk completely. And even that scheme is prone to break as soon as anything will do I/O. FYI, as for Emacs: I just fire up "emacs -nw" inside tmux and let it run for months. I call make-frame-on-display to add a window on my X session but I can close that or restart X without having to kill my Emacs inside tmux, along with a few other long-running processes such as mail reader and IRC clients.

jblow 10y ago

Zaretskii's stance is weird. If you are going to run out of people who can work on the core of the editor's source code, then the editor will die. So the lack of ability to work on the code is the real problem. This is probably because it has accreted way too much complexity at this point, and way too many hacks. Shedding some of those hacks is a very good idea. If you wanted to keep it the old way, and depend on the nuances of how an allocator stores memory, then ship your own allocator. Video game people do this as a matter of course; it's not a big deal.

dzdt 10y ago

I am actually surprised the dumper paradigm doesn't get more love. Startup time is an issue for most large programs. The dumper route is a generally applicable way to drastically improve startup time. Think of it as splitting your code into two parts: setup phase which is run at compile time (i.e. pre-dump) and run phase which runs at run time. Undumping substitutes a simple load of a file for the setup phase. What is not to love?

tener 10y ago
> What is not to love? The complexity of that solution. By nature it is very fragile and leads to nasty bugs. But I can see certainly see the benefit of this approach - would be interesting to see this applied to some big frameworks/VMs, such as Java or .NET.
- mobilerisotto 10y ago
  I've read something about dumping JVM state to decrease Clojure startup time (don't recall the details and I'm on mobile right now to check), so if I'm not wrong it was tried before...
barrkel 10y ago
At one point Borland C++ precompiled header files used this approach: dump the in-memory object graph for the header file whole. Makes more sense than for a whole executable.

dpc_pw 10y ago

I don't get it. In 2016, on i7 with 32GB of RAM, 2 stripped SSDs, emacs (spacemacs actually) still takes around 1-2s to start which forces me to use emacsclient, and that is already the second stage of it's initialization started from the snapshot? I'm impressed.

drfuchs 10y ago

OK, if you promise to stay off my lawn, I'll explain the history behind undump. Back in the 70's, the big CS departments typically had DEC 36-bit mainframes (PDP-10, PDP-20) running the Tops10/Tops20/Tenex/Waits/Sail family of operating systems. These are what Knuth used to do all of TeX, McCarthy LISP, and Stallman and Steele EMACS. Not Unix; and Linus hadn't touched a computer yet. Executable program files were not much more than memory images; to run a program, the OS pretty much just mapped the executable image into your address space and jumped to the start. But when the program stopped, your entire state was still there, sitting in your address space. If the program had stopped due to a crash of some sort, or if it had been in an infinite loop and you had hit control-C to interrupt it, the program was still sitting there, even though you were staring at the command prompt. And the OS had a basic debugging capability built-in, so you could simply start snooping around at the memory state of the halted program. You could continue a suspended program, or you could even restart it without the OS having to reload it from disk. It was kind of a work-space model. Translating into Linux-ish, it's as if you always used control-Z instead of control-C, and the exit() system call also behaved like control-Z; and gdb was a builtin function of the shell that you could invoke no matter how your program happened to have been paused, and it worked on the current paused process rather than a core file (which didn't exist). The OS also had a built-in command to allow you to SAVE the current memory image back into a new executable file. There wasn't much to this command, either, since executables weren't much more than a memory image to begin with. So, the equivalent of dump/undump was really just built into the OS, and wasn't considered any big deal or super-special feature. Of course, all language runtimes knew all about this, so they were always written to understand as a matter of course that they had to be able to deal with it properly. It pretty much came naturally if you were used to that environment, and wasn't a burden. Thus, when TeX (and I presume the various Lisp and Emacs and etc. that were birthed on these machines) were designed, it was completely expected that they'd work this way. Cycles were expensive, as was IO; so in TeX's case, for example, it took many seconds to read in the basic macro package and standard set of font metric files and to preprocess the hyphenation patterns into their data structure. By doing a SAVE of the resulting preloaded executable once during installation, everyone then saved these many seconds each time they ran TeX. But when TeX was ported over to Unix (and then Linux), it came as a bit of a surprise that the model was different, and that there was no convenient, predefined way to get this functionality, and that the runtimes weren't typically set up to make it easy to do. The undump stuff was created to deal with it, but it was never pretty, since it was bolted on. And many of use from those days wonder why there's still no good solution in the *nix world when there are still plenty of programs that take too damn long to start up.

gumby 10y ago

The dumper approach in Emacs predates GNU Emacs; it was how the original TECO Emacs worked. The real fix would be to make dynamically linkable compiled elisp files (i.e. .so files) and let the system linker make Emacs start quickly just like any other program.

sedachv 10y ago
> The real fix would be to make dynamically linkable compiled elisp files (i.e. .so files) and let the system linker make Emacs start quickly just like any other program. Dynamic linking is not quick! https://en.wikipedia.org/wiki/Prelink https://en.wikipedia.org/wiki/Prelink
- gumby 10y ago
  That's just the cacheing that makes it fast. Obviously it's slower at some level than a static binary like a dump, but no big deal.

gkafkg8y8 10y ago

I've loved and used Emacs for ~20 years, but if Emacs were to become slow, then if I were to have a replacement editor that could do the following (w/no X or window manager) without additional config in Linux, I'd use it instead: arrow keys to move add and delete text anywhere paste from terminal buffer ctrl-s -> search (and continue to find next match) ctrl-v -> down ctrl-esc -> up ctrl-k -> kill line ctrl-x ctrl-s -> save ctrl-x ctrl-c -> quit ctrl-a -> goto beginning of line ctrl-e -> goto end of line I don't even use selection anymore, because I can just use the terminal window copy/paste.

stevekemp 10y ago
Antirez, of redis fame, wrote a minimal editor in C which got a lot of love here on hacker news a while back: https://news.ycombinator.com/item?id=12065217 https://news.ycombinator.com/item?id=12065217 I took that release and added Lua scripting support to it, which also seemed reasonably popular: https://news.ycombinator.com/item?id=12137698 https://news.ycombinator.com/item?id=12137698 It pretty much meets your requirements, and was a fun project. But it did make me realize that writing editors is sometimes harder than you'd think.
- gkafkg8y8 10y ago
  Kilo's supported commands are[1]: CTRL-S: Save CTRL-Q: Quit CTRL-F: Find string in file (ESC to exit search, arrows to navigate) It's available in a lot of well-used distros: https://pkgs.org/search/kilo https://pkgs.org/search/kilo but doesn't look like it's in Arch, etc. Kilua looks cool also as it has more similar keybindings to Emacs[2]: Ctrl-x Ctrl-o Open a new file in the current buffer. Ctrl-x Ctrl-s Save the current file. Ctrl-x Ctrl-c Quit. Ctrl-x c Create a new buffer Ctrl-x n Move to the next buffer. Ctrl-x p Move to the previous buffer. Ctrl-x b Select buffer from a list M-x Evaluate lua at the prompt. Ctrl-r: Regular expression search. but the goal would be to have that available in a package manager in a default install, so that after logging into any server where I'm a sudoer, I could: sudo apt-get install some_package sudo yum -y install some_package pacman -S some_package ... [1]: https://github.com/antirez/kilo https://github.com/antirez/kilo [2]: https://github.com/skx/kilua https://github.com/skx/kilua

gonzo 10y ago

I ported emacs undump to the Convex machines 29 years ago. http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/1.0-RELEASE/ports/emacs/src/unexconvex.c http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-relea... Emacs is old and moldy. Let it die.

gomijacogeo 10y ago

Why not use LLVM and create a proper .so or binary?