9 ms·

Removing support for Emacs unexec from Glibc

106 points by jordigh 11y ago

_yosefk 11y ago
Actually, having a generic way of saving the program state into an executable that then runs from that point could be really useful, and some Unix versions had it (and Perl used a Unix utility called undump to implement a "Perl compiler" where it dumped core that could be "undumped" into a running program later; that's the same idea as "unexecing" program state into an executable.)
- protopete 11y ago
  I logged in to say the same thing. More info here: http://perldoc.perl.org/functions/dump.html http://perldoc.perl.org/functions/dump.html
- johntb86 11y ago
  http://criu.org/Main_Page http://criu.org/Main_Page
  - Freaky 11y ago
    Also: https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpoint&section=2 https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpo...
- Freaky 11y ago
  It's notable that the Solaris unexec() implementation[1] is basically a thin wrapper around dldump()[2]. 1: http://git.savannah.gnu.org/cgit/emacs.git/tree/src/unexsol.c http://git.savannah.gnu.org/cgit/emacs.git/tree/src/unexsol.... 2: https://docs.oracle.com/cd/E19455-01/806-0627/6j9vhfmop/index.html https://docs.oracle.com/cd/E19455-01/806-0627/6j9vhfmop/inde...
- deleted 11y ago
  [deleted]
fidget 11y ago
> LWN subscriber-only content kinda shitty
- lultimouomo 11y ago
  If you try scrolling down, you'll note you can actually read the article. LWN lets subscribers share (temporary) paywalled articles with anyone they like, even on public forums like HN. They are nice like that.
  - fidget 11y ago
    Oh, I'm aware I can read it. I'm just not convinced about sharing it on such a broad public forum.
    - lultimouomo 11y ago
      Sorry, I misunderstood what you meant. Links to LWN subscriber content appear on HN from time to time - not too often, and I would guess they do more good than harm to them.
      - markrages 11y ago
        Often such links are posted by LWN staff, so I believe they favor the practice.
    - cesarb 11y ago
      https://lwn.net/op/FAQ.lwn#slinks https://lwn.net/op/FAQ.lwn#slinks > Where is it appropriate to post a subscriber link? > Almost anywhere. Private mail, messages to project mailing lists, and blog entries are all appropriate. As long as people do not use subscriber links as a way to defeat our attempts to gain subscribers, we are happy to see them shared.
    - wtallis 11y ago
      https://news.ycombinator.com/item?id=10236304 https://news.ycombinator.com/item?id=10236304 https://news.ycombinator.com/item?id=9540182 https://news.ycombinator.com/item?id=9540182 https://news.ycombinator.com/item?id=9486938 https://news.ycombinator.com/item?id=9486938 If LWN minded, they'd say so.
deleted 11y ago
[deleted]
tenfingers 11y ago
Approaches similar to unexec() are quite common in lisp-like environments, so emacs doesn't come as a "surprise" here. There's no reason emacs needs to use the glibc's malloc implementation though, especially given emacs has it's own built-in (and probably neglected) malloc for other platforms.
- mcguire 11y ago
  Yep. Emacs builds and runs (and presumably unexecs) on many non-glibc platforms. This is more likely an optimization for it on glibc. "According to Paul Eggert, making unexec more portable has been on the to-do list for a while, "and this will light more of a fire under it". Concerns that Emacs might not build using a new Glibc API (which has not even been written yet) that came up earlier in the thread are not a problem, he said. "Emacs should still build and run even if the glibc API is changed, as Emacs ./configure probes for the glibc malloc-related API and falls back on its own malloc implementation otherwise."" And there's the dreaded autoconf doing it's job.
  - justincormack 11y ago
    Well no it does not unexec on other platforms. And the figures in the article suggest it is annoyingly slow to start on other platforms. This is all about making the Gnu editor work well on the Gnu OS. It builds and runs elsewhere but startup is slow.
mwcampbell 11y ago
I think it's wrong for any application to be this intimate with its host platform. In my opinion, Emacs should treat libc as a black box on GNU/Linux, just as it presumably does on Windows, OS X, and other proprietary platforms. But I admit that my opinion is shaped by being a user and developer of proprietary software, where binary compatibility is important. So I'm not surprised that RMS apparently disagrees.
- zaphar 11y ago
  You aren't wrong given the current development landscape but Emacs is a very old piece of software and this sort of thing wasn't unheard of when it first was given life. (I wasn't really around but anecdotal stories from the time period seem to support this.) This is just an example of technical debt handled well actually. The loan came due and now the emacs devs are doing the right thing and paying it off.
- ma2rten 11y ago
  I think it's common for applications that are developed by same organization as the OS. Windows also has private APIs for some MSFT applications.
  - afarrell 11y ago
    > I first heard about this from one of the developers of the hit game SimCity, who told me that there was a critical bug in his application: it used memory right after freeing it, a major no-no that happened to work OK on DOS but would not work under Windows where memory that is freed is likely to be snatched up by another running application right away. The testers on the Windows team were going through various popular applications, testing them to make sure they worked OK, but SimCity kept crashing. They reported this to the Windows developers, who disassembled SimCity, stepped through it in a debugger, found the bug, and added special code that checked if SimCity was running, and if it did, ran the memory allocator in a special mode in which you could still use memory after freeing it. http://www.joelonsoftware.com/articles/APIWar.html http://www.joelonsoftware.com/articles/APIWar.html
  - asveikau 11y ago
ma2rten 11y ago
I found it interesting that RMS wanted to discuss the issue in private given his position as a almost religious defender of free software. Isn't a part of free software that it is developed in the open?
- mwcampbell 11y ago
  Not necessarily. Read Eric Raymond's description of cathedral-style development in "The Cathedral and the Bazaar" and keep in mind that he had Emacs in mind. It wasn't until Linux that an open, public development process became closely associated with free software.
- deleted 11y ago
  [deleted]
- jordigh 11y ago
  rms doesn't think "free software" has anything to do with a responsibility for collaboration or openness. Those are slogans of the open source movement. He sees software as an ethical exchange between two parties. rms doesn't have a problem with privacy, secrecy, or refusal to collaborate. As long as Bob has the freedom to do whatever he wants with the software the Alice gave him (except, perhaps, restrict the freedom of others), rms doesn't think that Alice has any further moral obligations.
camgunz 11y ago
Summary: emacs devs want glibc to maintain backwards compat in their heap layout so building emacs is 5 seconds faster.
- mwcampbell 11y ago
  IIUC, the 5-second difference is in starting Emacs, not building it.
- deleted 11y ago
  [deleted]
- roel_v 11y ago
  No, starting emacs is 5 seconds faster - at least that's what I got from it.
  - ams6110 11y ago
    Agree -- but doesn't strike me as a big deal. For me, emacs is something I start and will often have running for months. 5 second startup time is irrelevant. That's no slower than starting Word or Excel, either. Even browsers take a couple of seconds to start from cold.
    - lukeschlather 11y ago
      Personally, I use Emacs as a Notepad as well as a coding environment. Also, if I'm screwing with my .emacs it's nice to have instant feedback beyond eval'ing any changes I make as I go.
    - borkabrak 11y ago
      I agree that a five-second startup can hardly be said to break usability, especially for something that many people will leave running for a long time. That said, though, if I was accustomed to one of my primary tools taking a half-second to start, I think I'd miss it pretty bad if it went away. Just the fact that people on both sides (glibc and emacs) are treating this so seriously seems to indicate a rather admirable dedication to keeping the quality of the software as high as possible.
jbssm 11y ago
I wonder if there is any project starting to re-write Emacs according to modern practices like we now have NeoVIM for VIM. I really like editing in VIM but I moved to Emacs using Spacemacs to try and get the best of both worlds (org mode for scientific research seems a really cool approach). Still I see Emacs as being quite more bug prone than VIM and VIM than NeoVIm. Perhaps this is what's needed for someone to start a complete refactor of Emacs and bring it (I mean the code not the editor) to the modern age.
- dmm 11y ago
  I don't really see that as being the case. Emacs development is very active, even at the low levels. The Guile scheme implementation now has an elisp implementation and there's a good chance that emacs will be moved to it eventually. If you encounter bugs you should submit bug reports.
  - wtbob 11y ago
    > The Guile scheme implementation now has an elisp implementation and there's a good chance that emacs will be moved to it eventually. FWIW, I think that's entirely the wrong direction to go. Emacs should be ported to Common Lisp, with an elisp compatibility layer. Scheme's a neat didactic tool, but anyone who wants to produce production-level software in Lisp should write it in Common Lisp. Heck, even Schemers recognise that, which is why the RnRS controversy exists.
    - pmoriarty 11y ago
      My own view is that Scheme is exactly the right way to go and CL would have been the wrong way to go. Modern Schemes are fully as production-ready as CL. The view of Scheme as merely an educational tool is outdated, and usually based on limited experience with ancient, bare-bones Scheme implementations such as MIT Scheme. Modern Schemes like Chicken have fairly extensive collections of practical libraries and features that barebones Scheme implementations lack. To add to that, Scheme is far more elegant than CL, and doesn't contain all of the ancient crud of CL, so is far more pleasant and easy to program in. I'm not thrilled that Guile (rather than Chicken) is the Scheme of choice for Emacs, but it's a far better choice than CL. That said, even CL would have been an enormous improvement over elisp. So the sooner the migration from elisp starts (whether to Scheme or CL), the better.
quotemstr 11y ago
IMHO, the XEmacs approach of using a portable dumper is the better option. I prefer that one not only because it frees us from having to worry about the details of executable formats, but because the same machinery that lets us find and relocate all the intra-lisp-heap pointers would also let us write a compacting GC, which is important for long-running programs like Emacs.
weinzierl 11y ago
> [..] fixing Emacs, which is seen as the only user of the interfaces: If I remember correctly TeX uses the same trick. I don't know if it depends on unexec() though.
- coliveira 11y ago
  From what I remember TeX dumps the processed macros into a binary file. Later, TeX loads those preprocessed formats quickly, without the need to parse again something as big as LaTeX.
nuxi7 11y ago
For the curious, the need to change the malloc behavior is not hypothetical: https://sourceware.org/bugzilla/show_bug.cgi?id=6527 https://sourceware.org/bugzilla/show_bug.cgi?id=6527 Basically for many years now glibc has been knowingly doing the wrong thing just to keep malloc_set_state/malloc_get_state working. Which as far as anyone knows, is used only by emacs. Keeping this interface alive for the sole benefit of an emacs optimization when it is blocking the fixes for fairly serious bugs is a pretty big bar to meet for justification. In the end this is largely a non-issue as an Emacs developer noted that their configure script probes for the special glibc malloc API and if it doesn't exist then emacs will use its own malloc implementation. The conclusion is that once the glibc devs get their changes up, everyone can meet up again to make sure existing emacs binaries still run and that the API detection in the emacs configure script works right. So props to Paul Eggert for being the only sane man in the room and pointing out that the drama was all for nothing.
chris_wot 11y ago
So much for rms standing up to principle. When the chips are down, he resorts to a private mailing list to discuss and influence matters that impact multiple users. I though he was into transparency?

106 points by jordigh 11y ago

Actually, having a generic way of saving the program state into an executable that then runs from that point could be really useful, and some Unix versions had it (and Perl used a Unix utility called undump to implement a "Perl compiler" where it dumped core that could be "undumped" into a running program later; that's the same idea as "unexecing" program state into an executable.)

protopete 11y ago
I logged in to say the same thing. More info here: http://perldoc.perl.org/functions/dump.html http://perldoc.perl.org/functions/dump.html
johntb86 11y ago
http://criu.org/Main_Page http://criu.org/Main_Page
- Freaky 11y ago
  Also: https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpoint&section=2 https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpo...
Freaky 11y ago
It's notable that the Solaris unexec() implementation[1] is basically a thin wrapper around dldump()[2]. 1: http://git.savannah.gnu.org/cgit/emacs.git/tree/src/unexsol.c http://git.savannah.gnu.org/cgit/emacs.git/tree/src/unexsol.... 2: https://docs.oracle.com/cd/E19455-01/806-0627/6j9vhfmop/index.html https://docs.oracle.com/cd/E19455-01/806-0627/6j9vhfmop/inde...
deleted 11y ago
[deleted]

fidget 11y ago

> LWN subscriber-only content kinda shitty

lultimouomo 11y ago
If you try scrolling down, you'll note you can actually read the article. LWN lets subscribers share (temporary) paywalled articles with anyone they like, even on public forums like HN. They are nice like that.
- fidget 11y ago
  Oh, I'm aware I can read it. I'm just not convinced about sharing it on such a broad public forum.
  - lultimouomo 11y ago
    Sorry, I misunderstood what you meant. Links to LWN subscriber content appear on HN from time to time - not too often, and I would guess they do more good than harm to them.
    - markrages 11y ago
      Often such links are posted by LWN staff, so I believe they favor the practice.
  - cesarb 11y ago
    https://lwn.net/op/FAQ.lwn#slinks https://lwn.net/op/FAQ.lwn#slinks > Where is it appropriate to post a subscriber link? > Almost anywhere. Private mail, messages to project mailing lists, and blog entries are all appropriate. As long as people do not use subscriber links as a way to defeat our attempts to gain subscribers, we are happy to see them shared.
  - wtallis 11y ago
    https://news.ycombinator.com/item?id=10236304 https://news.ycombinator.com/item?id=10236304 https://news.ycombinator.com/item?id=9540182 https://news.ycombinator.com/item?id=9540182 https://news.ycombinator.com/item?id=9486938 https://news.ycombinator.com/item?id=9486938 If LWN minded, they'd say so.

deleted 11y ago

[deleted]

tenfingers 11y ago

Approaches similar to unexec() are quite common in lisp-like environments, so emacs doesn't come as a "surprise" here. There's no reason emacs needs to use the glibc's malloc implementation though, especially given emacs has it's own built-in (and probably neglected) malloc for other platforms.

mcguire 11y ago
Yep. Emacs builds and runs (and presumably unexecs) on many non-glibc platforms. This is more likely an optimization for it on glibc. "According to Paul Eggert, making unexec more portable has been on the to-do list for a while, "and this will light more of a fire under it". Concerns that Emacs might not build using a new Glibc API (which has not even been written yet) that came up earlier in the thread are not a problem, he said. "Emacs should still build and run even if the glibc API is changed, as Emacs ./configure probes for the glibc malloc-related API and falls back on its own malloc implementation otherwise."" And there's the dreaded autoconf doing it's job.
- justincormack 11y ago
  Well no it does not unexec on other platforms. And the figures in the article suggest it is annoyingly slow to start on other platforms. This is all about making the Gnu editor work well on the Gnu OS. It builds and runs elsewhere but startup is slow.

mwcampbell 11y ago

I think it's wrong for any application to be this intimate with its host platform. In my opinion, Emacs should treat libc as a black box on GNU/Linux, just as it presumably does on Windows, OS X, and other proprietary platforms. But I admit that my opinion is shaped by being a user and developer of proprietary software, where binary compatibility is important. So I'm not surprised that RMS apparently disagrees.

zaphar 11y ago
You aren't wrong given the current development landscape but Emacs is a very old piece of software and this sort of thing wasn't unheard of when it first was given life. (I wasn't really around but anecdotal stories from the time period seem to support this.) This is just an example of technical debt handled well actually. The loan came due and now the emacs devs are doing the right thing and paying it off.
ma2rten 11y ago
I think it's common for applications that are developed by same organization as the OS. Windows also has private APIs for some MSFT applications.
- afarrell 11y ago
  > I first heard about this from one of the developers of the hit game SimCity, who told me that there was a critical bug in his application: it used memory right after freeing it, a major no-no that happened to work OK on DOS but would not work under Windows where memory that is freed is likely to be snatched up by another running application right away. The testers on the Windows team were going through various popular applications, testing them to make sure they worked OK, but SimCity kept crashing. They reported this to the Windows developers, who disassembled SimCity, stepped through it in a debugger, found the bug, and added special code that checked if SimCity was running, and if it did, ran the memory allocator in a special mode in which you could still use memory after freeing it. http://www.joelonsoftware.com/articles/APIWar.html http://www.joelonsoftware.com/articles/APIWar.html
- asveikau 11y ago

ma2rten 11y ago

I found it interesting that RMS wanted to discuss the issue in private given his position as a almost religious defender of free software. Isn't a part of free software that it is developed in the open?

mwcampbell 11y ago
Not necessarily. Read Eric Raymond's description of cathedral-style development in "The Cathedral and the Bazaar" and keep in mind that he had Emacs in mind. It wasn't until Linux that an open, public development process became closely associated with free software.
deleted 11y ago
[deleted]
jordigh 11y ago
rms doesn't think "free software" has anything to do with a responsibility for collaboration or openness. Those are slogans of the open source movement. He sees software as an ethical exchange between two parties. rms doesn't have a problem with privacy, secrecy, or refusal to collaborate. As long as Bob has the freedom to do whatever he wants with the software the Alice gave him (except, perhaps, restrict the freedom of others), rms doesn't think that Alice has any further moral obligations.

camgunz 11y ago

Summary: emacs devs want glibc to maintain backwards compat in their heap layout so building emacs is 5 seconds faster.

mwcampbell 11y ago
IIUC, the 5-second difference is in starting Emacs, not building it.
deleted 11y ago
[deleted]
roel_v 11y ago
No, starting emacs is 5 seconds faster - at least that's what I got from it.
- ams6110 11y ago
  Agree -- but doesn't strike me as a big deal. For me, emacs is something I start and will often have running for months. 5 second startup time is irrelevant. That's no slower than starting Word or Excel, either. Even browsers take a couple of seconds to start from cold.
  - lukeschlather 11y ago
    Personally, I use Emacs as a Notepad as well as a coding environment. Also, if I'm screwing with my .emacs it's nice to have instant feedback beyond eval'ing any changes I make as I go.
  - borkabrak 11y ago
    I agree that a five-second startup can hardly be said to break usability, especially for something that many people will leave running for a long time. That said, though, if I was accustomed to one of my primary tools taking a half-second to start, I think I'd miss it pretty bad if it went away. Just the fact that people on both sides (glibc and emacs) are treating this so seriously seems to indicate a rather admirable dedication to keeping the quality of the software as high as possible.

jbssm 11y ago

I wonder if there is any project starting to re-write Emacs according to modern practices like we now have NeoVIM for VIM. I really like editing in VIM but I moved to Emacs using Spacemacs to try and get the best of both worlds (org mode for scientific research seems a really cool approach). Still I see Emacs as being quite more bug prone than VIM and VIM than NeoVIm. Perhaps this is what's needed for someone to start a complete refactor of Emacs and bring it (I mean the code not the editor) to the modern age.

dmm 11y ago
I don't really see that as being the case. Emacs development is very active, even at the low levels. The Guile scheme implementation now has an elisp implementation and there's a good chance that emacs will be moved to it eventually. If you encounter bugs you should submit bug reports.
- wtbob 11y ago
  > The Guile scheme implementation now has an elisp implementation and there's a good chance that emacs will be moved to it eventually. FWIW, I think that's entirely the wrong direction to go. Emacs should be ported to Common Lisp, with an elisp compatibility layer. Scheme's a neat didactic tool, but anyone who wants to produce production-level software in Lisp should write it in Common Lisp. Heck, even Schemers recognise that, which is why the RnRS controversy exists.
  - pmoriarty 11y ago
    My own view is that Scheme is exactly the right way to go and CL would have been the wrong way to go. Modern Schemes are fully as production-ready as CL. The view of Scheme as merely an educational tool is outdated, and usually based on limited experience with ancient, bare-bones Scheme implementations such as MIT Scheme. Modern Schemes like Chicken have fairly extensive collections of practical libraries and features that barebones Scheme implementations lack. To add to that, Scheme is far more elegant than CL, and doesn't contain all of the ancient crud of CL, so is far more pleasant and easy to program in. I'm not thrilled that Guile (rather than Chicken) is the Scheme of choice for Emacs, but it's a far better choice than CL. That said, even CL would have been an enormous improvement over elisp. So the sooner the migration from elisp starts (whether to Scheme or CL), the better.

quotemstr 11y ago

IMHO, the XEmacs approach of using a portable dumper is the better option. I prefer that one not only because it frees us from having to worry about the details of executable formats, but because the same machinery that lets us find and relocate all the intra-lisp-heap pointers would also let us write a compacting GC, which is important for long-running programs like Emacs.

weinzierl 11y ago

> [..] fixing Emacs, which is seen as the only user of the interfaces: If I remember correctly TeX uses the same trick. I don't know if it depends on unexec() though.

coliveira 11y ago
From what I remember TeX dumps the processed macros into a binary file. Later, TeX loads those preprocessed formats quickly, without the need to parse again something as big as LaTeX.

nuxi7 11y ago

For the curious, the need to change the malloc behavior is not hypothetical: https://sourceware.org/bugzilla/show_bug.cgi?id=6527 https://sourceware.org/bugzilla/show_bug.cgi?id=6527 Basically for many years now glibc has been knowingly doing the wrong thing just to keep malloc_set_state/malloc_get_state working. Which as far as anyone knows, is used only by emacs. Keeping this interface alive for the sole benefit of an emacs optimization when it is blocking the fixes for fairly serious bugs is a pretty big bar to meet for justification. In the end this is largely a non-issue as an Emacs developer noted that their configure script probes for the special glibc malloc API and if it doesn't exist then emacs will use its own malloc implementation. The conclusion is that once the glibc devs get their changes up, everyone can meet up again to make sure existing emacs binaries still run and that the API detection in the emacs configure script works right. So props to Paul Eggert for being the only sane man in the room and pointing out that the drama was all for nothing.

chris_wot 11y ago

So much for rms standing up to principle. When the chips are down, he resorts to a private mailing list to discuss and influence matters that impact multiple users. I though he was into transparency?