4 ms·

Since you mentioned the single-threadedness (and it’s relevant to the discussion of IDE/Emacs): what’s the status of any initiative to move Emacs to multiple co

Since you mentioned the single-threadedness (and it’s relevant to the discussion of IDE/Emacs): what’s the status of any initiative to move Emacs to multiple cores? I’m single, and I hope to one day have a family etc.. If I end up staying single for long enough, helping move Emacs onto multiple cores is something I fantasize about contributing back to the community that has helped my life so much. Does anyone have any input?

db48x 4y ago
Most of what Emacs does just won’t benefit from massive parallelism. CPUs are ridiculously fast at processing text (AVX2 instructions can process vectors of 32 bytes at a time), and most of the stuff that _can_ benefit from parallelism should just be moved to an external process. For example, querying your compiler for a list of methods that apply to the current object, or a list of functions that start with “Foo” are mostly moving to external processes using LSP as the communication protocol. It would be nice for some things to use background threads, like Gnus polling your IMAP server for email, and Emacs does have the threading mechanisms to handle that now. It’s just a matter of time before someone does the work :)
- comfypotato 4y ago
  Hey! Exactly the response I was looking for. Thanks for this. Your response reminded me of an Intel project about regular expressions. I think it’s called hyperscan, but I’m not sure. It’s a regex engine (I think that’s the correct terminology) that makes dedicated use of very specific Intel instructions to reach incredible regex speeds on Intel processors. Would be cool if Emacs could make use of this tech when configured properly. I bet there are some internals that rely on regex.
- ykonstant 4y ago
  Does emacs use specialized instruction sets like AVX2?
  - lallysingh 4y ago
    Ha! No. Native compilation just snuck in a little while ago. SIMD will take... longer.
  - db48x 4y ago
    Alas, no. But the power is right there, all we have to is stretch forth our hands and use it.
- the-lazy-guy 4y ago
  I disagree. When I am running a compilation (with output being dumped into a visible buffer) + query magit for large commit. Over tramp. Things noticeably freeze. Technically it all is async. Practically, it is implemented as polling things on main thread with some witing happening in non-async fashion. > For example, querying your compiler for a list of methods that apply to the current object, or a list of functions that start with “Foo” are mostly moving to external processes using LSP as the communication protocol. That's why we have lsp-bridge and lsp-mode emacs fork :) Both of which build some infrastructure to avoid doing communication work with lsp-mode work in main emacs thread. So, heavy emacs users are building some async machinery which wraps another already async and relatively lightweight protocol, because core emacs facilities can't keep up with it. Architecturally it is kind of insane. I think, lsp-mode fork is doing the right thing (from practical POV; it goes against "emacs is just an elisp interpreter" ideology though) and hope it gets into core at some point. A better solution would have being having first class async and background threads support at the elisp level. Which would never happen due to elisp messiness. https://github.com/emacs-lsp/emacs https://github.com/emacs-lsp/emacs https://github.com/manateelazycat/lsp-bridge https://github.com/manateelazycat/lsp-bridge
tmtvl 4y ago
Emacs has support for multithreading (threads landed in 26), there was an article about the multithreaded model earlier this year (https://news.ycombinator.com/item?id=31559818 https://news.ycombinator.com/item?id=31559818), and the Elisp manual has a section on concurrency (section "Threads"). If you want to contribute to the future (pun intended) of multithreaded Emacs, your best bet may be to make a library that adds the nice concurrency facilities that will help library authors move to a multithreaded model. An actor model, green threads, or supervision trees may be avenues worth pursuing. A big blocker is probably the dynamic binding, but trying to "fix" that would break a lot, if not all, of the existing libraries.
- comfypotato 4y ago
  Thanks for the pleasant surprise of encouraging feedback. I’m going to send this comment chain to my email inbox so that I can find it later. This is great. It’s such a monumental task to hack on the basics of the text editing. I listened to a tts book; I think it was titled “The Art of the Text Editor” that delved a bit into some of the original philosophy of the text rendering etc. in Emacs. You might enjoy it If you ever need a boring book to read.
  - tmtvl 4y ago
    You probably mean The Craft of Text Editing -or- A Cookbook for an Emacs. I've skimmed it a bit before and it's on my reading list. First I'm still working my way through CRLS's Introduction to Algorithms and Aspnes' Notes on Data Structures and Programming Techniques.
    - bloopernova 4y ago
      Links to the electronic versions of this out of print book: https://www.finseth.com/craft/ https://www.finseth.com/craft/
- bloopernova 4y ago
  Warning, low self-esteem and anxiety ahead: I really wish I had the capacity and capability to pursue this effort. I barely have the energy to be good at shitty DevOps and Terraform, writing basic Python and Node is so time consuming. I can do some Lisp, but only at the basic level of init.el and adapting a few configuration examples. I do the good thing and donate to the FSF monthly but I wish I could donate directly to Emacs because I use it every day. Even better would be to sponsor the development of a concurrency library and port of major things like magit or org.

db48x 4y ago

Most of what Emacs does just won’t benefit from massive parallelism. CPUs are ridiculously fast at processing text (AVX2 instructions can process vectors of 32 bytes at a time), and most of the stuff that _can_ benefit from parallelism should just be moved to an external process. For example, querying your compiler for a list of methods that apply to the current object, or a list of functions that start with “Foo” are mostly moving to external processes using LSP as the communication protocol. It would be nice for some things to use background threads, like Gnus polling your IMAP server for email, and Emacs does have the threading mechanisms to handle that now. It’s just a matter of time before someone does the work :)

comfypotato 4y ago
Hey! Exactly the response I was looking for. Thanks for this. Your response reminded me of an Intel project about regular expressions. I think it’s called hyperscan, but I’m not sure. It’s a regex engine (I think that’s the correct terminology) that makes dedicated use of very specific Intel instructions to reach incredible regex speeds on Intel processors. Would be cool if Emacs could make use of this tech when configured properly. I bet there are some internals that rely on regex.
ykonstant 4y ago
Does emacs use specialized instruction sets like AVX2?
- lallysingh 4y ago
  Ha! No. Native compilation just snuck in a little while ago. SIMD will take... longer.
- db48x 4y ago
  Alas, no. But the power is right there, all we have to is stretch forth our hands and use it.
the-lazy-guy 4y ago
I disagree. When I am running a compilation (with output being dumped into a visible buffer) + query magit for large commit. Over tramp. Things noticeably freeze. Technically it all is async. Practically, it is implemented as polling things on main thread with some witing happening in non-async fashion. > For example, querying your compiler for a list of methods that apply to the current object, or a list of functions that start with “Foo” are mostly moving to external processes using LSP as the communication protocol. That's why we have lsp-bridge and lsp-mode emacs fork :) Both of which build some infrastructure to avoid doing communication work with lsp-mode work in main emacs thread. So, heavy emacs users are building some async machinery which wraps another already async and relatively lightweight protocol, because core emacs facilities can't keep up with it. Architecturally it is kind of insane. I think, lsp-mode fork is doing the right thing (from practical POV; it goes against "emacs is just an elisp interpreter" ideology though) and hope it gets into core at some point. A better solution would have being having first class async and background threads support at the elisp level. Which would never happen due to elisp messiness. https://github.com/emacs-lsp/emacs https://github.com/emacs-lsp/emacs https://github.com/manateelazycat/lsp-bridge https://github.com/manateelazycat/lsp-bridge

tmtvl 4y ago

Emacs has support for multithreading (threads landed in 26), there was an article about the multithreaded model earlier this year (https://news.ycombinator.com/item?id=31559818 https://news.ycombinator.com/item?id=31559818), and the Elisp manual has a section on concurrency (section "Threads"). If you want to contribute to the future (pun intended) of multithreaded Emacs, your best bet may be to make a library that adds the nice concurrency facilities that will help library authors move to a multithreaded model. An actor model, green threads, or supervision trees may be avenues worth pursuing. A big blocker is probably the dynamic binding, but trying to "fix" that would break a lot, if not all, of the existing libraries.

comfypotato 4y ago
Thanks for the pleasant surprise of encouraging feedback. I’m going to send this comment chain to my email inbox so that I can find it later. This is great. It’s such a monumental task to hack on the basics of the text editing. I listened to a tts book; I think it was titled “The Art of the Text Editor” that delved a bit into some of the original philosophy of the text rendering etc. in Emacs. You might enjoy it If you ever need a boring book to read.
- tmtvl 4y ago
  You probably mean The Craft of Text Editing -or- A Cookbook for an Emacs. I've skimmed it a bit before and it's on my reading list. First I'm still working my way through CRLS's Introduction to Algorithms and Aspnes' Notes on Data Structures and Programming Techniques.
  - bloopernova 4y ago
    Links to the electronic versions of this out of print book: https://www.finseth.com/craft/ https://www.finseth.com/craft/
bloopernova 4y ago
Warning, low self-esteem and anxiety ahead: I really wish I had the capacity and capability to pursue this effort. I barely have the energy to be good at shitty DevOps and Terraform, writing basic Python and Node is so time consuming. I can do some Lisp, but only at the basic level of init.el and adapting a few configuration examples. I do the good thing and donate to the FSF monthly but I wish I could donate directly to Emacs because I use it every day. Even better would be to sponsor the development of a concurrency library and port of major things like magit or org.