5 ms·

Maybe I'm missing something obvious here, but calling "mu index" from the offlineimap postsynchook is going to suck, because mu4e invokes "mu server"[1], that a

by avar 8y ago

Maybe I'm missing something obvious here, but calling "mu index" from the offlineimap postsynchook is going to suck, because mu4e invokes "mu server"[1], that aquires a Xapian lock on the index, which means you can't run the "mu index" while reading your E-Mail. This is the worst thing about mu/mu4e. You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI. After trying to get it to work in some automated way I gave up. I manually issue a key-combo to re-index and then read my E-Mail. Another comment: I recommend not using Emacs to send E-Mail, but to outsource this to a local smtpd such as Exim (which I use with mu4e on Debian). Then if you don't have WiFi or whatever the smtpd just queues those up for later sending, and has queue inspection tools etc. All of the solutions for doing this in Emacs itself are dirty hacks full of edge cases and bugs smtpds have solved decades ago. Despite some rough edges, mu+mu4 (plus Exim, or similar). is an amazing tool, and the most useful E-Mail client I've ever used. 1. https://www.djcbsoftware.nl/code/mu/mu4e/mu-server.html https://www.djcbsoftware.nl/code/mu/mu4e/mu-server.html

fusiongyro 8y ago
The locking thing has not turned out to be a really serious problem for me, and I have several gigabytes of email. mu index is really fast. Also, using the Emacs sendmail hasn't been a real problem. It turns out I don't need to be productive during the ~1s it takes to send the email and I have not run into any edge cases or bugs in the ~5 years I've been rocking this setup.
- avar 8y ago
  It doesn't matter how fast the index is, only one process can hold the Xapian lock, if I'm reading my E-Mail in Emacs with mu4e (which invokes "mu server") and try to index on the command-line I get: $ mu index [...] mu: mu_store_new_writable: xapian error 'Unable to get write lock on <Maildir>/xapian: already locked' (11) How do you send E-Mail with no network connection? Last I checked this was some ad-hoc thing where you neede dto queue them up in some queue managed by Emacs, and remember to turn that mode off when you were back on WiFi (or hack up your own hooks to automate this), v.s. sending it to a local smtpd which already has retry logic built-in.
  - fusiongyro 8y ago
    Why on earth would I try and index it twice concurrently? I never have a scenario where I need to send email without network.
    - avar 8y ago
      No, you wouldn't index from two processes concurrently, but for Emacs to read your mail in mu4e and for you to concurrently issue "mu index" on the command-line two processes ("mu server" and "mu index") would need to poen the same Xapian database read-write at the same time, and just like SQLite Xapian doesn't support that.
      - fusiongyro 8y ago
        ... I still don't see why I would do that. Emacs runs offlineimap and then Emacs runs mu index. Why would I do it again myself from the command line?
      - ams6110 8y ago
        Yeah I use notmuch with Emacs and it's the same. If you try to search, refresh search results, or read email while the index process is running you get an error. It's not a big deal for me, because I download and index my mail from the command line, so I'll never be reading or sending email at the same time.
        donio 8y ago
        This used to be true but notmuch has added support for Xapian's DB_RETRY_LOCK feature a couple of years ago so if you have non-ancient notmuch (0.23+) and Xapian (1.3.2+) then the second process will wait for the lock instead of erroring out. The feature can be disabled at compile time if you don't want it for some reason.
pulisse 8y ago
> You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI. It's worth noting that multithreading recently landed in Emacs master, so it's now possible to avoid this sort of problem.
- avar 8y ago
  Now possible as in mu4e would need to use some API that's not in a release version of Emacs, or is it more transparent than that?
  - pulisse 8y ago
    The former. I wasn't suggesting that mu4e should be converted right away to use this feature, but more concerned that an unqualified statement that long-running IO must block in Emacs would create the wrong impression.
kkylin 8y ago
You can configure mu4e to queue mail to send later: https://www.djcbsoftware.nl/code/mu/mu4e/Queuing-mail.html https://www.djcbsoftware.nl/code/mu/mu4e/Queuing-mail.html
- avar 8y ago
  Yeah, that's the dirty hack I was referring to. I don't need to tell an smtpd "now's when I expect not to have an Internet connection", which this method relies on, and could happen as I walk into another room and lose WiFi. I just give it mail to deliver, and if it doesn't work it keeps retrying until it works (with configurable backoff and a setting saying when to give up for good).
  - kkylin 8y ago
    Yup, I see where you're coming from. I've actually gotten used to mu4e's queuing behavior, and just leave it on all the time. For one thing, I like having a chance to edit/delete mail after hitting send (sort of like gmail's "undo"), and for another, I like the satisfaction of having a dozen or so messages sent all at once at the end of an email session.
dima55 8y ago
The locking is only a problem because he's doing it wrong, I think. I'm doing this: http://notes.secretsauce.net/notes/2014/12/12_mu4e-and-desktop-notifications.html I.e. there's an external script that invokes IMAP IDLE. And when IMAP tells it there's new mail, the script tells EMACS there's new mail. Then the reindexing happens within the emacs, and it uses the lock it already has. Makes this particular issue go away completely.
- f6e9ed793b 8y ago
  That must be it. I also trigger mu4e indexing using emacsclient, after IMAP sync. BTW you do not need to have mu4e open to index mail (and single-window mu4e mode will not have a * mu4e-main * buffer), it is enough to (progn (require 'mu4e) (mu4e-update-mail-and-index)).
jfb 8y ago
I must also be missing something because I never explicitly index my mail. I fetch it with mbsync every two minutes and then use mu4e to read it and new mail shows up as one would expect. ¯\_(ツ)_/¯
- avar 8y ago
  How many tens of GB of mail do you have? I think this problem only tends to show up on big mailboxes.
  - jfb 8y ago
    10-25GB. But a fast computer, so?
    - tom4000 8y ago
      Good for you. To save power and use smaller machines should always be an option. Indexing saves much resources here.
      - jfb 8y ago
        Agreed 100%. I'm just stating my case, not denying others'.
f6e9ed793b 8y ago
> Maybe I'm missing something obvious here, but calling "mu index" from the offlineimap postsynchook is going to suck, because mu4e invokes "mu server"[1], that aquires a Xapian lock on the index, which means you can't run the "mu index" while reading your E-Mail. > This is the worst thing about mu/mu4e. You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI. After trying to get it to work in some automated way I gave up. I manually issue a key-combo to re-index and then read my E-Mail. I tried both approaches (actually, I had mu4e reindex on `g` query refresh, no need for separate keybindings), and prefer having indexing run after IMAP sync. I don't notice any problems (and I mostly use very slow Atom hardware). > Another comment: I recommend not using Emacs to send E-Mail, but to outsource this to a local smtpd such as Exim (which I use with mu4e on Debian). Then if you don't have WiFi or whatever the smtpd just queues those up for later sending, and has queue inspection tools etc. All of the solutions for doing this in Emacs itself are dirty hacks full of edge cases and bugs smtpds have solved decades ago. FUD. smtpmail.el can do this. smtpmail.el can queue mail, and you can run it in a separate Emacs process periodically or on demand, to attempt to deliver those emails. The emails are queued as files in a directory, so you don't need separate queue inspection tools. I run smtpmail.el this way with mu4e, and think it is a far superior solution to having to install and configure yet another external program.
michaelmrose 8y ago
I use imapnotify to connect to my imap server and respond only when a mail is received by running offlineimap to sync the mail and telling mu4e via emacsclient to sync. Since mu is never ever called except by emacs and only exactly when an email has come in there is never contention for the lock.
nickysielicki 8y ago
I recommend notmuch in place of mu/mu4e and msmtp in place of a full-blown smtpd implementation. I also recommend isync's mbsync over offlineimap
- tom4000 8y ago
  Agree with using Isync. Notmuch lacks the backward synchronization to the IMAP server and to another mail client then notmuch. I primarily use mu4e but want to see changes on my mobile as well as on any remote computer using a web client in a browser. As far as I know notmuch can just be synchronized to another notmuch instance.
  - nickysielicki 8y ago
    If you use https://github.com/gauteh/gmailieer https://github.com/gauteh/gmailieer you can have bidirectional sync between gmail tags and notmuch tags. Personally, I don't bother with it. When I have to do mail on my phone, it's usually so limited in scope that I don't need to retain state when I sit down at my main computer. I can imagine that's not for everyone, though.
arnoooooo 8y ago
My mail fetching script kills mu before calling "mu index", like so : $ pgrep mu && kill `pgrep mu` $ mu index It works perfectly. Mu4e then restarts mu when it needs it. I think I saw that as the recommended setup somewhere in the mu docs.
- u801e 8y ago
  > $ pgrep mu && kill `pgrep mu` This could be rewritten as: $ pkill mu
- tom4000 8y ago
  With `pkill $(pgrep mu)` or even with `pkill mu` you can kill mutt and any mu$ process as well. `pkill -x mu` matches the whole name without substrings.

by avar 8y ago

fusiongyro 8y ago

The locking thing has not turned out to be a really serious problem for me, and I have several gigabytes of email. mu index is really fast. Also, using the Emacs sendmail hasn't been a real problem. It turns out I don't need to be productive during the ~1s it takes to send the email and I have not run into any edge cases or bugs in the ~5 years I've been rocking this setup.

avar 8y ago
It doesn't matter how fast the index is, only one process can hold the Xapian lock, if I'm reading my E-Mail in Emacs with mu4e (which invokes "mu server") and try to index on the command-line I get: $ mu index [...] mu: mu_store_new_writable: xapian error 'Unable to get write lock on <Maildir>/xapian: already locked' (11) How do you send E-Mail with no network connection? Last I checked this was some ad-hoc thing where you neede dto queue them up in some queue managed by Emacs, and remember to turn that mode off when you were back on WiFi (or hack up your own hooks to automate this), v.s. sending it to a local smtpd which already has retry logic built-in.
- fusiongyro 8y ago
  Why on earth would I try and index it twice concurrently? I never have a scenario where I need to send email without network.
  - avar 8y ago
    No, you wouldn't index from two processes concurrently, but for Emacs to read your mail in mu4e and for you to concurrently issue "mu index" on the command-line two processes ("mu server" and "mu index") would need to poen the same Xapian database read-write at the same time, and just like SQLite Xapian doesn't support that.
    - fusiongyro 8y ago
      ... I still don't see why I would do that. Emacs runs offlineimap and then Emacs runs mu index. Why would I do it again myself from the command line?
    - ams6110 8y ago
      Yeah I use notmuch with Emacs and it's the same. If you try to search, refresh search results, or read email while the index process is running you get an error. It's not a big deal for me, because I download and index my mail from the command line, so I'll never be reading or sending email at the same time.
      - donio 8y ago
        This used to be true but notmuch has added support for Xapian's DB_RETRY_LOCK feature a couple of years ago so if you have non-ancient notmuch (0.23+) and Xapian (1.3.2+) then the second process will wait for the lock instead of erroring out. The feature can be disabled at compile time if you don't want it for some reason.

pulisse 8y ago

> You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI. It's worth noting that multithreading recently landed in Emacs master, so it's now possible to avoid this sort of problem.

avar 8y ago
Now possible as in mu4e would need to use some API that's not in a release version of Emacs, or is it more transparent than that?
- pulisse 8y ago
  The former. I wasn't suggesting that mu4e should be converted right away to use this feature, but more concerned that an unqualified statement that long-running IO must block in Emacs would create the wrong impression.

kkylin 8y ago

You can configure mu4e to queue mail to send later: https://www.djcbsoftware.nl/code/mu/mu4e/Queuing-mail.html https://www.djcbsoftware.nl/code/mu/mu4e/Queuing-mail.html

avar 8y ago
Yeah, that's the dirty hack I was referring to. I don't need to tell an smtpd "now's when I expect not to have an Internet connection", which this method relies on, and could happen as I walk into another room and lose WiFi. I just give it mail to deliver, and if it doesn't work it keeps retrying until it works (with configurable backoff and a setting saying when to give up for good).
- kkylin 8y ago
  Yup, I see where you're coming from. I've actually gotten used to mu4e's queuing behavior, and just leave it on all the time. For one thing, I like having a chance to edit/delete mail after hitting send (sort of like gmail's "undo"), and for another, I like the satisfaction of having a dozen or so messages sent all at once at the end of an email session.

dima55 8y ago

The locking is only a problem because he's doing it wrong, I think. I'm doing this: http://notes.secretsauce.net/notes/2014/12/12_mu4e-and-desktop-notifications.html I.e. there's an external script that invokes IMAP IDLE. And when IMAP tells it there's new mail, the script tells EMACS there's new mail. Then the reindexing happens within the emacs, and it uses the lock it already has. Makes this particular issue go away completely.

f6e9ed793b 8y ago
That must be it. I also trigger mu4e indexing using emacsclient, after IMAP sync. BTW you do not need to have mu4e open to index mail (and single-window mu4e mode will not have a * mu4e-main * buffer), it is enough to (progn (require 'mu4e) (mu4e-update-mail-and-index)).

jfb 8y ago

I must also be missing something because I never explicitly index my mail. I fetch it with mbsync every two minutes and then use mu4e to read it and new mail shows up as one would expect. ¯\_(ツ)_/¯

avar 8y ago
How many tens of GB of mail do you have? I think this problem only tends to show up on big mailboxes.
- jfb 8y ago
  10-25GB. But a fast computer, so?
  - tom4000 8y ago
    Good for you. To save power and use smaller machines should always be an option. Indexing saves much resources here.
    - jfb 8y ago
      Agreed 100%. I'm just stating my case, not denying others'.

f6e9ed793b 8y ago

> Maybe I'm missing something obvious here, but calling "mu index" from the offlineimap postsynchook is going to suck, because mu4e invokes "mu server"[1], that aquires a Xapian lock on the index, which means you can't run the "mu index" while reading your E-Mail. > This is the worst thing about mu/mu4e. You need to do the indexing with a timer in Emacs, and because Emacs is single-threaded this blocks things / slows down the UI. After trying to get it to work in some automated way I gave up. I manually issue a key-combo to re-index and then read my E-Mail. I tried both approaches (actually, I had mu4e reindex on `g` query refresh, no need for separate keybindings), and prefer having indexing run after IMAP sync. I don't notice any problems (and I mostly use very slow Atom hardware). > Another comment: I recommend not using Emacs to send E-Mail, but to outsource this to a local smtpd such as Exim (which I use with mu4e on Debian). Then if you don't have WiFi or whatever the smtpd just queues those up for later sending, and has queue inspection tools etc. All of the solutions for doing this in Emacs itself are dirty hacks full of edge cases and bugs smtpds have solved decades ago. FUD. smtpmail.el can do this. smtpmail.el can queue mail, and you can run it in a separate Emacs process periodically or on demand, to attempt to deliver those emails. The emails are queued as files in a directory, so you don't need separate queue inspection tools. I run smtpmail.el this way with mu4e, and think it is a far superior solution to having to install and configure yet another external program.

michaelmrose 8y ago

I use imapnotify to connect to my imap server and respond only when a mail is received by running offlineimap to sync the mail and telling mu4e via emacsclient to sync. Since mu is never ever called except by emacs and only exactly when an email has come in there is never contention for the lock.

nickysielicki 8y ago

I recommend notmuch in place of mu/mu4e and msmtp in place of a full-blown smtpd implementation. I also recommend isync's mbsync over offlineimap

tom4000 8y ago
Agree with using Isync. Notmuch lacks the backward synchronization to the IMAP server and to another mail client then notmuch. I primarily use mu4e but want to see changes on my mobile as well as on any remote computer using a web client in a browser. As far as I know notmuch can just be synchronized to another notmuch instance.
- nickysielicki 8y ago
  If you use https://github.com/gauteh/gmailieer https://github.com/gauteh/gmailieer you can have bidirectional sync between gmail tags and notmuch tags. Personally, I don't bother with it. When I have to do mail on my phone, it's usually so limited in scope that I don't need to retain state when I sit down at my main computer. I can imagine that's not for everyone, though.

arnoooooo 8y ago

My mail fetching script kills mu before calling "mu index", like so : $ pgrep mu && kill `pgrep mu` $ mu index It works perfectly. Mu4e then restarts mu when it needs it. I think I saw that as the recommended setup somewhere in the mu docs.

u801e 8y ago
> $ pgrep mu && kill `pgrep mu` This could be rewritten as: $ pkill mu
tom4000 8y ago
With `pkill $(pgrep mu)` or even with `pkill mu` you can kill mutt and any mu$ process as well. `pkill -x mu` matches the whole name without substrings.