7 ms·

this is interesting, but in a way it's also really scary from my quick look at it it seems that is sql to store the emails. any reason to use that over mbox, o

this is interesting, but in a way it's also really scary from my quick look at it it seems that is sql to store the emails. any reason to use that over mbox, or milder, which you can then easily index? it also implements a small imap client(which may work fine for just reading gmail, but imap is a scary thing, everyone uses it a little differently) https://github.com/naspeh/mailr/blob/master/mailr/syncer.py https://github.com/naspeh/mailr/blob/master/mailr/syncer.py that said i guess it's a nice example for writing a mail client in python. you might want to use mailpile which had it's first release a while back: http://mailpile.is http://mailpile.is but even for that, if you don't care about fancy guis, you're probably still better off running mutt, imap, mu on a shell you have somewhere edit: i didn't pay enough attention, as was mentioned it's using imaplib, which is the same library offlineimap uses. i do believe though that a lot the work that happens is how people deal with the rfcs. i remember reading about frustrations of the offlineimap author with different servers, and clients. but basically imaplib itself is just a basic protocol implementation, everything else happens on top of it. and while you can indeed easily store your terabytes of data in postgresql and you can easily setup full text search, it's still much more portable and easily to handle backup and syncing purposes in different formats. Archiveopteryx looks interesting though, so thanks for the hint. if i never write a service to handle mails for other people i'll revisit it.

naspeh 13y ago
I'm author of Mailr. PostgreSQL has full text search, has transactions and I like working with them :). I had tried mailpile of course, but I don't like their way. I want conversations which really like gmail :). And I want to write less code for that. The first priority to work with gmail, so syncer just work with gmail :). there is a lot of work, as I said in my post
skorgu 13y ago
If you choose your layout carefully SQL should be fine. Certainly you can index on whatever you like that way. You also get the potential to do clever things like storing message bodies or mime parts once and foreign-keying them into individual mailboxes (if I email 30 people a single 1M file I should only really need to store it once). This actually looks almost like what I want out of a webmail client except it's lacking hotkeys. I don't know how people manage email without keyboard shortcuts.
andor 13y ago
it also implements a small imap client(which may work fine for just reading gmail, but imap is a scary thing, everyone uses it a little differently) It uses imaplib from the Python standard library. Doesn't look scary to me.
- tonyarkles 13y ago
  How deep have you gone with the library and with an imap client implementation? There is definitely a lot of variation from imap server to imap server. Things come back with weird and broken encodings sometimes (the server will tell you that a message is unicode and then return invalid unicode), commands that seem to be read-only will have persistent effects, etc. IMAP is nowhere near as nice to work with as its standard and interface imply.
  - naspeh 13y ago
    I know IMAP is hard way, but it most used nowadays. Now I try to work with gmail imap, so it is easier for support just one server for first. BTW. I had seen unicode issues... And yes, there is a lot of work...
    - XorNot 13y ago
      Gmail IMAP's IDLE support is broken (probably intentionally). It will report new messages, but it won't report changes to the number of unread messages (or which messages). This means your clients don't automatically update read message stats, which is a pain (most surprising thing I noticed moving to Gmail from hosting IMAP on my home server).
  - vidarh 13y ago
    > IMAP is nowhere near as nice to work with as its standard and interface imply. I prefer to say that it is just as horribly awful as it's abomination of a ill conceived standard implies. Pretty much "everything" is wrong about IMAP, from allowing the use of message sequence numbers that can change under you, to the horribly overcomplicated support for addressing mime parts, to the overcomplicated grammar...
feld 13y ago
All of my email is in Postgres. Ever heard of the IMAP server called Archiveopteryx? http://aox.org http://aox.org Arnt and Ahbijit know more about the email RFCs than probably anyone else on the planet. Arnt has been involved in writing several RFCs himself. Aox is blazing fast and your mailboxes scale as large as you want. Millions of emails in a single mail folder is not a problem. It's also really nice that it handles deduplication of your mail at several levels.
- darklajid 13y ago
  Thanks, I've never heard about that project, imap was cyrus, courier or dovecot for me (I run the latter). The dovecot developers claimed in various mailing list threads, that a SQL based backend would not make sense, not even performance wise. Interesting to see a different approach. What irks me is the FAQ entry claiming "If you have trouble with user authentication, edit the pg_hba.conf to allow trust or md5"... I - don't think that this is a good idea?
  - feld 13y ago
    I've never had to do that before. That would require a very poorly configured Postgres server.
    - Arnt 13y ago
      Yes, it would require that. Such servers do exist, I'm sorry to say.
  - Arnt 13y ago
    I wrote that sentence. Trust is not a good idea, in my opinion. But I know that some people disagree with me, and I have better things to do than discuss the issue with them. MD5 is fine, I think. If you allow randoms from the entire internet to connect, allowing MD5 won't hurt you much more. If you restrict connections to e.g. your own IP addresses using a packet filter or firewall, MD5 definitely will not hurt you.
- 616c 13y ago
otterley 13y ago
> it seems that is sql to store the emails. any reason to use that over mbox, or milder, which you can then easily index? If you do that, synchronizing the index with the data becomes your responsibility. And you're most likely indexing with a database anyway. So the next logical step is to put the message itself in the database, and then you get synchronization for free. It might actually be better to use ElasticSearch as the datastore for this particular application instead of MySQL or the like.
616c 13y ago
You seem to have reservations about email in SQL, but the MailDB has been building a usable solution around this for years. http://maildb.sourceforge.net/ http://maildb.sourceforge.net/
- cpach 13y ago
  That project doesn’t seem to have been updated since 2002.
  - cpach 13y ago
    Perhaps it was http://www.dbmail.org http://www.dbmail.org that '616c meant to refer to.
    - 616c 13y ago
      Woosh. Yes, that is what I had meant. Thanks for correcting that.
jfb 13y ago
Storing mail in mbox is such a terrible, terrible, terrible idea that anything would be better.
- mkr-hn 13y ago
  Can you elaborate for people like me who have no clue about mbox and why it might not be a good format?
  - ams6110 13y ago
    mbox is just all your email concatenated into one big text file. You can I'm sure imagine that gets unwieldy at some point.
    - gkya 13y ago
      Well, there are MTAs, which one may use to distribute mail to different mailboxes. Also, no one is obliged to not truncate their mbox files after a while. I just encrypt my set of files and tuck them in a tarball, and store that in a specific folder with older tarballs.
    - e12e 13y ago
      It's not so much that it's one file (although that makes for fun times if the underlying filesystem doesn't support proper write locks), after all most databases also use just one file -- it's more the fact that individual messages are only recognized by lines starting with "From: " -- which presents some obvious challenges with escaping (that some programs manage to get wrong). I also imagine it'd be challenging to maintain an effective index as mails are deleted...
ams6110 13y ago
If it were me I might look at building somthing on top of notmuch. I use the emacs package for notmuch and it's awesome, it has search and tagging capabilities very similar to gmail and handles conversations nicely. http://notmuchmail.org/ http://notmuchmail.org/
- naspeh 13y ago
  I have tried it, while was researching. It was exciting, but I chose PostgreSQL as storage :).
rodgerd 13y ago
> but even for that, if you don't care about fancy guis, you're probably still better off running mutt, imap, mu on a shell you have somewhere I do like fancy GUIs and use roundcube as one option. For me, > it's still much more portable and easily to handle backup and syncing purposes in different formats. ...the portability is very important. I don't want a new MUA to force me into changing my whole mailstore/MTA chain.

naspeh 13y ago

I'm author of Mailr. PostgreSQL has full text search, has transactions and I like working with them :). I had tried mailpile of course, but I don't like their way. I want conversations which really like gmail :). And I want to write less code for that. The first priority to work with gmail, so syncer just work with gmail :). there is a lot of work, as I said in my post

skorgu 13y ago

If you choose your layout carefully SQL should be fine. Certainly you can index on whatever you like that way. You also get the potential to do clever things like storing message bodies or mime parts once and foreign-keying them into individual mailboxes (if I email 30 people a single 1M file I should only really need to store it once). This actually looks almost like what I want out of a webmail client except it's lacking hotkeys. I don't know how people manage email without keyboard shortcuts.

andor 13y ago

it also implements a small imap client(which may work fine for just reading gmail, but imap is a scary thing, everyone uses it a little differently) It uses imaplib from the Python standard library. Doesn't look scary to me.

tonyarkles 13y ago
How deep have you gone with the library and with an imap client implementation? There is definitely a lot of variation from imap server to imap server. Things come back with weird and broken encodings sometimes (the server will tell you that a message is unicode and then return invalid unicode), commands that seem to be read-only will have persistent effects, etc. IMAP is nowhere near as nice to work with as its standard and interface imply.
- naspeh 13y ago
  I know IMAP is hard way, but it most used nowadays. Now I try to work with gmail imap, so it is easier for support just one server for first. BTW. I had seen unicode issues... And yes, there is a lot of work...
  - XorNot 13y ago
    Gmail IMAP's IDLE support is broken (probably intentionally). It will report new messages, but it won't report changes to the number of unread messages (or which messages). This means your clients don't automatically update read message stats, which is a pain (most surprising thing I noticed moving to Gmail from hosting IMAP on my home server).
- vidarh 13y ago
  > IMAP is nowhere near as nice to work with as its standard and interface imply. I prefer to say that it is just as horribly awful as it's abomination of a ill conceived standard implies. Pretty much "everything" is wrong about IMAP, from allowing the use of message sequence numbers that can change under you, to the horribly overcomplicated support for addressing mime parts, to the overcomplicated grammar...

feld 13y ago

All of my email is in Postgres. Ever heard of the IMAP server called Archiveopteryx? http://aox.org http://aox.org Arnt and Ahbijit know more about the email RFCs than probably anyone else on the planet. Arnt has been involved in writing several RFCs himself. Aox is blazing fast and your mailboxes scale as large as you want. Millions of emails in a single mail folder is not a problem. It's also really nice that it handles deduplication of your mail at several levels.

darklajid 13y ago
Thanks, I've never heard about that project, imap was cyrus, courier or dovecot for me (I run the latter). The dovecot developers claimed in various mailing list threads, that a SQL based backend would not make sense, not even performance wise. Interesting to see a different approach. What irks me is the FAQ entry claiming "If you have trouble with user authentication, edit the pg_hba.conf to allow trust or md5"... I - don't think that this is a good idea?
- feld 13y ago
  I've never had to do that before. That would require a very poorly configured Postgres server.
  - Arnt 13y ago
    Yes, it would require that. Such servers do exist, I'm sorry to say.
- Arnt 13y ago
  I wrote that sentence. Trust is not a good idea, in my opinion. But I know that some people disagree with me, and I have better things to do than discuss the issue with them. MD5 is fine, I think. If you allow randoms from the entire internet to connect, allowing MD5 won't hurt you much more. If you restrict connections to e.g. your own IP addresses using a packet filter or firewall, MD5 definitely will not hurt you.
616c 13y ago

otterley 13y ago

> it seems that is sql to store the emails. any reason to use that over mbox, or milder, which you can then easily index? If you do that, synchronizing the index with the data becomes your responsibility. And you're most likely indexing with a database anyway. So the next logical step is to put the message itself in the database, and then you get synchronization for free. It might actually be better to use ElasticSearch as the datastore for this particular application instead of MySQL or the like.

616c 13y ago

You seem to have reservations about email in SQL, but the MailDB has been building a usable solution around this for years. http://maildb.sourceforge.net/ http://maildb.sourceforge.net/

cpach 13y ago
That project doesn’t seem to have been updated since 2002.
- cpach 13y ago
  Perhaps it was http://www.dbmail.org http://www.dbmail.org that '616c meant to refer to.
  - 616c 13y ago
    Woosh. Yes, that is what I had meant. Thanks for correcting that.

jfb 13y ago

Storing mail in mbox is such a terrible, terrible, terrible idea that anything would be better.

mkr-hn 13y ago
Can you elaborate for people like me who have no clue about mbox and why it might not be a good format?
- ams6110 13y ago
  mbox is just all your email concatenated into one big text file. You can I'm sure imagine that gets unwieldy at some point.
  - gkya 13y ago
    Well, there are MTAs, which one may use to distribute mail to different mailboxes. Also, no one is obliged to not truncate their mbox files after a while. I just encrypt my set of files and tuck them in a tarball, and store that in a specific folder with older tarballs.
  - e12e 13y ago
    It's not so much that it's one file (although that makes for fun times if the underlying filesystem doesn't support proper write locks), after all most databases also use just one file -- it's more the fact that individual messages are only recognized by lines starting with "From: " -- which presents some obvious challenges with escaping (that some programs manage to get wrong). I also imagine it'd be challenging to maintain an effective index as mails are deleted...

ams6110 13y ago

If it were me I might look at building somthing on top of notmuch. I use the emacs package for notmuch and it's awesome, it has search and tagging capabilities very similar to gmail and handles conversations nicely. http://notmuchmail.org/ http://notmuchmail.org/

naspeh 13y ago
I have tried it, while was researching. It was exciting, but I chose PostgreSQL as storage :).

rodgerd 13y ago

> but even for that, if you don't care about fancy guis, you're probably still better off running mutt, imap, mu on a shell you have somewhere I do like fancy GUIs and use roundcube as one option. For me, > it's still much more portable and easily to handle backup and syncing purposes in different formats. ...the portability is very important. I don't want a new MUA to force me into changing my whole mailstore/MTA chain.