5 ms·

Ask HN: Is There an IMAP2Git?

6 points by l0b0 5y ago

After using about half a dozen different email clients I don't trust any of them to keep my messages safe indefinitely. I would much rather have several copies of every email all over the globe in a system which is fast and easy to sync, search, and reorganize, and which has easy recovery of deleted files: A self-hosted Git repository. A quick search didn't come up with any obvious candidates. There seems to be some projects out there for saving IMAP contents to various file formats, but none of the ones I found seem to have a big community, and I've no idea which file formats would be easily amenable to version control. Also, should I save the attachments embedded in the emails or as separate files (better version control and smaller email files, but breaks the explicit connection to the email file, and makes relocating them harder)? Finally, is there any reason why this is a terrible idea altogether, and I should just accept that personal messages from many years ago are just going to randomly disappear at some point?

jrootabega 5y ago
What would a commit represent? I've been thinking idly about converting certain media to a vcs format myself.
- toomuchtodo 5y ago
  Most likely a document in Internet Message Format, as described in RFC5322. https://datatracker.ietf.org/doc/html/rfc5322 https://datatracker.ietf.org/doc/html/rfc5322
  - jrootabega 5y ago
    The repository would contain documents like that, but I'm wondering what thoughts the OP had about how a mail structure would map to a graph structure. Each commit could represent the addition of such documents to a list of documents. Would it be the addition of exactly one message? Or each commit could be changing the entire content of the repo to be that document. Would mail threads map one-to-one to branches?
    - unqueued 5y ago
      Git is a really great format for email archives, because it provides free duplication and compression via packfiles, although I think it is a tilting at windmills to try to map specific commits to emails. I explained my git/imap workflow in another comment here. However, git is perfect for calendars. It is used as a backing store for these caldav servers: https://github.com/jelmer/xandikos https://github.com/jelmer/xandikos https://github.com/Kozea/Radicale https://github.com/Kozea/Radicale
      - jrootabega 5y ago
        Very interesting!
outsomnia 5y ago
How does your IMAP implmentation store the emails? If it's maildir format (it is for me on Postfix + Dovecot) you can just rsync the maildirs for users you care about, it also contains the IMAP folders and the original RFC822 emails.
- LinuxBender 5y ago
  To add to this, one could also make use of rsnapshot [1] perl script locally and/or remotely to get the rollback/forward capabilities of git with minimal disk space overhead. [1] - https://linux.die.net/man/1/rsnapshot https://linux.die.net/man/1/rsnapshot
seifferth 5y ago
While I don't know of any imap to git sync tool specifically, there are tons of tools for storing email locally in plain text and for keeping these local email stores in sync with imap servers. The most common local email store formats seem to be mbox[1] and maildir[2]. Mbox stores one "folder" of emails per file, while maildir stores one email per file, which does seem to be the more modern approach (exceeding your file quota does not seem to be a particularly urgent problem anymore). I personally use mbsync[3] to pull my email into a local maildir. I also use notmuch[4] and a nice tui[5] to search, read and write it. Note that filenames in maildir are unique, but provide little value beyond that. Some emails are also encoded in ways that will hide content from grep and friends (base64 being among the most popular ones). This makes notmuch especially valuable, as it abstracts all those encoding issues away and provides an amazing search interface. I do not use a git repository for email myself, yet since a maildir is just a directory tree of plain text files, storing it in a repo is a no-brainer: mbsync -a # Configured via dotfiles cd my_local_maildir git add . git commit [1]: https://en.wikipedia.org/wiki/Mbox https://en.wikipedia.org/wiki/Mbox [2]: https://en.wikipedia.org/wiki/Maildir https://en.wikipedia.org/wiki/Maildir [3]: https://isync.sourceforge.io/ https://isync.sourceforge.io/ [4]: https://notmuchmail.org/ https://notmuchmail.org/ [5]: https://github.com/wangp/bower https://github.com/wangp/bower
- unqueued 5y ago
  I solved my email problem recently, and I will probably clean it up and put it on Github. I have a Dovecot Docker container that I spin up and mirror my imap emails with imapsync. You can easily configure imapsync to use a friendly mapping for the imap layouts of different providers (they have good documentation with examples) Git worked pretty well for storage of the dovecot emails, but then I got even more sophisticated. I now use a bup repo inside a git-annex repo[1]. This is because I get the benefit of being able to track which computer I have backups on, and the myriad of git-annex backends[2], while bup provides compression, reduplication, and par2 error correction data. [1]: https://git-annex.branchable.com/tips/Bup_repositories_in_git-annex/ https://git-annex.branchable.com/tips/Bup_repositories_in_gi... [2]: https://git-annex.branchable.com/todo/external_backends/ https://git-annex.branchable.com/todo/external_backends/
nextos 5y ago
Have you looked into mbsync (isync) and notmuch?