4 ms·

I have been using this for years and it's great software. One tip, store each different email account in it's own "database" my crontab looks like this: #E

I have been using this for years and it's great software. One tip, store each different email account in it's own "database" my crontab looks like this: #Email backup 0 0,6,12,18 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/joshstrange --resume josh@joshstrange.com >> ~/mail-backup-logs/joshstrange.com.log 2>&1 0 1,7,13,19 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/otheremail --resume other@otheremail.com >> ~/mail-backup-logs/otheremail.com.log 2>&1 .... I sync it all locally to my house then back it up to Dropbox as well. The reason to store them in different datebases is you cannot "filter" them out when restoring so if they all go to the same DB if you restore you are restoring ALL your email across all accounts to one new account.

nicholasjarnold 10y ago
Chiming in to second this comment for anyone who is skeptical of using gmvault. I too have used it for years with great success. Thanks to the author for creating it!
laumars 10y ago
Thank you for your advice, that's handy to know. What's the database format it uses, sqlite3? (I tried looking on the repo but couldn't any obvious reference)
- joshstrange 10y ago
  I believe it is all flat-file. When I say "Database" I am just referencing what they call it: -d DB_DIR, --db-dir DB_DIR But from what I see the structure looks like this: db/ YYYY-MM/ 1234554543262346.eml.gz - I assume the meat and potatoes of the email along with attachments, not sure 1234554543262346.meta - JSON file with msg_id, thread_id, flags, labels, subject, etc
- leeoniya 10y ago
  if you want to try something that downloads Gmail via imap and indexes it into an sqlite3 db (with FTS5 fulltext of from/to/cc/bcc/subject/body fields) and extracts attachments to filesystem, take a look at a recent project of mine: https://github.com/good-3g-apps/email/tree/master/imap-importer https://github.com/good-3g-apps/email/tree/master/imap-impor... It also saves the raw .eml files to disk. No support for labels (yet), but it does properly link up threads in the db using `References` from the parsed headers (setting both MPTT and adjacency-list fields) It's WIP, so contributions welcome :)
  - leeoniya 10y ago
    FWIW regarding speed, i was able to download, index & extract my entire INBOX + Sent Items (14k emails, 3.5GB total) in < 10min on a fast connection. the limiting factor by far was connection/imap speed.
  - leeoniya 10y ago
    moved to https://github.com/Good-3G/email https://github.com/Good-3G/email

nicholasjarnold 10y ago

Chiming in to second this comment for anyone who is skeptical of using gmvault. I too have used it for years with great success. Thanks to the author for creating it!

laumars 10y ago

Thank you for your advice, that's handy to know. What's the database format it uses, sqlite3? (I tried looking on the repo but couldn't any obvious reference)

joshstrange 10y ago
I believe it is all flat-file. When I say "Database" I am just referencing what they call it: -d DB_DIR, --db-dir DB_DIR But from what I see the structure looks like this: db/ YYYY-MM/ 1234554543262346.eml.gz - I assume the meat and potatoes of the email along with attachments, not sure 1234554543262346.meta - JSON file with msg_id, thread_id, flags, labels, subject, etc
leeoniya 10y ago
if you want to try something that downloads Gmail via imap and indexes it into an sqlite3 db (with FTS5 fulltext of from/to/cc/bcc/subject/body fields) and extracts attachments to filesystem, take a look at a recent project of mine: https://github.com/good-3g-apps/email/tree/master/imap-importer https://github.com/good-3g-apps/email/tree/master/imap-impor... It also saves the raw .eml files to disk. No support for labels (yet), but it does properly link up threads in the db using `References` from the parsed headers (setting both MPTT and adjacency-list fields) It's WIP, so contributions welcome :)
- leeoniya 10y ago
  FWIW regarding speed, i was able to download, index & extract my entire INBOX + Sent Items (14k emails, 3.5GB total) in < 10min on a fast connection. the limiting factor by far was connection/imap speed.
- leeoniya 10y ago
  moved to https://github.com/Good-3G/email https://github.com/Good-3G/email