12 ms·
Gmvault: Backup and restore your Gmail account
- josefresco 10y agoI use Thunderbird Portable to backup our Google Apps/GSuite email accounts. Open once a week - Bob's your uncle.
- pawadu 10y agoGiven how easy and carelessly Google can close your account and ruin your digital life, I guess periodic backups of your cloud accounts will soon be considered a good practice.
- Demcox 10y agoThis - and having no unique data anywhere. Having your files distributed this day and age should be general approach.
- scholia 10y ago> I guess periodic backups of your cloud accounts will soon be considered a good practice. It has been good practice for a couple of decades. Of course, getting people to follow good practices is another matter....
- deleted 10y ago[deleted]
- bartvk 10y agoIn this case, it's not as good as a normal backup. To my knowledge, Google Docs/Sheets/etc. is exported in other file formats (MS Word format, PDF format, etc), not their internal file format. So it's not a real backup, just an export.
- scholia 10y agoGood point. And we have no way of knowing what Google's internal format looks like...
- adam12 10y agoI'm afraid this tool might trigger my account to be disabled.
- chongli 10y agoThen use Google Takeout [0], Google's official tool for backing up all your data. [0] https://takeout.google.com/settings/takeout https://takeout.google.com/settings/takeout
- tomschlick 10y agoIt uses the same IMAP protocol that your mail clients do. I have been using it for years and not a single issue with google closing down any of the 5-6 accounts I have archived.
- dharma1 10y agodoesn't http://www.google.com/takeout http://www.google.com/takeout include gmail?
- lima 10y agoNo
- cdubzzz 10y agoYep! And much more. But gmvault does also restore emails to an account, interestingly. I am (slowly) working on a project to pull some statistics from Takeout's mbox file for Mail. Also want to play around with the Location History, Chrome data and Hangouts exports.
- pokemongoaway 10y agoThat would be very useful. I'd like to use takeout incrementally - and have some tools to use those backups at any time (like you describe).
- eli 10y agoTakeout results in an MBOX file. Seems like it be possible to push that to a new account via IMAP.
- tedmiston 10y agoYes but I don't believe there's any way to automate Takeout. Also each new takeout is a full backup vs incremental.
- ljosa 10y agoI have been using both GYB (https://github.com/jay0lee/got-your-back https://github.com/jay0lee/got-your-back) and mbsync (http://isync.sourceforge.net/mbsync.html http://isync.sourceforge.net/mbsync.html) for this. How is gmvault better/different?
- wyldfire 10y agoWhy do you need to use both? Or are you saying that they're both equivalent and you'd had success with each?
- wielebny 10y agoI've been using gmvault for only one feature: ability to fetch e-mails that match a certain filter.
- chrislund 10y agoGYB does that, too: https://github.com/jay0lee/got-your-back/wiki#selective-backups-with-gmail-searching https://github.com/jay0lee/got-your-back/wiki#selective-back...
- tyingq 10y agoReading over the details, gmvault supports two things I don't see in GYB and one thing mbsync doesn't support: - XOauth - Sync to another gmail account (incrementally) Edit: Also supports only backing up emails with a specific gmail tag.
- amq 10y agoWhy should I use this over an email client like Thunderbird?
- ozim 10y agoI think there is no point, but if someone uses only web interface this might be valuable.
- sudshekhar 10y agoOff-topic. What are some of the reasons you use desktop clients? I have tried using thunderbird several times but the habit never stuck. Anything I might be missing out on (apart from local IMAP backups)?
- detaro 10y agoWorks offline, has its own OS-level windows (so I don't have to hunt around for the tab, can close the browser but keep e-mail open or vice vers), multiple e-mail accounts under one application while keeping them entirely separated (I don't want to give one provider access to my mailboxes at another provider, and I don't want to have to use multiple different web UIs). Also a matter of habit.
- favadi 10y agoMultiple accounts and GPG support
- scholia 10y ago(1) Desktop clients used to be faster and more efficient than using a web interface, and probably still are in most cases. However, Gmail is impressively snappy so I don't think it applies in that case. (2) You can do things with a desktop client that you can't do with a web interface. That includes sort by subject and sort by size. (3) Desktop clients can have real folders, which Gmail doesn't. (4) Works off-line. This is still useful, though not as useful as it used to be. (5) Easy to backup. Easy to automate backup.
- 10y ago
- lexalizer 10y agoPlease don't forget to encrypt your backups in case your local drive gets stolen or your cloud backup service is hacked.
- laurent123456 10y agoSince Gmail supports imap and pop3, one could simply use any proven email client to backup the emails. I don't think a special tool that or may not work is needed for this.
- cookiecaper 10y agoWhich "proven email client" do you recommend? There's an Import/Export extension for Thunderbird, one of the last standing desktop mail apps, but it's not good at handling huge, multi-thousand message exports. gmvault is purpose-built and quite simple. I've used it before with success. It backs up your whole mailbox in one command. Why should we fiddle with a desktop mail client?
- laurent123456 10y agoThunderbird works fine for me to handle several GB of emails (the mailbox archives that it generates are plain text and standard so there's no need to export them). For the command line there are also tools like `getmail`[0], with the added benefit that they work with any email provider. [0] http://pyropus.ca/software/getmail/ http://pyropus.ca/software/getmail/
- kitcar 10y agoFWIW I imported a 6GB .mbox file with Thunderbird last week, worked fine -
- cookiecaper 10y agoI stopped using Thunderbird because it was choking under my 20GB of mailboxes.
- pi-rat 10y agofetchmail, getmail and offlineimap are all "proven email clients" that work great for this.
- Twirrim 10y ago
- deleted 10y ago[deleted]
- emilburzo 10y agoAfter yesterday's post[1], I've setup a cron job to regularly backup Gmail/Google Drive. I'm using Gmvault for Gmail (emails and chats), pretty effortless and with enough features for me, especially the export to maildir. And rclone[2] for Google Drive, in "create a local mirror" mode (sync). Any other tools to backup your Google life? [1] - https://news.ycombinator.com/item?id=12972554 https://news.ycombinator.com/item?id=12972554 [2] - http://rclone.org/ http://rclone.org/
- jdc0589 10y agoI am actively trying to get off of dropbox. I really want a similar native application experience that syncs to-and-from S3. Not a cron job using the s3 cli, not something that only works on osx/windows/etc... So far owncloud enterprise is the only polished looking solution I've found, but thats a bit overkill...
- x1798DE 10y agoI've been using syncthing and I'm reasonably pleased with it. I don't use it heavily and only over my local network, though, so I'm not sure how well it handles archiving changes and file conflicts.
- mda590 10y agoYou may want to have a look at odrive (https://www.odrive.com https://www.odrive.com). Use it to sync a bunch of different storage accounts via this single app. Works alright for what it does, and they have an S3 option.
- shade23 10y agoWon't forklift serve your purpose?
- ReAzem 10y agoYou are looking for Syncany: https://www.syncany.org/ https://www.syncany.org/ But really, you may not need S3 at all and just sync between your devices with Syncthing: https://syncthing.net/ https://syncthing.net/ Shameless plug: Syncthing is in official Debian repositories! https://tracker.debian.org/pkg/syncthing https://tracker.debian.org/pkg/syncthing
- AdmiralAsshat 10y agoThe 'restore' feature looks nice. But would it be of any value if Google decides to close your account? That is, can I take my backup emails from Gmail and 'restore' it to myemail@self-hosted-domain.com in order to migrate all those emails over?
- timdafweak 10y agoGood question, I wonder if anyone’s tried this before. It would be invaluable if that were the case.
- whyagaindavid 10y agoI have opted to have 3 gmail accounts with various names; 1 outlook, 1 yahoo account. Every email sent to my gmail is forwarded > outlook > yahoo. I use 2 other gmail accounts for newsletters and etc. I am sure at least 1 company will remain free to access data! BTW thanks for the thunderbird tip.
- icebraining 10y agoI've tried and yes, you can.
- bshep 10y agoFrom reading the docs it seems possible Go to the website click on documentation then indepth then restore. You can also export to other formats ex. mbox
- hatchoo 10y agoI have used this in the past to backup email accounts of resigned employees so we continue to stay well within the maximum number of active accounts for our free Google Apps. It generally works fine and it allows you to restore the emails to a different account name. (I sometimes temporarily restore an account to search for old emails). It seems to have some issues with restoring accounts with a lot of large emails (large or multiple attachments) especially those that have reached the 15GB quota.
- hengheng 10y agoSince yesterday I have a few Google Takeout zip files in my backup ( https://takeout.google.com/settings/takeout https://takeout.google.com/settings/takeout ). I've used gmvault in the past, but this looks superior from the outside. Haven't delved into the data I'll admit.
- cdubzzz 10y agoI am working on digging around this data right now, actually. Some details on my website (see profile) and I will post on HN about it at some point when I get things further along (also want to look at Location History and Chrome data exports).
- jrcii 10y agoI use mutt setup according to Steve Losh's advice "The Homely Mutt" to achieve this with great success.
- deleted 10y ago[deleted]
- fgpwd 10y agoIs it considered safe to leave your oauth ID and Secret public in open source projects?
- honkhonkpants 10y agoIronically using some of these buggy tools can be an effective and swift way to get your account locked out or at least rate limited by anti-abuse systems. Read the code and make sure they are using official APIs and are coded sanely.
- equalunique 10y ago+1
- tomschlick 10y agoThey are. The first thing gmvault does is have you authenticate with oauth. From there it's just using the gmail apis / imap to download the data.
- kylehotchkiss 10y agoI was trying to push some people off my Google Apps accounts since google won't allow you to "split" some users off the account. Wish I would have seen this then!
- Roritharr 10y agoAre there similar tools for google docs documents? drive files? contacts?
- tomschlick 10y agoFor drive: https://github.com/prasmussen/gdrive https://github.com/prasmussen/gdrive
- SadWebDeveloper 10y agoOfftopic: I have Gsuite (free) for my company (<50), is it possible to backup all the emails of my users without knowing their passwords (with XOauth tokens?).
- rrggrr 10y agoNo Python 3 support (at least under Anaconda), but interested when its supported.
- joshstrange 10y agoI have been using this for years and it's great software. One tip, store each different email account in it's own "database" my crontab looks like this: #Email backup 0 0,6,12,18 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/joshstrange --resume josh@joshstrange.com >> ~/mail-backup-logs/joshstrange.com.log 2>&1 0 1,7,13,19 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/otheremail --resume other@otheremail.com >> ~/mail-backup-logs/otheremail.com.log 2>&1 .... I sync it all locally to my house then back it up to Dropbox as well. The reason to store them in different datebases is you cannot "filter" them out when restoring so if they all go to the same DB if you restore you are restoring ALL your email across all accounts to one new account.
- nicholasjarnold 10y agoChiming in to second this comment for anyone who is skeptical of using gmvault. I too have used it for years with great success. Thanks to the author for creating it!
- laumars 10y agoThank you for your advice, that's handy to know. What's the database format it uses, sqlite3? (I tried looking on the repo but couldn't any obvious reference)
- joshstrange 10y agoI believe it is all flat-file. When I say "Database" I am just referencing what they call it: -d DB_DIR, --db-dir DB_DIR But from what I see the structure looks like this: db/ YYYY-MM/ 1234554543262346.eml.gz - I assume the meat and potatoes of the email along with attachments, not sure 1234554543262346.meta - JSON file with msg_id, thread_id, flags, labels, subject, etc
- leeoniya 10y agoif you want to try something that downloads Gmail via imap and indexes it into an sqlite3 db (with FTS5 fulltext of from/to/cc/bcc/subject/body fields) and extracts attachments to filesystem, take a look at a recent project of mine: https://github.com/good-3g-apps/email/tree/master/imap-importer https://github.com/good-3g-apps/email/tree/master/imap-impor... It also saves the raw .eml files to disk. No support for labels (yet), but it does properly link up threads in the db using `References` from the parsed headers (setting both MPTT and adjacency-list fields) It's WIP, so contributions welcome :)
- equalunique 10y agoCool, but I'm going to have to do a source code review before I trust any new tool with my Gmail messages.
- hmart 10y agoSeems legit. In the past I have used Imapsync to transfer hundreds of email accounts out of gmail / google apps. https://imapsync.lamiral.info/ https://imapsync.lamiral.info/
- handruin 10y agoI'm interested to know what the pros and cons are of this utility vs using the Google takeout functionality? I like the idea of this project but I don't know what it would gain me over Google's native export? Is it the restoration that's missing from Google's service? https://takeout.google.com/settings/takeout https://takeout.google.com/settings/takeout
- tonyplee 10y agoI just started a takeout session on all my google drive, emails data too. I wonder how to use them once I have the backup data?
- JonathonW 10y agoExactly what you get from Takeout varies from service to service; for email you get an MBOX-format mailbox file (that you can then import into a desktop email client of your choice).
- ajennings 10y agoHere's one I wrote: https://github.com/abjennings/gmail-backup https://github.com/abjennings/gmail-backup Not as full featured (can't restore), but it's just a 77-line Python script. You could audit it yourself to make sure it doesn't upload your creds to another server.
- andydoan 10y agoDoes this only backup attachments and images as well as text? I can't seem to find any reference to this on the site. EDIT: Ok it appears gmvault encodes attachments into the eml default, cool.
- prplhaz4 10y agoI just searched this thread, the site, and github...Here's the best I've found...apparently it supports saving attachments... https://github.com/gaubert/gmvault/issues/73 https://github.com/gaubert/gmvault/issues/73 >> Users asked about attachments. Put somewhere in the doc that they are saved.
- kingnight 10y agoMy backup of Gmail is to use Mail.app via IMAP + download all attachments. I then have a backup of my computer -> 1. Time Machine + 2. Arq->S3/Glacier Given that this keeps mail locally in a constantly readible format (offline, copied in mbox)... is there something missing in my basic solution that this cli utility adds?
- jxy 10y agoOr any IMAP client that saves messages locally. I guess this utility benefits people who only use gmail's web interface.
- tonyplee 10y agoDoes it work on Raspberry Pi? I have an always on PI server as git server. I wouldn't mind put this on as one more cron job on it.
- icebraining 10y agoIt's written in Python, so it should work.
- falsedan 10y agoHow does this persist/restore filters?
- aq3cn 10y agoI have been planning to switch to mutt, this tool will also come in handy. Are there similar tools for MS Outlook or Protonmail or if it is possible to modify gmvault to work with either one?
- ajonit 10y agoI have been using Spanning backup for several years now. It backs up our Gmail, documents , calendars, contacts, sites It is one of those set-it-and-forget-it type services. The company is acquired by EMC now. The service costs us around $35/year/email account; for us it is small price for peace of mind. I wrote a review of them few years back - http://reviewofweb.com/gmail/backupify-vs-spanning/ http://reviewofweb.com/gmail/backupify-vs-spanning/
- bit_logic 10y agoData is only half the problem. The other is that email address has become the "primary key" for everything. Banking websites, random forums, everything. It's sometimes impossible to change it because it's the "primary key" for identity on that website. And email addresses are not portable like phone numbers.
- eigenvector 10y agoEmail addresses are portable to any webmail, mail server or other email infrastructure provider if you have your own domain and then forward to the service of your choice. This way you can use Gmail if you like or maintain your own full email stack, or anything in between, while still addressing your mail to something you control.
- bit_logic 10y agoThis is true if you're starting from nothing. However, Gmail is over a decade old at this point and is the primary email for millions of users. And something like custom domain is beyond the technical skills of average users.
- icebraining 10y agoHowever, Gmail is over a decade old at this point and is the primary email for millions of users. Switching is annoying, but not impossible. You just forward the old address to the new and progressively replace it on any accounts you might have. I still have access to my Gmail account, it just hasn't received a non-spam email in years.
- scholia 10y agoPlus you may be able to get your new email address to import all your old Gmail. Outlook.com does this, if you want...
- scholia 10y agoOddly enough, it was really easy to set up a custom domain on Gmail: I did it. However, Google stopped offering that service for free, which will stop most ordinary people from using it...
- mikebay 10y agoYes, we need to backup our emails before google delete our accounts. Googles operations start to remains me of obamas administration, where nobody can actually say anything out of template..
- AWildDHHAppears 10y agoI would love to do a backup, then delete everything older than a few months online.
- wfunction 10y agoWARNING/fun fact: it doesn't download all emails properly. Last time I tried it, it seemed that when the Gmail server randomly closed a connection (or maybe some other time, but I think it was in these instances), the program would just keep whatever partial results it had and then move on to the next email. Which meant I had a lot of partial emails on my drive (only a small fraction of all the emails, but still), and no way to detect them.
- joshfraser 10y agoA lazy alternative is to forward all incoming email to a yahoo account. I've been doing this for years and it's come in handy on the rare occasions when Gmail is unavailable.
- hobarrera 10y ago> Handle all Gmail IMAP hiccups. It's sad that after all these years, google still hasn't gotten around to properly implementing IMAP.
- tscs37 10y agoBest timing for posting this. Even before this I've been looking to get a bit independent from Google, atm I'm trying to get Camlistore to replicate my data over various places. (I wish it had some erasure encoding)
- xianshanyue 10y agogood