8 ms·

Back That Gmail Up

106 points by sant0sk1 14y ago

Rudism 14y ago
Another good one is http://gmvault.org http://gmvault.org. I used this just recently to backup and transfer all of my Google Apps emails to a separate gmail account since I'm transferring my apps domain to a different provider. Going just off of what the article says about BaGoMa, Gmvault seems to be comparable in features.
- waffle_ss 14y ago
  From a quick glance Gmvault looks preferable from a security standpoint as your credentials get saved in ~/.gmvault rather than being passed as a command-line argument, which could be viewed by any user on the system.
  - deleted 14y ago
    [deleted]
- manveru 14y ago
  Been trying to get gmvault to do at least one successful run since it showed up on HN maybe a week or so ago. It always stops somewhere at mails from 2008 with no error message, just dies. Guess I'll wait until it's a bit more mature. I only have 6GB of mails too.
  - zoobert 14y ago
    Gmvault can fail for multiple reasons, for example Gmail can cut the connection and throttle you for a while. There are also bugs left or GMail emails that cannot be downloaded (email corruption ?). For this reason, there is a --restart option for the sync and restore mode (sync being the backup). The restart mode is a resume mode that will restart the backup or restore near from where it failed. With that mode you should be able to download your 6 GB on disk.
yurisagalov 14y ago
https://www.backupify.com/ https://www.backupify.com/ is a pretty good alternative as well
stevenleeg 14y ago
My main problem with this is that it would require me to store my password, in clear text, in a shell script. I'm no security expert but that raises a red flag for me. I'd rather have some way of entering my password each time I want to back up, but this would also make the cron process impossible.
- ben1040 14y ago
  See gmvault as mentioned in the comments here, it just saves an oauth token. Still can be used to access your mail but at least it can be easily revoked and isn't your clear text password.
- yaix 14y ago
  Just chmod the shell script to 700, so that only your account can read it, together with an encrypted HDD that should be quite save.
  - waffle_ss 14y ago
    If it's a shared box, you'd still have to hide the --pwd argument from `ps` process listings. You'd have to do something like make a ~/.gmail-pass file with the content "--pwd foobar", and call the script like this: cat ~/.gmail-pass | xargs python bagoma.py --email=foo@bar
    - yaix 14y ago
      Good point, hadn't thought of the process list.
johnbrayton 14y ago
My Mac app, CloudPull, does something similar. CloudPull backs up Google Mail, Contacts, Calendars, Drive (formery Docs) and Reader to your Mac. http://www.goldenhillsoftware.com/ http://www.goldenhillsoftware.com/ CloudPull stores your password securely in the Mac OS X keychain.
- DenisM 14y ago
  I looked through your web site and one question remained unanswered: Does it also do cloud push? In other words, can I use your app to restore the backups to a different Gmail account later?
  - johnbrayton 14y ago
    No, CloudPull does not restore directly back to Google. You could do this independently by restoring to Apple Mail, and then importing into Gmail or another IMAP account.
    - DenisM 14y ago
      Two problems with that: 1. I'd rather have one app do the whole thing, so that I know it's tested and it works. 2. How about labels, contact, calendars, etc? I'd pay for this app, but only if it allowed full roundtrip.
      - johnbrayton 14y ago
        Sorry, CloudPull has no ability to restore directly to Google. My emphasis has been restoring to local apps. In all cases, there are ways of importing back into Google, but none that the app does on its own. The ability to restore directly to Google is on my roadmap for future versions.
- frou_dh 14y ago
  Nice! That looks very well executed. Personally I'm glad to see you explicitly mention that it's read-only. Separation of concerns and all that.
Camillo 14y ago
This may be a naive question, but why is it not enough to set your mail client to save a copy of all messages?
- icebraining 14y ago
  For many people, Gmail is their email client...
  - slantyyz 14y ago
    For many, it might be easier to run a mail client than a python script though.
- deleted 14y ago
  [deleted]
- replax 14y ago
  The problem which might arise is also, that being locked out of your email account will most likely result in being locked out of any "recover password" functions anywhere on the web. Thus, you need a backup of your email and preferably a working email address, too!
there 14y ago
I'm not leery of Google shuttering its Gmail service, but I can imagine a scenario where they cut off my access to Gmail and I have no way to plead my case to the search behemoth. I am baffled at how people continue using such a service for something as important as e-mail in 2012, while still being fully aware that their access can be completely cut off at random with no reason or recourse. Does everyone just assume "it won't happen to me"?
- icebraining 14y ago
  Shit happens with every provider, free or paid; hoping it won't because you pay $400/month for it is foolish. The smart thing is to make sure it doesn't affect you much, by decoupling from the service and turning it into a replaceable commodity as much as possible. That means using your own domain, keeping backups, etc. Then, when shit happens, you can just sign up for another provider and keep going.
  - there 14y ago
    Shit happens, but with nearly every other provider, you can at least talk to someone to find out what happened and get your service restored, or at least get your data.
    - icebraining 14y ago
      You can always get your data if you do backups - like you should in any case - so that point is moot. So, you're paying and giving up on a nice webmail client with two-factor auth (how many providers offer this?) for "finding out what happen". I don't think it's a great deal. I'm not a Gmail apologist, by the way - I think there are valid reasons not to use it, like not giving Google a copy of your whole life - but I don't find their policy for when "shit happens" a real problem.
      - replax 14y ago
        You do not have to give GMail up, not sure where you read that... The point was, that exclusively relying on it is not smart, if email is important to you. You can easily use the full benefits of your gmail convenience while still having a fully working fallback. There are even two ways of doing it, one would be to simply forward your email, or using your own domain on google apps.
sneak 14y ago
On most stock OSes, this will show your password to anyone who views the list of processes on the machine. Try 'imapsync' [1] instead, as it uses a config file for your credentials instead of command-line arguments, keeping the secret(s) out of the process list. Also, the author of this blog post is showing his inexperience: > BaGoMa's output is pretty informative so I send it to log a file in case I ever want to check up on it and make sure the backups are still working. You won't ever do that (no reason to check the logs when it's working), and it will fail (effectively-silently) one day without you noticing, and then you won't have backups when you actually find that you need them. Don't do it that way. Instead, use a cronjob wrapper script (e.g. [2]) that is silent on success and outputs on error, so the "cronjob-output-as-email" feature works to notify you when it breaks. [1] http://imapsync.lamiral.info/ http://imapsync.lamiral.info/ [2] https://github.com/sneak/misc/blob/master/cronify/cronify https://github.com/sneak/misc/blob/master/cronify/cronify
- MikeW 14y ago
  imapsync doesn't work perfectly with gmails imap server. There are multiple scripts out there just to work around as many edge cases as possible.
- cynwoody 14y ago
  > Try 'imapsync' [1] instead, as it uses a config file for your credentials instead of command-line arguments, keeping the secret(s) out of the process list. Given where he said the box resides, I'd say he's got a major problem if unfriendlies are in s position to run ps -eF on it, no matter what is in the command strings, LOL! And, if they can run ps, they can more than likely read the config files, too! > Instead, use a cronjob wrapper script (e.g. [2]) that is silent on success and outputs on error, so the "cronjob-output-as-email" feature works to notify you when it breaks. Excellent suggestion.
- chuckbot 14y ago
  gmvault opens a browser window, where you confirm you want to give an auth token to gmvault. All without a password anywhere.
  - 14y ago
gouranga 14y ago
If you're going to go through that level of effort and you're going to have a computer on at 3am, just run your own SMTP/imap setup and back that up. You'll never have to worry about getting kicked off or them going under then. I've been running my own mail server for 15 years and I've probably spent a couple of hours administering it over that time.
- jarofgreen 14y ago
  Then you just have a bunch of other worries; are your servers up and secure; not an open relay; has an anti-spam measure kicked in against your server so your emails aren't being read .... not saying you shouldn't do it, just that there are pluses and minuses to both.
  - gouranga 14y ago
    Most of which is just paranoia and will never shoot you. Are my servers up? Don't really care - SMTP will be redelivered if I discover they are down. They were down for about 2 days once thanks to the idiots at BT digging through the cable. No major effect other than some peace. Is it an open relay? no it's not - I checked it. Anti spam? I deal with it manually. I don't get much as I don't publicise my email address.
    - jarofgreen 14y ago
      > Are my servers up? Fair point on SMTP retries, but "up" also includes "not under attack, not already hacked, fully patched & secure". > Is it an open relay? no it's not - I checked it. Good for you, you clearly have the skills to do so and to be confident in it. Not everyone does or even knows what that means. > Anti spam? Actually, I wasn't talking about the receiving spam problem. You can deal with that on your own server in many ways. I was talking about the email you sent being marked as spam problem, so your intended receiver never reads it. Maybe your reverse DNS records are wrong or missing, or SPF records, or one of a thousand little things. This is much harder to fix, as you have to do it on each mail host you send to, and worse, you may not even realise it's happening. Again, not saying you or anyone shouldn't, just that there are downsides that need to be considered. I did it for years then realised my time was better spent elsewhere and moved to Google Apps.
masnick 14y ago
Instead of putting the password in as a plaintext argument, you can store it in the OS X keychain and access it with the command described here: http://www.maclovin.de/2010/02/access-os-x-keychain-from-terminal/ http://www.maclovin.de/2010/02/access-os-x-keychain-from-ter... I use this in my .zshrc to create Bitbucket repos on the fly: function bitbucket() { curl -X POST -u masnick:`security 2>&1 >/dev/null find-internet-password -gs masnick.bitbucket.org | cut -d '"' -f 2` https://api.bitbucket.org/1.0/repositories/ -d name=$@ -d scm=git -d is_private=True }; (Running "bitbucket newrepo" will create a new repo in my Bitbucket acct called "newrepo".)
muyuu 14y ago
Does this generate some sort of "standard" mbox file? something one can open with Thunderbird or similar? Probably more convenient than periodically running a POP3 client.
jarofgreen 14y ago
I like it. Just got it going. Specifically I like that it stores all your emails as plain text, one file per email, so you can easily use/write other tools to search/browse your backups if you need to. Maybe someone already has? (Well, it stores the raw email contents as text. Most emails have a text/plain bit but not all, so you can almost just use grep.)
- icebraining 14y ago
  Plain text one file per email is the standard Maildir[1] format. It's extremely well supported by email programs and utilities. [1]: http://en.wikipedia.org/wiki/Maildir http://en.wikipedia.org/wiki/Maildir
  - jarofgreen 14y ago
    BaGoMa puts your email in folders based on the first 2 characters of the file name. eg 00/00a87... 00/00aa8... 0b/0b0a9... So you'd have to do some folder shuffling before dumping the data into a Maildir server filesystem but that's not to hard.
DigitalSea 14y ago
People were raving about it on a forum - well I'm sold on the idea. In all serious though, if you assume that your email service will be around forever you probably shouldn't be on the Internet. Cudos to the author, I haven't really seen the point stressed enough that you should backup your email, especially on a service as large as Gmail.
geoffhill 14y ago
> "It's hosted on SourceForge, just kidding that was weird and scary" How the world has changed...
codyjames 14y ago
Great idea! I need to do this!
danbmil99 14y ago
"X It's hosted on SourceForge, just kidding that was weird and scary" It is hosted on SF, and that IS weird and scary. In this day and age, why would your new cutting-edge FOSS offering be delivered on SF rather than github?
andybak 14y ago
So. GMVault looks like the best solution except: 1. It saves one big inscrutable .db file rather than individual messages in some compatible format or other. 2. It doesn't seem to play nice with laptop suspend. If I'm going to run it as a scheduled task then it needs to not get confused if I shut my laptop lid in the middle of a run. 3. It could be clearer about the correct workflow for continual incremental backups. It sounds like you probably want to do one full backup, continual partial backups and then occasionally a full backup in case your labels have changed. I don't like choices.
- zoobert 14y ago
  I am the Gmvault developer. 1. Gmvault doesn't save all your emails in a .db file. It creates a db directory which contains sub directories containing your emails in eml format. So it is using a standard email format and allow users to access individual emails. 2. I have already closed my laptop and reopen it and Gmvault restarted from where it stopped (tried on windows). Still there might be a problem so please report it following the instructions here: http://gmvault.org/report_pb.html http://gmvault.org/report_pb.html. 3. Regarding the workflow, the recommended mode is to do a full backup and then daily do a quick backup. The quick backup is an incremental backup that only check emails for the last two months. It takes only 2 to 3 minutes normally.
  - andybak 14y ago
    Thanks for replying. I'm going to check again. I must have been doing something wrong. I'll contact you by a more appropriate means when I've figured out what.

106 points by sant0sk1 14y ago

Another good one is http://gmvault.org http://gmvault.org. I used this just recently to backup and transfer all of my Google Apps emails to a separate gmail account since I'm transferring my apps domain to a different provider. Going just off of what the article says about BaGoMa, Gmvault seems to be comparable in features.

waffle_ss 14y ago
From a quick glance Gmvault looks preferable from a security standpoint as your credentials get saved in ~/.gmvault rather than being passed as a command-line argument, which could be viewed by any user on the system.
- deleted 14y ago
  [deleted]
manveru 14y ago
Been trying to get gmvault to do at least one successful run since it showed up on HN maybe a week or so ago. It always stops somewhere at mails from 2008 with no error message, just dies. Guess I'll wait until it's a bit more mature. I only have 6GB of mails too.
- zoobert 14y ago
  Gmvault can fail for multiple reasons, for example Gmail can cut the connection and throttle you for a while. There are also bugs left or GMail emails that cannot be downloaded (email corruption ?). For this reason, there is a --restart option for the sync and restore mode (sync being the backup). The restart mode is a resume mode that will restart the backup or restore near from where it failed. With that mode you should be able to download your 6 GB on disk.

yurisagalov 14y ago

https://www.backupify.com/ https://www.backupify.com/ is a pretty good alternative as well

stevenleeg 14y ago

My main problem with this is that it would require me to store my password, in clear text, in a shell script. I'm no security expert but that raises a red flag for me. I'd rather have some way of entering my password each time I want to back up, but this would also make the cron process impossible.

ben1040 14y ago
See gmvault as mentioned in the comments here, it just saves an oauth token. Still can be used to access your mail but at least it can be easily revoked and isn't your clear text password.
yaix 14y ago
Just chmod the shell script to 700, so that only your account can read it, together with an encrypted HDD that should be quite save.
- waffle_ss 14y ago
  If it's a shared box, you'd still have to hide the --pwd argument from `ps` process listings. You'd have to do something like make a ~/.gmail-pass file with the content "--pwd foobar", and call the script like this: cat ~/.gmail-pass | xargs python bagoma.py --email=foo@bar
  - yaix 14y ago
    Good point, hadn't thought of the process list.

johnbrayton 14y ago

My Mac app, CloudPull, does something similar. CloudPull backs up Google Mail, Contacts, Calendars, Drive (formery Docs) and Reader to your Mac. http://www.goldenhillsoftware.com/ http://www.goldenhillsoftware.com/ CloudPull stores your password securely in the Mac OS X keychain.

DenisM 14y ago
I looked through your web site and one question remained unanswered: Does it also do cloud push? In other words, can I use your app to restore the backups to a different Gmail account later?
- johnbrayton 14y ago
  No, CloudPull does not restore directly back to Google. You could do this independently by restoring to Apple Mail, and then importing into Gmail or another IMAP account.
  - DenisM 14y ago
    Two problems with that: 1. I'd rather have one app do the whole thing, so that I know it's tested and it works. 2. How about labels, contact, calendars, etc? I'd pay for this app, but only if it allowed full roundtrip.
    - johnbrayton 14y ago
      Sorry, CloudPull has no ability to restore directly to Google. My emphasis has been restoring to local apps. In all cases, there are ways of importing back into Google, but none that the app does on its own. The ability to restore directly to Google is on my roadmap for future versions.
frou_dh 14y ago
Nice! That looks very well executed. Personally I'm glad to see you explicitly mention that it's read-only. Separation of concerns and all that.

Camillo 14y ago

This may be a naive question, but why is it not enough to set your mail client to save a copy of all messages?

icebraining 14y ago
For many people, Gmail is their email client...
- slantyyz 14y ago
  For many, it might be easier to run a mail client than a python script though.
deleted 14y ago
[deleted]
replax 14y ago
The problem which might arise is also, that being locked out of your email account will most likely result in being locked out of any "recover password" functions anywhere on the web. Thus, you need a backup of your email and preferably a working email address, too!

there 14y ago

I'm not leery of Google shuttering its Gmail service, but I can imagine a scenario where they cut off my access to Gmail and I have no way to plead my case to the search behemoth. I am baffled at how people continue using such a service for something as important as e-mail in 2012, while still being fully aware that their access can be completely cut off at random with no reason or recourse. Does everyone just assume "it won't happen to me"?

icebraining 14y ago
Shit happens with every provider, free or paid; hoping it won't because you pay $400/month for it is foolish. The smart thing is to make sure it doesn't affect you much, by decoupling from the service and turning it into a replaceable commodity as much as possible. That means using your own domain, keeping backups, etc. Then, when shit happens, you can just sign up for another provider and keep going.
- there 14y ago
  Shit happens, but with nearly every other provider, you can at least talk to someone to find out what happened and get your service restored, or at least get your data.
  - icebraining 14y ago
    You can always get your data if you do backups - like you should in any case - so that point is moot. So, you're paying and giving up on a nice webmail client with two-factor auth (how many providers offer this?) for "finding out what happen". I don't think it's a great deal. I'm not a Gmail apologist, by the way - I think there are valid reasons not to use it, like not giving Google a copy of your whole life - but I don't find their policy for when "shit happens" a real problem.
    - replax 14y ago
      You do not have to give GMail up, not sure where you read that... The point was, that exclusively relying on it is not smart, if email is important to you. You can easily use the full benefits of your gmail convenience while still having a fully working fallback. There are even two ways of doing it, one would be to simply forward your email, or using your own domain on google apps.

sneak 14y ago

On most stock OSes, this will show your password to anyone who views the list of processes on the machine. Try 'imapsync' [1] instead, as it uses a config file for your credentials instead of command-line arguments, keeping the secret(s) out of the process list. Also, the author of this blog post is showing his inexperience: > BaGoMa's output is pretty informative so I send it to log a file in case I ever want to check up on it and make sure the backups are still working. You won't ever do that (no reason to check the logs when it's working), and it will fail (effectively-silently) one day without you noticing, and then you won't have backups when you actually find that you need them. Don't do it that way. Instead, use a cronjob wrapper script (e.g. [2]) that is silent on success and outputs on error, so the "cronjob-output-as-email" feature works to notify you when it breaks. [1] http://imapsync.lamiral.info/ http://imapsync.lamiral.info/ [2] https://github.com/sneak/misc/blob/master/cronify/cronify https://github.com/sneak/misc/blob/master/cronify/cronify

MikeW 14y ago
imapsync doesn't work perfectly with gmails imap server. There are multiple scripts out there just to work around as many edge cases as possible.
cynwoody 14y ago
> Try 'imapsync' [1] instead, as it uses a config file for your credentials instead of command-line arguments, keeping the secret(s) out of the process list. Given where he said the box resides, I'd say he's got a major problem if unfriendlies are in s position to run ps -eF on it, no matter what is in the command strings, LOL! And, if they can run ps, they can more than likely read the config files, too! > Instead, use a cronjob wrapper script (e.g. [2]) that is silent on success and outputs on error, so the "cronjob-output-as-email" feature works to notify you when it breaks. Excellent suggestion.
chuckbot 14y ago
gmvault opens a browser window, where you confirm you want to give an auth token to gmvault. All without a password anywhere.
- 14y ago

gouranga 14y ago

If you're going to go through that level of effort and you're going to have a computer on at 3am, just run your own SMTP/imap setup and back that up. You'll never have to worry about getting kicked off or them going under then. I've been running my own mail server for 15 years and I've probably spent a couple of hours administering it over that time.

jarofgreen 14y ago
Then you just have a bunch of other worries; are your servers up and secure; not an open relay; has an anti-spam measure kicked in against your server so your emails aren't being read .... not saying you shouldn't do it, just that there are pluses and minuses to both.
- gouranga 14y ago
  Most of which is just paranoia and will never shoot you. Are my servers up? Don't really care - SMTP will be redelivered if I discover they are down. They were down for about 2 days once thanks to the idiots at BT digging through the cable. No major effect other than some peace. Is it an open relay? no it's not - I checked it. Anti spam? I deal with it manually. I don't get much as I don't publicise my email address.
  - jarofgreen 14y ago
    > Are my servers up? Fair point on SMTP retries, but "up" also includes "not under attack, not already hacked, fully patched & secure". > Is it an open relay? no it's not - I checked it. Good for you, you clearly have the skills to do so and to be confident in it. Not everyone does or even knows what that means. > Anti spam? Actually, I wasn't talking about the receiving spam problem. You can deal with that on your own server in many ways. I was talking about the email you sent being marked as spam problem, so your intended receiver never reads it. Maybe your reverse DNS records are wrong or missing, or SPF records, or one of a thousand little things. This is much harder to fix, as you have to do it on each mail host you send to, and worse, you may not even realise it's happening. Again, not saying you or anyone shouldn't, just that there are downsides that need to be considered. I did it for years then realised my time was better spent elsewhere and moved to Google Apps.

masnick 14y ago

Instead of putting the password in as a plaintext argument, you can store it in the OS X keychain and access it with the command described here: http://www.maclovin.de/2010/02/access-os-x-keychain-from-terminal/ http://www.maclovin.de/2010/02/access-os-x-keychain-from-ter... I use this in my .zshrc to create Bitbucket repos on the fly: function bitbucket() { curl -X POST -u masnick:`security 2>&1 >/dev/null find-internet-password -gs masnick.bitbucket.org | cut -d '"' -f 2` https://api.bitbucket.org/1.0/repositories/ -d name=$@ -d scm=git -d is_private=True }; (Running "bitbucket newrepo" will create a new repo in my Bitbucket acct called "newrepo".)

muyuu 14y ago

Does this generate some sort of "standard" mbox file? something one can open with Thunderbird or similar? Probably more convenient than periodically running a POP3 client.

jarofgreen 14y ago

I like it. Just got it going. Specifically I like that it stores all your emails as plain text, one file per email, so you can easily use/write other tools to search/browse your backups if you need to. Maybe someone already has? (Well, it stores the raw email contents as text. Most emails have a text/plain bit but not all, so you can almost just use grep.)

icebraining 14y ago
Plain text one file per email is the standard Maildir[1] format. It's extremely well supported by email programs and utilities. [1]: http://en.wikipedia.org/wiki/Maildir http://en.wikipedia.org/wiki/Maildir
- jarofgreen 14y ago
  BaGoMa puts your email in folders based on the first 2 characters of the file name. eg 00/00a87... 00/00aa8... 0b/0b0a9... So you'd have to do some folder shuffling before dumping the data into a Maildir server filesystem but that's not to hard.

DigitalSea 14y ago

People were raving about it on a forum - well I'm sold on the idea. In all serious though, if you assume that your email service will be around forever you probably shouldn't be on the Internet. Cudos to the author, I haven't really seen the point stressed enough that you should backup your email, especially on a service as large as Gmail.

geoffhill 14y ago

> "It's hosted on SourceForge, just kidding that was weird and scary" How the world has changed...

codyjames 14y ago

Great idea! I need to do this!

danbmil99 14y ago

"X It's hosted on SourceForge, just kidding that was weird and scary" It is hosted on SF, and that IS weird and scary. In this day and age, why would your new cutting-edge FOSS offering be delivered on SF rather than github?

andybak 14y ago

So. GMVault looks like the best solution except: 1. It saves one big inscrutable .db file rather than individual messages in some compatible format or other. 2. It doesn't seem to play nice with laptop suspend. If I'm going to run it as a scheduled task then it needs to not get confused if I shut my laptop lid in the middle of a run. 3. It could be clearer about the correct workflow for continual incremental backups. It sounds like you probably want to do one full backup, continual partial backups and then occasionally a full backup in case your labels have changed. I don't like choices.

zoobert 14y ago
I am the Gmvault developer. 1. Gmvault doesn't save all your emails in a .db file. It creates a db directory which contains sub directories containing your emails in eml format. So it is using a standard email format and allow users to access individual emails. 2. I have already closed my laptop and reopen it and Gmvault restarted from where it stopped (tried on windows). Still there might be a problem so please report it following the instructions here: http://gmvault.org/report_pb.html http://gmvault.org/report_pb.html. 3. Regarding the workflow, the recommended mode is to do a full backup and then daily do a quick backup. The quick backup is an incremental backup that only check emails for the last two months. It takes only 2 to 3 minutes normally.
- andybak 14y ago
  Thanks for replying. I'm going to check again. I must have been doing something wrong. I'll contact you by a more appropriate means when I've figured out what.