3 ms·

Kind of related: What do you use to backup binary files like photos? At the moment I've just got copies on old USB drives, but I'd like to put something on AWS

Kind of related: What do you use to backup binary files like photos? At the moment I've just got copies on old USB drives, but I'd like to put something on AWS Glacier too.

rlpb 12y ago
git-annex. It supports multiple external remotes, including USB drives, Glacier[1], bup and ddar[2]. You can keep files on any combination of the external remotes for redundancy and cost management. It keeps track of what is where, so you don't have to. The only catch is that you have to make sure to back up the (metadata-only) git repository itself also, and maintaining this backup on anything that is not a direct filesystem is painful. [1] Through my tool, glacier-cli, for which git-annex has native support. [2] I wrote ddar and the git-annex support for it.
- icebraining 12y ago
  +1 for git annex. For the git repo, if you don't have privacy concerns, you can just push to a private bitbucket repository. But I just had a script with: git clone /my/repo /tmp/backup-git-annex tar cvf /tmp/backup-git-annex.tar /tmp/backup-git-annex That way, you're left with a single tar file to backup.
- rcthompson 12y ago
  So git-annex now has support for efficiently deduplicating backends? Might be time for me to give it another look.
  - rlpb 12y ago
    For "local" storage as supported by bup and ddar, yes. Generally for external remotes: not yet, AFAIK. http://git-annex.branchable.com/design/assistant/deltas/ http://git-annex.branchable.com/design/assistant/deltas/ has some design thoughts, so I think it is planned.
- sytse 12y ago
  GitLab CEO here, as you mention backing up the git repository is very important. We just announced free hosting for git-annex files and repositories on GitLab.com that might be good for your use case, see https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-... for an overview.
- n3t 12y ago
  git-annex website says[1] it is not supposed to be a backup solution: > git-annex is not a backup system. It may be a useful component of an archival system, or a way to deliver files to a backup system With this in mind, can we treat it as a backup tool? Assuming backing up the git repository itself as you mentioned. [1] http://git-annex.branchable.com/not/ http://git-annex.branchable.com/not/
borgel 12y ago
I have a layered photo backup process: - New photos live on my laptop (in my Dropbox and covered by Backblaze) - I manually copy photos from my laptop to a RAID array on a server elsewhere in my house when I'm done editing/processing them - The server backs up the photos to S3 S3 is the last bastion, so to speak. I expect that normally, I pull any photos I need back off my server. But the house burns down, I go to S3. I run a cron job on my backup server that picks up all changed files (in my photo storage dir) and pushes them to S3 with s3cmd[1]. It's a pretty 'dumb' process, but works well. The S3 <-> Glacier lifecycle stuff is pretty cool, so the plan is to eventually enable it to deep-archive older photos. [1] http://s3tools.org/s3cmd http://s3tools.org/s3cmd

Kind of related: What do you use to backup binary files like photos? At the moment I've just got copies on old USB drives, but I'd like to put something on AWS Glacier too.

rlpb 12y ago

git-annex. It supports multiple external remotes, including USB drives, Glacier[1], bup and ddar[2]. You can keep files on any combination of the external remotes for redundancy and cost management. It keeps track of what is where, so you don't have to. The only catch is that you have to make sure to back up the (metadata-only) git repository itself also, and maintaining this backup on anything that is not a direct filesystem is painful. [1] Through my tool, glacier-cli, for which git-annex has native support. [2] I wrote ddar and the git-annex support for it.

icebraining 12y ago
+1 for git annex. For the git repo, if you don't have privacy concerns, you can just push to a private bitbucket repository. But I just had a script with: git clone /my/repo /tmp/backup-git-annex tar cvf /tmp/backup-git-annex.tar /tmp/backup-git-annex That way, you're left with a single tar file to backup.
rcthompson 12y ago
So git-annex now has support for efficiently deduplicating backends? Might be time for me to give it another look.
- rlpb 12y ago
  For "local" storage as supported by bup and ddar, yes. Generally for external remotes: not yet, AFAIK. http://git-annex.branchable.com/design/assistant/deltas/ http://git-annex.branchable.com/design/assistant/deltas/ has some design thoughts, so I think it is planned.
sytse 12y ago
GitLab CEO here, as you mention backing up the git repository is very important. We just announced free hosting for git-annex files and repositories on GitLab.com that might be good for your use case, see https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-... for an overview.
n3t 12y ago
git-annex website says[1] it is not supposed to be a backup solution: > git-annex is not a backup system. It may be a useful component of an archival system, or a way to deliver files to a backup system With this in mind, can we treat it as a backup tool? Assuming backing up the git repository itself as you mentioned. [1] http://git-annex.branchable.com/not/ http://git-annex.branchable.com/not/

borgel 12y ago

I have a layered photo backup process: - New photos live on my laptop (in my Dropbox and covered by Backblaze) - I manually copy photos from my laptop to a RAID array on a server elsewhere in my house when I'm done editing/processing them - The server backs up the photos to S3 S3 is the last bastion, so to speak. I expect that normally, I pull any photos I need back off my server. But the house burns down, I go to S3. I run a cron job on my backup server that picks up all changed files (in my photo storage dir) and pushes them to S3 with s3cmd[1]. It's a pretty 'dumb' process, but works well. The S3 <-> Glacier lifecycle stuff is pretty cool, so the plan is to eventually enable it to deep-archive older photos. [1] http://s3tools.org/s3cmd http://s3tools.org/s3cmd