5 ms·

I use Arq[1] on my macs and it does the same thing but handles all the "complicated" bits that I'd rather be programmatically dealt with. I have 6 macs, about

by aroch 13y ago

I use Arq[1] on my macs and it does the same thing but handles all the "complicated" bits that I'd rather be programmatically dealt with. I have 6 macs, about 2TB of backups in Glacier and a ~20GB "system image" backup in hot-storage on S3. [1] http://www.haystacksoftware.com/arq/ http://www.haystacksoftware.com/arq/

makmanalp 13y ago
How do you like glacier? It seems it's fairly cheap until you want your data back. Any insights?
- aidos 13y ago
  Personally I've factored that in to my cost of recovery. It's going to cost me about £120 to restore my data over the course of a week. It depends on your use case but for me - in the event that glacier is the last point I can get my data from, £120 is a small price to pay to get it back. I used backblaze before and it was about the same to get them to post out a hdd (which was an awesome service on the one occasion I had to use it).
  - toomuchtodo 13y ago
    It's too bad Amazon doesn't offer a discounted rate to restore from Glacier if you pay upfront.
    - zz1 13y ago
      Sorry to chase you between threads, but yes, I'd like to get in touch for the Getty files! Care to share contact info?
      - toomuchtodo 13y ago
        m8r-fhf8r1@mailinator.com (apologies, would prefer not to share my public email address here) I'll reply back from my personal email account.
  - _red 13y ago
    In addition, Glacier should be thought of as "offline" storage , like a tape rotation scheme. When thought of that way, the cost for restoration is non-existent. Its not for "I may need this file next week" (thats what your local NAS / Time Machine is for...Glacier is for "what happens if this place burns down". At that point a week and £120 is not high on the list of your concerns...
- aroch 13y ago
  I like it quite a bit, Arq is my "backup of last resort" and what I use for provision new macs if they're not with me. I have daily/weekly/monthly local backups and semi-local, bi-weekly backups (stored offsite or in a building with a different mac). Setting up a new Mac remotely costs about $30 for the 60GB provisioning profile to be downloaded and restored (maybe more or less depending on the internet connection at the place). I pay a few hundred yearly for backups that I have no doubt will be there if I need them.
aidos 13y ago
I also use Arq and recommend it to everyone on OS X. It has a sane transparent format (modelled on git), it can store to Glacier and it doesn't periodically explode like Back Blaze used to. Interested to hear more about your "system image". How do you do it and what do you store in it? Documents / preferences / Library etc? How would you use it in a recovery situation?
- aroch 13y ago
  It's a separate backup vault in Arq that's maintained by clean OS install + my essential documents, dotfiles, and an encrypted PII volume with scans of passport/SSC/Drivers license/Birth certificate. It's all on a 60GB SSD that's booted every week to update to the latest versions of everything and then to be Arq backed up. Restoration is downloading the vault to a new SSD and shoving it in a computer. The "root" user is included in the backup as an encrypted volume. My setup's a little strange in that the OS disks on my computers are filevaulted and each user's homedir is also in an encrypted volume only they have the password too that's mounted to the correct /Users/folder with fstab. (It'd probably be helpful to note that I have a hackintosh this is done on with 10 internal bays, in which one resides this SSD with a clone of this SSD elsewhere)
  - sreitshamer 13y ago
    Could you write that up in a blog post please? I'd love to read how that works. (I'm the author of Arq)
    - aroch 13y ago
      I could probably whack something together, but what specifically were you wondering about?
      - sreitshamer 13y ago
        I'm wondering about the whole process. It sounds like an external drive that you periodically copy your documents, dot files etc to, and then let Arq back up? Or is it something else? (You talk about "booting" the disk -- do you mean booting the computer from that disk, or just mounting it?) Also I didn't understand the "root" user part. And is this a multi-user computer? You mentioned something about "each user".
apaprocki 13y ago
Another happy Arq user here -- I just use it for Glacier backup, no need for hot S3 storage.
OWaz 13y ago
Has anyone been able to accurately figure out how much Glacier will costs once storage is retrieved from it?
- aroch 13y ago
  Here's my last bill: $0.010 per GB / month - Storage 2112.146 GB-Mo 21.12 $0.050 per 1,000 Requests 40,447 Requests 2.02 My last "restoration" was actually setting up a new iMac at a remote site involved downloading ~60GB over ~20hour and cost me about $30
- mjn 13y ago
  Above the free retrieval quota (5% of your storage per month), pricing is based entirely on your peak hourly retrieval rate for the month, billed at $7.20 x [peak rate measured in GB/hr]. So for example, if you retrieve 100 GB in one hour, that's $720. If you retrieve 10 TB at the rate of 100 GB/hr for 100 hours, that's also $720. Translated into bandwidth (assuming you're pulling at this rate for at least an hour), it's about $3 per Mbps. So if you limit your retrieval to 10 Mbps, you'll pay $30, regardless of how much data you retrieve. This makes it fairly easy to cap your expenditure, if you have a client that can stagger its requests appropriately (you need to throttle the Glacier retrieval requests, possibly using Glacier's range requests if you have very large files, not throttle at the network level).
ivanhoe 13y ago
Arq is simple and works great, although it seems that a half of my S3 bills are for the thousands of requests that Arq makes back and forth. Simpler upload & forget backup model probably could save a few bucks...
- aroch 13y ago
  I make, on average, 40K requests a month which is like $2 while my storage costs are ~$22. At the low end you'll be paying what seems like a lot for requests but as things scale your requests costs become a much smaller percentage. You an cut down on requests by moving to daily backups (what I do) and narrowing down what you're backing up. Backing up your applications folders will take a ton of requests (Some apps have upwards of 1000 assets whose bless values may change if you open the application and this trigger Arq)
aesptux 13y ago
Do you recommend Glacier over Crashplan? I'm thinking about switching
- ye 13y ago
  They are very different. Glacier takes hours just to get the listing of files, or start the file download. Their pricing is very different. Crashplan charges per computer. Glacier charges per GB of storage and per GB of transfer out.
  - aesptux 13y ago
    In what cases would be better to use Glacier? I don't have too many GB, ~30GB.
    - TkTech 13y ago
      Glacier is for when you want to store essentially never-changing backups that you expect to very, very rarely access. You can store a couple GB or petabytes. Think an archive of family videos and photos spanning 30 years, or all the tax documents for a large enterprise.
    - aroch 13y ago
      A 30GB backup, that's updated nightly, on AWS Glacier will cost somewhere in the neighborhood of $1 a month. A 300GB backup somewhere around $5 on AWS.
    - ye 13y ago
      Glacier is cheaper for small backups that you almost never touch.
dewiz 13y ago
Is there something like Arq compatible with both OSX and Windows ? Features I'm interested in: - must use S3 or similar, as long as it's private and "unlimited" - must support multiple clients, even though I don't need real time sync - should encrypt on the client without need of a separate sw I've been using Dropbox + CloudFogger so far, but dropbox doesn't scale anymore with the quantity of data I have, mostly due to the number of files (and I don't like depending on two different sw for one task).
- aroch 13y ago
  The only two I have used are git-annex [1] and duplicati [2]. I much prefer git-annex but the windows version isn't so stable. [1] http://git-annex.branchable.com/ http://git-annex.branchable.com/ [2] http://www.duplicati.com/ http://www.duplicati.com/

by aroch 13y ago

makmanalp 13y ago

How do you like glacier? It seems it's fairly cheap until you want your data back. Any insights?

aidos 13y ago
Personally I've factored that in to my cost of recovery. It's going to cost me about £120 to restore my data over the course of a week. It depends on your use case but for me - in the event that glacier is the last point I can get my data from, £120 is a small price to pay to get it back. I used backblaze before and it was about the same to get them to post out a hdd (which was an awesome service on the one occasion I had to use it).
- toomuchtodo 13y ago
  It's too bad Amazon doesn't offer a discounted rate to restore from Glacier if you pay upfront.
  - zz1 13y ago
    Sorry to chase you between threads, but yes, I'd like to get in touch for the Getty files! Care to share contact info?
    - toomuchtodo 13y ago
      m8r-fhf8r1@mailinator.com (apologies, would prefer not to share my public email address here) I'll reply back from my personal email account.
- _red 13y ago
  In addition, Glacier should be thought of as "offline" storage , like a tape rotation scheme. When thought of that way, the cost for restoration is non-existent. Its not for "I may need this file next week" (thats what your local NAS / Time Machine is for...Glacier is for "what happens if this place burns down". At that point a week and £120 is not high on the list of your concerns...
aroch 13y ago
I like it quite a bit, Arq is my "backup of last resort" and what I use for provision new macs if they're not with me. I have daily/weekly/monthly local backups and semi-local, bi-weekly backups (stored offsite or in a building with a different mac). Setting up a new Mac remotely costs about $30 for the 60GB provisioning profile to be downloaded and restored (maybe more or less depending on the internet connection at the place). I pay a few hundred yearly for backups that I have no doubt will be there if I need them.

aidos 13y ago

I also use Arq and recommend it to everyone on OS X. It has a sane transparent format (modelled on git), it can store to Glacier and it doesn't periodically explode like Back Blaze used to. Interested to hear more about your "system image". How do you do it and what do you store in it? Documents / preferences / Library etc? How would you use it in a recovery situation?

aroch 13y ago
It's a separate backup vault in Arq that's maintained by clean OS install + my essential documents, dotfiles, and an encrypted PII volume with scans of passport/SSC/Drivers license/Birth certificate. It's all on a 60GB SSD that's booted every week to update to the latest versions of everything and then to be Arq backed up. Restoration is downloading the vault to a new SSD and shoving it in a computer. The "root" user is included in the backup as an encrypted volume. My setup's a little strange in that the OS disks on my computers are filevaulted and each user's homedir is also in an encrypted volume only they have the password too that's mounted to the correct /Users/folder with fstab. (It'd probably be helpful to note that I have a hackintosh this is done on with 10 internal bays, in which one resides this SSD with a clone of this SSD elsewhere)
- sreitshamer 13y ago
  Could you write that up in a blog post please? I'd love to read how that works. (I'm the author of Arq)
  - aroch 13y ago
    I could probably whack something together, but what specifically were you wondering about?
    - sreitshamer 13y ago
      I'm wondering about the whole process. It sounds like an external drive that you periodically copy your documents, dot files etc to, and then let Arq back up? Or is it something else? (You talk about "booting" the disk -- do you mean booting the computer from that disk, or just mounting it?) Also I didn't understand the "root" user part. And is this a multi-user computer? You mentioned something about "each user".

apaprocki 13y ago

Another happy Arq user here -- I just use it for Glacier backup, no need for hot S3 storage.

OWaz 13y ago

Has anyone been able to accurately figure out how much Glacier will costs once storage is retrieved from it?

aroch 13y ago
Here's my last bill: $0.010 per GB / month - Storage 2112.146 GB-Mo 21.12 $0.050 per 1,000 Requests 40,447 Requests 2.02 My last "restoration" was actually setting up a new iMac at a remote site involved downloading ~60GB over ~20hour and cost me about $30
mjn 13y ago
Above the free retrieval quota (5% of your storage per month), pricing is based entirely on your peak hourly retrieval rate for the month, billed at $7.20 x [peak rate measured in GB/hr]. So for example, if you retrieve 100 GB in one hour, that's $720. If you retrieve 10 TB at the rate of 100 GB/hr for 100 hours, that's also $720. Translated into bandwidth (assuming you're pulling at this rate for at least an hour), it's about $3 per Mbps. So if you limit your retrieval to 10 Mbps, you'll pay $30, regardless of how much data you retrieve. This makes it fairly easy to cap your expenditure, if you have a client that can stagger its requests appropriately (you need to throttle the Glacier retrieval requests, possibly using Glacier's range requests if you have very large files, not throttle at the network level).

ivanhoe 13y ago

Arq is simple and works great, although it seems that a half of my S3 bills are for the thousands of requests that Arq makes back and forth. Simpler upload & forget backup model probably could save a few bucks...

aroch 13y ago
I make, on average, 40K requests a month which is like $2 while my storage costs are ~$22. At the low end you'll be paying what seems like a lot for requests but as things scale your requests costs become a much smaller percentage. You an cut down on requests by moving to daily backups (what I do) and narrowing down what you're backing up. Backing up your applications folders will take a ton of requests (Some apps have upwards of 1000 assets whose bless values may change if you open the application and this trigger Arq)

aesptux 13y ago

Do you recommend Glacier over Crashplan? I'm thinking about switching

ye 13y ago
They are very different. Glacier takes hours just to get the listing of files, or start the file download. Their pricing is very different. Crashplan charges per computer. Glacier charges per GB of storage and per GB of transfer out.
- aesptux 13y ago
  In what cases would be better to use Glacier? I don't have too many GB, ~30GB.
  - TkTech 13y ago
    Glacier is for when you want to store essentially never-changing backups that you expect to very, very rarely access. You can store a couple GB or petabytes. Think an archive of family videos and photos spanning 30 years, or all the tax documents for a large enterprise.
  - aroch 13y ago
    A 30GB backup, that's updated nightly, on AWS Glacier will cost somewhere in the neighborhood of $1 a month. A 300GB backup somewhere around $5 on AWS.
  - ye 13y ago
    Glacier is cheaper for small backups that you almost never touch.

dewiz 13y ago

Is there something like Arq compatible with both OSX and Windows ? Features I'm interested in: - must use S3 or similar, as long as it's private and "unlimited" - must support multiple clients, even though I don't need real time sync - should encrypt on the client without need of a separate sw I've been using Dropbox + CloudFogger so far, but dropbox doesn't scale anymore with the quantity of data I have, mostly due to the number of files (and I don't like depending on two different sw for one task).

aroch 13y ago
The only two I have used are git-annex [1] and duplicati [2]. I much prefer git-annex but the windows version isn't so stable. [1] http://git-annex.branchable.com/ http://git-annex.branchable.com/ [2] http://www.duplicati.com/ http://www.duplicati.com/