5 ms·

Typically they are tiered. There'll be a near-line HDD array. This is for the recent content and content they profile as being common-access. Then there'll be

by tezza 14y ago

Typically they are tiered. There'll be a near-line HDD array. This is for the recent content and content they profile as being common-access. Then there'll be a robotic tape library. Any restore request will go in a queue annd when an arm-tapedrive becomes free they'll seek to the data and read it into the HDD array. Waiting for a slot with the robot arm - tape drive is what will take 4 hours. EMC(kinda), Fujitsu etc make these. http://en.wikipedia.org/wiki/Tape_library http://en.wikipedia.org/wiki/Tape_library http://www.theregister.co.uk/2012/06/26/emc_tape_sucks_no_more/ http://www.theregister.co.uk/2012/06/26/emc_tape_sucks_no_mo...

nodata 14y ago
Wouldn't there also need to be a lot of logic to prevent fragmentation? You'd probably want data from one user near other data from that user, i.e. on the same tape.
- omh 14y ago
  The multiple-hour window could give you a lot of wiggle room here though. It's unlikely to take 3 hours to restore from a single tape, so even if you have to visit 2-3 tapes then you have plenty of time. I'm sure that there is a general tiered storage platform (as mentioned above) which keeps some of the data online as well. That would let you run a "defrag" algorithm later if you find you need it.
- dagw 14y ago
  I'd guess that they ignore that problem and have baked the time it takes to get data from several tapes into the 3-4 hour estimate. If you think about it, writes are more common than reads on average, so it's more efficient to just write to whatever tape is online and deal with the fragmentation problem on the read end, as opposed to queueing writes until the 'correct' tape can be brought online just save some time reading. Also in backup situations like this, it's more important to get the backup done in a timely manner.
  - jvdh 14y ago
    That's true, but they're obviously using a multi-tier solution, with about a 90 day buffer before things go to tape: In addition, there is a pro-rated charge of $0.033 per gigabyte for items deleted prior to 90 days. So it may just be feasible to organise writes so they end up together. But, yes, ultimately it is probably not worth it to do so.
- kondro 14y ago
  Given it takes 3.5-4 hours to retrieve a backup before you can download it, probably not.
phil 14y ago
Thanks -- I had no idea you could just buy a tape library off the rack (more or less).
look_lookatme 14y ago
Do you think there is redundancy baked into the tape drive system?
sintaks 14y ago
Close. Tiered, yes. But remember who we're talking about. First, no tape. The areal storage density of tape is lower than hard disks. Too many moving parts involved. Too hard to perform integrity checks on in a scalable, automated fashion without impacting incoming work. Second, in order to claim the durability that they do (99.999999999%), that means every spot along the pipe needs to meet those requirements. That means the "near-line HDD array" for warm, incoming data needs to meet those requirements. Additionally, if the customer has specified that the data be encrypted, it needs to be encrypted during this staging period as well. It also needs to be able to scale to tens if not hundreds of thousands of concurrent requests per second (though, for something like Glacier, this might be overkill). They've already built something that does all that. It's called S3. The upload operations likely proxy to S3 internally (with a bit of magic), and use that as staging space. After that, the bottleneck is likely I/O to Glacier's underlying storage - but again, not tapes. See this post for deets: http://news.ycombinator.com/item?id=4416065 http://news.ycombinator.com/item?id=4416065

by tezza 14y ago

nodata 14y ago

Wouldn't there also need to be a lot of logic to prevent fragmentation? You'd probably want data from one user near other data from that user, i.e. on the same tape.

omh 14y ago
The multiple-hour window could give you a lot of wiggle room here though. It's unlikely to take 3 hours to restore from a single tape, so even if you have to visit 2-3 tapes then you have plenty of time. I'm sure that there is a general tiered storage platform (as mentioned above) which keeps some of the data online as well. That would let you run a "defrag" algorithm later if you find you need it.
dagw 14y ago
I'd guess that they ignore that problem and have baked the time it takes to get data from several tapes into the 3-4 hour estimate. If you think about it, writes are more common than reads on average, so it's more efficient to just write to whatever tape is online and deal with the fragmentation problem on the read end, as opposed to queueing writes until the 'correct' tape can be brought online just save some time reading. Also in backup situations like this, it's more important to get the backup done in a timely manner.
- jvdh 14y ago
  That's true, but they're obviously using a multi-tier solution, with about a 90 day buffer before things go to tape: In addition, there is a pro-rated charge of $0.033 per gigabyte for items deleted prior to 90 days. So it may just be feasible to organise writes so they end up together. But, yes, ultimately it is probably not worth it to do so.
kondro 14y ago
Given it takes 3.5-4 hours to retrieve a backup before you can download it, probably not.

phil 14y ago

Thanks -- I had no idea you could just buy a tape library off the rack (more or less).

look_lookatme 14y ago

Do you think there is redundancy baked into the tape drive system?

sintaks 14y ago

Close. Tiered, yes. But remember who we're talking about. First, no tape. The areal storage density of tape is lower than hard disks. Too many moving parts involved. Too hard to perform integrity checks on in a scalable, automated fashion without impacting incoming work. Second, in order to claim the durability that they do (99.999999999%), that means every spot along the pipe needs to meet those requirements. That means the "near-line HDD array" for warm, incoming data needs to meet those requirements. Additionally, if the customer has specified that the data be encrypted, it needs to be encrypted during this staging period as well. It also needs to be able to scale to tens if not hundreds of thousands of concurrent requests per second (though, for something like Glacier, this might be overkill). They've already built something that does all that. It's called S3. The upload operations likely proxy to S3 internally (with a bit of magic), and use that as staging space. After that, the bottleneck is likely I/O to Glacier's underlying storage - but again, not tapes. See this post for deets: http://news.ycombinator.com/item?id=4416065 http://news.ycombinator.com/item?id=4416065