5 ms·

I decided to invest in an LTE drive to periodically archive my ZFS based NAS. This relatively old drive support 50-150 MB/s with a small internal buffer. My big

by sly010 6y ago

I decided to invest in an LTE drive to periodically archive my ZFS based NAS. This relatively old drive support 50-150 MB/s with a small internal buffer. My biggest problem ended up being that the default tar util on ZFS on spinning rust cannot even saturate the 50MB/s consistently, so the drive had so stop and rewind (called shoe-shining). It would take ages to backup parts of my filesystem with lot of small files (i.e. git repos). I ended up writing a custom software with an adaptive scheduler to avoid sudden speed changes. That is to say as far as I am concerned my ancient LTO6 drive is too fast...

throw0101a 6y ago
It's been best practice for a while in the 'enterprise' realm to do disk-to-disk backup first so that the streams are all sequential and then copy to tape. Enterprise-y software can also generally multiplex multiple clients to a single tape.
- kstrauser 6y ago
  I used Amanda Backup[0] for years on a Unix farm, and that's exactly how it works. Client apps on all of the machines being backed up ran tar and streamed the output to a backup server where it was written to disk. Then the completed files were streamed out to the tape. [0] http://www.amanda.org http://www.amanda.org
  - TedDoesntTalk 6y ago
    I don’t understand how this scales. If you have 1 PT to backup to tape, your first have to write it to disks on a backup server? So you have to buy 1PT worth of backup disks first, is that correct?
    - kstrauser 6y ago
      You wouldn't usually backup the whole thing first. That 1 PB is probably a bunch of smaller projects, so you'd create a tar of the first project, then flush it to tape. Tar up the second project, then flush it to tape. It's not so bad if you're writing 23 tarballs out to tape and pausing between each one. What you really want to avoid is where the tape drive is pausing many times during a single archive, perhaps because tar can't keep the drive fed while it's traversing a big directory of teensy files.
    - dragontamer 6y ago
      You'd never buy a 1PB tape drive if you only had 1PB of data to backup. A 1 PB (1000TB) tape drive would probably cost $10,000+ to $100,000. The tapes themselves would be cheap (cheaper than hard drives or SSDs). So you'd buy many tapes, perhaps 50 of them, to be cost-effective. That's why a lot of comments around here are talking about tape libraries (entire boxes of tapes). You must plan to use more than 50 tapes (aka: 50PBs) before you reach any level of cost-effectiveness. --------- From there, we see that 1PB of data is simply "infrastructure", to help you lay out the data before it gets to the tape. There are 20TB hard drives today: a single machine with 50 x 20TB hard drives (and maybe some flash storage to accelerate the write to hard drives) is what we're looking at to feed the Tape Drive.
      - tharkun__ 6y ago
        When I was a kid and we had to do a school internship, I went to my dad's company. Since we were just school kids, they didn't really give us anything to do. Except: Loading and unloading tapes and moving tape lists from the main tape library room to the backup safe. The backup safe was several hundred meters away in a different building (on the same lot). This was basically their WORM archival system. The tape library was a room full of tapes, wall to wall with shelves in the middle. Those were the oldest tapes. The in-use tapes were inside "Robbie", the circular tape robot. Not sure it it's this model exactly, but it sure looked like this: https://bjgreenberg.files.wordpress.com/2012/05/tape-library.jpg?w=800 https://bjgreenberg.files.wordpress.com/2012/05/tape-library... The tape robot would periodically spit out tapes on one end that we'd collect and then file away in the correct location on the shelves and vice versa, got a list of tapes to get from the shelves and insert into "Robbie". I don't remember typical tape sizes at the time, except that obviously it was much more capacity in a little tape than I had in my hard drives at home at the time :) Now imagine, Robbie filled to the brim with these 1PB tapes... Robbie and his extended tape library was in a room right adjacent to the actual data center floor, which had an IBM 390 mainframe and lots and lots of EMC^2s, AIXs, Suns and everything else you can think of being in a data center ca. 2000. You obviously had to get into and out of the data center through a single person entrance gate (a vertical tube that literally only one person could fit in, opening one side at a time only). First day I'd just walk through the data center and look at everything, reading labels and such. Found the server that all my personal emails were going through, as my ISP was hosting there! Heaven for a kid like me.
    - NovemberWhiskey 6y ago
      Sure, why not? It's an incremental cost of the backup system. Look at the current generation stuff. An LTO-8 drive puts 30TB onto a tape, but that's assuming 2.5:1 compression, so the native capacity is only 12TB. Let's assume you're actually going to do the compression as you build the virtual tape. What's a 12TB hard disk cost these days? A lot less than the $4,000 that the tape drive costs. Heck, 12TB of SSD is going to be cheaper.
    - throw0101a 6y ago
      60x18TB drives and a server to put them in would cost about US$ 45K: * https://www.45drives.com/products/storinator-xl60-configurations.php https://www.45drives.com/products/storinator-xl60-configurat... That gets you about 1PB raw.
    - deleted 6y ago
      [deleted]
  - deleted 6y ago
    [deleted]
kmeisthax 6y ago
I would highly recommend jacking up the --blocking-factor on tar (or your custom software) as high as your drive will accept. AFAIK the default on tar is 512b which is utterly inadequate, and even sequential data on a fast SSD will shoe-shine when written directly to tape. AFAIK most large businesses wind up writing to disk and then queuing the resulting serialized virtual tapes for actual writing at a later time. This basically turns the virtual tape library into a massive write buffer. It's absolutely insane that we have to do this - but a lot of archival utilities aren't built for async/parallel IO and thus aren't taking advantage of modern storage devices.
- sly010 6y ago
  The tool I ended up writing does essentially something very similar: - It uses a block size that matches the disk - It uses 50% of my RAM as a ring buffer - On start (and on buffer underrun) it waits for the buffer to fill before it starts writing to tape. - It adjust the tape write speed as an exponential function of the buffer fill level. I am proud of this last one. It is very simple and automatically converges to the best average speed to keep both the tape speed and the buffer level relatively stable. I have been considering making the reading multi threaded to make use of more disks, but that would be a bit of an overkill for my personal 2 disk NAS. edit:formatting
  - js2 6y ago
    - How many TB are you backing up? - How frequently do you perform backups? - How often do you do fulls vs incremental? - What drive/autoloader are you using, and what was the total investment in drive + tapes? I keep considering a used LTO but I've only got 20 TB and it doesn't seem worth the trouble. I should just buy another 20 TB of drives and keep them offline when not backing up.
    - foobiekr 6y ago
      I back up around 20TB and the easiest solution I found was to just buy 4x12TB drives and mirror to cloud (yielding 2x offline copies). For stuff that's really really important, around half of that, and which if restore is needed fast access is required, I also back it up to SSD. It's not the greatest solution, but when you're hovering at some small multiple of common single drives, it's certainly the easy answer.
snuxoll 6y ago
It’s not exactly best practice, but instead of tarballing the files themselves maybe you would have better performance just writing a snapshot via zfs send direct to tape?
- sly010 6y ago
  I admit to have thought of that, but using a zfs stream for archival would make eventual restore that much more complicated. A single bit error at the beginning of the stream would make the rest of the archive un-accessible, whereas with tar I can literally slice the tape and still recover data from it.
  - vorpalhex 6y ago
    You can now tell ZFS to still process bad snapshots (with the risk of still losing some corrupted data).
h2odragon 6y ago
long ago there was a utility called `buffer` that solved that problem. i know i've seen a "buffering netcat" go by here recently, that should serve too.

by sly010 6y ago

throw0101a 6y ago

It's been best practice for a while in the 'enterprise' realm to do disk-to-disk backup first so that the streams are all sequential and then copy to tape. Enterprise-y software can also generally multiplex multiple clients to a single tape.

kstrauser 6y ago
I used Amanda Backup[0] for years on a Unix farm, and that's exactly how it works. Client apps on all of the machines being backed up ran tar and streamed the output to a backup server where it was written to disk. Then the completed files were streamed out to the tape. [0] http://www.amanda.org http://www.amanda.org
- TedDoesntTalk 6y ago
  I don’t understand how this scales. If you have 1 PT to backup to tape, your first have to write it to disks on a backup server? So you have to buy 1PT worth of backup disks first, is that correct?
  - kstrauser 6y ago
    You wouldn't usually backup the whole thing first. That 1 PB is probably a bunch of smaller projects, so you'd create a tar of the first project, then flush it to tape. Tar up the second project, then flush it to tape. It's not so bad if you're writing 23 tarballs out to tape and pausing between each one. What you really want to avoid is where the tape drive is pausing many times during a single archive, perhaps because tar can't keep the drive fed while it's traversing a big directory of teensy files.
  - dragontamer 6y ago
    You'd never buy a 1PB tape drive if you only had 1PB of data to backup. A 1 PB (1000TB) tape drive would probably cost $10,000+ to $100,000. The tapes themselves would be cheap (cheaper than hard drives or SSDs). So you'd buy many tapes, perhaps 50 of them, to be cost-effective. That's why a lot of comments around here are talking about tape libraries (entire boxes of tapes). You must plan to use more than 50 tapes (aka: 50PBs) before you reach any level of cost-effectiveness. --------- From there, we see that 1PB of data is simply "infrastructure", to help you lay out the data before it gets to the tape. There are 20TB hard drives today: a single machine with 50 x 20TB hard drives (and maybe some flash storage to accelerate the write to hard drives) is what we're looking at to feed the Tape Drive.
    - tharkun__ 6y ago
      When I was a kid and we had to do a school internship, I went to my dad's company. Since we were just school kids, they didn't really give us anything to do. Except: Loading and unloading tapes and moving tape lists from the main tape library room to the backup safe. The backup safe was several hundred meters away in a different building (on the same lot). This was basically their WORM archival system. The tape library was a room full of tapes, wall to wall with shelves in the middle. Those were the oldest tapes. The in-use tapes were inside "Robbie", the circular tape robot. Not sure it it's this model exactly, but it sure looked like this: https://bjgreenberg.files.wordpress.com/2012/05/tape-library.jpg?w=800 https://bjgreenberg.files.wordpress.com/2012/05/tape-library... The tape robot would periodically spit out tapes on one end that we'd collect and then file away in the correct location on the shelves and vice versa, got a list of tapes to get from the shelves and insert into "Robbie". I don't remember typical tape sizes at the time, except that obviously it was much more capacity in a little tape than I had in my hard drives at home at the time :) Now imagine, Robbie filled to the brim with these 1PB tapes... Robbie and his extended tape library was in a room right adjacent to the actual data center floor, which had an IBM 390 mainframe and lots and lots of EMC^2s, AIXs, Suns and everything else you can think of being in a data center ca. 2000. You obviously had to get into and out of the data center through a single person entrance gate (a vertical tube that literally only one person could fit in, opening one side at a time only). First day I'd just walk through the data center and look at everything, reading labels and such. Found the server that all my personal emails were going through, as my ISP was hosting there! Heaven for a kid like me.
  - NovemberWhiskey 6y ago
    Sure, why not? It's an incremental cost of the backup system. Look at the current generation stuff. An LTO-8 drive puts 30TB onto a tape, but that's assuming 2.5:1 compression, so the native capacity is only 12TB. Let's assume you're actually going to do the compression as you build the virtual tape. What's a 12TB hard disk cost these days? A lot less than the $4,000 that the tape drive costs. Heck, 12TB of SSD is going to be cheaper.
  - throw0101a 6y ago
    60x18TB drives and a server to put them in would cost about US$ 45K: * https://www.45drives.com/products/storinator-xl60-configurations.php https://www.45drives.com/products/storinator-xl60-configurat... That gets you about 1PB raw.
  - deleted 6y ago
    [deleted]
- deleted 6y ago
  [deleted]

kmeisthax 6y ago

I would highly recommend jacking up the --blocking-factor on tar (or your custom software) as high as your drive will accept. AFAIK the default on tar is 512b which is utterly inadequate, and even sequential data on a fast SSD will shoe-shine when written directly to tape. AFAIK most large businesses wind up writing to disk and then queuing the resulting serialized virtual tapes for actual writing at a later time. This basically turns the virtual tape library into a massive write buffer. It's absolutely insane that we have to do this - but a lot of archival utilities aren't built for async/parallel IO and thus aren't taking advantage of modern storage devices.

sly010 6y ago
The tool I ended up writing does essentially something very similar: - It uses a block size that matches the disk - It uses 50% of my RAM as a ring buffer - On start (and on buffer underrun) it waits for the buffer to fill before it starts writing to tape. - It adjust the tape write speed as an exponential function of the buffer fill level. I am proud of this last one. It is very simple and automatically converges to the best average speed to keep both the tape speed and the buffer level relatively stable. I have been considering making the reading multi threaded to make use of more disks, but that would be a bit of an overkill for my personal 2 disk NAS. edit:formatting
- js2 6y ago
  - How many TB are you backing up? - How frequently do you perform backups? - How often do you do fulls vs incremental? - What drive/autoloader are you using, and what was the total investment in drive + tapes? I keep considering a used LTO but I've only got 20 TB and it doesn't seem worth the trouble. I should just buy another 20 TB of drives and keep them offline when not backing up.
  - foobiekr 6y ago
    I back up around 20TB and the easiest solution I found was to just buy 4x12TB drives and mirror to cloud (yielding 2x offline copies). For stuff that's really really important, around half of that, and which if restore is needed fast access is required, I also back it up to SSD. It's not the greatest solution, but when you're hovering at some small multiple of common single drives, it's certainly the easy answer.

snuxoll 6y ago

It’s not exactly best practice, but instead of tarballing the files themselves maybe you would have better performance just writing a snapshot via zfs send direct to tape?

sly010 6y ago
I admit to have thought of that, but using a zfs stream for archival would make eventual restore that much more complicated. A single bit error at the beginning of the stream would make the rest of the archive un-accessible, whereas with tar I can literally slice the tape and still recover data from it.
- vorpalhex 6y ago
  You can now tell ZFS to still process bad snapshots (with the risk of still losing some corrupted data).

h2odragon 6y ago

long ago there was a utility called `buffer` that solved that problem. i know i've seen a "buffering netcat" go by here recently, that should serve too.