7 ms·

Managing tape drives and libraries with the Unix/Linux CLI

68 points by wazoox 4y ago

wazoox 4y ago
Hi guys, reposting because it may interest you. In particular using LTFS on the command line get little love on the internets...
wazoox 4y ago
Also this post exists in French here https://blogs.intellique.com/tech/2021/08/20#BandesCLI https://blogs.intellique.com/tech/2021/08/20#BandesCLI
bandrami 4y ago
It's always weird to remember that what we use tar for every day (packing and unpacking archives on disk) is by design a strange corner case and not its actual purpose.
Karunamon 4y ago
I'm surprised still that so many people sleep on tape. It is a very mature and stable technology with the best dollars/gigabyte ratio of any storage technology out there. Start up costs are a bit unpleasant but purchasing secondhand is reasonable and economical, and unlike the cloud the costs are mostly one time instead of forever. The tapes are durable and last for decades, so that initial outlay will last you a very long time.
- zozbot234 4y ago
  The tapes might last for decades, but support for them won't. Newer LTO standard drives can only read tapes a couple generations back at most. So you're pretty much forced to copy your stuff over if you want to stay current.
  - Karunamon 4y ago
    That's just it, there is no need whatsoever to "stay current" unless you are unhappy with the capacity of the tapes. For multiple generations now, that has been the only difference. The last major feature was slightly better compression followed by LTFS, and that came out on LTO 5 and 6… In 2010 and 2012.
    - Hackbraten 4y ago
      If your drive breaks down while you’re trying to do a restore, you’re going to want to get a spare one quickly. That may become a problem if you choose not to stay current.
      - Karunamon 4y ago
        In which case I'm going to eBay where I probably got the original drive in the first place over 10 years ago, and a quick look finds thousands of listings for both standalone and internal drives, at good prices, from reputable sellers with good history, going all the way back to LTO2 (which is almost 2 decades old!). This is what I mean, LTO sits in this niche where it is/was popular in the enterprise and so the secondhand market is big. I would be willing to wager that I will be able to find LTO 6 drives in another 10 years, with no problem.
    - wazoox 4y ago
      There's the very interesting new feature of LTO7M and 8M, which adds 50% capacity to your LTO7/8 tapes when formatting them with a - respectively - LTO8/9 drive. A 30$, 9TB LTO7 tape is the cheapest storage you can buy.
      - zozbot234 4y ago
        It's a one-way conversion, and then you're stuck with that one LTO generation; a LTO7M tape won't work anymore in a LTO7 drive.
        justsomehnguy 4y ago
- matheusmoreira 4y ago
  It's not just the expense. Backwards compatibility is only guaranteed up to the last two generations of tape media. If you hold onto these things for decades, you might find the drives required to read them no longer exist.
  - justsomehnguy 4y ago
    > If you hold onto these things for decades It's always amusing when people bash tapes for the lack of backwards compatibility, but... Hey, try to find a working system with PATA drives support? Try to connect SATA150 drive to SATA-III port? How about SCSI-320? People like to talk, but I always said what if your data has a value then it's your responsibility to have means to backup and to restore it. If your data "is extremely valuable" but all your C-level are approving is a single drive decade old library - you can write your backups straight to /dev/null, because your data is worthless for your C-level.
    - matheusmoreira 4y ago
      I'm not "bashing" tapes, I'm pointing out the fact they have limited backwards compatibility guarantees. Given the massive expense of tape drives and the fact you need to keep replacing them and the tapes just to keep up with new LTO standards puts this almost perfect solution out of reach of mere mortals like me. I'm not an enterprise, there is no "C-level" approving anything.
      - justsomehnguy 4y ago
        > I'm not "bashing" tapes Oh, this wasn't pointed at you at all, sorry for the misunderstanding. > Given the massive expense of tape drives and the fact you need to keep replacing them and the tapes just to keep up with new LTO standards Well, you don't need to keep replacing them. The rolling upgrade (gradually replacing the tapes and drives for a new generation) makes sense only if you have tens to hundreds of drives and thousands and thousands of tapes - ie for those insane enterprise libraries. For everything else, be it a small enterprise or a "mere mortals" it would be always cheaper to just buy a whole new library/autoloader. And if the cost of initial buyout is too much then of course tapes aren't for you... along with the compatibility woes.
adepressedthrow 4y ago
I was able to purchase a LTO-7 library full of tapes a few years ago for basically nothing, and still haven't gotten around to using it. There seems to be very little in terms of free, automated software for managing the system and performing tasks such as backups. tar doesn't exactly cut it for hands-off usage. The best I've found is using Veeam Backup & Replication, but it's Windows only and not great still.
- trynewideas 4y ago
  > free, automated software for managing the system and performing tasks such as backups Did you come across Amanda[1] or bacula + bacularis[2]? 1: http://www.amanda.org/ http://www.amanda.org/ 2: https://bacularis.app/doc/index.html https://bacularis.app/doc/index.html
  - adepressedthrow 4y ago
    I hadn't seen Amanda before. Bacularis looks like it might help with the complexity of setting up Backula. I might try that out. It's hard to get the motivation to mess with these tools that have relatively little documentation/examples and are purely config file based.
    - trynewideas 4y ago
      That's fair. I used to use Ark, which was mostly a proprietary front-end for bacula, and that exposed me to the kind of futility of a "user-friendly" tape interface. Every tape operation from picking to winding to r/w takes so much time, and has so many subtle ways that it can fail, that it defies attempts at friendly user interfaces.
    - herewulf 4y ago
      Bacula is great software, but yes, fiddly to set up and SCSI tape drives / changers aren't really straightforward either. It will serve you well once all is set up though. I worked for a company using Bacula quite successfully for many years with a 24 tape system. It was much better than the proprietary software the system came with.
squarefoot 4y ago
I currently use removable hard drives for backups, but was considering the idea of adding a refurbished LTO tape drive to my XigmaNAS based NAS, however I couldn't find any information about feasibility, compatibility etc. Does anyone have some pointers, assuming it is doable?
funstuff007 4y ago
LTO does seem to be the last bastion of closed source vendor lock-in. That being said, even if prices dropped by 75%, I'm not sure our shop would look to do LTO instead of a series of hard drives. 20 TB hard drives are now $400.
- EvanAnderson 4y ago
  If you fall in the right place on the curve $75 / LTO-8 cartridge still makes a lot of sense. It doesn’t take many $400 hard drives to make up the price for a tape element at that per-cartridge price. (Assuming you can deal with sequential access, etc.)
- waynesonfire 4y ago
  Hdds are soo fragile. They constantly break. A proper 3-2-1 backup solution is incomplete without tape IMHO.
  - Hamuko 4y ago
    At least HDDs are fast, cheap, don't have massive up-front investments. You can take a couple of HDDs, RAID them together for reliability and you can still write and read data at humane speeds.
    - wazoox 4y ago
      A RAID array isn't a backup. Your data is still live on it, an "rm -rf" away from destruction. HDDs fare poorly when shelved for a long time, RAID are even worse in that regard. OTOH a tape on a shelf is perfectly safe for 5 to 10 years or more (I recently restored 1998/99 sensitive data from LTO2 tapes written in 2003).
      - Hamuko 4y ago
        >A RAID array isn't a backup. Who said it was? >Your data is still live on it, an "rm -rf" away from destruction. How is accidental deletion not a problem with tapes?
memco 4y ago
I didn’t know about sg_logs: that could be useful. One issue I have encountered and one that isn’t really addressed is that there are different paths (sg, st and nst) for a tape drive, but it isn’t always easy to know which of those device paths is associated with which drive in the changer. It is also possible that the /dev/sg* path number will be different from the st and or nst number. IBM has a lin_tape daemon that reassigns /dev/ns* to /dev/IBMtape* so that’s extra fun if you need to access this information in a repeatable fashion. Beyond the tools mentioned in the article udevadm -a can provide helpful information about a drive.
Eleison23 4y ago
A tape drive actually led to my adoption of Linux. Around 1999, I decided that a QIC tape was the best backup solution for my home network. At the time, I had a 486 running Windows, and two Apollo 425t running OpenBSD. The 486 may have also dual-booted into OpenBSD. Anyway, I had a bare-bones 386 sitting around and I had upgraded the 486 so much that the spare parts were available to reconstitute the 386. I found a nice floppy tape drive, consumer grade, and wouldn't you know that OpenBSD didn't have floppy tape drivers. So I installed Linux on the 386 and it was a dedicated backup server. At the same time I had a SyQuest EZ 135 that I was very fond of, and those 135MB cartridges could hold good stuff, like a complete complement of Windows updates. But its capacity was too low to be a backup solution, of course; it was storage on-the-go.
permalac 4y ago
I feel is missing a clear way to identify self rewind tapes,otherwise you may be reading the first file multiple times.
Caitlynbranson 4y ago
[dead]
- wazoox 4y ago
  They're simple but for some reason they don't seem to be packaged in most distros.
wazoox 4y ago
One thing I forgot to mention is the M format. Starting with LTO7 media used with LTO8 drives, if you label the tapes with labels ending with M7 instead of L7 the drive will format the tape when first writing it in the 8M format, adding 3TB of capacity to it. Of course these media can only be read using drives of the next generation. Notice that first formatting a M9 tape can take up to 45 minutes. That can be a surprise when you first try it! Take this into account if you're time-constrained. Confusingly, LTO7 tapes with "M7"-ending barcodes are called "8M", and LTO8 with "M8"-ending barcodes are called "9M". Don't ask...
dabiged 4y ago
Can anyone comment on the brands of tape you can purchase? Given it is an open format what sets the brands apart? Are there records of failure rates like backblaze has done with HDDs? Currently for LTO8 tapes you can buy the following brands: * Fujifilm * Quantum * IBM * HPE * Oracle * Dell EMC
- wazoox 4y ago
  Actually only Sony and Fuji make tape. All the brands use one or the other. However they use two different technologies (metal oxide and metal evaporate) so the general recommendation is to use both for long-term archive. I generally use HPE or Overland (Sony-based) and Fujifilm.

68 points by wazoox 4y ago

Hi guys, reposting because it may interest you. In particular using LTFS on the command line get little love on the internets...

wazoox 4y ago

Also this post exists in French here https://blogs.intellique.com/tech/2021/08/20#BandesCLI https://blogs.intellique.com/tech/2021/08/20#BandesCLI

bandrami 4y ago

It's always weird to remember that what we use tar for every day (packing and unpacking archives on disk) is by design a strange corner case and not its actual purpose.

Karunamon 4y ago

I'm surprised still that so many people sleep on tape. It is a very mature and stable technology with the best dollars/gigabyte ratio of any storage technology out there. Start up costs are a bit unpleasant but purchasing secondhand is reasonable and economical, and unlike the cloud the costs are mostly one time instead of forever. The tapes are durable and last for decades, so that initial outlay will last you a very long time.

zozbot234 4y ago
The tapes might last for decades, but support for them won't. Newer LTO standard drives can only read tapes a couple generations back at most. So you're pretty much forced to copy your stuff over if you want to stay current.
- Karunamon 4y ago
  That's just it, there is no need whatsoever to "stay current" unless you are unhappy with the capacity of the tapes. For multiple generations now, that has been the only difference. The last major feature was slightly better compression followed by LTFS, and that came out on LTO 5 and 6… In 2010 and 2012.
  - Hackbraten 4y ago
    If your drive breaks down while you’re trying to do a restore, you’re going to want to get a spare one quickly. That may become a problem if you choose not to stay current.
    - Karunamon 4y ago
      In which case I'm going to eBay where I probably got the original drive in the first place over 10 years ago, and a quick look finds thousands of listings for both standalone and internal drives, at good prices, from reputable sellers with good history, going all the way back to LTO2 (which is almost 2 decades old!). This is what I mean, LTO sits in this niche where it is/was popular in the enterprise and so the secondhand market is big. I would be willing to wager that I will be able to find LTO 6 drives in another 10 years, with no problem.
  - wazoox 4y ago
    There's the very interesting new feature of LTO7M and 8M, which adds 50% capacity to your LTO7/8 tapes when formatting them with a - respectively - LTO8/9 drive. A 30$, 9TB LTO7 tape is the cheapest storage you can buy.
    - zozbot234 4y ago
      It's a one-way conversion, and then you're stuck with that one LTO generation; a LTO7M tape won't work anymore in a LTO7 drive.
      - justsomehnguy 4y ago
matheusmoreira 4y ago
It's not just the expense. Backwards compatibility is only guaranteed up to the last two generations of tape media. If you hold onto these things for decades, you might find the drives required to read them no longer exist.
- justsomehnguy 4y ago
  > If you hold onto these things for decades It's always amusing when people bash tapes for the lack of backwards compatibility, but... Hey, try to find a working system with PATA drives support? Try to connect SATA150 drive to SATA-III port? How about SCSI-320? People like to talk, but I always said what if your data has a value then it's your responsibility to have means to backup and to restore it. If your data "is extremely valuable" but all your C-level are approving is a single drive decade old library - you can write your backups straight to /dev/null, because your data is worthless for your C-level.
  - matheusmoreira 4y ago
    I'm not "bashing" tapes, I'm pointing out the fact they have limited backwards compatibility guarantees. Given the massive expense of tape drives and the fact you need to keep replacing them and the tapes just to keep up with new LTO standards puts this almost perfect solution out of reach of mere mortals like me. I'm not an enterprise, there is no "C-level" approving anything.
    - justsomehnguy 4y ago
      > I'm not "bashing" tapes Oh, this wasn't pointed at you at all, sorry for the misunderstanding. > Given the massive expense of tape drives and the fact you need to keep replacing them and the tapes just to keep up with new LTO standards Well, you don't need to keep replacing them. The rolling upgrade (gradually replacing the tapes and drives for a new generation) makes sense only if you have tens to hundreds of drives and thousands and thousands of tapes - ie for those insane enterprise libraries. For everything else, be it a small enterprise or a "mere mortals" it would be always cheaper to just buy a whole new library/autoloader. And if the cost of initial buyout is too much then of course tapes aren't for you... along with the compatibility woes.

adepressedthrow 4y ago

I was able to purchase a LTO-7 library full of tapes a few years ago for basically nothing, and still haven't gotten around to using it. There seems to be very little in terms of free, automated software for managing the system and performing tasks such as backups. tar doesn't exactly cut it for hands-off usage. The best I've found is using Veeam Backup & Replication, but it's Windows only and not great still.

trynewideas 4y ago
> free, automated software for managing the system and performing tasks such as backups Did you come across Amanda[1] or bacula + bacularis[2]? 1: http://www.amanda.org/ http://www.amanda.org/ 2: https://bacularis.app/doc/index.html https://bacularis.app/doc/index.html
- adepressedthrow 4y ago
  I hadn't seen Amanda before. Bacularis looks like it might help with the complexity of setting up Backula. I might try that out. It's hard to get the motivation to mess with these tools that have relatively little documentation/examples and are purely config file based.
  - trynewideas 4y ago
    That's fair. I used to use Ark, which was mostly a proprietary front-end for bacula, and that exposed me to the kind of futility of a "user-friendly" tape interface. Every tape operation from picking to winding to r/w takes so much time, and has so many subtle ways that it can fail, that it defies attempts at friendly user interfaces.
  - herewulf 4y ago
    Bacula is great software, but yes, fiddly to set up and SCSI tape drives / changers aren't really straightforward either. It will serve you well once all is set up though. I worked for a company using Bacula quite successfully for many years with a 24 tape system. It was much better than the proprietary software the system came with.

squarefoot 4y ago

I currently use removable hard drives for backups, but was considering the idea of adding a refurbished LTO tape drive to my XigmaNAS based NAS, however I couldn't find any information about feasibility, compatibility etc. Does anyone have some pointers, assuming it is doable?

funstuff007 4y ago

LTO does seem to be the last bastion of closed source vendor lock-in. That being said, even if prices dropped by 75%, I'm not sure our shop would look to do LTO instead of a series of hard drives. 20 TB hard drives are now $400.

EvanAnderson 4y ago
If you fall in the right place on the curve $75 / LTO-8 cartridge still makes a lot of sense. It doesn’t take many $400 hard drives to make up the price for a tape element at that per-cartridge price. (Assuming you can deal with sequential access, etc.)
waynesonfire 4y ago
Hdds are soo fragile. They constantly break. A proper 3-2-1 backup solution is incomplete without tape IMHO.
- Hamuko 4y ago
  At least HDDs are fast, cheap, don't have massive up-front investments. You can take a couple of HDDs, RAID them together for reliability and you can still write and read data at humane speeds.
  - wazoox 4y ago
    A RAID array isn't a backup. Your data is still live on it, an "rm -rf" away from destruction. HDDs fare poorly when shelved for a long time, RAID are even worse in that regard. OTOH a tape on a shelf is perfectly safe for 5 to 10 years or more (I recently restored 1998/99 sensitive data from LTO2 tapes written in 2003).
    - Hamuko 4y ago
      >A RAID array isn't a backup. Who said it was? >Your data is still live on it, an "rm -rf" away from destruction. How is accidental deletion not a problem with tapes?

memco 4y ago

I didn’t know about sg_logs: that could be useful. One issue I have encountered and one that isn’t really addressed is that there are different paths (sg, st and nst) for a tape drive, but it isn’t always easy to know which of those device paths is associated with which drive in the changer. It is also possible that the /dev/sg* path number will be different from the st and or nst number. IBM has a lin_tape daemon that reassigns /dev/ns* to /dev/IBMtape* so that’s extra fun if you need to access this information in a repeatable fashion. Beyond the tools mentioned in the article udevadm -a can provide helpful information about a drive.

Eleison23 4y ago

A tape drive actually led to my adoption of Linux. Around 1999, I decided that a QIC tape was the best backup solution for my home network. At the time, I had a 486 running Windows, and two Apollo 425t running OpenBSD. The 486 may have also dual-booted into OpenBSD. Anyway, I had a bare-bones 386 sitting around and I had upgraded the 486 so much that the spare parts were available to reconstitute the 386. I found a nice floppy tape drive, consumer grade, and wouldn't you know that OpenBSD didn't have floppy tape drivers. So I installed Linux on the 386 and it was a dedicated backup server. At the same time I had a SyQuest EZ 135 that I was very fond of, and those 135MB cartridges could hold good stuff, like a complete complement of Windows updates. But its capacity was too low to be a backup solution, of course; it was storage on-the-go.

permalac 4y ago

I feel is missing a clear way to identify self rewind tapes,otherwise you may be reading the first file multiple times.

Caitlynbranson 4y ago

[dead]

wazoox 4y ago
They're simple but for some reason they don't seem to be packaged in most distros.

wazoox 4y ago

One thing I forgot to mention is the M format. Starting with LTO7 media used with LTO8 drives, if you label the tapes with labels ending with M7 instead of L7 the drive will format the tape when first writing it in the 8M format, adding 3TB of capacity to it. Of course these media can only be read using drives of the next generation. Notice that first formatting a M9 tape can take up to 45 minutes. That can be a surprise when you first try it! Take this into account if you're time-constrained. Confusingly, LTO7 tapes with "M7"-ending barcodes are called "8M", and LTO8 with "M8"-ending barcodes are called "9M". Don't ask...

dabiged 4y ago

Can anyone comment on the brands of tape you can purchase? Given it is an open format what sets the brands apart? Are there records of failure rates like backblaze has done with HDDs? Currently for LTO8 tapes you can buy the following brands: * Fujifilm * Quantum * IBM * HPE * Oracle * Dell EMC

wazoox 4y ago
Actually only Sony and Fuji make tape. All the brands use one or the other. However they use two different technologies (metal oxide and metal evaporate) so the general recommendation is to use both for long-term archive. I generally use HPE or Overland (Sony-based) and Fujifilm.