5 ms·

Kind of semi related: What do you buy if you want to do tape backups at home?

by axx 5y ago

scheme271 5y ago
Probably a LTO drive a few generations old and a SAS card. Current generations cost too much and you aren't going to be able to keep up with the data rate that they prefer.
- Dylan16807 5y ago
  > Current generations cost too much and you aren't going to be able to keep up with the data rate that they prefer. Maybe. Keeping up only really matters if your slowest data rate is going to be between 50MB/s and 120MB/s. If it's below that then you have a strong need for a buffer drive no matter what tape generation. And if it's above that then you shouldn't have any real trouble no matter what tape generation.
- aduitsis 5y ago
  Very insightful comment on the data rate, if you can't feed the LTO properly, it may be stopping and starting all the time, which could have a detrimental effect to its mechanism. But this is probably easily fixed by spooling (using bacula terminology). You buffer like 100Gbytes, write them in one batch. Plus, some drives support variable speeds. Also, in addition to SAS, a fibre channel card would also fit the bill, albeit probably a trifle more expensive. But if you go for low speeds (e.g. 8Gbps), those cards can be less than 100$.
  - adrian_b 5y ago
    Yes, that is how it should be done. For example, I collect all the files that I send to backup in archive files whose size is approximately 50 GB, then I copy to the tape the 50 GB files. Doing like this, the write speed to a LTO-7 drive is always constant at 300 MB per second, which is the maximum possible for that standard (which is much higher than what a HDD can sustain, so the 50 GB files must come from a fast SSD or from a RAM disk).
    - at_a_remove 5y ago
      I didn't know -- or perhaps remember -- that you had to match the feed against the speed, sort of like the way you had to be very careful about burning CD-Rs in the old days, but that makes perfect sense. I have been fearfully realizing that I will soon need tape backup for my next project and this is helpful. Now I am wondering if a RAID 0 of multiple HDDs could provide the ~300 MB/s speed needed.
      - adrian_b 5y ago
        You do not have to match the speed, but you should. Sending the data fast enough to the drive will reduce both the total time required for backup, when the drive is active, and it will also avoid starting and stopping a lot the drive during the transfer. Both eliminating the start-stop cycles and reducing the total active time will increase the life of your tape drive. On my server I have 128 GB of DRAM, which makes it extremely easy to ensure maximum speed without wearing a SSD. When the backup starts, I create an 80-GB RAM disk, larger than the up to 60 GB archives in which I collect the files sent to backup. With less memory one could either use smaller chunks or use a SSD instead of the RAM disk, but that seems wasteful. Then the archive files are buffered in the RAM disk and written in one command, without pauses.
- kmeisthax 5y ago
  From personal experience you are not going to be able to saturate even a horrifically obsolete LTO drive and SAS interface without specialized software or purely sequential data. Most of the cheap/free ways of running tape drives aren't optimized for parallel I/O and will horrifically shoe-shine (that's the rev-up and rev-down sound you hear when the tape drive isn't getting enough data), which isn't great for the tapes and massively increases backup time and drive usage. AFAIK most commercial tape deployments nowadays are disk-to-disk-to-tape arrangements. All the actual data is serialized to an archive on disk first, and then that serialized archive is written to tape at full speed. This minimizes tape wear and ensures your very expensive tape drives are being used efficiently.
  - bombcar 5y ago
    It may even be possible to find on eBay a multi tape loader that has a disk inside and appears to the system as just a dumb tape drive.
  - adrian_b 5y ago
    I am using a LTO-7 drive connected to a FreeBSD server. FreeBSD always succeeds to write the tape continuously at about 300 MB per second, which is the maximum speed for LTO-7. All the files send to the tape are grouped into large archive files and for the dd command that writes to tape I use a block size of 128 kB. The tape commands from FreeBSD are more convenient than those from Linux, which have not seen much maintenance in recent years. Obviously, you cannot reach tape speed when making the backup directly from a HDD or from a 1 Gb/s Ethernet. You must write the backup to tape either from a fast SSD, or from 10 Gb/s Ethernet coming from a fast SSD at the other end, or from a RAM disk configured on the server, if you have enough memory. To not wear unnecessarily the SSDs on my server where the tape drive is located, whenever I write the backup, I configure a large RAM disk on the server. The backup files coming through Ethernet to the server are written to the RAM disk on the server, then they are copied to the tape. With this arrangement it is very easy to ensure that the tape drive is written at maximum speed without any hiccups.
    - vermaden 5y ago

Kind of semi related: What do you buy if you want to do tape backups at home?

by axx 5y ago

Kind of semi related: What do you buy if you want to do tape backups at home?

Probably a LTO drive a few generations old and a SAS card. Current generations cost too much and you aren't going to be able to keep up with the data rate that they prefer.

Dylan16807 5y ago
> Current generations cost too much and you aren't going to be able to keep up with the data rate that they prefer. Maybe. Keeping up only really matters if your slowest data rate is going to be between 50MB/s and 120MB/s. If it's below that then you have a strong need for a buffer drive no matter what tape generation. And if it's above that then you shouldn't have any real trouble no matter what tape generation.
aduitsis 5y ago
Very insightful comment on the data rate, if you can't feed the LTO properly, it may be stopping and starting all the time, which could have a detrimental effect to its mechanism. But this is probably easily fixed by spooling (using bacula terminology). You buffer like 100Gbytes, write them in one batch. Plus, some drives support variable speeds. Also, in addition to SAS, a fibre channel card would also fit the bill, albeit probably a trifle more expensive. But if you go for low speeds (e.g. 8Gbps), those cards can be less than 100$.
- adrian_b 5y ago
  Yes, that is how it should be done. For example, I collect all the files that I send to backup in archive files whose size is approximately 50 GB, then I copy to the tape the 50 GB files. Doing like this, the write speed to a LTO-7 drive is always constant at 300 MB per second, which is the maximum possible for that standard (which is much higher than what a HDD can sustain, so the 50 GB files must come from a fast SSD or from a RAM disk).
  - at_a_remove 5y ago
    I didn't know -- or perhaps remember -- that you had to match the feed against the speed, sort of like the way you had to be very careful about burning CD-Rs in the old days, but that makes perfect sense. I have been fearfully realizing that I will soon need tape backup for my next project and this is helpful. Now I am wondering if a RAID 0 of multiple HDDs could provide the ~300 MB/s speed needed.
    - adrian_b 5y ago
      You do not have to match the speed, but you should. Sending the data fast enough to the drive will reduce both the total time required for backup, when the drive is active, and it will also avoid starting and stopping a lot the drive during the transfer. Both eliminating the start-stop cycles and reducing the total active time will increase the life of your tape drive. On my server I have 128 GB of DRAM, which makes it extremely easy to ensure maximum speed without wearing a SSD. When the backup starts, I create an 80-GB RAM disk, larger than the up to 60 GB archives in which I collect the files sent to backup. With less memory one could either use smaller chunks or use a SSD instead of the RAM disk, but that seems wasteful. Then the archive files are buffered in the RAM disk and written in one command, without pauses.
kmeisthax 5y ago
From personal experience you are not going to be able to saturate even a horrifically obsolete LTO drive and SAS interface without specialized software or purely sequential data. Most of the cheap/free ways of running tape drives aren't optimized for parallel I/O and will horrifically shoe-shine (that's the rev-up and rev-down sound you hear when the tape drive isn't getting enough data), which isn't great for the tapes and massively increases backup time and drive usage. AFAIK most commercial tape deployments nowadays are disk-to-disk-to-tape arrangements. All the actual data is serialized to an archive on disk first, and then that serialized archive is written to tape at full speed. This minimizes tape wear and ensures your very expensive tape drives are being used efficiently.
- bombcar 5y ago
  It may even be possible to find on eBay a multi tape loader that has a disk inside and appears to the system as just a dumb tape drive.
- adrian_b 5y ago
  I am using a LTO-7 drive connected to a FreeBSD server. FreeBSD always succeeds to write the tape continuously at about 300 MB per second, which is the maximum speed for LTO-7. All the files send to the tape are grouped into large archive files and for the dd command that writes to tape I use a block size of 128 kB. The tape commands from FreeBSD are more convenient than those from Linux, which have not seen much maintenance in recent years. Obviously, you cannot reach tape speed when making the backup directly from a HDD or from a 1 Gb/s Ethernet. You must write the backup to tape either from a fast SSD, or from 10 Gb/s Ethernet coming from a fast SSD at the other end, or from a RAM disk configured on the server, if you have enough memory. To not wear unnecessarily the SSDs on my server where the tape drive is located, whenever I write the backup, I configure a large RAM disk on the server. The backup files coming through Ethernet to the server are written to the RAM disk on the server, then they are copied to the tape. With this arrangement it is very easy to ensure that the tape drive is written at maximum speed without any hiccups.
  - vermaden 5y ago