4 ms·

Excellent help. To clarify a few items: - yes I mean drives with autoloader. for example: https://www.backupworks.com/qualstar-Q24-LTO-9-SAS-Library.aspx https

Excellent help. To clarify a few items: - yes I mean drives with autoloader. for example: https://www.backupworks.com/qualstar-Q24-LTO-9-SAS-Library.aspx https://www.backupworks.com/qualstar-Q24-LTO-9-SAS-Library.a... it’s basically a hard requirement as we don’t have staff time to enter data centers frequently. we are a bit unusual in being certainly not big, but not really a small business either when looking at budgets available. unless there is something wrong with qualstar product linked above perhaps autoloaders are cheaper than you believed? - understood your rebooting trick. however being full automated (apart from blank tape rotations) is a requirement also. it’s a production infrastructure. if FreeBSD provides significant value it seems safer to spec a dedicated 1U server to use for backups. there is a management node currently that might work though that has to run Linux as it currently does and I need to check if the SAS on it can be used. It has an bunch of SAS ssd drives currently and I would have assumed there is a way to cable up the qualstar drive … but again I’m still early in researching. and the SAS compatibility issue you raise is perfect example of stuff I need to figure out. - love par2cmdline and our burner with mdisc for IP backup uses that on git repo files and then seqbox as an outer container for data to guard against potential fs metadata corruption issues. there was a newer low level tool (rust rewrite I think) with many bitrot protection features that I can’t recall it’s name currently and isn’t immediately coming up in my notes, but I know it exists and have been meaning to look into it. it has a newer erasure encoding like raptorq and also block metadata like seqbox, I think can replace the par2 seqbox combo we are currently using on MDISC physical backup for IP. I don’t trust a 100% cloud as one can imagine somehow getting all accounts hacked and deleted. - yes on compressed. the 2PB is already highly highly compressed. so it means 18TB/tape. Do you have any vendor/distributors you can recommend? I always recommend 45drives to people and I was planning to ask them about LTO when we order next storinator which is coming up soon also. There is this interesting blog post from a couple of years ago that probably was the seed of my plan to embark on LTO. Our monthly backblaze invoice is totally out of control. But we need a full backup of our data as it’s simply not replaceable and at the heart of the business. https://blog.benjojo.co.uk/post/lto-tape-backups-for-linux-nerds https://blog.benjojo.co.uk/post/lto-tape-backups-for-linux-n... they talked me into it. not out of it given our specific situation.

adrian_b 2y ago
That is indeed a cheap autoloader. If you would use the full configuration with 2 tape drives, the cost of the system might be around $15k, which is very reasonable for a tape library with autoloader. I think that this autoloader is a good choice, especially if the price includes "1 x IBM LTO-9 SAS Tape Drive Installed". As I have said, I believe that it is better to choose the option of also including the second tape drive. For the tapes, there is no reason to worry about specific distributors. I have always bought them from Amazon, but shops that are specialized in storage products should be OK, unless they charge a premium price over what can be found at Amazon or Newegg. While the tapes are made by Fuji or Sony, they are usually easier to find and at at lower prices as IBM, HP or Quantum branded tapes. The prices vary, so whichever vendor is cheaper when you buy a batch of tapes should be fine. An LTO-9 cartridge should be only slightly over $100. In time the prices of LTO-9 cartridges should drop. For now they are more expensive than the older cartridges, because they are still relatively new. I store the tapes in Turtle cases: https://turtlecase.com/collections/lto https://turtlecase.com/collections/lto You must check the tape drive requirements for the SAS HBA PCIe card that must be installed in the server, which must have compatible connectors, and you must buy an appropriate SAS cable. I believe that the LTO-9 drives require the newer 12 Gb/s SAS standard and also the newer variant of the external SAS connectors (perhaps SAS HD SFF-8644 connectors). If you already have a 12 Gb/s SAS HBA that has only internal connectors for SSDs, it is possible to reuse it by buying a SAS internal to external adapter of the appropriate connector types, which must occupy one of the empty expansion slots of the server case and which plugs into the internal connectors, while providing external connectors. Such adapters can also be used with server motherboards that have on-board SAS controllers. If you have a SAS HBA card that has external connectors, but different from those on the tape drive, e.g. SAS SFF-8088, there are cables with mixed SAS connectors that can connect the tape drives. The HBA cards usually have at least 2 external SAS connectors, suitable for 2 tape drives. With the autoloader, it should be easy to make the backup or retrieval process completely automatic, so that an operator should not have to visit the tape autoloader more often than at a few months interval, except for the initial phase when you would have to write 2 PB on almost 120 tapes (or a double number for improved redundancy, beyond the redundancy added per each archive file; 2 copies can be stored in 2 different geographic locations, to avoid the catastrophic loss of all tapes), so you would want to keep the tape autoloader in an easily accessible place for that time. The initial cost for writing 2 copies of 2 PB of data, i.e. 4 PB of data, would be not much less than $30k for the tapes. This, together with the autoloader with 2 tape drives, HBA card, cases, cables and maybe adapters, would be in the range of $45k to $50k, so within your estimated budget. As I have said, it is convenient to have a database with the metadata (including content hashes, made e.g. with BLAKE2b-512 or with BLAKE3-256) of all the files that have ever been archived, which shall be used whenever information must be retrieved and which can also be used for deduplication (for which the content hashes are handy), to check whether a file is already present in some earlier archive, so there is no need for its backup.

adrian_b 2y ago

That is indeed a cheap autoloader. If you would use the full configuration with 2 tape drives, the cost of the system might be around $15k, which is very reasonable for a tape library with autoloader. I think that this autoloader is a good choice, especially if the price includes "1 x IBM LTO-9 SAS Tape Drive Installed". As I have said, I believe that it is better to choose the option of also including the second tape drive. For the tapes, there is no reason to worry about specific distributors. I have always bought them from Amazon, but shops that are specialized in storage products should be OK, unless they charge a premium price over what can be found at Amazon or Newegg. While the tapes are made by Fuji or Sony, they are usually easier to find and at at lower prices as IBM, HP or Quantum branded tapes. The prices vary, so whichever vendor is cheaper when you buy a batch of tapes should be fine. An LTO-9 cartridge should be only slightly over $100. In time the prices of LTO-9 cartridges should drop. For now they are more expensive than the older cartridges, because they are still relatively new. I store the tapes in Turtle cases: https://turtlecase.com/collections/lto https://turtlecase.com/collections/lto You must check the tape drive requirements for the SAS HBA PCIe card that must be installed in the server, which must have compatible connectors, and you must buy an appropriate SAS cable. I believe that the LTO-9 drives require the newer 12 Gb/s SAS standard and also the newer variant of the external SAS connectors (perhaps SAS HD SFF-8644 connectors). If you already have a 12 Gb/s SAS HBA that has only internal connectors for SSDs, it is possible to reuse it by buying a SAS internal to external adapter of the appropriate connector types, which must occupy one of the empty expansion slots of the server case and which plugs into the internal connectors, while providing external connectors. Such adapters can also be used with server motherboards that have on-board SAS controllers. If you have a SAS HBA card that has external connectors, but different from those on the tape drive, e.g. SAS SFF-8088, there are cables with mixed SAS connectors that can connect the tape drives. The HBA cards usually have at least 2 external SAS connectors, suitable for 2 tape drives. With the autoloader, it should be easy to make the backup or retrieval process completely automatic, so that an operator should not have to visit the tape autoloader more often than at a few months interval, except for the initial phase when you would have to write 2 PB on almost 120 tapes (or a double number for improved redundancy, beyond the redundancy added per each archive file; 2 copies can be stored in 2 different geographic locations, to avoid the catastrophic loss of all tapes), so you would want to keep the tape autoloader in an easily accessible place for that time. The initial cost for writing 2 copies of 2 PB of data, i.e. 4 PB of data, would be not much less than $30k for the tapes. This, together with the autoloader with 2 tape drives, HBA card, cases, cables and maybe adapters, would be in the range of $45k to $50k, so within your estimated budget. As I have said, it is convenient to have a database with the metadata (including content hashes, made e.g. with BLAKE2b-512 or with BLAKE3-256) of all the files that have ever been archived, which shall be used whenever information must be retrieved and which can also be used for deduplication (for which the content hashes are handy), to check whether a file is already present in some earlier archive, so there is no need for its backup.