8 ms·

Well, professionally, tape is it - there is technology and it lasts more than 5 years. Unfortunately, the market for tape has evolved such that it's not very fr

by creer 2y ago

Well, professionally, tape is it - there is technology and it lasts more than 5 years. Unfortunately, the market for tape has evolved such that it's not very friendly to the non-pros. Not impossible but not friendly. That probably has to do with the lack of perceived market for that among non-corporate - or perhaps the impression that clown storage is where it's at for non-corporate. To be fair some more, JBOD/RAID and hard drives does work pretty well. Past the 5 year horizon to be sure. Product mgt and corp finance has also fallen in love with subscriptions - and clown storage is such an awesome match for that! Who needs to sell long term terabyte solutions when you can rent it out. Easy to argue against that logic of course, but not easy to fight.

tombert 2y ago
I have an LTO-6 tape drive. It works fine, but it is a pain in the ass to set up on Linux. It only connects via SAS, you need to load a lot of arcane kernel modules, the logs are non-standardized and often misleading, and the entire interface is command line based. I don't mind living in the command line and I don't even mind fighting to get everything up, but I don't see most people putting up with it. It's also a huge pain to get working with a laptop, since I don't think most laptops have a SAS connector, so you have to use an eGPU case with a Host Bus Adapter, which is its own share of headaches.
- MisterTea 2y ago
  And here I just bought a NIB LTO-5 SAS drive for home file server backup.
- creer 2y ago
  Hehe. If you're trying to live with just a laptop and an LTO drive, I can certainly see it! I'd expect most people who get into LTO drives have a massive set of hard drives, some convoluted desktop-and-up case(s) to run them, a couple laptops and random peripherals such as cameras, scanners and whatnot. USB all over the place. - So that for them both the drive and the command line interface are deeper in a technology map. But that goes back to the minimal market for LTO among amateurs. They will scratch their itch and write software for it but it's not exactly critical mass.
  - tombert 2y ago
    I used to have a rack mounted server that I used for Jellyfin and as a NAS so it was easy to install a SAS card, but it was sucking up way too much power and costing me about $50/month to run, so I got rid of it, and replaced it with one of those tiny gaming PCs you can get on Amazon. I've gotten it to work over thunderbolt with an eGPU case on both the mini computer and my laptop, so it can be done, at least with NixOS. I still have the tapes, and I use them to archive really important stuff, but honestly what I mostly do now is have a pretty resilient ZFS RAID, which is 24 16TB drives, three RAID-Z2's chained together, so I can lose up to six drives (two from each RAID) before I risk losing data. I run a scrub once a month, so if a drive dies, hopefully I can catch it before I lose anything.
  - rbanffy 2y ago
    To keep those drives happy and minimize wear, you need to push data at a constant rate so the tape can move continuously while writing. You'll need to push a sustained stream of up to 400 MBps (for LTO-9) to the drive (and, therefore, read that from the sources). A common pattern is to send files for archival to a disk volume and dump from there to tape. To fill an LTO-9 tape, expect to set up from 20 to 50TB of fast storage. I would assume a RAID-0 of large-ish HDDs with an SSD cache in front can handle it. I'd prefer a data-center grade SSD with the full size of the batch you want to record. Nothing forces you to fill the tape though. If your datasets are in the 100GB range, Blu-ray archival might be a better option. The disks are supposed to last 100 years if well stored and the drivers are much, much, much cheaper.
    - Tepix 2y ago
      400 MB/s, not Mbps. That‘s getting near S-ATA III (SATA-600) limits. Also a LTO-9 tape can store up to 45TB of uncompressed data (18 TB uncompressed), so you better have a fast storage volume ready with your data when you’re writing your next tape.
      - rbanffy 2y ago
        The one thing worth remembering though is that you are not forced to write all of the 18TB of data, so your staging drive can be significantly smaller, but considering the tape alone costs about half of what a large hard disk costs (at least at my local office supplies provider), unless you are using more than 50% of the capacity, you'll be better off by writing it to a HDD (at least for backup purposes). Long-term archival, however, is a different thing and tape is a much better medium for that. It's a shame DOTS [1] doesn't seem to be going anywhere... 1- https://group47.com/what-is-dots/ https://group47.com/what-is-dots/
deleted 2y ago
[deleted]
pdimitar 2y ago
> or perhaps the impression that clown storage is where it's at for non-corporate. Clown storage. Thanks for the giggle. :D Needed it, had a pretty rough last couple of days.
EGreg 2y ago
Why is tape the best thing? One would think two- or three-dimensional storage could pack way more data!
- codemac 2y ago
  Good thinking - there are many projects around this. Ceramic, DNA, 3d glass, etc. They all run into the same IO problems tape has though.
- wtallis 2y ago
  Nobody needs to worry about storage density in the sense of how much shelf space you need for cold storage. Tape, NAND flash, and even high-capacity hard drives are all dense enough that the incremental cost of more shelf space is not the most important part of the problem. And NAND flash is already pretty 3D, with hundreds of layers of memory cells fabricated on the surface of each wafer, and several such dies stacked in each BGA package, and it's not uncommon for U.2 drives to contain two PCBs each with NAND packages on each side.
- creer 2y ago
  Tape is a way to get volume. Through easily spooled layers. Not fast access but otherwise not unreasonable. Plus tapes stored outside of the drive. Another way to look at it: "and yet here we are". Tape is still what has led to the highest density so far (depending how much storage you look at, and cost tradeoff). But also cost tradeoff of separating the drive from the medium. which hasn't really worked: LTO tapes are pro supplies - and so, expensive.
- UltraSane 2y ago
  Tape on a spool is 3D volumetric storage
- mrweasel 2y ago
  Tape is generally available, I don't think in of those more esoteric storage medium, like 3D glass cubes are commercially available.

by creer 2y ago

tombert 2y ago

I have an LTO-6 tape drive. It works fine, but it is a pain in the ass to set up on Linux. It only connects via SAS, you need to load a lot of arcane kernel modules, the logs are non-standardized and often misleading, and the entire interface is command line based. I don't mind living in the command line and I don't even mind fighting to get everything up, but I don't see most people putting up with it. It's also a huge pain to get working with a laptop, since I don't think most laptops have a SAS connector, so you have to use an eGPU case with a Host Bus Adapter, which is its own share of headaches.

MisterTea 2y ago
And here I just bought a NIB LTO-5 SAS drive for home file server backup.
creer 2y ago
Hehe. If you're trying to live with just a laptop and an LTO drive, I can certainly see it! I'd expect most people who get into LTO drives have a massive set of hard drives, some convoluted desktop-and-up case(s) to run them, a couple laptops and random peripherals such as cameras, scanners and whatnot. USB all over the place. - So that for them both the drive and the command line interface are deeper in a technology map. But that goes back to the minimal market for LTO among amateurs. They will scratch their itch and write software for it but it's not exactly critical mass.
- tombert 2y ago
  I used to have a rack mounted server that I used for Jellyfin and as a NAS so it was easy to install a SAS card, but it was sucking up way too much power and costing me about $50/month to run, so I got rid of it, and replaced it with one of those tiny gaming PCs you can get on Amazon. I've gotten it to work over thunderbolt with an eGPU case on both the mini computer and my laptop, so it can be done, at least with NixOS. I still have the tapes, and I use them to archive really important stuff, but honestly what I mostly do now is have a pretty resilient ZFS RAID, which is 24 16TB drives, three RAID-Z2's chained together, so I can lose up to six drives (two from each RAID) before I risk losing data. I run a scrub once a month, so if a drive dies, hopefully I can catch it before I lose anything.
- rbanffy 2y ago
  To keep those drives happy and minimize wear, you need to push data at a constant rate so the tape can move continuously while writing. You'll need to push a sustained stream of up to 400 MBps (for LTO-9) to the drive (and, therefore, read that from the sources). A common pattern is to send files for archival to a disk volume and dump from there to tape. To fill an LTO-9 tape, expect to set up from 20 to 50TB of fast storage. I would assume a RAID-0 of large-ish HDDs with an SSD cache in front can handle it. I'd prefer a data-center grade SSD with the full size of the batch you want to record. Nothing forces you to fill the tape though. If your datasets are in the 100GB range, Blu-ray archival might be a better option. The disks are supposed to last 100 years if well stored and the drivers are much, much, much cheaper.
  - Tepix 2y ago
    400 MB/s, not Mbps. That‘s getting near S-ATA III (SATA-600) limits. Also a LTO-9 tape can store up to 45TB of uncompressed data (18 TB uncompressed), so you better have a fast storage volume ready with your data when you’re writing your next tape.
    - rbanffy 2y ago
      The one thing worth remembering though is that you are not forced to write all of the 18TB of data, so your staging drive can be significantly smaller, but considering the tape alone costs about half of what a large hard disk costs (at least at my local office supplies provider), unless you are using more than 50% of the capacity, you'll be better off by writing it to a HDD (at least for backup purposes). Long-term archival, however, is a different thing and tape is a much better medium for that. It's a shame DOTS [1] doesn't seem to be going anywhere... 1- https://group47.com/what-is-dots/ https://group47.com/what-is-dots/

deleted 2y ago

[deleted]

pdimitar 2y ago

> or perhaps the impression that clown storage is where it's at for non-corporate. Clown storage. Thanks for the giggle. :D Needed it, had a pretty rough last couple of days.

EGreg 2y ago

Why is tape the best thing? One would think two- or three-dimensional storage could pack way more data!

codemac 2y ago
Good thinking - there are many projects around this. Ceramic, DNA, 3d glass, etc. They all run into the same IO problems tape has though.
wtallis 2y ago
Nobody needs to worry about storage density in the sense of how much shelf space you need for cold storage. Tape, NAND flash, and even high-capacity hard drives are all dense enough that the incremental cost of more shelf space is not the most important part of the problem. And NAND flash is already pretty 3D, with hundreds of layers of memory cells fabricated on the surface of each wafer, and several such dies stacked in each BGA package, and it's not uncommon for U.2 drives to contain two PCBs each with NAND packages on each side.
creer 2y ago
Tape is a way to get volume. Through easily spooled layers. Not fast access but otherwise not unreasonable. Plus tapes stored outside of the drive. Another way to look at it: "and yet here we are". Tape is still what has led to the highest density so far (depending how much storage you look at, and cost tradeoff). But also cost tradeoff of separating the drive from the medium. which hasn't really worked: LTO tapes are pro supplies - and so, expensive.
UltraSane 2y ago
Tape on a spool is 3D volumetric storage
mrweasel 2y ago
Tape is generally available, I don't think in of those more esoteric storage medium, like 3D glass cubes are commercially available.