4 ms·

As my NAS expands, I get more and more cold data. Meanwhile large SSD's are getting cheap-ish (even TLC variants). Are there any good options right now for mul

As my NAS expands, I get more and more cold data. Meanwhile large SSD's are getting cheap-ish (even TLC variants). Are there any good options right now for multi-tiered storage for the home lab? LVM has writeback cache as an option, but I couldn't quite figure out how reliable that is, found some old posts with disturbing issues but not a lot of recent talk. Would also need to run ZFS on top of it for the ZFS features I rely on like snapshots and such, so feels like a Jenga tower solution. I know Ceph has this option, but Ceph performance is abysmal for small installations from what I could benchmark. Bcachefs looks like it'll be a winner, but it's still WIP so won't trust it with my precious data quite yet I think.

codemac 3y ago
The idea you want tiering... at home.. just seems so unlikely to save you any serious money, or gain you significant performance. "tiering" in the traditional sense isn't even popular in the DC with things like Vast, Pure, etc all doing well. Is there any reason you don't know the data set that is generally cold? The vast majority of home users basically have data that is trivially super cold that needs reasonable read performance - this is just HDDs. Then have a second mount point with things that you need greater performance, and have that be just SSDs.
- the8472 3y ago
  Having frequently used file metadata on SSDs while the cold parts are on HDDs would be noticeable even at home; the moment you start browsing the directory tree.
  - seized 3y ago
    ZFS on Linux has exactly this, you can specify specific devices to hold metadata. Caveats apply, you need to mirror those devices for redundancy. https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7.html#Special_Allocation_Class https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7...
- magicalhippo 3y ago
  It's not about saving money, more about convenience and noise. With large SSDs it seems alluring that one could have the majority of hot data on SSDs, essentially making the cold data even colder. As you mention I could split my data in two, but that's a chore. If I need new data on the hot pool, I might have to first decide what got cold enough to move to the cold pool if there's not enough room left on the hot pool. That means I have to wait for that before I can save the new stuff on the hot pool. Searching for stuff means searching both places etc. I forgot I stumbled over AutoTier[1] which could work, but it seems abandoned so again, not ideal for precious data. And it's FUSE based, so performance is likely not the best. [1]: https://github.com/45Drives/autotier https://github.com/45Drives/autotier
sitkack 3y ago
You need a buy a copy of StorNext. https://www.quantum.com/en/products/file-system/stornext/ https://www.quantum.com/en/products/file-system/stornext/
leptons 3y ago
I love my LTO-5 tape drive. 1.5TB on a tape, encryption, they have a write protect notch, and I add parity files to each tape to combat bitrot. The price per TB is pretty low. I keep a backup of important stuff off-site. As datacenters upgrade, I'll gladly take their hand-me-downs, when the price is right.
- sacnoradhq 3y ago
  Why? Hard drives are as cheap as tape now: $10/TB. Tapes are sequential access media, wear out, and malfunction, whereas hard drives are random access media. I know all about StorageTek silos and 4U tape robots but still don't bother with them.
  - adrian_b 3y ago
    Tape is $5 per TB at retail and you can safely store data on it for at least 10 years, after which you may need to make a new copy not due to the risk of aging, but due to the risk of no longer finding new tape drives compatible with the tape format. Your price of $10/TB might apply when HDDs are purchased in bulk, because at retail I see prices between $15/TB and $20/TB. No HDD may be trusted to store data for more than 5 years and this duration is valid only for the more expensive models. Assuming your price of $10/TB, one must buy at least a double HDD capacity than tape capacity, due to the short lifetime, so tape is at least 4 times cheaper. At the prices that I see at retail the difference is even greater. The sequential transfer speed of tape is greater than that of HDDs, so archiving or retrieving many GB of data takes less time with tapes. HDDs wear out and malfunction more frequently than tapes. The only real disadvantage of tapes is the high cost of the tape drives, which makes tapes preferable only when more than 100 or 200 TB of data have to be stored.
    - riobard 3y ago
      Where do you buy $5/TB tape at retail?
      - adrian_b 3y ago
sacnoradhq 3y ago
I have more storage than I know what to do with. mdadm + LVM + XFS is reliable and works.
maccam94 3y ago
Why would you use LVM with ZFS? ZFS should handle all of the features LVM has (as far as I can think of, at least).
- magicalhippo 3y ago
  The idea would be to use LVM to get writeback caching on the block-device level, then use those cached LVM devices as backing devices for a ZFS pool. I've tested this on a small scale prototype level (one HDD and one NVME SSD) and there it worked quite well. But like I said it feels a bit fragile, lots of moving parts.
matheusmoreira 3y ago
> As my NAS expands How are you expanding capacity? With traditional software raid I'd need to fail every drive and replace them one by one with identical higher capacity drives which requires several rebuilds from parity which means a long time and massive I/O loads and high risk of unrecoverable read errors just destroying the whole array in the process. It's easier to make a new array and move data over...
- KingMachiavelli 3y ago
  I this it's just ZFS that has that "limitation", LVM and Btrfs just let you mix drives of arbitrary sizes.
  - matheusmoreira 3y ago
    Everything raid has that limitation, including LVM, mdadm. https://raid.wiki.kernel.org/index.php/Growing#Expanding_existing_partitions https://raid.wiki.kernel.org/index.php/Growing#Expanding_exi... Btrfs has a more flexible block allocator but its parity implementation is still unreliable.
  - zorrotorro 3y ago
    ZFS will also let you mix drives of different sizes. And the pool would grow as soon as you remove the smallest one.
- magicalhippo 3y ago
  Yes this is also a thing. I used to do what you're doing. Last 6-7 years I've been running mirrored setup so I could expand by just adding a new vdev. Rebuilt it twice when I upgraded to significantly larger disks.
  - matheusmoreira 3y ago
    What are you doing now? Surely there are better ways...

codemac 3y ago

The idea you want tiering... at home.. just seems so unlikely to save you any serious money, or gain you significant performance. "tiering" in the traditional sense isn't even popular in the DC with things like Vast, Pure, etc all doing well. Is there any reason you don't know the data set that is generally cold? The vast majority of home users basically have data that is trivially super cold that needs reasonable read performance - this is just HDDs. Then have a second mount point with things that you need greater performance, and have that be just SSDs.

the8472 3y ago
Having frequently used file metadata on SSDs while the cold parts are on HDDs would be noticeable even at home; the moment you start browsing the directory tree.
- seized 3y ago
  ZFS on Linux has exactly this, you can specify specific devices to hold metadata. Caveats apply, you need to mirror those devices for redundancy. https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7.html#Special_Allocation_Class https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7...
magicalhippo 3y ago
It's not about saving money, more about convenience and noise. With large SSDs it seems alluring that one could have the majority of hot data on SSDs, essentially making the cold data even colder. As you mention I could split my data in two, but that's a chore. If I need new data on the hot pool, I might have to first decide what got cold enough to move to the cold pool if there's not enough room left on the hot pool. That means I have to wait for that before I can save the new stuff on the hot pool. Searching for stuff means searching both places etc. I forgot I stumbled over AutoTier[1] which could work, but it seems abandoned so again, not ideal for precious data. And it's FUSE based, so performance is likely not the best. [1]: https://github.com/45Drives/autotier https://github.com/45Drives/autotier

sitkack 3y ago

You need a buy a copy of StorNext. https://www.quantum.com/en/products/file-system/stornext/ https://www.quantum.com/en/products/file-system/stornext/

leptons 3y ago

I love my LTO-5 tape drive. 1.5TB on a tape, encryption, they have a write protect notch, and I add parity files to each tape to combat bitrot. The price per TB is pretty low. I keep a backup of important stuff off-site. As datacenters upgrade, I'll gladly take their hand-me-downs, when the price is right.

sacnoradhq 3y ago
Why? Hard drives are as cheap as tape now: $10/TB. Tapes are sequential access media, wear out, and malfunction, whereas hard drives are random access media. I know all about StorageTek silos and 4U tape robots but still don't bother with them.
- adrian_b 3y ago
  Tape is $5 per TB at retail and you can safely store data on it for at least 10 years, after which you may need to make a new copy not due to the risk of aging, but due to the risk of no longer finding new tape drives compatible with the tape format. Your price of $10/TB might apply when HDDs are purchased in bulk, because at retail I see prices between $15/TB and $20/TB. No HDD may be trusted to store data for more than 5 years and this duration is valid only for the more expensive models. Assuming your price of $10/TB, one must buy at least a double HDD capacity than tape capacity, due to the short lifetime, so tape is at least 4 times cheaper. At the prices that I see at retail the difference is even greater. The sequential transfer speed of tape is greater than that of HDDs, so archiving or retrieving many GB of data takes less time with tapes. HDDs wear out and malfunction more frequently than tapes. The only real disadvantage of tapes is the high cost of the tape drives, which makes tapes preferable only when more than 100 or 200 TB of data have to be stored.
  - riobard 3y ago
    Where do you buy $5/TB tape at retail?
    - adrian_b 3y ago

sacnoradhq 3y ago

I have more storage than I know what to do with. mdadm + LVM + XFS is reliable and works.

maccam94 3y ago

Why would you use LVM with ZFS? ZFS should handle all of the features LVM has (as far as I can think of, at least).

magicalhippo 3y ago
The idea would be to use LVM to get writeback caching on the block-device level, then use those cached LVM devices as backing devices for a ZFS pool. I've tested this on a small scale prototype level (one HDD and one NVME SSD) and there it worked quite well. But like I said it feels a bit fragile, lots of moving parts.

matheusmoreira 3y ago

> As my NAS expands How are you expanding capacity? With traditional software raid I'd need to fail every drive and replace them one by one with identical higher capacity drives which requires several rebuilds from parity which means a long time and massive I/O loads and high risk of unrecoverable read errors just destroying the whole array in the process. It's easier to make a new array and move data over...

KingMachiavelli 3y ago
I this it's just ZFS that has that "limitation", LVM and Btrfs just let you mix drives of arbitrary sizes.
- matheusmoreira 3y ago
  Everything raid has that limitation, including LVM, mdadm. https://raid.wiki.kernel.org/index.php/Growing#Expanding_existing_partitions https://raid.wiki.kernel.org/index.php/Growing#Expanding_exi... Btrfs has a more flexible block allocator but its parity implementation is still unreliable.
- zorrotorro 3y ago
  ZFS will also let you mix drives of different sizes. And the pool would grow as soon as you remove the smallest one.
magicalhippo 3y ago
Yes this is also a thing. I used to do what you're doing. Last 6-7 years I've been running mirrored setup so I could expand by just adding a new vdev. Rebuilt it twice when I upgraded to significantly larger disks.
- matheusmoreira 3y ago
  What are you doing now? Surely there are better ways...