5 ms·
>The main issue with OpenZFS performance is its write speed. At first I thought you were talking about actual raw read/write speed and how things like ARC or w
by xoa 5y ago
>The main issue with OpenZFS performance is its write speed.
At first I thought you were talking about actual raw read/write speed and how things like ARC or write caches can actually become bottlenecks when using NVMe storage, which can easily get to 200 Gbps with a small number of devices. That's being worked on via efforts like adding Direct IO.
Instead though I think you've fallen into one of those niche holes that always takes longer to get filled because there isn't much demand. Big ZFS users simply have tons and tons of drives, and with large enough arrays to spread across even rust can do alright. They'll also have more custom direction of hot/cold to different pools entirely. Smaller scale users, depending on size vs budget, may just be using pure solid state at this point, or multiple pools with more manual management. Basic 2TB SATA SSDs are down to <$170 and 2TB NVMe drives are <$190. 10 Gbps isn't much to hit, and there are lots of ways to script things up for management. RAM for buffer is pretty cheap now too. Using a mirrored NVMe-based ZIL and bypassing cache might also get many people where they want to be on sync writes.
I can see why it'd be a nice bit of polish to have hybrid pool management built-in, but I can also see how there wouldn't be a ton of demand to implement it given the various incentives in play. Might be worth seeing if there is any open work on such a thing or at least feature request.
Also to your lower comment:
>I am using TrueNAS for my ZFS solution and it doesn't offer anything out of the box to address the issues I am bringing up.
While not directly related to precisely what you're asking for, I recently started using TrueNAS myself for the first time after over a decade of using ZFS full time and one thing that immediately surprised me is how unexpectedly limited the GUI is. TONS and tons of normal ZFS functionality is not exposed there for no reason I can understand. However, it's all still supported, it's still FreeBSD and normal ZFS underneath. You can still pull up a shell and manipulate things via the commandline, or (better for some stuff) modify the GUI framework helpers to customize things like pool creation options. The power is still there at least, though other issues like inexplicably shitty user support in the GUI show it's clearly aimed at home/SoHo, maybe with some SMB roles.
- bhouston 5y agoYou are correct -- I am in a niche area. I have less than 12 HDDs in a RAIDZ2 configuration. If I had more HDDs in the RAID I would see better write speeds. But I think most home users with TrueNAS ZFS do not have massive arrays, thus my envy of the tiering or true SSD caching. You are correct that prices of small SSD are near HDD prices, but most people would want larger drives like 12 to 16TB drives in their array, and SSDs can not compete with these on price at all. Maybe we are in the twilight of HDDs? And SSDs will compete on price across the whole storage capacity range soon? Maybe...
- rsync 5y ago"You are correct -- I am in a niche area. I have less than 12 HDDs in a RAIDZ2 configuration. If I had more HDDs in the RAID I would see better write speeds." That is incorrect. RaidZ(x) vdev write speed is equal to the write speed of a single drive in the vdev. If you want a raidZ zpool to have faster writes, you add a second vdev - that would double your write speed. A wider vdev, on the other hand (15 or 18 drives, etc.) would not have faster write speeds.
- walrus01 5y agoanother consideration is that the "cheap" 4TB SSDs are things like quad-level cells, such as the samsung 870 QVO, which are slightly better in a $ per GB ratio, but also suffer in ultimate GB written lifespan for cell endurance wear-out. Thus the reason why the samsung "pro" are almost double the cost. if you needed to write a LOT (let's say something like a sustained 70MB per second all the time) but the data rate was not particularly high, you might easily wear out a RAIDZ2 composed of 2TB-4TB sized cheap SSDs and kill them in a fairly short period of time, when the same RAIDZ2 composed of 2.5" 15mm height 5TB spinning drives, or 3.5" 14-16TB drives would successfully run for many years. also re: cheap SSDs, the quad level cell tech and write cache is a big limitation in sustained write speeds: https://www.firstpost.com/tech/news-analysis/samsung-870-qvo-sata-ssd-review-possibly-the-best-qlc-drive-you-can-get-but-it-isnt-for-everyone-9145371.html https://www.firstpost.com/tech/news-analysis/samsung-870-qvo...
- laumars 5y agoWhat larger HDDs would you recommend? I need to upgrade a bunch of 3TB and 5TB drives that have been running for several years but really concerned about failure rates. I used to swear by WD Red however I’m hearing some horror stories about them in recent years.
- blancNoir 5y agohttps://www.backblaze.com/blog/backblaze-drive-stats-for-q3-2021/ https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-...
- vladvasiliu 5y ago
- walrus01 5y ago> Basic 2TB SATA SSDs are down to <$170 and 2TB NVMe drives are <$190. You sure wouldn't want to use them in any write heavy zfs application however, because you'll kill them quick in their ultimate TB write endurance. Those are triple level and quad level cell SSDs. Double the dollar figures there for anything with a ultimate lifespan TB written figure that you would want to try in a ZFS array.