3 ms·

> the WD Red's firmware was up to the challenge of handling a conventional RAID rebuild, which amounts to an enormous, very large block sequential write test. (

by acqq 6y ago

> the WD Red's firmware was up to the challenge of handling a conventional RAID rebuild, which amounts to an enormous, very large block sequential write test. (...) the Red is only 16.7 percent slower than its non-SMR Ironwolf competition > we can model an ideal ZFS resilvering workload with a massive sequential write — and we did exactly that, using 32KiB blocks of incompressible pseudorandom data (...) With this test workload, we achieved a throughput of 209.3MiB/sec on the Ironwolf, but only 13.2MiB/sec on the Red — a 15.9:1 slowdown So if I understood correctly "very large block sequential write test" is only 16% slower, but reducing the size of the written block to 32KiB, like apparently ZFS does, it becomes 16 times slower? Couldn't that be fixed in ZFS? 32 KiB block is too small for SSDs too, if I understand correctly. (Edit: I refer to: https://tytso.livejournal.com/2009/02/20/ https://tytso.livejournal.com/2009/02/20/ "However, with SSD’s (...) you need to align partitions on at least 128k boundaries for maximum efficiency." -- it seems that 128K was an important number?). If the writes are sequential, some kind of caching (i.e., blocking the sequential writes) could be made to avoid the slowdown? How big were the blocks written when the 16% slowdown happened? And what is the minimal needed block size to avoid many-times slowdown?

dannyw 6y ago
You can change the block size with ZFS easily, it is a supported function: sudo zfs set recordsize=[size] data/media/series This can be done at any time, but will only apply to new files. ZFS is a Copy on Write system, which means that if you take a snapshot (or duplicate) a file and modify one byte, the modifications (min size of a record size) is created then (so 128KiB default). Each block is then distributed reduced to about ~32KiB of writes because of striping, parity, etc. If you made the record size something large, let's say 1MB, then it would mean that changing 1 byte of a duplicated or snapshotted 1GB file would cause an additional 1MB of storage. Very, very very inefficient. However, still better than a non-CoW system, where duplicates and snapshots would occupy an additional 1GB irrespective of modifications. So to summarize, ZFS is intentionally designed and benefits from small block sizes, and rather than changing it, you should simply buy a non-SMR drive esp. because they are available at comparable prices. Also keep in mind that an advanced format HDD sector is 4096 bytes. I haven't heard of any drives with more than ~4096 bytes per sector. So as long as your block size is an integer multiple of 4096 bytes, you are getting all the benefits of your HDD medium. I believe SSD native sectors are similar, so again, there's not reason to go for huge sectors. SMR's abysmal performance with small sectors is a defect, something you should RMA the moment you receive a SMR drive (WD is honoring replacements, just mention the magic word of 'ZFS rebuild' to support).
- RNCTX 6y ago
  > Also keep in mind that an advanced format HDD sector is 4096 bytes. I haven't heard of any drives with more than ~4096 bytes per sector. There are some NVMEs which are 8192. Not sure which, because, we haven't learned a damn thing in ~15 years. I distinctly remember back in the 2000s Adaptec giving Theo at OpenBSD a run-around about driver code because they had lawyers in their ear trying to convince them that they should be patent and licensing trolls rather than hardware vendors (worked great for Lucent) and finally, in retaliation, he shipped a version of OpenBSD without Adaptec RAID support. https://openzfs.org/wiki/Performance_tuning#Alignment_Shift_.28ashift.29 https://openzfs.org/wiki/Performance_tuning#Alignment_Shift_... Why is this relevant? Because some of these NVME drives report wrong values for their sector size, so we're still in that same boat of being flat out lied to by the people we buy this hardware from. Some NVME SSDs, per testing, report 4096 when they're actually 8192. So you don't know what value to use until you set up a pool/dataset, test it for performance, and then if the performance seems off versus say... Anandtech benchmarks, destroy the pool/dataset and try the other value (this becomes ashift=12 or ashift=13 currently). You can find discussion on the ZFS subreddit about this pretty commonly, and also on the PostGIS and OpenStreetMap user groups and github issue forums since those people tend to be on the bleeding edge of the disk performance market.
- zozbot234 6y ago
  > I believe SSD native sectors are similar, so again, there's not reason to go for huge sectors. SSD "erase" operations involve fairly large sectors but these are something you'd want to limit anyway, for the sake of data resilience. So using small sectors for SSD's may make sense too.
- secabeen 6y ago
  > If you made the record size something large, let's say 1MB, then it would mean that changing 1 byte of a duplicated or snapshotted 1GB file would cause an additional 1MB of storage. Very, very very inefficient. However, still better than a non-CoW system, where duplicates and snapshots would occupy an additional 1GB irrespective of modifications. 1MB recordsizes make a lot of sense for drives that primarily hold large files for archive or preservation. That lines up with the intended use of SMR drives. The problem here is that these were stealth-SMR drives, saving WD money at the customer's pain. For the small % of use cases where you have large files that are written sequentially, and never edited internally, labeled-SMR with large records can be a good solution that saves money.

by acqq 6y ago

dannyw 6y ago

You can change the block size with ZFS easily, it is a supported function: sudo zfs set recordsize=[size] data/media/series This can be done at any time, but will only apply to new files. ZFS is a Copy on Write system, which means that if you take a snapshot (or duplicate) a file and modify one byte, the modifications (min size of a record size) is created then (so 128KiB default). Each block is then distributed reduced to about ~32KiB of writes because of striping, parity, etc. If you made the record size something large, let's say 1MB, then it would mean that changing 1 byte of a duplicated or snapshotted 1GB file would cause an additional 1MB of storage. Very, very very inefficient. However, still better than a non-CoW system, where duplicates and snapshots would occupy an additional 1GB irrespective of modifications. So to summarize, ZFS is intentionally designed and benefits from small block sizes, and rather than changing it, you should simply buy a non-SMR drive esp. because they are available at comparable prices. Also keep in mind that an advanced format HDD sector is 4096 bytes. I haven't heard of any drives with more than ~4096 bytes per sector. So as long as your block size is an integer multiple of 4096 bytes, you are getting all the benefits of your HDD medium. I believe SSD native sectors are similar, so again, there's not reason to go for huge sectors. SMR's abysmal performance with small sectors is a defect, something you should RMA the moment you receive a SMR drive (WD is honoring replacements, just mention the magic word of 'ZFS rebuild' to support).

RNCTX 6y ago
> Also keep in mind that an advanced format HDD sector is 4096 bytes. I haven't heard of any drives with more than ~4096 bytes per sector. There are some NVMEs which are 8192. Not sure which, because, we haven't learned a damn thing in ~15 years. I distinctly remember back in the 2000s Adaptec giving Theo at OpenBSD a run-around about driver code because they had lawyers in their ear trying to convince them that they should be patent and licensing trolls rather than hardware vendors (worked great for Lucent) and finally, in retaliation, he shipped a version of OpenBSD without Adaptec RAID support. https://openzfs.org/wiki/Performance_tuning#Alignment_Shift_.28ashift.29 https://openzfs.org/wiki/Performance_tuning#Alignment_Shift_... Why is this relevant? Because some of these NVME drives report wrong values for their sector size, so we're still in that same boat of being flat out lied to by the people we buy this hardware from. Some NVME SSDs, per testing, report 4096 when they're actually 8192. So you don't know what value to use until you set up a pool/dataset, test it for performance, and then if the performance seems off versus say... Anandtech benchmarks, destroy the pool/dataset and try the other value (this becomes ashift=12 or ashift=13 currently). You can find discussion on the ZFS subreddit about this pretty commonly, and also on the PostGIS and OpenStreetMap user groups and github issue forums since those people tend to be on the bleeding edge of the disk performance market.
zozbot234 6y ago
> I believe SSD native sectors are similar, so again, there's not reason to go for huge sectors. SSD "erase" operations involve fairly large sectors but these are something you'd want to limit anyway, for the sake of data resilience. So using small sectors for SSD's may make sense too.
secabeen 6y ago
> If you made the record size something large, let's say 1MB, then it would mean that changing 1 byte of a duplicated or snapshotted 1GB file would cause an additional 1MB of storage. Very, very very inefficient. However, still better than a non-CoW system, where duplicates and snapshots would occupy an additional 1GB irrespective of modifications. 1MB recordsizes make a lot of sense for drives that primarily hold large files for archive or preservation. That lines up with the intended use of SMR drives. The problem here is that these were stealth-SMR drives, saving WD money at the customer's pain. For the small % of use cases where you have large files that are written sequentially, and never edited internally, labeled-SMR with large records can be a good solution that saves money.