I haven't had any issues with performance, but then again my requirement was just "reasonable performance". I ran some quick benchmarks (data below). Obviously

I haven't had any issues with performance, but then again my requirement was just "reasonable performance". I ran some quick benchmarks (data below). Obviously this is far from rigorous, but maybe it'll be useful. In previous tests I found that volblocksize=128K was optimal for my stack -- which is why the last benchmarks use that setting. Every additional ZFS filesystem in the stack may reduce storage efficiency (minimum free space requirements [1]; metadata & checksum overhead [2][3]) -- that's why I used ext4 as the top layer instead of another ZFS. [1] (as mentioned before) https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_happens_when_you_go_over_80_on/cxqu7uh/ https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_ha... [2] https://news.ycombinator.com/item?id=14756360 https://news.ycombinator.com/item?id=14756360 [3] https://forums.freenas.org/index.php?threads/what-is-the-exact-checksum-size-overhead.28187/#post-183802 https://forums.freenas.org/index.php?threads/what-is-the-exa... Test setup: debian stable kernel 4.9.0-3-amd64 zfs 0.6.5.9-5 ZFS "pool": mirror with 2x 7200rpm drives Benchmark command: for i in `seq 1 10`; do sync; dd if=/dev/zero of=DEST bs=1M count=1024 conv=fdatasync; done zfs mirror -> dataset Data (MB/s): 125,115,104,135,148,170,135,151,118,119 Mean (MB/s): 132.0 Std.dev.: 19.9 zfs mirror -> zvol (volblocksize=8K [default]) Data (MB/s): 150,115,127,125,122,118,105,118,124,128 Mean (MB/s): 123.2 Std.dev.: 11.6 zfs mirror -> zvol (volblocksize=128K) Data (MB/s): 68.5,112,115,114,94.3,85.1,83.1,98.4,120,108 Mean (MB/s): 99.8 Std.dev.: 16.9 zfs mirror -> zvol (volblocksize=128K) -> luks -> ext4 (my stack) Data (MB/s): 130,94.4,109,139,138,125,94.9,124,134,133 Mean (MB/s): 122.1 Std.dev.: 16.8 edit: formatting