Interesting, thanks for the perspective. Myself, I treat MDADM/ZFS RAID failure more of a headache than RAID cards. As I mentioned earlier, RAID cards allow f

Interesting, thanks for the perspective. Myself, I treat MDADM/ZFS RAID failure more of a headache than RAID cards. As I mentioned earlier, RAID cards allow for easy hot-swap without any OS intervention. I am always nervous when I have to replace drives out of a ZFS pool - mainly because I forget the exact commands that need to be run (in order) for a successful swap. With hardware RAID - no commands :-) Thus far, we don't have any high-density NVMe servers so I can't really test high-end servers. My experience is limited to 8x SSDs or 16x spinning drives in our arrays. Finally, I avoided ZFS for a long time as well. Early on (OpenZFS 0.6, 0.7, etc), I spent an enormous amount of time trying to tweak/tune the system just to get on par with our HW RAID devices. No matter what I did, I simply could not get the server (typical Linux NFS NAS) to get any decent performance. In fact, I clearly remember HW RAID hitting 1.8GB/sec on our SSDs while OpenZFS could only get around 400MB/sec. Only until OpenZFS 2.1 mark did I see any real performance gains. ...Speaking of ZFS... I recently worked on a project to replace XFS with ZFS for Postgresql servers. I learned a lot about OpenZFS - specifically the memory latency when using compression. I had a lengthy discussion with the OpenZFS "gang" (https://zfsonlinux.topicbox.com/groups/zfs-discuss/T5122ffd3e191f75f https://zfsonlinux.topicbox.com/groups/zfs-discuss/T5122ffd3...) to identify some large latency numbers. Turns out, you need to disable ARC compression and "ADB Scatter Enabled" otherwise you will definitely hit some performance issues. Take a look at that thread - especially the very end where I publish some tuning suggestions. It was a fun learning exercise :-)