One thing to consider is that when running on-prem, you will be able to procure disks that advertise 512B sector size, and actually performing well when doing s

One thing to consider is that when running on-prem, you will be able to procure disks that advertise 512B sector size, and actually performing well when doing so. With that, you can use 8KB record size on ZFS, and still obtain the same 5.5x compress ratio, assuming you are currently using 4KB sector size with 64KB record size on ZFS. With 8KB record size, there would be no read/write amplification caused by postgres. More importantly, ZFS fragmentation would be much less of an issue. Of course, disks procurement can be a nightmare process, which is why AWS is printing money.