4 ms·

3-Way Disk Mirrors with ZFS on Linux

28 points by bhoey 11y ago

psophis 11y ago
A much easier way to test ZFS configurations is to use files instead[0]: $ mkfile 128m /home/user/disk1 $ mkfile 128m /home/user/disk2 $ mkfile 128m /home/user/disk3 $ zfs create tank mirror /home/user/disk1 /home/user/disk2 /home/user/disk3 [0]: https://flux.org.uk/tech/2007/03/zfs_tutorial_1.html https://flux.org.uk/tech/2007/03/zfs_tutorial_1.html
- deleted 11y ago
  [deleted]
  - psophis 11y ago
    Yes they would. But you should never use this in production. Only for the purpose of learning/testing ZFS.
    - gvb 11y ago
      Sorry, I deleted my comment about the same time you replied to it. I realized both you and TFA were discusing testing setups, not actual hard drive redundancy setups.
- bhoey 11y ago
  Author here, thanks for the heads up. It doesn't look like mkfile is included with the Jessie base system I was using to test. For those wishing to use this method it looks like the xfsprogs package contains xfs_mkfile which operates the same as parent's example.
  - psophis 11y ago
    mkfile is standard on Solaris. On Debain use fallocate: $ fallocate -l 128MB /home/user/disk1
    - voidz 11y ago
      there is also 'truncate'. Especially nice about truncate is the ability to create sparse files. So if you want to test how big a pool would be with three 1TB disks without actually using 3TB of allocated space: 'truncate -s 1T file1.img'. voila, a 1tb file that only takes kilobytes upon creation of a zpool.
    - NeutronBoy 11y ago
      Is there an advantage of using fallocate over say, dd to pipe random or zeroes to a file?
      - asimilator 11y ago
        fallocate doesn't do any IO; it's much faster. From the fallocate man page: > For filesystems which support the fallocate system call, preallocation is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeros.
voidz 11y ago
One thing to note: the '-o ashift' option is a per-drive (vdev) attribute according to the manual. So, when replacing a vdev, be sure to specify this option again if you do not use the default: zpool replace -o ashift=12 old new.
- bhoey 11y ago
  Good point, post updated, thanks!
- jlgaddis 11y ago
  I realize this article is aimed at Linux, but on FreeBSD you can simply set vfs.zfs.min_auto_ashift=12 in /etc/sysctl.conf to handle this automatically.
radiowave 11y ago
Another nice property of ZFS mirroring is the read performance. Since data integrity is verified by checksums rather than by reading the data back from all disks and comparing it, the disks within a ZFS mirror are able to serve read requests in parallel.
booi 11y ago
Also, standard linux raid (mdadm) will allow 3 or more disks in RAID 1 and will read from all of them for better performance. ZFS has a lot of overhead compared to mdadm+LVM but it depends on your specific use case.

28 points by bhoey 11y ago

A much easier way to test ZFS configurations is to use files instead[0]: $ mkfile 128m /home/user/disk1 $ mkfile 128m /home/user/disk2 $ mkfile 128m /home/user/disk3 $ zfs create tank mirror /home/user/disk1 /home/user/disk2 /home/user/disk3 [0]: https://flux.org.uk/tech/2007/03/zfs_tutorial_1.html https://flux.org.uk/tech/2007/03/zfs_tutorial_1.html

deleted 11y ago
[deleted]
- psophis 11y ago
  Yes they would. But you should never use this in production. Only for the purpose of learning/testing ZFS.
  - gvb 11y ago
    Sorry, I deleted my comment about the same time you replied to it. I realized both you and TFA were discusing testing setups, not actual hard drive redundancy setups.
bhoey 11y ago
Author here, thanks for the heads up. It doesn't look like mkfile is included with the Jessie base system I was using to test. For those wishing to use this method it looks like the xfsprogs package contains xfs_mkfile which operates the same as parent's example.
- psophis 11y ago
  mkfile is standard on Solaris. On Debain use fallocate: $ fallocate -l 128MB /home/user/disk1
  - voidz 11y ago
    there is also 'truncate'. Especially nice about truncate is the ability to create sparse files. So if you want to test how big a pool would be with three 1TB disks without actually using 3TB of allocated space: 'truncate -s 1T file1.img'. voila, a 1tb file that only takes kilobytes upon creation of a zpool.
  - NeutronBoy 11y ago
    Is there an advantage of using fallocate over say, dd to pipe random or zeroes to a file?
    - asimilator 11y ago
      fallocate doesn't do any IO; it's much faster. From the fallocate man page: > For filesystems which support the fallocate system call, preallocation is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeros.

voidz 11y ago

One thing to note: the '-o ashift' option is a per-drive (vdev) attribute according to the manual. So, when replacing a vdev, be sure to specify this option again if you do not use the default: zpool replace -o ashift=12 old new.

bhoey 11y ago
Good point, post updated, thanks!
jlgaddis 11y ago
I realize this article is aimed at Linux, but on FreeBSD you can simply set vfs.zfs.min_auto_ashift=12 in /etc/sysctl.conf to handle this automatically.

radiowave 11y ago

Another nice property of ZFS mirroring is the read performance. Since data integrity is verified by checksums rather than by reading the data back from all disks and comparing it, the disks within a ZFS mirror are able to serve read requests in parallel.

booi 11y ago

Also, standard linux raid (mdadm) will allow 3 or more disks in RAID 1 and will read from all of them for better performance. ZFS has a lot of overhead compared to mdadm+LVM but it depends on your specific use case.