5 ms·

Same, for at least 6 years in a 4 drive zraid array. It always reads and writes at full gigabit ethernet speeds and I haven't had any downtime other than mainta

Same, for at least 6 years in a 4 drive zraid array. It always reads and writes at full gigabit ethernet speeds and I haven't had any downtime other than maintaining FreeBSD updates which are trivial even when going from 10.x to 11 to 12.

StavrosK 7y ago
Same here (4-drive raidz for many years), though I do have an issue where deleting large files (~1 GB) takes around a minute and nobody seems to know why (I have plenty free space and RAM)...
- dekhn 7y ago
  do you have lots of snapshots? every snapshotting FS I've worked with has really slow deletes, especially when the volume is near capacity.
  - StavrosK 7y ago
    I have 400 in total, though none on the slow volume :/ That shouldn't affect it, right?
  - rleigh 7y ago
    Snapshots are one thing ZFS is fast at. All the blocks for a given snapshot are placed on a "deadlist". Snapshot deletion is essentially just returning this list of blocks back to the free pool. A terabyte snapshot will take a short while (in the background) to recycle those blocks. But the deletion itself is near instantaneous.
    - dekhn 7y ago
      I think you misunderstand: file deletions are what is slow (I don't use ZFS, my reference is WAFL, but my understanding is that all snapshotting file systems have this problem).
      - rleigh 7y ago
        Even this should have minimal overhead. If the file is present in the snapshot, then it's simply moving the blocks over to the deadlist which is a very cheap operation. If it's not in the snapshot then the blocks will get recycled in the background. In both cases you should have the unlink complete almost immediately. All of the snapshot functionality is based upon simple transaction number comparisons plus the deadlist of blocks owned by the snapshot. Only the recycling of blocks should have a bit of overhead, and that's done by a background worker--you see the free space increase for a few minutes after a gargantuan snapshot or dataset deletion, but the actual deletion completed immediately.
        dekhn 7y ago
        I've been promised many things by vendors and they always fall back to "hey! look! cool CS file system theory". I test my systems carefully and report the results back; they often don't agree. I should point out again that I don't have enough direct experience with ZFS to say if this is the case, my experience was with an enterprise NetApp server at a large company that was filling the disk up (>95%) in addition to doing hourly snapshots.
glenneroo 7y ago
"Same" for the last ~4 years, starting with 8 disks and as of 2018, the 24-bay enclosure is full. Each vdev is a mirrored pair split across HBAs to sedate my paranoia. I've replaced a few drives after watching unreadable sector count slowly increase over a few months. I've also switched out most of the original 3TB pairs to 8TB and 10TB pairs. ~42TB usable and the box only has 16GB of RAM (because I can't get the used 32GB sticks to work, it's a picky mainboard and difficult to find matching ECC memory here in Europe). I haven't powered down much except to attempt to replace the RAM or during extremely hot days. Read/write speed is more or less max gigabit, even during rebuild after hot-swapping drives.
vetinari 7y ago
A single 5400 rpm drive (the like of wd red) should be able to saturate gigabit ethernet. 4 drive array should be basically idling.

StavrosK 7y ago

Same here (4-drive raidz for many years), though I do have an issue where deleting large files (~1 GB) takes around a minute and nobody seems to know why (I have plenty free space and RAM)...

dekhn 7y ago
do you have lots of snapshots? every snapshotting FS I've worked with has really slow deletes, especially when the volume is near capacity.
- StavrosK 7y ago
  I have 400 in total, though none on the slow volume :/ That shouldn't affect it, right?
- rleigh 7y ago
  Snapshots are one thing ZFS is fast at. All the blocks for a given snapshot are placed on a "deadlist". Snapshot deletion is essentially just returning this list of blocks back to the free pool. A terabyte snapshot will take a short while (in the background) to recycle those blocks. But the deletion itself is near instantaneous.
  - dekhn 7y ago
    I think you misunderstand: file deletions are what is slow (I don't use ZFS, my reference is WAFL, but my understanding is that all snapshotting file systems have this problem).
    - rleigh 7y ago
      Even this should have minimal overhead. If the file is present in the snapshot, then it's simply moving the blocks over to the deadlist which is a very cheap operation. If it's not in the snapshot then the blocks will get recycled in the background. In both cases you should have the unlink complete almost immediately. All of the snapshot functionality is based upon simple transaction number comparisons plus the deadlist of blocks owned by the snapshot. Only the recycling of blocks should have a bit of overhead, and that's done by a background worker--you see the free space increase for a few minutes after a gargantuan snapshot or dataset deletion, but the actual deletion completed immediately.
      - dekhn 7y ago
        I've been promised many things by vendors and they always fall back to "hey! look! cool CS file system theory". I test my systems carefully and report the results back; they often don't agree. I should point out again that I don't have enough direct experience with ZFS to say if this is the case, my experience was with an enterprise NetApp server at a large company that was filling the disk up (>95%) in addition to doing hourly snapshots.

glenneroo 7y ago

"Same" for the last ~4 years, starting with 8 disks and as of 2018, the 24-bay enclosure is full. Each vdev is a mirrored pair split across HBAs to sedate my paranoia. I've replaced a few drives after watching unreadable sector count slowly increase over a few months. I've also switched out most of the original 3TB pairs to 8TB and 10TB pairs. ~42TB usable and the box only has 16GB of RAM (because I can't get the used 32GB sticks to work, it's a picky mainboard and difficult to find matching ECC memory here in Europe). I haven't powered down much except to attempt to replace the RAM or during extremely hot days. Read/write speed is more or less max gigabit, even during rebuild after hot-swapping drives.

vetinari 7y ago

A single 5400 rpm drive (the like of wd red) should be able to saturate gigabit ethernet. 4 drive array should be basically idling.