5 ms·

It's not just the rebuild,some storage software (CEPH for example) also validate the data on the disks from time to time, and since the iops is quite limited on

It's not just the rebuild,some storage software (CEPH for example) also validate the data on the disks from time to time, and since the iops is quite limited on spindles, it takes more and more of that iops bucket just to verify the data that you have on the disks.

bayindirh 6y ago
ZFS also periodically re-silvers the disks to keep the FS in top shape. IIRC ZFS tries to re-silver disks when the traffic is low but, it's not always possible. I bet that the disks is 16+ TB range will be used for colder tiers of the storage. Also, they should be useful in the OST of the LustreFS since the random read storm hits the MDT more severely.
- karamanolev 6y ago
  A friendly minor correction - a resilver is only when a drive is replaced, the periodic checksum check is a scrub in ZFS parlance.
  - magicalhippo 6y ago
    And the scrub has to be initiated somehow, typically cron job, it's not automatic in ZFS. Though NAS distributions like TrueNAS/FreeNAS sets this up for you by default.
    - bayindirh 6y ago
      > And the scrub has to be initiated somehow, typically cron job, it's not automatic in ZFS. The devices were Oracle/Sun ZFS appliances (I tortured a 7320, we liked it & bought a full-out 7420) so, maybe it was set up to scrub/resilver automatically in some cases. As I've aforementioned, we're more of a Lustre shop and retired the systems some years ago.
  - bayindirh 6y ago
    > A friendly minor correction... Thank you. I'm no expert in ZFS TBH (We use Lustre much more) but, IIRC, when I was benchmarking the then new Sun Oracle ZFS 7320, I remember it was resilvering the disks after especially torturous loads, at night. Maybe it was specific to the appliances (Our behemoth 7420 did the same) or, something was wrong. I remember Oracle/Sun guys jokingly asking me whether I succeeded to make it resilver the disks and, hearing it did indeed resilver the disks a dozen times visibly upset them. They've only said that "Pack it up, we need to go". Fun times, it was.
    - Fnoord 6y ago
      Hmm, if you had to resilver the disks a dozen times, I would assume that is indication of a hardware failure somewhere in the device (perhaps RAM?).
rubiquity 6y ago
Yes that’s a good call out. I was being lazy but there are quite a few different reasons to need to scan the entire disk.

bayindirh 6y ago

ZFS also periodically re-silvers the disks to keep the FS in top shape. IIRC ZFS tries to re-silver disks when the traffic is low but, it's not always possible. I bet that the disks is 16+ TB range will be used for colder tiers of the storage. Also, they should be useful in the OST of the LustreFS since the random read storm hits the MDT more severely.

karamanolev 6y ago
A friendly minor correction - a resilver is only when a drive is replaced, the periodic checksum check is a scrub in ZFS parlance.
- magicalhippo 6y ago
  And the scrub has to be initiated somehow, typically cron job, it's not automatic in ZFS. Though NAS distributions like TrueNAS/FreeNAS sets this up for you by default.
  - bayindirh 6y ago
    > And the scrub has to be initiated somehow, typically cron job, it's not automatic in ZFS. The devices were Oracle/Sun ZFS appliances (I tortured a 7320, we liked it & bought a full-out 7420) so, maybe it was set up to scrub/resilver automatically in some cases. As I've aforementioned, we're more of a Lustre shop and retired the systems some years ago.
- bayindirh 6y ago
  > A friendly minor correction... Thank you. I'm no expert in ZFS TBH (We use Lustre much more) but, IIRC, when I was benchmarking the then new Sun Oracle ZFS 7320, I remember it was resilvering the disks after especially torturous loads, at night. Maybe it was specific to the appliances (Our behemoth 7420 did the same) or, something was wrong. I remember Oracle/Sun guys jokingly asking me whether I succeeded to make it resilver the disks and, hearing it did indeed resilver the disks a dozen times visibly upset them. They've only said that "Pack it up, we need to go". Fun times, it was.
  - Fnoord 6y ago
    Hmm, if you had to resilver the disks a dozen times, I would assume that is indication of a hardware failure somewhere in the device (perhaps RAM?).

rubiquity 6y ago

Yes that’s a good call out. I was being lazy but there are quite a few different reasons to need to scan the entire disk.