6 ms·

>Most systems that include ZFS schedule scrubs once per month. This frequency is appropriate for many environments, but high churn systems may require more freq

>Most systems that include ZFS schedule scrubs once per month. This frequency is appropriate for many environments, but high churn systems may require more frequent scrubs. Is there a more specific 'rule of thumb' for scrub frequency? What variables should one consider?

nubinetwork 9mo ago
Total pool size and speed. Less data scrubs faster, as do faster disks or disk topology (a 3 way stripe of nvme will scrub faster than a single sata ssd) For what its worth, I scrub daily mostly because I can. It's completely overkill, but if it only takes half an hour, then it can run in the middle of the night while I'm sleeping.
ssl-3 9mo ago
The cost of a scrub is just a flurry of disk reads and a reduction in performance during a scrub. If this cost is affordable on a daily basis, then do a scrub daily. If it's only affordable less often, then do it less often. (Whatever the case: It's not like a scrub causes any harm to the hardware or the data. It can run as frequently as you elect to tolerate.)
- agapon 9mo ago
  With HDDs, it's also mechanical wear and increased chance of a failure. SSDs are not fully immune to increased load either.
  - ssl-3 9mo ago
    Is there any evidence that suggests that reading from a hard drive (instead of it just spinning idle) increases physical wear in any meaningful way? Likewise, is there any evidence of this for solid-state storage?
    - rcxdude 9mo ago
      Yes. Hard drives have published "Annualized Workload Rate" ratings, which are in TB/year, and the manufacturers state there is no difference between reads and writes for the purpose of this rating. (https://www.toshiba-storage.com/trends-technology/mttf-what-hard-drive-reliability-really-means/ https://www.toshiba-storage.com/trends-technology/mttf-what-...) For SSDs, writes matter a lot more. Reads may increase the temperature of the drive, so they'll have some effect, but I don't think I've seen a read endurance rating for an SSD.
    - digiown 9mo ago
      Reading from it requires the read head to move, as opposed to spinning idle where the heads are parked on the side. Moving parts generally wear out over time.
toast0 9mo ago
Once a month seems like a reasonable rule of thumb. But you're balancing the cost of the scrub vs the benefit of learning about a problem as soon as possible. A scrub does a lot of I/O and a fair amount of computing. The scrub load competes with your application load and depending on the size of your disk(s) and their read bandwidth, it may take quite some time to do the scrub. There's even maybe some potential that the read load could push a weak drive over the edge to failure. On my personal servers, application load is nearly meaningless, so I do an about monthly scrub from cron which I think will only scrub one zpool at a time per machine, which seems reasonable enough to me. I run relatively large spinning disks, so if I scrubbed on a daily basis, the drives would spend most of the day scrubbing and that doesn't seem reasonable. I haven't run ZFS in a work environment... I'd have to really consider how the read load impacted the production load and if scrubbing with limits to reduce production impact would complete in a reasonable amount of time... I've run some systems that are essentially alwayd busy and if a scrub would take several months, I'd probably only scrub when other systems indicate a problem and I can take the machine out of rotation to examine it. If I had very high reliability needs or a long time to get replacement drives, I might scrub more often? If I was worried about power consumption, I might scrub less often (and also let my servers and drives go into standby). The article's recommendation to scan at least once every 4 months seems pretty reasonable, although if you have seriously offline disks, maybe once a year is more approachable. I don't think I'd push beyond that, lots of things don't like to sit for a year and then turn on correctly.
atmosx 9mo ago
Once a month is fine ("/etc/cron.monthly/zfs-scrub"): #!/bin/bash # # ZFS scrub script for monthly maintenance # Place in /etc/cron.monthly/zfs-scrub POOL="storage" TAG="zfs-scrub" # Log start logger -t "$TAG" -p user.notice "Starting ZFS scrub on pool: $POOL" # Run the scrub if /sbin/zpool scrub "$POOL"; then logger -t "$TAG" -p user.notice "ZFS scrub initiated successfully on pool: $POOL" else logger -t "$TAG" -p user.err "Failed to start ZFS scrub on pool: $POOL" exit 1 fi exit 0
- chungy 9mo ago
  That script might do with the "-w" parameter passed to scrub. Then "zpool scrub" won't return until the scrub is finished.
- k_bx 9mo ago
  Didn't know about the logger script, looks nice. Can it wrap the launch of the scrub itself so that it logs like logger too, or do you separately track its stdout/stderr when something happens? update: figured how you can improve that call to add logs to logger
  - nubinetwork 9mo ago
    Scrub doesn't log anything by default, you run it and it returns quickly... you have to get the results out of zpool status or through zed.
  - atmosx 9mo ago
    The status of the pool is tracked by Prometheus though an ZFS exporter and faults are reported via alert manager & pushover.
kanbankaren 9mo ago
Once a month might be too high because HDDs are rated at ~ 180 TB workload/year. Remember, the workload/year limit includes read & writes and doesn't vary much by capacity, so a 10 TB HDD scrubbed monthly consumes 67% of the workload, let alone any other usage. Scrubbing every quarter is usually sufficient without putting high wear on the HDD.
- Hakkin 9mo ago
  A scrub only reads allocated space, so in your 10TB example, a scrub would only read whatever portion of that 10TB is actually occupied by data. It's also usually recommended to keep your usage below 80% of the total pool size to avoid performance issues, so the worst case in your scenario would be more like ~53% assuming you follow the 80% rule.
  - formerly_proven 9mo ago
    Is the 80% rule real or just passed down across decades like other “x% free” rules? Those waste enormous amounts of resources on modern systems and I kind of doubt ZFS actually needs a dozen terabytes or more of free space in order to not shit the bed. Just like Linux doesn’t actually need >100 GB of free memory to work properly.
    - cornonthecobra 9mo ago
      Speaking strictly about ZFS internal operations, the free space requirement is closer to 5% on current ZFS versions. That allows for CoW and block reallocations in real-world pools. Heavy churn and very large files will increase that margin.
    - barrkel 9mo ago
      In practice you see noticeable degradation of performance for streaming reads of large files written after 85% or so. Files you used to be able to expect to get 500+MB/sec could be down to 50MB/sec. It's fragmentation, and it's fairly scale invariant, in my experience.
    - 9mo ago
barrkel 9mo ago
I scrub once a quarter because scrubs take 11 days to complete. I have 8x 18TB raidz2 pool, and I keep a couple of spare drives on hand so I can start a resilver as soon as an issue crops up. In the past, I've gone for a few years between scrubs. One system had a marginal I/O setup and was unreliable for high streaming load. When copying the pool off of it, I had to throttle the I/O to keep it reliable. No data loss though. Scrubs are intensive. They will IMO provoke failure in drives sooner than not doing them. But they're the kind of failures you want to bring forward if you can afford the replacements (and often the drives are under warranty anyway). If you don't scrub, eventually you generally start seeing one of two things: delays in reads and writes because drive error recovery is reading and rereading to recover data; or, if you have that disk behaviour disabled via firmware flags (and you should, unless you're reslivering and on your last disk of redundancy), you see zfs kicking a drive out of the pool during normal operations. If I start seeing unrecoverable errors, or a drive dropping out of the pool, I'll disable scrubs if I don't have a spare drive on hand to start mirroring straight away. But it's better to have the spares. At least two, because often a second drive shows weakness during resilver. There is a specific failure mode that scrubs defend against: silent disk corruption that only shows up when you read a file, but for files you almost never read. This is a pretty rare occurrence - it's never happened to me in about 50 drives worth of pools over 15 years or so. The way I think about this is, how is it actionable? If it's not a failing disk, you need to check your backups. And thus your scrub interval should be tied to your backup retention.

nubinetwork 9mo ago

Total pool size and speed. Less data scrubs faster, as do faster disks or disk topology (a 3 way stripe of nvme will scrub faster than a single sata ssd) For what its worth, I scrub daily mostly because I can. It's completely overkill, but if it only takes half an hour, then it can run in the middle of the night while I'm sleeping.

ssl-3 9mo ago

The cost of a scrub is just a flurry of disk reads and a reduction in performance during a scrub. If this cost is affordable on a daily basis, then do a scrub daily. If it's only affordable less often, then do it less often. (Whatever the case: It's not like a scrub causes any harm to the hardware or the data. It can run as frequently as you elect to tolerate.)

agapon 9mo ago
With HDDs, it's also mechanical wear and increased chance of a failure. SSDs are not fully immune to increased load either.
- ssl-3 9mo ago
  Is there any evidence that suggests that reading from a hard drive (instead of it just spinning idle) increases physical wear in any meaningful way? Likewise, is there any evidence of this for solid-state storage?
  - rcxdude 9mo ago
    Yes. Hard drives have published "Annualized Workload Rate" ratings, which are in TB/year, and the manufacturers state there is no difference between reads and writes for the purpose of this rating. (https://www.toshiba-storage.com/trends-technology/mttf-what-hard-drive-reliability-really-means/ https://www.toshiba-storage.com/trends-technology/mttf-what-...) For SSDs, writes matter a lot more. Reads may increase the temperature of the drive, so they'll have some effect, but I don't think I've seen a read endurance rating for an SSD.
  - digiown 9mo ago
    Reading from it requires the read head to move, as opposed to spinning idle where the heads are parked on the side. Moving parts generally wear out over time.

toast0 9mo ago

Once a month seems like a reasonable rule of thumb. But you're balancing the cost of the scrub vs the benefit of learning about a problem as soon as possible. A scrub does a lot of I/O and a fair amount of computing. The scrub load competes with your application load and depending on the size of your disk(s) and their read bandwidth, it may take quite some time to do the scrub. There's even maybe some potential that the read load could push a weak drive over the edge to failure. On my personal servers, application load is nearly meaningless, so I do an about monthly scrub from cron which I think will only scrub one zpool at a time per machine, which seems reasonable enough to me. I run relatively large spinning disks, so if I scrubbed on a daily basis, the drives would spend most of the day scrubbing and that doesn't seem reasonable. I haven't run ZFS in a work environment... I'd have to really consider how the read load impacted the production load and if scrubbing with limits to reduce production impact would complete in a reasonable amount of time... I've run some systems that are essentially alwayd busy and if a scrub would take several months, I'd probably only scrub when other systems indicate a problem and I can take the machine out of rotation to examine it. If I had very high reliability needs or a long time to get replacement drives, I might scrub more often? If I was worried about power consumption, I might scrub less often (and also let my servers and drives go into standby). The article's recommendation to scan at least once every 4 months seems pretty reasonable, although if you have seriously offline disks, maybe once a year is more approachable. I don't think I'd push beyond that, lots of things don't like to sit for a year and then turn on correctly.

atmosx 9mo ago

Once a month is fine ("/etc/cron.monthly/zfs-scrub"): #!/bin/bash # # ZFS scrub script for monthly maintenance # Place in /etc/cron.monthly/zfs-scrub POOL="storage" TAG="zfs-scrub" # Log start logger -t "$TAG" -p user.notice "Starting ZFS scrub on pool: $POOL" # Run the scrub if /sbin/zpool scrub "$POOL"; then logger -t "$TAG" -p user.notice "ZFS scrub initiated successfully on pool: $POOL" else logger -t "$TAG" -p user.err "Failed to start ZFS scrub on pool: $POOL" exit 1 fi exit 0

chungy 9mo ago
That script might do with the "-w" parameter passed to scrub. Then "zpool scrub" won't return until the scrub is finished.
k_bx 9mo ago
Didn't know about the logger script, looks nice. Can it wrap the launch of the scrub itself so that it logs like logger too, or do you separately track its stdout/stderr when something happens? update: figured how you can improve that call to add logs to logger
- nubinetwork 9mo ago
  Scrub doesn't log anything by default, you run it and it returns quickly... you have to get the results out of zpool status or through zed.
- atmosx 9mo ago
  The status of the pool is tracked by Prometheus though an ZFS exporter and faults are reported via alert manager & pushover.

kanbankaren 9mo ago

Once a month might be too high because HDDs are rated at ~ 180 TB workload/year. Remember, the workload/year limit includes read & writes and doesn't vary much by capacity, so a 10 TB HDD scrubbed monthly consumes 67% of the workload, let alone any other usage. Scrubbing every quarter is usually sufficient without putting high wear on the HDD.

Hakkin 9mo ago
A scrub only reads allocated space, so in your 10TB example, a scrub would only read whatever portion of that 10TB is actually occupied by data. It's also usually recommended to keep your usage below 80% of the total pool size to avoid performance issues, so the worst case in your scenario would be more like ~53% assuming you follow the 80% rule.
- formerly_proven 9mo ago
  Is the 80% rule real or just passed down across decades like other “x% free” rules? Those waste enormous amounts of resources on modern systems and I kind of doubt ZFS actually needs a dozen terabytes or more of free space in order to not shit the bed. Just like Linux doesn’t actually need >100 GB of free memory to work properly.
  - cornonthecobra 9mo ago
    Speaking strictly about ZFS internal operations, the free space requirement is closer to 5% on current ZFS versions. That allows for CoW and block reallocations in real-world pools. Heavy churn and very large files will increase that margin.
  - barrkel 9mo ago
    In practice you see noticeable degradation of performance for streaming reads of large files written after 85% or so. Files you used to be able to expect to get 500+MB/sec could be down to 50MB/sec. It's fragmentation, and it's fairly scale invariant, in my experience.
  - 9mo ago

barrkel 9mo ago

I scrub once a quarter because scrubs take 11 days to complete. I have 8x 18TB raidz2 pool, and I keep a couple of spare drives on hand so I can start a resilver as soon as an issue crops up. In the past, I've gone for a few years between scrubs. One system had a marginal I/O setup and was unreliable for high streaming load. When copying the pool off of it, I had to throttle the I/O to keep it reliable. No data loss though. Scrubs are intensive. They will IMO provoke failure in drives sooner than not doing them. But they're the kind of failures you want to bring forward if you can afford the replacements (and often the drives are under warranty anyway). If you don't scrub, eventually you generally start seeing one of two things: delays in reads and writes because drive error recovery is reading and rereading to recover data; or, if you have that disk behaviour disabled via firmware flags (and you should, unless you're reslivering and on your last disk of redundancy), you see zfs kicking a drive out of the pool during normal operations. If I start seeing unrecoverable errors, or a drive dropping out of the pool, I'll disable scrubs if I don't have a spare drive on hand to start mirroring straight away. But it's better to have the spares. At least two, because often a second drive shows weakness during resilver. There is a specific failure mode that scrubs defend against: silent disk corruption that only shows up when you read a file, but for files you almost never read. This is a pretty rare occurrence - it's never happened to me in about 50 drives worth of pools over 15 years or so. The way I think about this is, how is it actionable? If it's not a failing disk, you need to check your backups. And thus your scrub interval should be tied to your backup retention.