4 ms·

I used to run a small hosting service and we standardized on Ultrastar disks. We would regularly have drives fall out of the RAID arrays, but when we ran badbl

I used to run a small hosting service and we standardized on Ultrastar disks. We would regularly have drives fall out of the RAID arrays, but when we ran badblocks on them to verify they were having problems they would always show up as good. After we put them back in production, they would always be happy from that point. So we started going on the theory that there were some marginal parts of the disk during manufacture, and exercising them would allow them to be remapped to good parts of the disk. Once we started including "badblocks -svw -p 5" in the burnin procedure, our drive failure rates dropped to basically 0. I wonder if backblaze does anything similar to provision drives. Recently I replaced some drives in our old Dell R720 servers (5 years old). They were showing media errors over several months but never went degraded. During the replacement, each server had another drive fail. So I ran badblocks on the RAID arrays over the weekend, just to make sure any other marginal drives or sectors had an opportunity to remap. No other drive failures. Aside: I didn't lose any data, the arrays never went FAIL, but also I had migrated all our services off these machines. Love virtualization!

chrisseaton 7y ago
Is it easy to standardise on anything and then reliably get that product in the future? I would have thought supply would start and stop, prices would go up and down, quality would vary, etc.
- linsomniac 7y ago
  For drives it was relatively easy, though we would have to eventually just use the next bigger size. All of our RAID systems would handle that just fine. Drive quality never really was an issue, even in the "Deathstar" days... The bigger standardizing issue was the system. We use Supermicro barebones, and between memory and CPU changes, we would have to qualify a couple-three new chassis every couple of years. And those chassis were much more expensive than a disk drive. Mostly, Supermicro quality was very good. With the exception of around a year or 18 months where the power supplies they were using all failed roughly a year after we bought them. We had older and newer servers with no problem, but around 30 in the middle all failed.
e40 7y ago
Thanks. I just found that macports has badblocks in the e2fsprogs package. I'm using it right now to scan a 3TB disk I had in a RAID6 array that has had 2 errors in the last 4 years. I'm currently using it as a cold standby, and this will give me more confidence in it! EDIT: the RAID6 array is not on macOS, but Linux. I have a TB2 external enclosure that allows me to easily insert drives on my mac. Hopefully the macOS version is as good as the Linux version.
brianwski 7y ago
Disclaimer: I work at Backblaze. > I wonder if Backblaze does anything similar to provision drives. We do different things for different drives, but ALWAYS have a "burn in" period for new vaults (group of 20 computers that files are Reed-Solomon encoded across) that come online. When I say "different things", when we deployed a vault full of recent Toshiba (?) drives it came up "too slow" and we figured out the OS block size and drive block size wasn't lined up "by default". It "works" but essentially a single write required reading two blocks and then writing two blocks. (sigh) I always wonder how many mis-configured PCs exist in the world with little problems like this quietly slowing down some user who never figures it out.
- VectorLock 7y ago
  >I always wonder how many mis-configured PCs exist in the world with little problems like this quietly slowing down some user who never figures it out. All of them.
  - deleted 7y ago
    [deleted]
- devwastaken 7y ago
  How would one go about finding the right block size?
  - brianwski 7y ago
    > How would one go about finding the right block size? From our IT guy who figured out the problem...... "If it's 4K native, then a multiple of 4K is a good starting point. If it's not then a multiple of 512 is ok. If it's SMR based then it's more complicated and depends on whether it's host managed or drive managed and the number of conventional PMR zones. The second part of the equation is alignment. If there's a partition table or logical volume mapping, then the partition or logical volume's offset needs to be a multiple of the native geometry otherwise the block size doesn't really matter because everything will be misaligned but it might be possible to use math to figure out a block size that cancels out the misalignment." Sooooo..... consumers are hopelessly screwed and better hope Dell or Apple got it correct out of the factory. More info from other IT guys....... "These days I think parted gets it right most of the time, at least for individual spinning drives. SSDs are a bit weird because they tend to not want to share the erase block size, but I think the recent-gen ones tend to have enough firmware magic that it's OK. ... parted also has a flag for alignment type (--align optimal) which is supposed to choose an optimal multiple of physical block size."
vbezhenar 7y ago
I just tried to run that command on by 4TB HDD and 0,05% took 5 minutes, or like 7 days of continuous operation for the entire drive. And I suspect that it's just the fist pass, so 1.5 months of drive scrubbing. Is that expected behaviour?
- linsomniac 7y ago
  That seems awfully slow. If I calculate that correctly, that is around 7MB/sec. Maybe that'd be right for an SMR, but I don't think so (from what little I know of SMR). I'd expect more like 100MB/sec, which would take around a day for one read/write cycle. We started that burn in process when we could get "-p5" run in a day or two, maybe 120GB drives... Towards the end we were reducing it down to "-p1" and that took a week. My 5 year old Dell R720s were running more like 2GB/sec, but that was across a RAID-10 array of 16x 10K drives.
zzzcpan 7y ago
It's not a particularly researched area, but my impression is that HDDs can die prematurely or slow down to a crawl if you let them remap bad blocks. You should manage this in software yourself, don't risk overwriting bad blocks and letting the drive remap them.
- linsomniac 7y ago
  Whaaaa?!? I've never heard this, or heard of anyone doing it, or even can imagine how I would approach this. References?
  - zzzcpan 7y ago
    First time I suspected that problem, I wrote a tiny program that scanned the disk for slow blocks then marked as bad each block and extra megabyte to prevent disk's readahead from hitting bad blocks too. You are probably using a kernel filesystem, some of them let you provide a list of bad blocks to mark and avoid, I haven't tried that though.

chrisseaton 7y ago

Is it easy to standardise on anything and then reliably get that product in the future? I would have thought supply would start and stop, prices would go up and down, quality would vary, etc.

linsomniac 7y ago
For drives it was relatively easy, though we would have to eventually just use the next bigger size. All of our RAID systems would handle that just fine. Drive quality never really was an issue, even in the "Deathstar" days... The bigger standardizing issue was the system. We use Supermicro barebones, and between memory and CPU changes, we would have to qualify a couple-three new chassis every couple of years. And those chassis were much more expensive than a disk drive. Mostly, Supermicro quality was very good. With the exception of around a year or 18 months where the power supplies they were using all failed roughly a year after we bought them. We had older and newer servers with no problem, but around 30 in the middle all failed.

e40 7y ago

Thanks. I just found that macports has badblocks in the e2fsprogs package. I'm using it right now to scan a 3TB disk I had in a RAID6 array that has had 2 errors in the last 4 years. I'm currently using it as a cold standby, and this will give me more confidence in it! EDIT: the RAID6 array is not on macOS, but Linux. I have a TB2 external enclosure that allows me to easily insert drives on my mac. Hopefully the macOS version is as good as the Linux version.

brianwski 7y ago

Disclaimer: I work at Backblaze. > I wonder if Backblaze does anything similar to provision drives. We do different things for different drives, but ALWAYS have a "burn in" period for new vaults (group of 20 computers that files are Reed-Solomon encoded across) that come online. When I say "different things", when we deployed a vault full of recent Toshiba (?) drives it came up "too slow" and we figured out the OS block size and drive block size wasn't lined up "by default". It "works" but essentially a single write required reading two blocks and then writing two blocks. (sigh) I always wonder how many mis-configured PCs exist in the world with little problems like this quietly slowing down some user who never figures it out.

VectorLock 7y ago
>I always wonder how many mis-configured PCs exist in the world with little problems like this quietly slowing down some user who never figures it out. All of them.
- deleted 7y ago
  [deleted]
devwastaken 7y ago
How would one go about finding the right block size?
- brianwski 7y ago
  > How would one go about finding the right block size? From our IT guy who figured out the problem...... "If it's 4K native, then a multiple of 4K is a good starting point. If it's not then a multiple of 512 is ok. If it's SMR based then it's more complicated and depends on whether it's host managed or drive managed and the number of conventional PMR zones. The second part of the equation is alignment. If there's a partition table or logical volume mapping, then the partition or logical volume's offset needs to be a multiple of the native geometry otherwise the block size doesn't really matter because everything will be misaligned but it might be possible to use math to figure out a block size that cancels out the misalignment." Sooooo..... consumers are hopelessly screwed and better hope Dell or Apple got it correct out of the factory. More info from other IT guys....... "These days I think parted gets it right most of the time, at least for individual spinning drives. SSDs are a bit weird because they tend to not want to share the erase block size, but I think the recent-gen ones tend to have enough firmware magic that it's OK. ... parted also has a flag for alignment type (--align optimal) which is supposed to choose an optimal multiple of physical block size."

vbezhenar 7y ago

I just tried to run that command on by 4TB HDD and 0,05% took 5 minutes, or like 7 days of continuous operation for the entire drive. And I suspect that it's just the fist pass, so 1.5 months of drive scrubbing. Is that expected behaviour?

linsomniac 7y ago
That seems awfully slow. If I calculate that correctly, that is around 7MB/sec. Maybe that'd be right for an SMR, but I don't think so (from what little I know of SMR). I'd expect more like 100MB/sec, which would take around a day for one read/write cycle. We started that burn in process when we could get "-p5" run in a day or two, maybe 120GB drives... Towards the end we were reducing it down to "-p1" and that took a week. My 5 year old Dell R720s were running more like 2GB/sec, but that was across a RAID-10 array of 16x 10K drives.

zzzcpan 7y ago

It's not a particularly researched area, but my impression is that HDDs can die prematurely or slow down to a crawl if you let them remap bad blocks. You should manage this in software yourself, don't risk overwriting bad blocks and letting the drive remap them.

linsomniac 7y ago
Whaaaa?!? I've never heard this, or heard of anyone doing it, or even can imagine how I would approach this. References?
- zzzcpan 7y ago
  First time I suspected that problem, I wrote a tiny program that scanned the disk for slow blocks then marked as bad each block and extra megabyte to prevent disk's readahead from hitting bad blocks too. You are probably using a kernel filesystem, some of them let you provide a list of bad blocks to mark and avoid, I haven't tried that though.