7 ms·

No. It absolutely cannot be trusted. I buy drives in pairs, and RAID them. If you can't afford to do that with drives the size you need, buy smaller drives and

No. It absolutely cannot be trusted. I buy drives in pairs, and RAID them. If you can't afford to do that with drives the size you need, buy smaller drives and keep less data. It's like being a half-decade behind on the Moore's Law curve. You get used to it and survive.

greenshackle2 6y ago
Do you buy pairs of identical drives? If you want to be extra paranoid, you can use a pair of different models or at least different batches to reduce the risk that they both fail in a short time window.
- cbhl 6y ago
  Latency is worse for non-identical drives, though, no?
  - pixl97 6y ago
    Latency is as bad as the slowest drive operation. This said, even two identical drives can have different performance due to things like error rates or heat differences.
- rbanffy 6y ago
  I had a RAID-5 array of five identical, same batch and close serial numbers. One went bad. I ordered replacement and procurement blocked the purchase until I got it approved by my manager. I warned it was urgent. They delayed it for 3 days. On day 2, the second drive died. We had backups and could replay all db transactions up to the minute. Still, the service stopped for more than 24 hours. It took a week to clear the backlog.
  - jonpurdy 6y ago
    Delayed approval of an urgent 3-figure purchase resulting in a 5-6 figure loss in productivity, customer sat, etc. Great example of penny-wise, pound-foolish!
    - rbanffy 6y ago
      Indeed. Small powers.
    - Spooky23 6y ago
      Buying individual drives to shave a few pennies is pretty foolish to begin with, so why stop there! Make it an OEM problem.
      - rbanffy 6y ago
        In their defense, we had a bunch of drives coming in in a month. It's just that we ran out of spares faster than expected (we had to increase capacity in a couple servers on short notice).
  - driverdan 6y ago
    Hopefully they let you keep at least one new drive around as a backup after that experience. If you're not running RAID6 you should at least have one on the shelf, preferably from a different batch.
    - rbanffy 6y ago
      That was in 2003 or so. IIRC, mdadm didn't have raid6 support back then and the chassis wouldn't support more than 4 devices. Those were the heroic days.
  - pilsetnieks 6y ago
    Honestly though, 1) the array rebuild might take that long time anyway; 2) the stress of the array rebuilding would probably make the other drive fail anyway.
- yellowapple 6y ago
  That's exactly what I do, except I take it a step further and go with different vendors.
mceachen 6y ago
RAID won't save you from bit rot, and after I added bit rot detection in PhotoStructure, I was surprised to see how many of my files on older spindles had succumbed to minor corruption. Using a filesystem that supports data scrubbing doesn't require an "enterprise class" budget anymore. After fielding this question from several of my beta users, I wrote it up the why's and how's on my blog: https://photostructure.com/faq/how-do-i-safely-store-files/ https://photostructure.com/faq/how-do-i-safely-store-files/
- skoskie 6y ago
  You are Mr. Photostructure? Cool. Thanks so much for making an awesome piece of software.
  - mceachen 6y ago
    Thanks! BTW, I just released v0.8 last night: make sure you check it out, it's quite an upgrade.
- ksec 6y ago
  I do wish there are Consumer NAS that has all these built in and does it automatically. Btrfs is not available on All Synology Model. And I am waiting for QNAP's ZFS based OS. It is god damn annoying.
  - pronoiac 6y ago
    Huh? My Synology NAS runs btrfs.
    - ksec 6y ago
      Sorry. I missed the word on All Synology Model.
- ars 6y ago
  > RAID won't save you from bit rot Sure it can. You use mdadm to schedule a monthly full scan of the RAID array, comparing both disks. It's the default in Debian. You should also use smartmontools to schedule weekly long self tests of the disk, and nightly short self tests. That way you find out about problems when they are still small.
deelowe 6y ago
With larger drive sizes and failures tending to follow a rather normal distribution w.r.t. mtbf, raid is becoming less of a safeguard these days. There are some good calculators online that will do the math for you, but at these sizes, you're very likely to see the second and even third redundant drive fail before the rebuild is complete.
- StillBored 6y ago
  What your saying is a falsehood, and a persistent rumor started by a bad paper and easily disproved. The individual disk capacity has nothing to do with the drive failure rate. First, while RAID isn't a "safeguard", the idea that there is a high probability of raid failure during a rebuild has been repeatedly proven to be false as long as proper data hygiene is maintained. For the most part the chance of a second disk falling out during the rebuild is exactly the same as any other day regardless of capacity. Put another way, if your scrubbing (or just accessing the data) frequently enough to avoid having silent bad sectors the idea that there is a high probability of a drive loss during the day or two it takes to rebuild an array would also result in drive losses on a nearly daily basis. So unless your constructing a single RAID5 array out of ten thousand plus disks, if the MTBF numbers are anywhere close to realistic, drives in a single array aren't going to be failing at a rate high enough that the chance of a double fault not caused by something symptomatic across the entire line of drives (say a firmware that wipes the drive after X power on hours, in which case no level of redundancy will save you) is a real worry. So, the rumor is based on the idea that your increasing the RAID rebuild time from a couple hours to days/weeks on end due to the larger capacity. But this is also false, because as density increases so has the sequential throughput of the drives. AKA comparing a ~ten year old 4TB drive doing ~100MB/sec vs a modern 8TB drive doing 200MB/sec. There are some non-linearities due to the number of platters and the rate of TPI vs BPI increases but generally platter count is somewhat constant and TPI and BPI are both increasing. Now if your hitting internal RAID limits because your running modern drives on old controllers which cannot rebuild at the drives peak throughput that is a different problem. Bottom line, your doing it wrong if RAID5/6 isn't increasing both actual as well as theoretical availability. The MTBF numbers are published to give a baseline on what the expected failure rates are.

greenshackle2 6y ago

Do you buy pairs of identical drives? If you want to be extra paranoid, you can use a pair of different models or at least different batches to reduce the risk that they both fail in a short time window.

cbhl 6y ago
Latency is worse for non-identical drives, though, no?
- pixl97 6y ago
  Latency is as bad as the slowest drive operation. This said, even two identical drives can have different performance due to things like error rates or heat differences.
rbanffy 6y ago
I had a RAID-5 array of five identical, same batch and close serial numbers. One went bad. I ordered replacement and procurement blocked the purchase until I got it approved by my manager. I warned it was urgent. They delayed it for 3 days. On day 2, the second drive died. We had backups and could replay all db transactions up to the minute. Still, the service stopped for more than 24 hours. It took a week to clear the backlog.
- jonpurdy 6y ago
  Delayed approval of an urgent 3-figure purchase resulting in a 5-6 figure loss in productivity, customer sat, etc. Great example of penny-wise, pound-foolish!
  - rbanffy 6y ago
    Indeed. Small powers.
  - Spooky23 6y ago
    Buying individual drives to shave a few pennies is pretty foolish to begin with, so why stop there! Make it an OEM problem.
    - rbanffy 6y ago
      In their defense, we had a bunch of drives coming in in a month. It's just that we ran out of spares faster than expected (we had to increase capacity in a couple servers on short notice).
- driverdan 6y ago
  Hopefully they let you keep at least one new drive around as a backup after that experience. If you're not running RAID6 you should at least have one on the shelf, preferably from a different batch.
  - rbanffy 6y ago
    That was in 2003 or so. IIRC, mdadm didn't have raid6 support back then and the chassis wouldn't support more than 4 devices. Those were the heroic days.
- pilsetnieks 6y ago
  Honestly though, 1) the array rebuild might take that long time anyway; 2) the stress of the array rebuilding would probably make the other drive fail anyway.
yellowapple 6y ago
That's exactly what I do, except I take it a step further and go with different vendors.

mceachen 6y ago

RAID won't save you from bit rot, and after I added bit rot detection in PhotoStructure, I was surprised to see how many of my files on older spindles had succumbed to minor corruption. Using a filesystem that supports data scrubbing doesn't require an "enterprise class" budget anymore. After fielding this question from several of my beta users, I wrote it up the why's and how's on my blog: https://photostructure.com/faq/how-do-i-safely-store-files/ https://photostructure.com/faq/how-do-i-safely-store-files/

skoskie 6y ago
You are Mr. Photostructure? Cool. Thanks so much for making an awesome piece of software.
- mceachen 6y ago
  Thanks! BTW, I just released v0.8 last night: make sure you check it out, it's quite an upgrade.
ksec 6y ago
I do wish there are Consumer NAS that has all these built in and does it automatically. Btrfs is not available on All Synology Model. And I am waiting for QNAP's ZFS based OS. It is god damn annoying.
- pronoiac 6y ago
  Huh? My Synology NAS runs btrfs.
  - ksec 6y ago
    Sorry. I missed the word on All Synology Model.
ars 6y ago
> RAID won't save you from bit rot Sure it can. You use mdadm to schedule a monthly full scan of the RAID array, comparing both disks. It's the default in Debian. You should also use smartmontools to schedule weekly long self tests of the disk, and nightly short self tests. That way you find out about problems when they are still small.

deelowe 6y ago

With larger drive sizes and failures tending to follow a rather normal distribution w.r.t. mtbf, raid is becoming less of a safeguard these days. There are some good calculators online that will do the math for you, but at these sizes, you're very likely to see the second and even third redundant drive fail before the rebuild is complete.

StillBored 6y ago
What your saying is a falsehood, and a persistent rumor started by a bad paper and easily disproved. The individual disk capacity has nothing to do with the drive failure rate. First, while RAID isn't a "safeguard", the idea that there is a high probability of raid failure during a rebuild has been repeatedly proven to be false as long as proper data hygiene is maintained. For the most part the chance of a second disk falling out during the rebuild is exactly the same as any other day regardless of capacity. Put another way, if your scrubbing (or just accessing the data) frequently enough to avoid having silent bad sectors the idea that there is a high probability of a drive loss during the day or two it takes to rebuild an array would also result in drive losses on a nearly daily basis. So unless your constructing a single RAID5 array out of ten thousand plus disks, if the MTBF numbers are anywhere close to realistic, drives in a single array aren't going to be failing at a rate high enough that the chance of a double fault not caused by something symptomatic across the entire line of drives (say a firmware that wipes the drive after X power on hours, in which case no level of redundancy will save you) is a real worry. So, the rumor is based on the idea that your increasing the RAID rebuild time from a couple hours to days/weeks on end due to the larger capacity. But this is also false, because as density increases so has the sequential throughput of the drives. AKA comparing a ~ten year old 4TB drive doing ~100MB/sec vs a modern 8TB drive doing 200MB/sec. There are some non-linearities due to the number of platters and the rate of TPI vs BPI increases but generally platter count is somewhat constant and TPI and BPI are both increasing. Now if your hitting internal RAID limits because your running modern drives on old controllers which cannot rebuild at the drives peak throughput that is a different problem. Bottom line, your doing it wrong if RAID5/6 isn't increasing both actual as well as theoretical availability. The MTBF numbers are published to give a baseline on what the expected failure rates are.