6 ms·

Yeah, with a 9+ year old server, i wouldn't trust those disks, plus the amount of power...

eric-hu 4y ago
Would the hard disks be unreliable too?
Nextgrid 4y ago
With enough redundancy anything can be made reliable. ZFS can do some crazy RAID levels that will pretty much make anything reliable given enough disks and CPU resources. Whether it's a good idea (especially in terms of power usage) is a good question though.
- ilikejam 4y ago
  You'd think so, but failing disks in raidz arrays can cause extreme performance problems, and failing disk electronics can cause lockups on SAS/SATA controllers. Old disks are always risky. Avoid.
olavgg 4y ago
I would love to have this. Old disks are no problem. They will most likely continue to run fine until they no longer are worth it due to either insane electricity prices or 50TB hard drives becomes available for a bargain. My backup server is still running almost 17-18 years old 500GB hard drives. They will be retired soon though, as they consume too much power and lack spindown when not in use.
- throw0101c 4y ago
  > […] and lack spindown when not in use. I was under the impression that constant on-off was what put the most wear on a drive, and that if it is constantly on it would last longer.
  - fluoridation 4y ago
    Yep. In a previous iteration of my file server I killed a WD green drive in about two years because I unwittingly left the head park feature on. It was parking after 8 seconds of inactivity, and in two years it had accumulated like 2.5 million parks.
  - olavgg 4y ago
    Not really. With minimum 50000 cycles, you can spin up and down 28 times every day for 5 years. It may not sound a lot, but in reality it is. I have my NAS set to spindown after 15 minutes idle time and have on average around 1000 cycles every year.
- gjulianm 4y ago
  I think there's a difference between the wear and tear in a personal backup server and in a Netflix cache server. We manage several storage servers which see intensive usage, and usually at around 5 years we start to see disk failures.
  - dachryn 4y ago
    its fair to assume netflix is mostly doing reads and not writes, and also fairly sequential. Thats very good for longevity of the disks
    - michaelt 4y ago
      True - but presumably they've been spinning and doing seeks pretty much 24/7. It's not like backblaze/coldline where the drives are practically powered down most of the time. And of course, 36x the drives means 36x the drive failures - and even if you avoid losing data, you've still got the chore of swapping each failed drive.
      - dijit 4y ago
        Constantly spinning is better for the motor than starting and stopping.
      - Maursault 4y ago
        > And of course, 36x the drives means 36x the drive failures I think you must mean 36x the chance of drive failure. Drive failure probability per year is published at around 1% though practically as high as 10%. 36 drives means there's somewhere between a 30.35% (1 - 0.99^36) and 97.7% (1 - 0.9^36) chance per year of a single drive failing.
    - justsomehnguy 4y ago
      > and also fairly sequential ... No. When you hit hundreds of concurrent operations there is no such thing as sequential. Also there is no need for sequential reads because a stripe of 7-15 disks would give you much more than streaming bitrate needs.
- aasasd 4y ago
  For some reason the make and model of those disks is absent from your comment, must've been lost in the wires.
  - olavgg 4y ago
    They are a mix of Samsung, WD and Seagate. 2 of them are PATA, while the rest is SATA
- dylan604 4y ago
  >or 50TB hard drives becomes available for a bargain. Maybe my old age is showing, but this just frightens me to no end. 1 50TB failure, and you've lost 50TB. Build an array of smaller drives to get to 50TB, and much less catastrophe if 1 drive dies.
  - NicoJuicy 4y ago
    Maintenance costs money too. What if you have to replace 3 times more often because of the smaller drives?
    - dylan604 4y ago
      if it costs some $ to be able to recover is still better than not being able to recover at all because the single point of failure fails
AnIdiotOnTheNet 4y ago
You should never trust any disk, that's why we have backups and redundancy in the first place.
stfp 4y ago
Also: noise. Those things were loud.
winrid 4y ago
Only about 180w average draw from the disks depending on workload
runlevel1 4y ago
According to this article, it consumes about 500W of power: https://www.phoronix.com/news/MTExNDM#:~:text=500w https://www.phoronix.com/news/MTExNDM#:~:text=500w Though I assume that's at typical Netflix workload levels.
js2 4y ago
I buy old SAS/SATA drives that have lots of hours on them. They tend to look like this: https://pastebin.com/raw/E39HuK59 https://pastebin.com/raw/E39HuK59 Run SMART, check the stats, make sure the grown defect list is 0, make sure it's never been over-temp, run a SMART long test, run bad blocks, run another SMART long test. If it passes all those I'd be fine continuing to use it. The HUH728080ALE600 drives in that server[1] idle at 5.1 watts[2], so it's 184 watts just on the drives. I'm guessing at idle the server runs around 300 watts. Which is not great, but it's not terrible. I pay ~ $1/watt/year in NC, so running that server 24x7 would cost me $300 annually. If it were me I'd disconnect at least half the drives to keep as spares. 1. https://imgur.com/a/zb6Mqty https://imgur.com/a/zb6Mqty 2. https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/data-center-drives/ultrastar-sas-series/data-sheet-ultrastar-he8.pdf https://documents.westerndigital.com/content/dam/doc-library...
- dylan604 4y ago
  Cold spares can sometimes disappoint. Lots of techs preferred hot spares not just for immediate availability for the rebuild, but just because the spin-up was just as likely to kill the drive as other failure options. With hot-spares, you could rotate the drives by migrating data around like the old farmers partitioning their fields to let sections rest.
- icedchai 4y ago
  I use HGST / old Hitachi drives in most of my servers, typically either 2 TB or 3 TB, though I do have some older 1 TBs still kicking (over 10 years, running 24x7.) In all the years, I've only had 1 fail. I just checked and one of them has over 91000 hours!

Would the hard disks be unreliable too?

With enough redundancy anything can be made reliable. ZFS can do some crazy RAID levels that will pretty much make anything reliable given enough disks and CPU resources. Whether it's a good idea (especially in terms of power usage) is a good question though.

ilikejam 4y ago
You'd think so, but failing disks in raidz arrays can cause extreme performance problems, and failing disk electronics can cause lockups on SAS/SATA controllers. Old disks are always risky. Avoid.

olavgg 4y ago

I would love to have this. Old disks are no problem. They will most likely continue to run fine until they no longer are worth it due to either insane electricity prices or 50TB hard drives becomes available for a bargain. My backup server is still running almost 17-18 years old 500GB hard drives. They will be retired soon though, as they consume too much power and lack spindown when not in use.

throw0101c 4y ago
> […] and lack spindown when not in use. I was under the impression that constant on-off was what put the most wear on a drive, and that if it is constantly on it would last longer.
- fluoridation 4y ago
  Yep. In a previous iteration of my file server I killed a WD green drive in about two years because I unwittingly left the head park feature on. It was parking after 8 seconds of inactivity, and in two years it had accumulated like 2.5 million parks.
- olavgg 4y ago
  Not really. With minimum 50000 cycles, you can spin up and down 28 times every day for 5 years. It may not sound a lot, but in reality it is. I have my NAS set to spindown after 15 minutes idle time and have on average around 1000 cycles every year.
gjulianm 4y ago
I think there's a difference between the wear and tear in a personal backup server and in a Netflix cache server. We manage several storage servers which see intensive usage, and usually at around 5 years we start to see disk failures.
- dachryn 4y ago
  its fair to assume netflix is mostly doing reads and not writes, and also fairly sequential. Thats very good for longevity of the disks
  - michaelt 4y ago
    True - but presumably they've been spinning and doing seeks pretty much 24/7. It's not like backblaze/coldline where the drives are practically powered down most of the time. And of course, 36x the drives means 36x the drive failures - and even if you avoid losing data, you've still got the chore of swapping each failed drive.
    - dijit 4y ago
      Constantly spinning is better for the motor than starting and stopping.
    - Maursault 4y ago
      > And of course, 36x the drives means 36x the drive failures I think you must mean 36x the chance of drive failure. Drive failure probability per year is published at around 1% though practically as high as 10%. 36 drives means there's somewhere between a 30.35% (1 - 0.99^36) and 97.7% (1 - 0.9^36) chance per year of a single drive failing.
  - justsomehnguy 4y ago
    > and also fairly sequential ... No. When you hit hundreds of concurrent operations there is no such thing as sequential. Also there is no need for sequential reads because a stripe of 7-15 disks would give you much more than streaming bitrate needs.
aasasd 4y ago
For some reason the make and model of those disks is absent from your comment, must've been lost in the wires.
- olavgg 4y ago
  They are a mix of Samsung, WD and Seagate. 2 of them are PATA, while the rest is SATA
dylan604 4y ago
>or 50TB hard drives becomes available for a bargain. Maybe my old age is showing, but this just frightens me to no end. 1 50TB failure, and you've lost 50TB. Build an array of smaller drives to get to 50TB, and much less catastrophe if 1 drive dies.
- NicoJuicy 4y ago
  Maintenance costs money too. What if you have to replace 3 times more often because of the smaller drives?
  - dylan604 4y ago
    if it costs some $ to be able to recover is still better than not being able to recover at all because the single point of failure fails

AnIdiotOnTheNet 4y ago

You should never trust any disk, that's why we have backups and redundancy in the first place.

stfp 4y ago

Also: noise. Those things were loud.

winrid 4y ago

Only about 180w average draw from the disks depending on workload

runlevel1 4y ago

According to this article, it consumes about 500W of power: https://www.phoronix.com/news/MTExNDM#:~:text=500w https://www.phoronix.com/news/MTExNDM#:~:text=500w Though I assume that's at typical Netflix workload levels.

js2 4y ago

I buy old SAS/SATA drives that have lots of hours on them. They tend to look like this: https://pastebin.com/raw/E39HuK59 https://pastebin.com/raw/E39HuK59 Run SMART, check the stats, make sure the grown defect list is 0, make sure it's never been over-temp, run a SMART long test, run bad blocks, run another SMART long test. If it passes all those I'd be fine continuing to use it. The HUH728080ALE600 drives in that server[1] idle at 5.1 watts[2], so it's 184 watts just on the drives. I'm guessing at idle the server runs around 300 watts. Which is not great, but it's not terrible. I pay ~ $1/watt/year in NC, so running that server 24x7 would cost me $300 annually. If it were me I'd disconnect at least half the drives to keep as spares. 1. https://imgur.com/a/zb6Mqty https://imgur.com/a/zb6Mqty 2. https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/data-center-drives/ultrastar-sas-series/data-sheet-ultrastar-he8.pdf https://documents.westerndigital.com/content/dam/doc-library...

dylan604 4y ago
Cold spares can sometimes disappoint. Lots of techs preferred hot spares not just for immediate availability for the rebuild, but just because the spin-up was just as likely to kill the drive as other failure options. With hot-spares, you could rotate the drives by migrating data around like the old farmers partitioning their fields to let sections rest.
icedchai 4y ago
I use HGST / old Hitachi drives in most of my servers, typically either 2 TB or 3 TB, though I do have some older 1 TBs still kicking (over 10 years, running 24x7.) In all the years, I've only had 1 fail. I just checked and one of them has over 91000 hours!