3 ms·

I think there's a difference between the wear and tear in a personal backup server and in a Netflix cache server. We manage several storage servers which see i

I think there's a difference between the wear and tear in a personal backup server and in a Netflix cache server. We manage several storage servers which see intensive usage, and usually at around 5 years we start to see disk failures.

dachryn 4y ago
its fair to assume netflix is mostly doing reads and not writes, and also fairly sequential. Thats very good for longevity of the disks
- michaelt 4y ago
  True - but presumably they've been spinning and doing seeks pretty much 24/7. It's not like backblaze/coldline where the drives are practically powered down most of the time. And of course, 36x the drives means 36x the drive failures - and even if you avoid losing data, you've still got the chore of swapping each failed drive.
  - dijit 4y ago
    Constantly spinning is better for the motor than starting and stopping.
  - Maursault 4y ago
    > And of course, 36x the drives means 36x the drive failures I think you must mean 36x the chance of drive failure. Drive failure probability per year is published at around 1% though practically as high as 10%. 36 drives means there's somewhere between a 30.35% (1 - 0.99^36) and 97.7% (1 - 0.9^36) chance per year of a single drive failing.
- justsomehnguy 4y ago
  > and also fairly sequential ... No. When you hit hundreds of concurrent operations there is no such thing as sequential. Also there is no need for sequential reads because a stripe of 7-15 disks would give you much more than streaming bitrate needs.
olavgg 4y ago
I think that is a good point, however MTBF is usually a lot higher on enterprise grade hard drives. My hard drives are consumer grade.
Hamuko 4y ago
>We manage several storage servers which see intensive usage, and usually at around 5 years we start to see disk failures. Which begs the question of how old the drives actually are. If intensive usage would result in lots of disk failures after 5 years, then a 9-year-old server would surely have a bunch of new(er) disks inside it? Or do they just reduce the amount of cache space they have with every failure?
- js2 4y ago
  The latter. The case design clearly doesn’t lend itself to replacing drives. https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_netflix_cache_server/ittpppw/ https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_...
- gjulianm 4y ago
  Of course, the timing depends on the usage pattern. In our case we write a lot to the disks, which means they fail faster. Time to failure also depends on manufacturer, batch, server conditions... But in general I wouldn't trust too much that have been continuously running under a significant load (I don't think a Netflix cache server had a lot of idle time) for so many years.

dachryn 4y ago

its fair to assume netflix is mostly doing reads and not writes, and also fairly sequential. Thats very good for longevity of the disks

michaelt 4y ago
True - but presumably they've been spinning and doing seeks pretty much 24/7. It's not like backblaze/coldline where the drives are practically powered down most of the time. And of course, 36x the drives means 36x the drive failures - and even if you avoid losing data, you've still got the chore of swapping each failed drive.
- dijit 4y ago
  Constantly spinning is better for the motor than starting and stopping.
- Maursault 4y ago
  > And of course, 36x the drives means 36x the drive failures I think you must mean 36x the chance of drive failure. Drive failure probability per year is published at around 1% though practically as high as 10%. 36 drives means there's somewhere between a 30.35% (1 - 0.99^36) and 97.7% (1 - 0.9^36) chance per year of a single drive failing.
justsomehnguy 4y ago
> and also fairly sequential ... No. When you hit hundreds of concurrent operations there is no such thing as sequential. Also there is no need for sequential reads because a stripe of 7-15 disks would give you much more than streaming bitrate needs.

olavgg 4y ago

I think that is a good point, however MTBF is usually a lot higher on enterprise grade hard drives. My hard drives are consumer grade.

Hamuko 4y ago

>We manage several storage servers which see intensive usage, and usually at around 5 years we start to see disk failures. Which begs the question of how old the drives actually are. If intensive usage would result in lots of disk failures after 5 years, then a 9-year-old server would surely have a bunch of new(er) disks inside it? Or do they just reduce the amount of cache space they have with every failure?

js2 4y ago
The latter. The case design clearly doesn’t lend itself to replacing drives. https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_netflix_cache_server/ittpppw/ https://old.reddit.com/r/homelab/comments/ydollm/so_i_got_a_...
gjulianm 4y ago
Of course, the timing depends on the usage pattern. In our case we write a lot to the disks, which means they fail faster. Time to failure also depends on manufacturer, batch, server conditions... But in general I wouldn't trust too much that have been continuously running under a significant load (I don't think a Netflix cache server had a lot of idle time) for so many years.