4 ms·

I was about to buy 3x 12TB Toshiba for Deep Learning datasets; now I need to reconsider... Does anyone know what are the current reliability stats for >10TB dri

by bitL 8y ago

I was about to buy 3x 12TB Toshiba for Deep Learning datasets; now I need to reconsider... Does anyone know what are the current reliability stats for >10TB drives? My old 6x 4TB HGST in NAS are running without a single problem for the past 3 years...

DavidVoid 8y ago
The best resource for that would probably be Backblaze's quarterly hard drive stats. Here are the ones for Q3 2018: https://www.backblaze.com/blog/2018-hard-drive-failure-rates/ https://www.backblaze.com/blog/2018-hard-drive-failure-rates... Scroll down a bit and you'll see the annualized failure rates (AFR). The 10TB and 12TB ones seem to be pretty excellent.
- mnw21cam 8y ago
  The counter-point to this is that since Backblaze uses consumer drives, they probably won't ever test these new drives, because they are enterprise.
  - DavidVoid 8y ago
    Would the 10TB Seagate ST10000NM0086 (of which they have 1,220 drives) not count as enterprise ones? Or the 12TB HGST HUH721212ALN604 for that matter.
wazoox 8y ago
Excellent. All recent big disks over 8 TB from all vendors have excellent reliability. HGST/WD Helium drives are particularly good; I configured and installed many hundreds from 6 to 12 TB in the past 4 years, and not a single one failed yet.
rootw0rm 8y ago
before my current crop of 8TB WD reds, I ran Toshiba enterprise drives and they were extremely reliable for me. none of them failed in a 24x7 hardware raid6 environment after a few years. only replaced them to upgrade capacity.
- bitL 8y ago
  I've read that Toshiba is based on old HGST process and those drives were extremely reliable, which is why I wanted to get them for the new Deep Learning workstation.
  - linker3000 8y ago
    Correct (ex HGST employee). As part of one of the acquisitions/mergers, HGST divested a 3.5" production line to Toshiba. https://www.tomshardware.com/news/wd-toshiba-hdd-hard-drive,14858.html https://www.tomshardware.com/news/wd-toshiba-hdd-hard-drive,...
worldexplorer 8y ago
How do people use such large HDDs when internet download speeds are still low as compared to downloading directly on cloud services (like aws)?
- detaro 8y ago
  What do you consider low? On a 100Mbit/s connection you can fill a TB in a day or two, and those are somewhat common. And of course some people actually create the data themselves and don't download it.
- aidenn0 8y ago
  My dad and I created backups locally, mailed each other hard-drives, and we just do a weekly rsync. Storage is large, but network traffic is relatively low.
  - astrodust 8y ago
    Having a "backup buddy" you can swap drives with once in a while is never a bad idea. Encrypted backup drives can save your bacon if you're ever caught in a bad situation.
- bitL 8y ago
  If you are doing serious Deep/Machine Learning you often get massive datasets, you might need to pre-process substantial portions of them for different models you want to try, you might want to do custom models that are trained e.g. on 4k/8k video footage (think DLSS, self-driving car or drone footage); space gets exhausted pretty quickly. You might also want multiple independent drives, as you might do this all in parallel and it would be too slow to do it on a single drive at the same time, and even RAID with two drives would have a large performance penalty for seeks.
- peterburkimsher 8y ago
  Internet speeds are slow in some places, such as France (specifically the Pays de Gex near Geneva, where my parents live). My dad uses iCloud, but he drives to CERN to upload (he just retired). I have 18 TB: 5 TB Seagate (x2), 4 TB WesternDigital, 2 TB WesternDigital, and 2 TB internal. Backups take the most space - I fix laptops for friends from church, and they don't back up but still want their files to be safe. I had to shuffle some files around to free up 650 GB for a recent repair, mostly photos & videos. Virtual machines use a lot of space too. I made VMWare Fusion images of every Mac OS version 10.5-10.13, Windows 95, 98, 2000, XP, 7, and 10, in several languages ( https://peterburk.github.com/i2018n https://peterburk.github.com/i2018n ), and some Linux distros. Another 1 TB is a dataset of Chinese characters from a machine learning project of mine ( https://blog.usejournal.com/making-of-a-chinese-characters-dataset-92d4065cc7cc https://blog.usejournal.com/making-of-a-chinese-characters-d... ). Music, mostly from repaired iPods back in high school, accounts for a lot as well. There's some movies too, though I missed a chance to get 2 TB from a friend because I didn't have enough space at the time. If I upload those, even those that I legally ripped from CDs & DVDs, I'm worried that it'll trigger content filters. For these, local disks are more useful than cloud services in my opinion.

by bitL 8y ago

DavidVoid 8y ago

The best resource for that would probably be Backblaze's quarterly hard drive stats. Here are the ones for Q3 2018: https://www.backblaze.com/blog/2018-hard-drive-failure-rates/ https://www.backblaze.com/blog/2018-hard-drive-failure-rates... Scroll down a bit and you'll see the annualized failure rates (AFR). The 10TB and 12TB ones seem to be pretty excellent.

mnw21cam 8y ago
The counter-point to this is that since Backblaze uses consumer drives, they probably won't ever test these new drives, because they are enterprise.
- DavidVoid 8y ago
  Would the 10TB Seagate ST10000NM0086 (of which they have 1,220 drives) not count as enterprise ones? Or the 12TB HGST HUH721212ALN604 for that matter.

wazoox 8y ago

Excellent. All recent big disks over 8 TB from all vendors have excellent reliability. HGST/WD Helium drives are particularly good; I configured and installed many hundreds from 6 to 12 TB in the past 4 years, and not a single one failed yet.

rootw0rm 8y ago

before my current crop of 8TB WD reds, I ran Toshiba enterprise drives and they were extremely reliable for me. none of them failed in a 24x7 hardware raid6 environment after a few years. only replaced them to upgrade capacity.

bitL 8y ago
I've read that Toshiba is based on old HGST process and those drives were extremely reliable, which is why I wanted to get them for the new Deep Learning workstation.
- linker3000 8y ago
  Correct (ex HGST employee). As part of one of the acquisitions/mergers, HGST divested a 3.5" production line to Toshiba. https://www.tomshardware.com/news/wd-toshiba-hdd-hard-drive,14858.html https://www.tomshardware.com/news/wd-toshiba-hdd-hard-drive,...

worldexplorer 8y ago

How do people use such large HDDs when internet download speeds are still low as compared to downloading directly on cloud services (like aws)?

detaro 8y ago
What do you consider low? On a 100Mbit/s connection you can fill a TB in a day or two, and those are somewhat common. And of course some people actually create the data themselves and don't download it.
aidenn0 8y ago
My dad and I created backups locally, mailed each other hard-drives, and we just do a weekly rsync. Storage is large, but network traffic is relatively low.
- astrodust 8y ago
  Having a "backup buddy" you can swap drives with once in a while is never a bad idea. Encrypted backup drives can save your bacon if you're ever caught in a bad situation.
bitL 8y ago
If you are doing serious Deep/Machine Learning you often get massive datasets, you might need to pre-process substantial portions of them for different models you want to try, you might want to do custom models that are trained e.g. on 4k/8k video footage (think DLSS, self-driving car or drone footage); space gets exhausted pretty quickly. You might also want multiple independent drives, as you might do this all in parallel and it would be too slow to do it on a single drive at the same time, and even RAID with two drives would have a large performance penalty for seeks.
peterburkimsher 8y ago
Internet speeds are slow in some places, such as France (specifically the Pays de Gex near Geneva, where my parents live). My dad uses iCloud, but he drives to CERN to upload (he just retired). I have 18 TB: 5 TB Seagate (x2), 4 TB WesternDigital, 2 TB WesternDigital, and 2 TB internal. Backups take the most space - I fix laptops for friends from church, and they don't back up but still want their files to be safe. I had to shuffle some files around to free up 650 GB for a recent repair, mostly photos & videos. Virtual machines use a lot of space too. I made VMWare Fusion images of every Mac OS version 10.5-10.13, Windows 95, 98, 2000, XP, 7, and 10, in several languages ( https://peterburk.github.com/i2018n https://peterburk.github.com/i2018n ), and some Linux distros. Another 1 TB is a dataset of Chinese characters from a machine learning project of mine ( https://blog.usejournal.com/making-of-a-chinese-characters-dataset-92d4065cc7cc https://blog.usejournal.com/making-of-a-chinese-characters-d... ). Music, mostly from repaired iPods back in high school, accounts for a lot as well. There's some movies too, though I missed a chance to get 2 TB from a friend because I didn't have enough space at the time. If I upload those, even those that I legally ripped from CDs & DVDs, I'm worried that it'll trigger content filters. For these, local disks are more useful than cloud services in my opinion.