6 ms·

I would caution against this. If you need the data to truly be protected, don't build it yourself. Here are some questions and issues that can come up: * Usi

I would caution against this. If you need the data to truly be protected, don't build it yourself. Here are some questions and issues that can come up: * Using desktop drives, when 24 are mounted in a single chassis, the vibration from those drives will cause a higher rate of failure. Desktop drives were not designed for this purpose and are not made to handle the vibration and heat created by 23 other drives in the same chassis. * When a single drive in your 24 drive chassis fails, how do you know which one it is? On an enterprise storage array, you have the ability to flash an LED so you know which drive to replace, or you have a direct mapping (shelf 2, bay 16) in the error message. If you built it yourself, how do you know which physical drive is /dev/sdzg? * Purchasing a large number of drives from the same batch means they are very likely to start failing around the same date. Can you handle a double disk failure without losing data? How about a triple disk failure? Do you have hot spares capability? * Are you scrubbing your data regularly to make sure those blocks on disk aren't getting silently corrupted over time? If you never read your data or scrub your entire disk regularly, the only time you'll find out if you've lost data is during a RAID rebuild, when you'll get read errors and have complete data loss (unable to read parity from another drive, double disk failure, complete loss of RAID). * Are you able to use RAID 6 or additional parity disk technology to protect against double disk failures? What if I want 3 parity disks to prevent against a triple disk failure? All of these considerations are things that will come up when you're designing a storage system. These are things that enterprise storage vendors spend months with engineers designing. Failure modes should be considered. In my experience with hand built PC server storage arrays, the time you are most likely to lose data is when you have a disk failure, and are rebuilding parity from another disk. If you have many disks from the same batch, a rebuild can stress the disks more than normal, and cause a second disk to fail. This is why RAID 6 is becoming more and more popular. I would caution you to consider that cheap hand built storage usually ends up expensive in the long run, especially when you lose data that you thought was protected.

wazoox 16y ago
> don't build it yourself. Of course not. I've built several petabytes worth of storage systems in the past 15 years, OTOH. > Using desktop drives, when 24 are mounted in a single chassis, the vibration from those drives will cause a higher rate of failure. Absolutely. However this is not always directly applicable to all desktop drives. In fact I'm going to reveal a secret : physically, desktop SATA drives are exactly the same as professional grade 24/24h operation-guaranteed SATA drives. When they fall off the assembly line, they're tested for reliability and performance. Over a certain threshold, a drive is deemed professional and receive the pro firmware; under the bar, it's a desktop drive. What is the most important difference between the desktop vs pro firmware? The desktop drive is supposed to be used alone. In case of an error, it will retry, retry and retry again endlessly to ecc-correct the error. OTOH, the pro drives "knows" it's part of a RAID array; at the slightest fault, it will simply cry "I'm failed" and give up. So in case of a transient error, your IOs won't be blocked. There are some other differences depending upon the maker: for instance all Hitachi GST drives carry an accelerometer, but the desktop drive firmware don't know how to use it to compensate for low-frequency vibrations. WD desktop drives have particular settings that make them extremely dangerous to use in RAID arrays. Note that the "pro/consumer" threshold can be moved to reach some volume threshold regardless of the actual drive quality. This happened, I've been bitten. Anyway, prefer the pro drives even if there 50% more expensive. It will save you some headaches. > When a single drive in your 24 drive chassis fails, how do you know which one it is? First, I suppose the maker (in my case, my team), carefully checked that all drives are properly labeled (so that drive 1 is labeled 1, etc); second, decent RAID controllers can flash a drive LED (Adaptec 5xx5 or 6xx5 series come to mind). > Can you handle a double disk failure without losing data? With drives of 1 TB or more, RAID-5 is definitely out and has been so for several years. RAID-6, RAID-6, RAID-6 is the way to go. > These are things that enterprise storage vendors spend months with engineers designing. E-xact-ly. BTW I'm an enterprise storage vendor. However I'm really too far from Louisville to be able to ship there and support it. I just mentioned what is possible; don't forget that nowadays, when you're buying an SGI (or many other big names) storage system you're buying Supermicro hardware.
- originalgeek 16y ago
  > In fact I'm going to reveal a secret : physically, desktop SATA drives are exactly the same as professional grade 24/24h operation-guaranteed SATA drives. When they fall off the assembly line, they're tested for reliability and performance. Over a certain threshold, a drive is deemed professional and receive the pro firmware; under the bar, it's a desktop drive. This may be true in some circumstances, however I have observed that enterprise drives from some vendors have a larger heatsink volume than the consumer drives, and also weigh a little more.
  - wazoox 16y ago
    Possibly; however Seagate Barracuda 7200.11 and ES2, and Constellation 1 and 2 TB; Hitachi Deskstar and Ultrastar 1, 2 TB, and Western Digital Caviar Green 2 TB desktop and pro are all physically exactly identical.
moe 16y ago
don't build it yourself Sorry, but what a bunch of FUD. Any seasoned sysadmin can build an array with such low requirements. It's a different story when you need high IOPS or extreme availability - but none of that was asked for here. Wazoox listed the off the shelf parts that you need, from there it's a matter of being able to operate a screwdriver and configuring your favorite RAID controller or soft-raid. Of course you can double-up and buy the feel-good solution from a storage-vendor (DELL MD1000 would be the standard in this range). But in this case it doesn't seem worth to pay a markup for essentially the same hardware.
- phamilton 16y ago
  Yes, but next day support makes all the difference. When I worked as an HPC sysadmin, we saw about 2-3 DIMM failures a week in our 10,000 core cluster. I'm sure we could have gotten the hardware cheaper, even with replacement DIMMs for the 3 years period of service, but Dell made it quite frictionless. A simple web login and form and the new DIMM would show up the next day. We ship the old one back (using their shipping label) and move on. Our time was worth more than we would have saved building it ourselves. 500 TB in 1 or 2 TB drives is a lot of hardware. <complete back of the hand calculation> If we have 250 drives with a 1 000 000 hour mtbf, then (according to an exponential distribution), you've got around ~5% chance of a drive failure each week. Not huge, but worth noting. </ complete back of the hand calculation>
  - moe 16y ago
    Well, a 10k core cluster is a far cry from a 500TB array. The latter fits comfortably into 2-3 racks (if your DC allows the power density). The former sounds like 15+ racks - entirely different ballgame. I'm absolutely not denying that you should have a dedicated admin to look after such a deployment. But by today's standards it's just not a lot of hardware anymore. And definitely not the size where you must spend a fortune on "platinum" support contracts that can easily cost as much (per year) as a fully fleshed out spare shelf... FWIW, I'm personally running a few moderately sized storage clusters (the largest is ~200T, 3x3 MD1000) and we're seeing disk failures at a rate of 2-3 per year.

wazoox 16y ago

> don't build it yourself. Of course not. I've built several petabytes worth of storage systems in the past 15 years, OTOH. > Using desktop drives, when 24 are mounted in a single chassis, the vibration from those drives will cause a higher rate of failure. Absolutely. However this is not always directly applicable to all desktop drives. In fact I'm going to reveal a secret : physically, desktop SATA drives are exactly the same as professional grade 24/24h operation-guaranteed SATA drives. When they fall off the assembly line, they're tested for reliability and performance. Over a certain threshold, a drive is deemed professional and receive the pro firmware; under the bar, it's a desktop drive. What is the most important difference between the desktop vs pro firmware? The desktop drive is supposed to be used alone. In case of an error, it will retry, retry and retry again endlessly to ecc-correct the error. OTOH, the pro drives "knows" it's part of a RAID array; at the slightest fault, it will simply cry "I'm failed" and give up. So in case of a transient error, your IOs won't be blocked. There are some other differences depending upon the maker: for instance all Hitachi GST drives carry an accelerometer, but the desktop drive firmware don't know how to use it to compensate for low-frequency vibrations. WD desktop drives have particular settings that make them extremely dangerous to use in RAID arrays. Note that the "pro/consumer" threshold can be moved to reach some volume threshold regardless of the actual drive quality. This happened, I've been bitten. Anyway, prefer the pro drives even if there 50% more expensive. It will save you some headaches. > When a single drive in your 24 drive chassis fails, how do you know which one it is? First, I suppose the maker (in my case, my team), carefully checked that all drives are properly labeled (so that drive 1 is labeled 1, etc); second, decent RAID controllers can flash a drive LED (Adaptec 5xx5 or 6xx5 series come to mind). > Can you handle a double disk failure without losing data? With drives of 1 TB or more, RAID-5 is definitely out and has been so for several years. RAID-6, RAID-6, RAID-6 is the way to go. > These are things that enterprise storage vendors spend months with engineers designing. E-xact-ly. BTW I'm an enterprise storage vendor. However I'm really too far from Louisville to be able to ship there and support it. I just mentioned what is possible; don't forget that nowadays, when you're buying an SGI (or many other big names) storage system you're buying Supermicro hardware.

originalgeek 16y ago
> In fact I'm going to reveal a secret : physically, desktop SATA drives are exactly the same as professional grade 24/24h operation-guaranteed SATA drives. When they fall off the assembly line, they're tested for reliability and performance. Over a certain threshold, a drive is deemed professional and receive the pro firmware; under the bar, it's a desktop drive. This may be true in some circumstances, however I have observed that enterprise drives from some vendors have a larger heatsink volume than the consumer drives, and also weigh a little more.
- wazoox 16y ago
  Possibly; however Seagate Barracuda 7200.11 and ES2, and Constellation 1 and 2 TB; Hitachi Deskstar and Ultrastar 1, 2 TB, and Western Digital Caviar Green 2 TB desktop and pro are all physically exactly identical.

moe 16y ago

don't build it yourself Sorry, but what a bunch of FUD. Any seasoned sysadmin can build an array with such low requirements. It's a different story when you need high IOPS or extreme availability - but none of that was asked for here. Wazoox listed the off the shelf parts that you need, from there it's a matter of being able to operate a screwdriver and configuring your favorite RAID controller or soft-raid. Of course you can double-up and buy the feel-good solution from a storage-vendor (DELL MD1000 would be the standard in this range). But in this case it doesn't seem worth to pay a markup for essentially the same hardware.

phamilton 16y ago
Yes, but next day support makes all the difference. When I worked as an HPC sysadmin, we saw about 2-3 DIMM failures a week in our 10,000 core cluster. I'm sure we could have gotten the hardware cheaper, even with replacement DIMMs for the 3 years period of service, but Dell made it quite frictionless. A simple web login and form and the new DIMM would show up the next day. We ship the old one back (using their shipping label) and move on. Our time was worth more than we would have saved building it ourselves. 500 TB in 1 or 2 TB drives is a lot of hardware. <complete back of the hand calculation> If we have 250 drives with a 1 000 000 hour mtbf, then (according to an exponential distribution), you've got around ~5% chance of a drive failure each week. Not huge, but worth noting. </ complete back of the hand calculation>
- moe 16y ago
  Well, a 10k core cluster is a far cry from a 500TB array. The latter fits comfortably into 2-3 racks (if your DC allows the power density). The former sounds like 15+ racks - entirely different ballgame. I'm absolutely not denying that you should have a dedicated admin to look after such a deployment. But by today's standards it's just not a lot of hardware anymore. And definitely not the size where you must spend a fortune on "platinum" support contracts that can easily cost as much (per year) as a fully fleshed out spare shelf... FWIW, I'm personally running a few moderately sized storage clusters (the largest is ~200T, 3x3 MD1000) and we're seeing disk failures at a rate of 2-3 per year.