5 ms·
Backblaze's environment is abnormal. They do very little to dampen vibration amongst other things, based on their last pod design. That puts drives under stre
by Twirrim 10y ago
Backblaze's environment is abnormal. They do very little to dampen vibration amongst other things, based on their last pod design. That puts drives under stresses that don't reflect what you're likely to see in your environments.
What you're seeing is not the most reliable, but the most reliable under their particular set of bad conditions. The reliability under your circumstances might be wildly different.
Their pod approach even results in worse performance: more vibration results in more effort spent by the drives trying to keep the head positioned in the right places, increasing latencies and decreasing read performance. The higher density they push for with their pods actually compounds this. Drives vibrate. Fans vibrate (chassis, PSU, CPU, the works) . If you don't pad them all appropriately, the vibration of the components ends up causing the entire rack to vibrate. By the time you've got several dozen of drives in the rack, the whole rack will be feeling the effect.
- Symbiote 10y agoDoes an average server have much to dampen vibration? Something like [1]. I occasionally replace drives in similar machines, and I haven't noticed anything that looks like it helps with vibration -- but I don't look very much. [1] http://www.dell.com/us/business/p/poweredge-r530/pd?ref=PD_OC http://www.dell.com/us/business/p/poweredge-r530/pd?ref=PD_O...
- Twirrim 10y agoAverage servers don't tend to have much. Storage servers do tend to do stuff to dampen vibration, but then it's a question of storage density. When you've got something like a backblaze pod that's 60 drives in 4U of space. It'd be more unusual to have 60 drives in total in a rack full of normal servers. Now consider 60 drives / 4U, and you'll have multiple of these storage pods in a rack. It adds up.
- jasode 10y ago>What you're seeing is not the most reliable, but the most reliable under their particular set of bad conditions. The reliability under your circumstances might be wildly different. You do have a valid point but I also wanted to add that my harddrive failures in my gentle home environment matches BackBlaze's more hostile environment. I have 50+ harddrives in various capacities from 2TB to 10TB and my differences from BB include drives being iso-mounted on rubber grommets and they also do not run 24/7. (I only power them up when I need archived files off of them.) Even with those stark operational differences, my harddrive failures exactly match the generalization of Backblaze's data: More failures with WD Green and Seagates especially the 3TB drives. My HGST drives purchased over the last 4 years have zero failures so far.
- gameshot911 10y agoCurious, what are you doing with so much storage at home?
- slewis 10y agoAt Google my colleague developed a simple test that made vibration effects very obvious (this was in 2008). In a server with a bunch of disks, do random reads or writes on all but one disk. On the final disk, do sequential reads. Back then you could get on the order of 100MB/s sequential read speed on the outer edge of a hard drive in ideal conditions. Under this test performance would dive to something like 10MB/s or worse. All of the seeking would cause vibration, which caused misreads on the disk under test. When a bad read happens a hard drive has to wait until the next rotation to try again. We eventually fixed this with better damping.
- deegles 10y agoI think it's been known for a while! Here's an old video of a guy inducing read latency spikes by shouting at some servers: https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4
- Twirrim 10y ago"a guy" in this case being Brendan Gregg, http://www.brendangregg.com/ http://www.brendangregg.com/, one of the prominent experts in server performance!
- sliken 10y agoI had an argument from a user who wanted consumer drives instead of "RAID edition drivers". So I ran a test in a 1U server with 8 or so 15k rpm fans. With consumer drive outside the case, on a bench: http://broadley.org/bill/consumer-no-vibration.png http://broadley.org/bill/consumer-no-vibration.png Inside: http://broadley.org/bill/consumer.png http://broadley.org/bill/consumer.png Server drive inside: http://broadley.org/bill/server.png http://broadley.org/bill/server.png We went with the RAID edition.
- deleted 10y ago[deleted]
- raubitsj 10y ago
- atYevP 10y ago> They do very little to dampen vibration amongst other things, based on their last pod design. Yev from Backblaze -> In Storage Pod 6 we actually did a lot more to work on the dampening (and 5.0 as well, but 6.0 builds on it). The drives are all placed in to guided rows and are held down by individuals lids w/ tension on them. You can take a gander at the latest build here -> https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv...
- vacri 10y agoWhere can I see a public stats breakdown of similar magnitude of your preferred environment?
- userbinator 10y ago"The reliability under your circumstances might be wildly different." No. I've seen this argument raised a few times whenever hard drive failure rates comparing manufacturers has occurred. (Previously on HN: https://news.ycombinator.com/item?id=7119323 https://news.ycombinator.com/item?id=7119323 ) Backblaze puts more stress on their drives than typical users, but a drive which works well in their environment is also very likely to do the same in a less stressful one. This is how accelerated life testing is done. https://en.wikipedia.org/wiki/Accelerated_life_testing https://en.wikipedia.org/wiki/Accelerated_life_testing Their pod approach even results in worse performance Not once in the data presented in the article is performance mentioned. This is not about performance, it's about reliability. It may be anecdotal, but the numbers I see correlate well with my experience and many others' experiences I've heard of. More amusingly, I remember coming across a forum discussing data recovery, which happened to be split by manufacturer, and noticing the Seagate forum had an overwhelmingly large number of topics containing people asking for help with their dead disks relative to the WD, Toshiba, Hitachi, etc., which was disproportionate to their marketshare.
- agumonkey 10y agoThat was my understanding too, but is it sure that there's no counter intuitive factor possible ?
- jackmott 10y agowell, if a vibration free environment led to failures being rare enoguh that the difference was immaterial, that would be good to know. you could save money or opt for better.perf at the same price.
- jackmott 10y agowell, if a vibration free environment led to failures being rare enoguh that the difference was immaterial, that would be good to know. you could save money or opt for better.perf at the same price.
- deleted 10y ago[deleted]