4 ms·

I was responsible for maintaining some large glusterfs clusters built on top of these drives, and they certainly added excitement to the experience. 1) At one

I was responsible for maintaining some large glusterfs clusters built on top of these drives, and they certainly added excitement to the experience. 1) At one point we were experiencing disk failures often enough that we wrote a cronjob to detect drive failures via smartctl and automatically send an email to our hosting company requesting them replace the drive. This saved engineering time, and more importantly reduced time to drive replacement, because 2) On at least one occasion, we had the third drive in a RAID6 array fail before we had rebuilt from the initial failure, leading to loss of the array. We think the increased load of the rebuild increased the chance of subsequent failures. Needless to say recovering from this destroyed all plans (and sleep) for the weekend it happened.

dstick 5y ago
These things always happen in the weekend don't they? When I managed a large network of servers, it seems that all the outages happened during parties and/or the weekend.
- lostlogin 5y ago
  Issues seem to happen as you implement the long desired upgrade/fix/repair too. Things hold on for years then crap out right at the final moment.
  - Operyl 5y ago
    This makes sense though doesn’t it? Rebuilding an array is a very heavy load op, and putting components that have been around for a while under heavy load seems like a good and fast way to expose failing parts.
- spookthesunset 5y ago
  > These things always happen in the weekend don't they? Thats because servers know.... oh they know. Tuesday morning when you get into the office? Naw... that's too easy. Let's shit the bed on Christmas morning at 3:49am. That will let those stupid humans know who is boss around here.
matclayton 5y ago
That serial number sends shivers down my spine. I also used to work on this system, and still do! At one point seagate admitted that the firmware was faulty. It would periodically stop responding under load, and cause the raid controller to think the drive had failed, causing it to remove it from the array. I’ve never seen this posted publicly by them, but I’m fairly sure a revised firmware was offered. We deemed it too risky to upgrade the drives one at a time, so built a new cluster on Western Digital drives.

dstick 5y ago

These things always happen in the weekend don't they? When I managed a large network of servers, it seems that all the outages happened during parties and/or the weekend.

lostlogin 5y ago
Issues seem to happen as you implement the long desired upgrade/fix/repair too. Things hold on for years then crap out right at the final moment.
- Operyl 5y ago
  This makes sense though doesn’t it? Rebuilding an array is a very heavy load op, and putting components that have been around for a while under heavy load seems like a good and fast way to expose failing parts.
spookthesunset 5y ago
> These things always happen in the weekend don't they? Thats because servers know.... oh they know. Tuesday morning when you get into the office? Naw... that's too easy. Let's shit the bed on Christmas morning at 3:49am. That will let those stupid humans know who is boss around here.

matclayton 5y ago

That serial number sends shivers down my spine. I also used to work on this system, and still do! At one point seagate admitted that the firmware was faulty. It would periodically stop responding under load, and cause the raid controller to think the drive had failed, causing it to remove it from the array. I’ve never seen this posted publicly by them, but I’m fairly sure a revised firmware was offered. We deemed it too risky to upgrade the drives one at a time, so built a new cluster on Western Digital drives.