4 ms·

Any idea why a power failure would cause (or reveal) hardware damage? Leading up to Y2K, I remember concerns about spinning hard disks not being able to start

Any idea why a power failure would cause (or reveal) hardware damage? Leading up to Y2K, I remember concerns about spinning hard disks not being able to start up again. And if the power is flaky with spikes and brown-outs, I understand that's a problem. But is either of those relevant to AWS?

kube-system 5y ago
If I were to make some WAGs about things that could generally be described this way: * increased failure rates of drives (or other hardware) due to thermal cycling * cache power failures leading to corrupted data * previously unknown failures in recovery processes (like firmware bugs that might be described as a "hardware problem") * cooling failures leading to hardware failure Most of these are pretty rare issues, but it becomes more probable to happen at least once when you're running freekin' us-east-1
SteveNuts 5y ago
In my experience it's common when there's a full power down of a large data center like this. In 2013 the company I was working at had an unplanned power outage which took out 200 spinning drives, and 18 servers. I think the motors in the drives are fine running at constant speed but don't have the juice to spin up from a dead stop. I remember IBM was renting private planes to get us replacement hardware from around the country (we were a very large customer).
- jbergens 5y ago
  A very long time ago I worked for a medium sized company and there was a power outage. The servers were in a room in our building but they had ups power. The problem was that the ups's were broken/malfunctioning and the servers needed to do a disk check after an unplanned reboot. It took hours before we could work again even though most drives survived. This was also before git so we could not reach the central version control system and could not commit any code. I think all code and workspaces were on servers that was not up yet.
- lamontcg 5y ago
  I worked at Amazon from 2001-2006 and there were a couple of datacenter-wide power outages during my time and there was always some machines that never came back. It is just a numbers game. When you have thousands of servers affected some small fraction just don't survive the forced power cycle.
dahfizz 5y ago
https://news.ycombinator.com/item?id=29666048 https://news.ycombinator.com/item?id=29666048 TLDR if electronics are near-death, a power cycle is likely to kick it over the edge.
michaelrpeskin 5y ago
This might be old lore, but I remember many years ago that someone told me that when a spinning disk has been up to temperature for a long time there's a bit of lubricant that is vaporized and evenly distributed everywhere (like the halides in your halogen headlamps). But when it stops spinning that stuff cools down and gets sticky and ends up sticking the heads to the disk so that it can't get going anymore. Don't know the truth to that, but it seemed like a reasonable explanation at the time.
_3u10 5y ago
Think of it like a car with a starter wearing out, at some point you turn the engine off and it never starts up again, until you replace the starter. Similarly bad batteries, corrosion on terminals, etc. Startup is a much different workload on electronics and in the case of HDDs mechanical devices than running. I have an old fan that will need replacing one day that takes about 15 minutes to reach full speed as the bearings are gone. Everytime I turn it off I wonder if the next time I turn it on it will be time to replace it.
wdfx 5y ago
It's just the numbers. Eventually all the hardware will fail. Some of that hardware will require a specific set of conditions to fail. Such a condition may be current inrush from being powercycled. Given the scale of a data centre and powercycling it all at once, then some will fail due to that condition. Further to that thought, there must be hardware in that centre which is failing and being replaced all the time due to other conditions, we just won't hear about it because it's part of normal maintenance.
emptybottle 5y ago
It can be a number of things, but often times it's the thermal cycle and wear on the components. After letting devices (especially that rarely do this) cool, and then bringing them back up to temperature some percentage of components may fail or operate out of spec. And mechanical parts have similar issue. A motor that's been happily running for months/years may not restart after being stopped. It's also possible that a power fault itself could damage hardware, although in my experience this type of issue is far less common.
- fy20 5y ago
  If you think of it in these terms ICEs are an engineering marvell.
dilyevsky 5y ago
I worked for one of the largest datacenter company in the world and significant portion of boxes would never come up after being powercycled
mgsouth 5y ago
Slamming on the brakes can be hard on the electronics. It causes large, high-speed voltage or current spikes that kill (old, tired) components before the spikes can be clamped. Can be made worse by old electolytic caps that no longer filter effectively. Especially hard on power supply diodes and mosfets. Here's a few references at random: https://vbn.aau.dk/ws/portalfiles/portal/108034396/ESREF2013_ConferenceVersion.pdf https://vbn.aau.dk/ws/portalfiles/portal/108034396/ESREF2013... https://www.infineon.com/dgdl/s30p5.pdf?fileId=5546d462533600a401535748559c3fcd https://www.infineon.com/dgdl/s30p5.pdf?fileId=5546d46253360... https://www.sciencedirect.com/science/article/pii/S0038110102003283 https://www.sciencedirect.com/science/article/pii/S003811010...
- Spivak 5y ago
  Which is why if you realize that your hard drive is failing the worst thing you can do is cut the power [1]. You get your data off that box as fast as possible. [1] Yes yes always have backups, and RAID, and replication something about not being able to count that low. Betcha most'a'ya don't have any of that on your MBP though except for a maybe a backup that lags hours behind your work.
- CoastalCoder 5y ago
  > Slamming on the brakes can be hard on the electronics. It causes large, high-speed voltage or current spikes that kill (old, tired) components before the spikes can be clamped. Suppose that the "slamming on the brakes" here is that the power lines connecting a data center to the grid get physically cut. I can see how that would cause a power spike on the grid side of the cut, but would it really create a spike on the datacenter side of the cut? Or assuming that the wires in a cable aren't cut exactly simultaneously, does it depend on the particular phases of the AC power at the time of each wire in the cable being cut?
  - R0b0t1 5y ago
    Any change in the state of an inductive load causes kickback. If you sever the lines running to a building, the generator side gets the most kick, but even a fully passive device gets a kick from the collapsing field in the wires servicing it. The motors generate appreciable back EMF when suddenly unpowered. Likewise an SMPS is an inductive load. Spikes from power supplies going off is probably what killed the disks, assuming they didn't just decide not to power on. Some disks will run for 10 years but power them off once and they don't come back up.
prpl 5y ago
I’m sure temperature variation has something to do with it in some cases. Most parts, even when cooled, are well above room temperature for years on end, a loss of power can drop them to room temp relatively quickly

kube-system 5y ago

If I were to make some WAGs about things that could generally be described this way: * increased failure rates of drives (or other hardware) due to thermal cycling * cache power failures leading to corrupted data * previously unknown failures in recovery processes (like firmware bugs that might be described as a "hardware problem") * cooling failures leading to hardware failure Most of these are pretty rare issues, but it becomes more probable to happen at least once when you're running freekin' us-east-1

SteveNuts 5y ago

In my experience it's common when there's a full power down of a large data center like this. In 2013 the company I was working at had an unplanned power outage which took out 200 spinning drives, and 18 servers. I think the motors in the drives are fine running at constant speed but don't have the juice to spin up from a dead stop. I remember IBM was renting private planes to get us replacement hardware from around the country (we were a very large customer).

jbergens 5y ago
A very long time ago I worked for a medium sized company and there was a power outage. The servers were in a room in our building but they had ups power. The problem was that the ups's were broken/malfunctioning and the servers needed to do a disk check after an unplanned reboot. It took hours before we could work again even though most drives survived. This was also before git so we could not reach the central version control system and could not commit any code. I think all code and workspaces were on servers that was not up yet.
lamontcg 5y ago
I worked at Amazon from 2001-2006 and there were a couple of datacenter-wide power outages during my time and there was always some machines that never came back. It is just a numbers game. When you have thousands of servers affected some small fraction just don't survive the forced power cycle.

dahfizz 5y ago

https://news.ycombinator.com/item?id=29666048 https://news.ycombinator.com/item?id=29666048 TLDR if electronics are near-death, a power cycle is likely to kick it over the edge.

michaelrpeskin 5y ago

This might be old lore, but I remember many years ago that someone told me that when a spinning disk has been up to temperature for a long time there's a bit of lubricant that is vaporized and evenly distributed everywhere (like the halides in your halogen headlamps). But when it stops spinning that stuff cools down and gets sticky and ends up sticking the heads to the disk so that it can't get going anymore. Don't know the truth to that, but it seemed like a reasonable explanation at the time.

_3u10 5y ago

Think of it like a car with a starter wearing out, at some point you turn the engine off and it never starts up again, until you replace the starter. Similarly bad batteries, corrosion on terminals, etc. Startup is a much different workload on electronics and in the case of HDDs mechanical devices than running. I have an old fan that will need replacing one day that takes about 15 minutes to reach full speed as the bearings are gone. Everytime I turn it off I wonder if the next time I turn it on it will be time to replace it.

wdfx 5y ago

It's just the numbers. Eventually all the hardware will fail. Some of that hardware will require a specific set of conditions to fail. Such a condition may be current inrush from being powercycled. Given the scale of a data centre and powercycling it all at once, then some will fail due to that condition. Further to that thought, there must be hardware in that centre which is failing and being replaced all the time due to other conditions, we just won't hear about it because it's part of normal maintenance.

emptybottle 5y ago

It can be a number of things, but often times it's the thermal cycle and wear on the components. After letting devices (especially that rarely do this) cool, and then bringing them back up to temperature some percentage of components may fail or operate out of spec. And mechanical parts have similar issue. A motor that's been happily running for months/years may not restart after being stopped. It's also possible that a power fault itself could damage hardware, although in my experience this type of issue is far less common.

fy20 5y ago
If you think of it in these terms ICEs are an engineering marvell.

dilyevsky 5y ago

I worked for one of the largest datacenter company in the world and significant portion of boxes would never come up after being powercycled

mgsouth 5y ago

Slamming on the brakes can be hard on the electronics. It causes large, high-speed voltage or current spikes that kill (old, tired) components before the spikes can be clamped. Can be made worse by old electolytic caps that no longer filter effectively. Especially hard on power supply diodes and mosfets. Here's a few references at random: https://vbn.aau.dk/ws/portalfiles/portal/108034396/ESREF2013_ConferenceVersion.pdf https://vbn.aau.dk/ws/portalfiles/portal/108034396/ESREF2013... https://www.infineon.com/dgdl/s30p5.pdf?fileId=5546d462533600a401535748559c3fcd https://www.infineon.com/dgdl/s30p5.pdf?fileId=5546d46253360... https://www.sciencedirect.com/science/article/pii/S0038110102003283 https://www.sciencedirect.com/science/article/pii/S003811010...

Spivak 5y ago
Which is why if you realize that your hard drive is failing the worst thing you can do is cut the power [1]. You get your data off that box as fast as possible. [1] Yes yes always have backups, and RAID, and replication something about not being able to count that low. Betcha most'a'ya don't have any of that on your MBP though except for a maybe a backup that lags hours behind your work.
CoastalCoder 5y ago
> Slamming on the brakes can be hard on the electronics. It causes large, high-speed voltage or current spikes that kill (old, tired) components before the spikes can be clamped. Suppose that the "slamming on the brakes" here is that the power lines connecting a data center to the grid get physically cut. I can see how that would cause a power spike on the grid side of the cut, but would it really create a spike on the datacenter side of the cut? Or assuming that the wires in a cable aren't cut exactly simultaneously, does it depend on the particular phases of the AC power at the time of each wire in the cable being cut?
- R0b0t1 5y ago
  Any change in the state of an inductive load causes kickback. If you sever the lines running to a building, the generator side gets the most kick, but even a fully passive device gets a kick from the collapsing field in the wires servicing it. The motors generate appreciable back EMF when suddenly unpowered. Likewise an SMPS is an inductive load. Spikes from power supplies going off is probably what killed the disks, assuming they didn't just decide not to power on. Some disks will run for 10 years but power them off once and they don't come back up.

prpl 5y ago

I’m sure temperature variation has something to do with it in some cases. Most parts, even when cooled, are well above room temperature for years on end, a loss of power can drop them to room temp relatively quickly