6 ms·

"As is often the case with a loss of power, there may be some hardware that is not recoverable..." No. Not even rarely. If they lost hardware because of this s

• Johnny555 5y ago
  If my home UPS failed, I'd be awfully surprised if my home fileserver that was plugged into it failed too. But if I lost power to thousands of servers, I'd expect some number of them to fail. I've even lost servers when losing power to a single rack.
• jdsully 5y ago
  Starting up is hard on electronics due to the inrush current. There may be marginal devices just barely operating that will not survive a reboot. Things like bad capacitors aren’t always detectable during steady state. At AWS scale it’s highly likely there are more than a few of these.
  • daneel_w 5y ago
    Did you think this over before hitting the reply button? Can you imagine any server or networking brand surviving if its products were at substantial risk of dying when powering up or cycling? Have you ever heard of common consumer grade computer equipment regularly dying from it? No? Not me either. It's really common to find thyristors in devices to limit in-rush current, even in cheap electronics. You can bet that PSUs in data center equipment use them.
    • emodendroket 5y ago
      > Have you ever heard of common consumer grade computer equipment regularly dying from powering up? Yes, absolutely.
      • daneel_w 5y ago
        You're romanticizing here. Electronics eventually dying is not the same as being at substantial risk from dying due to power-cycling itself.
        • emodendroket 5y ago
          No, I am not. I personally have experienced this. Pop, whoops, it doesn't work anymore.
          • daneel_w 5y ago
            Same. What you're romanticizing about is it being a common and regular occurrence. It's not. It's a fluke.
            • emodendroket 5y ago
              Now imagine you have one of the world's largest assemblages of electronic devices and they all turn on at once. How likely are some flukes?
            • dpratt 5y ago
              And when you have 500,000 copies of something, most of which have been in use for some appreciable fraction of their effective lifetime, it doesn’t have to be “common” to occur to some fraction of them. The post you’re responding to never implied that it happens to _most_ of them, or even many of them, just that when you have a gigantic farm, it’s not unreasonable to see a small handful of hardware instances release up their magic smoke when they’re all coming back from being powered down. MTBF is a thing.
            • ericd 5y ago
              A fluke x massive scale = regular occurrence.
            • deleted 5y ago
              [deleted]
            • 5y ago
    • kube-system 5y ago
      I am not sure whether or not inrush current (or thermal cycling, or something else) is to blame, but server equipment in data centers has been fairly well documented to exhibit increased hardware failure rates after outages.
    • organsnyder 5y ago
      Substantial? No. A low percentage that is still large enough to be impactful in a datacenter? Absolutely.
      • X-Istence 5y ago
        When you've got a massive datacenter packed, even 0.01% is a meaningful amount of hardware.
    • sonofhans 5y ago
      It's a scale problem. We're not talking about "substantial risk" or "regularly dying." At AWS scale even 0.01% cold start failure is noticeable. It's not possible to guarantee perfect operation of each system or component; physical things wear out. If you can find a way to avoid cold-start failure 100% of the time your product would be valuable to many businesses. Also: > Did you think this over before hitting the reply button? This is ad hominem, and we should avoid it here.
    • hhh 5y ago
      I don’t know if it’s intentional, but your comment comes off very hostile. Recently we has a violent power outage where I work due to the tornados in the midwest. A few of our facilities have industrial hardware with batteries that will last for 72 hours and after that point it’s a world of unknowns what will happen. The batteries died after a lot of effort to try and restore power, but everything came up just fine. However, a random Cisco switch elsewhere in the building had at least one power supply fail. This was in one facility where most of these have been running for 4+ years straight (12 years for the system with a 72 hour battery.) I find it hard to imagine that in the scale of even one AZ that it wouldn’t be possible that at least one system has this happen.
    • jeffbee 5y ago
      > Have you ever heard of common consumer grade computer equipment regularly dying from it? You have outed yourself as someone with zero experience operating computers, either individually or at scale. Computers have moving parts, whether fans or hard disk drives. Just because those were moving doesn't mean they will begin moving again from a dead stop. There are also things like dead cmos/nvram batteries that can prevent machines from automatically starting when power is applied.
    • acdha 5y ago
      > Did you think this over before hitting the reply button? Can you imagine any server or networking brand surviving if its products were at substantial risk of dying when powering up or cycling? This is hostile enough that I'd have trouble squaring it with the site guidelines. It's especially bad because, as others have been saying, your belief isn't supported by real lived experience for many of us. A data center has enough devices that even a low probability error rate will fairly reliably happen, and that doesn't usually hurt the manufacturer's reputation because they never promised 100% and will replace it under warranty. I've seen this with hard drives especially but also things like power supplies and on-board batteries, and even things like motherboards where a cooling/heating cycle was enough to unseat RAM or cause hairline fractures in solder traces. This can be especially bad with unplanned power outages if the power doesn't instantly go out and stay out for the duration. Especially for hard drives hitting the power up/down cycle a few times was a good way to have extra failures. Unscheduled power outages can be even worse
      • jacquesm 5y ago
        I've seen even planned outages and tests result in failed hardware.
        • acdha 5y ago
          Ditto — you never do a substantial shutdown without learning something.
• nicolaslem 5y ago
  It is common for failing hard drives to continue running until their are power cycled, at which point they never start again.
  • X-Istence 5y ago
    Having had to restore a NetApp cluster from backup because the disks would no longer spin up after having been running for 4+ years... yup. The hardware was fully working and according to all internal metrics was still perfectly good, but a power outage and they never came back up.
• jacquesm 5y ago
  To forestall that reply: I did think about it before hitting the reply button. I've seen a couple of large scale DC power outages in my 35 years of IT work. The vast majority of the hardware came through just fine. But older data centers with gear that has been running uninterrupted for many years tend to have at least some hardware that simply won't come up again, due to a variety of reasons. One can be that the on-board batteries have gone bad, which nobody noticed until the power cycle. Another is that some harddrives function well as long as they keep spinning but that has worn a nasty little spot on the bearing that keeps the spindle aloft. When it spins down, depending on the orientation it may not be able to start back up again, it may even crash while spinning down. Powercycle enough gear that has been running for years and you will most likely lose at least some small fraction of it. You could do it the next day after taking care of those and everything would likely be fine.
  • vlovich123 5y ago
    Wouldn’t a UPS + generator failover mean that the HW never spins down in the first place? That’s how I interpreted op’s statement anyway.
    • jacquesm 5y ago
      I've seen plenty of UPS's fail, and generator failover is that moment when everybody stands there with their fingers crossed. None of those things are fool proof and in case of a large scale outage, especially one that lasts more than a couple of minutes there is a fair chance that you will find the limitations of some of your contingency plans.
    • acdha 5y ago
      Hopefully, but those aren't perfect. Large data centers regularly test those because things can go wrong with any of the UPS, generator[1], or the distribution hardware which switches between the line and generator power. One of the longer outages I've seen was when the distribution hardware itself failed and burnt out multiple parts which the manufacturer did not have sufficient spares in our region. AWS has a lot of very skilled professionals who are quite familiar with those issues so I'd be quite surprised if it turned out to be something that simple but you always want to have a contingency plan you've tested for what happens if core infrastructure like that fails and takes multiple days to recover. 1. One nasty example: fuel issues which aren't immediately obvious so someone doing a 5 minute test wouldn't learn that the system wasn't going to handle an outage longer than that.
• X-Istence 5y ago
  Having worked in datacenter where servers were running for 5+ years in racks, when the datacenter went dark due to a UPS going up in flames there were definitely systems that did not come back online. Most notably we had a large NetApp array that would not boot, once we replaced the controllers we had lost 13 out of the 60 hard drives in the array. They would no longer spin up. Like physically seized. Because they had been spinning for so long, they would have likely kept spinning just fine, but with power gone, it was done. Fans are another fun one, anything with ball bearings really. Power supplies are another issue, due to the sudden large inrush of power when the switch was flipped back to on, some power supplies had their capacitors go up in smoke. This is not rare, this a common occurrence. When you have an absolutely massive footprint with thousands upon thousands of servers, that have been running for a long time, there will be things that just don't come back once they have stopped running or when the electricity is gone.
• joshuamorton 5y ago
  Yes, its a fairly well known issue at datacenter scale that if you power-cycle everything, some percentage of things won't turn back on (I recall something like 1% of HDDs just failing to work again being the number quoted at me). This obviously isn't the case for fresh new HDDs, but ones that have been spinning continuously for a years. Other hardware is similar.