5 ms·

Starting up is hard on electronics due to the inrush current. There may be marginal devices just barely operating that will not survive a reboot. Things like ba

Starting up is hard on electronics due to the inrush current. There may be marginal devices just barely operating that will not survive a reboot. Things like bad capacitors aren’t always detectable during steady state. At AWS scale it’s highly likely there are more than a few of these.

daneel_w 5y ago
Did you think this over before hitting the reply button? Can you imagine any server or networking brand surviving if its products were at substantial risk of dying when powering up or cycling? Have you ever heard of common consumer grade computer equipment regularly dying from it? No? Not me either. It's really common to find thyristors in devices to limit in-rush current, even in cheap electronics. You can bet that PSUs in data center equipment use them.
- emodendroket 5y ago
  > Have you ever heard of common consumer grade computer equipment regularly dying from powering up? Yes, absolutely.
  - daneel_w 5y ago
    You're romanticizing here. Electronics eventually dying is not the same as being at substantial risk from dying due to power-cycling itself.
    - emodendroket 5y ago
      No, I am not. I personally have experienced this. Pop, whoops, it doesn't work anymore.
      - daneel_w 5y ago
        Same. What you're romanticizing about is it being a common and regular occurrence. It's not. It's a fluke.
        emodendroket 5y ago
        Now imagine you have one of the world's largest assemblages of electronic devices and they all turn on at once. How likely are some flukes?
        dpratt 5y ago
        And when you have 500,000 copies of something, most of which have been in use for some appreciable fraction of their effective lifetime, it doesn’t have to be “common” to occur to some fraction of them. The post you’re responding to never implied that it happens to _most_ of them, or even many of them, just that when you have a gigantic farm, it’s not unreasonable to see a small handful of hardware instances release up their magic smoke when they’re all coming back from being powered down. MTBF is a thing.
        ericd 5y ago
        A fluke x massive scale = regular occurrence.
        deleted 5y ago
        [deleted]
        unionpivo 5y ago
        Pretty much anyone who has worked in datacenter disagrees with you. Your assumptions that you know better than people working there are arrogant to extreme. If electronics (not just computer) is going to fail its almost always at power on or power off. And since DC is one big electronics warehouse, stuff breaks. Old spinning drives were most prone to this, with power supply and motherboards being second.
    - deleted 5y ago
      [deleted]
  - KronisLV 5y ago
    Right now i run my homelab on consumer hardware in addition to my cloud VPSes. Due to me not having that much free space and also enjoying a good night's sleep, i have cron set up to put the servers offline (rtcwake) while i'm sleeping, so that they wouldn't just go on droning in the background and to start up in the morning. Since they do this every day with no exceptions, that's a decent amount of power cycling, in addition to them working all the time otherwise, which has been going on for about 5 years or so. So essentially each of them has been on for about 30'000 - 35'000 hours and has power cycled around 1800 - 3600 times, because for the longest time i've had a cron restart set up right after the rtcwake sleep period finishes, due to some odd bug in the kernel that otherwise caused their CPUs to get stuck at 100% usage after waking up. Over these 5 or so years, i've had: - 2 PSUs die (possibly one due to a power surge though, before i had UPSes in front of them), seemed to work fine but then one morning didn't start up - 2 HDDs die (Seagate Barracuda 1 TBs, each box has 2-3 of them), typically this isn't abrupt, but the HDDs lose power when trying to spin up and loop like that for a while before eventually deciding to work, before eventually not being able to do that anymore at all; oddly enough, Clonezilla could still mirror such a disk after a few attempts - 1 motherboard die (not sure precisely why, then again, for a while in my university i had the servers (without cases) up next to a slighty ajar window, even in winter as well as in summer, while they were otherwise only passively cooled (x86, but 35W TDP 200 GEs) - some problems with bearings, once in a cheap CPU fan before i had passive cooling and then again in one of the current PSU fans, sometimes when starting up it sounds like a miniature chainsaw (checked the fan, isn't actually seized up or anything, just cheap bearings) - some problems with the power button (well, more like the pins, since i didn't have a proper button at one point, just a screwdriver to close the circuit with), though i solved that by closing the circuit on the ATX connector itself by putting a little wire in there All of the failures have happened around power cycling, never in the middle of otherwise normal operation. To my surprise, nothing has actually melted or caught fire, the last time when anything close to that happened was ages ago with my 9800 GT GPU which for some reason started getting really bad thermals (90 degrees C) and so i had to reflow solder it in the oven a few times, before eventually it just refused to work and artifacted out on boot. Or maybe i could mention a MOLEX connector stopping working, but i might just have a splitter in there due to not enough SATA connections from the PSU, something that people typically advise against. So, to sum all of that up, yes, consumer hardware will definitely die from regular use under normal circumstances, probably moreso with regular power cycling of parts that aren't really made for that (especially the regular HDDs, which seemed to die right after restarts). Also, my case might be an outlier (and a sample size of 1), due to me not caring that much in university about keeping the servers in a good environment, but i bet that most consumer hardware has to deal with sub-optimal conditions in one way or another - accumulation of dust, bad airflow, air humidity, ambient temperature and power grid fluctuations all should also be taken into account. I actually had to stagger the startup times for my servers, since otherwise they'd max out their power draw at the same time in the morning (e.g. when launching Docker containers and spinning up VMs), which also lead to some weirdness. Of course, the quality of the parts themselves probably also matters (like a reputable brand PSU vs the cheaper options that i could afford) as much as how they're used (prolonged periods of full load while everything starts up), as do other factors, like vibrations of the HDDs, for example.
- kube-system 5y ago
  I am not sure whether or not inrush current (or thermal cycling, or something else) is to blame, but server equipment in data centers has been fairly well documented to exhibit increased hardware failure rates after outages.
- organsnyder 5y ago
  Substantial? No. A low percentage that is still large enough to be impactful in a datacenter? Absolutely.
  - X-Istence 5y ago
    When you've got a massive datacenter packed, even 0.01% is a meaningful amount of hardware.
- sonofhans 5y ago
  It's a scale problem. We're not talking about "substantial risk" or "regularly dying." At AWS scale even 0.01% cold start failure is noticeable. It's not possible to guarantee perfect operation of each system or component; physical things wear out. If you can find a way to avoid cold-start failure 100% of the time your product would be valuable to many businesses. Also: > Did you think this over before hitting the reply button? This is ad hominem, and we should avoid it here.
- hhh 5y ago
  I don’t know if it’s intentional, but your comment comes off very hostile. Recently we has a violent power outage where I work due to the tornados in the midwest. A few of our facilities have industrial hardware with batteries that will last for 72 hours and after that point it’s a world of unknowns what will happen. The batteries died after a lot of effort to try and restore power, but everything came up just fine. However, a random Cisco switch elsewhere in the building had at least one power supply fail. This was in one facility where most of these have been running for 4+ years straight (12 years for the system with a 72 hour battery.) I find it hard to imagine that in the scale of even one AZ that it wouldn’t be possible that at least one system has this happen.
- jeffbee 5y ago
  > Have you ever heard of common consumer grade computer equipment regularly dying from it? You have outed yourself as someone with zero experience operating computers, either individually or at scale. Computers have moving parts, whether fans or hard disk drives. Just because those were moving doesn't mean they will begin moving again from a dead stop. There are also things like dead cmos/nvram batteries that can prevent machines from automatically starting when power is applied.
- acdha 5y ago
  > Did you think this over before hitting the reply button? Can you imagine any server or networking brand surviving if its products were at substantial risk of dying when powering up or cycling? This is hostile enough that I'd have trouble squaring it with the site guidelines. It's especially bad because, as others have been saying, your belief isn't supported by real lived experience for many of us. A data center has enough devices that even a low probability error rate will fairly reliably happen, and that doesn't usually hurt the manufacturer's reputation because they never promised 100% and will replace it under warranty. I've seen this with hard drives especially but also things like power supplies and on-board batteries, and even things like motherboards where a cooling/heating cycle was enough to unseat RAM or cause hairline fractures in solder traces. This can be especially bad with unplanned power outages if the power doesn't instantly go out and stay out for the duration. Especially for hard drives hitting the power up/down cycle a few times was a good way to have extra failures. Unscheduled power outages can be even worse
  - jacquesm 5y ago
    I've seen even planned outages and tests result in failed hardware.
    - acdha 5y ago
      Ditto — you never do a substantial shutdown without learning something.

daneel_w 5y ago

Did you think this over before hitting the reply button? Can you imagine any server or networking brand surviving if its products were at substantial risk of dying when powering up or cycling? Have you ever heard of common consumer grade computer equipment regularly dying from it? No? Not me either. It's really common to find thyristors in devices to limit in-rush current, even in cheap electronics. You can bet that PSUs in data center equipment use them.

emodendroket 5y ago
> Have you ever heard of common consumer grade computer equipment regularly dying from powering up? Yes, absolutely.
- daneel_w 5y ago
  You're romanticizing here. Electronics eventually dying is not the same as being at substantial risk from dying due to power-cycling itself.
  - emodendroket 5y ago
    No, I am not. I personally have experienced this. Pop, whoops, it doesn't work anymore.
    - daneel_w 5y ago
      Same. What you're romanticizing about is it being a common and regular occurrence. It's not. It's a fluke.
      - emodendroket 5y ago
        Now imagine you have one of the world's largest assemblages of electronic devices and they all turn on at once. How likely are some flukes?
      - dpratt 5y ago
        And when you have 500,000 copies of something, most of which have been in use for some appreciable fraction of their effective lifetime, it doesn’t have to be “common” to occur to some fraction of them. The post you’re responding to never implied that it happens to _most_ of them, or even many of them, just that when you have a gigantic farm, it’s not unreasonable to see a small handful of hardware instances release up their magic smoke when they’re all coming back from being powered down. MTBF is a thing.
      - ericd 5y ago
        A fluke x massive scale = regular occurrence.
      - deleted 5y ago
        [deleted]
      - unionpivo 5y ago
        Pretty much anyone who has worked in datacenter disagrees with you. Your assumptions that you know better than people working there are arrogant to extreme. If electronics (not just computer) is going to fail its almost always at power on or power off. And since DC is one big electronics warehouse, stuff breaks. Old spinning drives were most prone to this, with power supply and motherboards being second.
  - deleted 5y ago
    [deleted]
- KronisLV 5y ago
  Right now i run my homelab on consumer hardware in addition to my cloud VPSes. Due to me not having that much free space and also enjoying a good night's sleep, i have cron set up to put the servers offline (rtcwake) while i'm sleeping, so that they wouldn't just go on droning in the background and to start up in the morning. Since they do this every day with no exceptions, that's a decent amount of power cycling, in addition to them working all the time otherwise, which has been going on for about 5 years or so. So essentially each of them has been on for about 30'000 - 35'000 hours and has power cycled around 1800 - 3600 times, because for the longest time i've had a cron restart set up right after the rtcwake sleep period finishes, due to some odd bug in the kernel that otherwise caused their CPUs to get stuck at 100% usage after waking up. Over these 5 or so years, i've had: - 2 PSUs die (possibly one due to a power surge though, before i had UPSes in front of them), seemed to work fine but then one morning didn't start up - 2 HDDs die (Seagate Barracuda 1 TBs, each box has 2-3 of them), typically this isn't abrupt, but the HDDs lose power when trying to spin up and loop like that for a while before eventually deciding to work, before eventually not being able to do that anymore at all; oddly enough, Clonezilla could still mirror such a disk after a few attempts - 1 motherboard die (not sure precisely why, then again, for a while in my university i had the servers (without cases) up next to a slighty ajar window, even in winter as well as in summer, while they were otherwise only passively cooled (x86, but 35W TDP 200 GEs) - some problems with bearings, once in a cheap CPU fan before i had passive cooling and then again in one of the current PSU fans, sometimes when starting up it sounds like a miniature chainsaw (checked the fan, isn't actually seized up or anything, just cheap bearings) - some problems with the power button (well, more like the pins, since i didn't have a proper button at one point, just a screwdriver to close the circuit with), though i solved that by closing the circuit on the ATX connector itself by putting a little wire in there All of the failures have happened around power cycling, never in the middle of otherwise normal operation. To my surprise, nothing has actually melted or caught fire, the last time when anything close to that happened was ages ago with my 9800 GT GPU which for some reason started getting really bad thermals (90 degrees C) and so i had to reflow solder it in the oven a few times, before eventually it just refused to work and artifacted out on boot. Or maybe i could mention a MOLEX connector stopping working, but i might just have a splitter in there due to not enough SATA connections from the PSU, something that people typically advise against. So, to sum all of that up, yes, consumer hardware will definitely die from regular use under normal circumstances, probably moreso with regular power cycling of parts that aren't really made for that (especially the regular HDDs, which seemed to die right after restarts). Also, my case might be an outlier (and a sample size of 1), due to me not caring that much in university about keeping the servers in a good environment, but i bet that most consumer hardware has to deal with sub-optimal conditions in one way or another - accumulation of dust, bad airflow, air humidity, ambient temperature and power grid fluctuations all should also be taken into account. I actually had to stagger the startup times for my servers, since otherwise they'd max out their power draw at the same time in the morning (e.g. when launching Docker containers and spinning up VMs), which also lead to some weirdness. Of course, the quality of the parts themselves probably also matters (like a reputable brand PSU vs the cheaper options that i could afford) as much as how they're used (prolonged periods of full load while everything starts up), as do other factors, like vibrations of the HDDs, for example.
kube-system 5y ago
I am not sure whether or not inrush current (or thermal cycling, or something else) is to blame, but server equipment in data centers has been fairly well documented to exhibit increased hardware failure rates after outages.
organsnyder 5y ago
Substantial? No. A low percentage that is still large enough to be impactful in a datacenter? Absolutely.
- X-Istence 5y ago
  When you've got a massive datacenter packed, even 0.01% is a meaningful amount of hardware.
sonofhans 5y ago
It's a scale problem. We're not talking about "substantial risk" or "regularly dying." At AWS scale even 0.01% cold start failure is noticeable. It's not possible to guarantee perfect operation of each system or component; physical things wear out. If you can find a way to avoid cold-start failure 100% of the time your product would be valuable to many businesses. Also: > Did you think this over before hitting the reply button? This is ad hominem, and we should avoid it here.
hhh 5y ago
I don’t know if it’s intentional, but your comment comes off very hostile. Recently we has a violent power outage where I work due to the tornados in the midwest. A few of our facilities have industrial hardware with batteries that will last for 72 hours and after that point it’s a world of unknowns what will happen. The batteries died after a lot of effort to try and restore power, but everything came up just fine. However, a random Cisco switch elsewhere in the building had at least one power supply fail. This was in one facility where most of these have been running for 4+ years straight (12 years for the system with a 72 hour battery.) I find it hard to imagine that in the scale of even one AZ that it wouldn’t be possible that at least one system has this happen.
jeffbee 5y ago
> Have you ever heard of common consumer grade computer equipment regularly dying from it? You have outed yourself as someone with zero experience operating computers, either individually or at scale. Computers have moving parts, whether fans or hard disk drives. Just because those were moving doesn't mean they will begin moving again from a dead stop. There are also things like dead cmos/nvram batteries that can prevent machines from automatically starting when power is applied.
acdha 5y ago
> Did you think this over before hitting the reply button? Can you imagine any server or networking brand surviving if its products were at substantial risk of dying when powering up or cycling? This is hostile enough that I'd have trouble squaring it with the site guidelines. It's especially bad because, as others have been saying, your belief isn't supported by real lived experience for many of us. A data center has enough devices that even a low probability error rate will fairly reliably happen, and that doesn't usually hurt the manufacturer's reputation because they never promised 100% and will replace it under warranty. I've seen this with hard drives especially but also things like power supplies and on-board batteries, and even things like motherboards where a cooling/heating cycle was enough to unseat RAM or cause hairline fractures in solder traces. This can be especially bad with unplanned power outages if the power doesn't instantly go out and stay out for the duration. Especially for hard drives hitting the power up/down cycle a few times was a good way to have extra failures. Unscheduled power outages can be even worse
- jacquesm 5y ago
  I've seen even planned outages and tests result in failed hardware.
  - acdha 5y ago
    Ditto — you never do a substantial shutdown without learning something.