4 ms·

I heard the devops guys at a previous job wrote a script to detect CPU fan failures. Then got asked to stop sending so many emails about CPU fan failures. Was s

I heard the devops guys at a previous job wrote a script to detect CPU fan failures. Then got asked to stop sending so many emails about CPU fan failures. Was still worth it because it's cheap, but they're cheap because it's not new and reliable hardware.

robertlagrant 4y ago
I'm no particular Hetzner fan, but is it possible there were a load of false positives in there?
- adql 4y ago
  "Oh look, fan stopped, means it's dead" CPU temperature: 40C
wg0 4y ago
Really? Been running few dedicated servers for several years straight without a fail. Even Herzner cloud just works and I don't know how they do it but it's dirt cheap.
- jonatron 4y ago
  How many is a few? If you've have 500 servers for a long time, and a new script discovers 5% have a failure and sends 25 emails at the same time, I can see why Hetzner might want a single email. Numbers made up, but you get the idea.
byefruit 4y ago
Is this any worse than not even knowing when parts of your AWS instance are failing?
- jonatron 4y ago
  If I search my emails: "EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance" It's just something you have to do yourself with Hetzner.
  - byefruit 4y ago
    Right but AWS gets to make that choice, you have no idea whether they consider a failing fan as "degradation".
  - renonce 4y ago
    You should compare EC2 with Hetzner Cloud. Hetzner Cloud do send such alerts occasionally.
adql 4y ago
Maybe he detected 0 RPM without looking at temperature.... some controllers will just stop fans when there is enough cooling. We had a bunch of racks and fan failures gotta be one of rarest one. Even on my personal junk I had zero actual failures, just one getting noisy > Was still worth it because it's cheap, but they're cheap because it's not new and reliable hardware. Not really different than cloud, they aren't buying top of the line servers, they are making their own just like hetzner, for cheapest per performance unit.
- lloydatkinson 4y ago
  Are controllers that throttle down to zero common in the data centre space though? I don't think I've ever heard a quiet or silent server, except for the ones tech people make themselves at home.
  - usrusr 4y ago
    Hetzner is known to do their own thing instead of blindly following datacenter conventions. E.g. having the physical architecture (how their buildings are designed) so optimized for convective flow that they get sufficient air flow without extra fans most hours of the year and only need active "heat pump" style cooling in rare exceptions. They do have board-level fans, but it it would be almost out-of-character if those did not throttle to zero whenever possible. Big german language report: https://www.golem.de/news/besuch-im-rechenzentrum-so-betreibt-hetzner-tausende-server-effizient-2211-169943.html https://www.golem.de/news/besuch-im-rechenzentrum-so-betreib...

robertlagrant 4y ago

I'm no particular Hetzner fan, but is it possible there were a load of false positives in there?

adql 4y ago
"Oh look, fan stopped, means it's dead" CPU temperature: 40C

wg0 4y ago

Really? Been running few dedicated servers for several years straight without a fail. Even Herzner cloud just works and I don't know how they do it but it's dirt cheap.

jonatron 4y ago
How many is a few? If you've have 500 servers for a long time, and a new script discovers 5% have a failure and sends 25 emails at the same time, I can see why Hetzner might want a single email. Numbers made up, but you get the idea.

byefruit 4y ago

Is this any worse than not even knowing when parts of your AWS instance are failing?

jonatron 4y ago
If I search my emails: "EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance" It's just something you have to do yourself with Hetzner.
- byefruit 4y ago
  Right but AWS gets to make that choice, you have no idea whether they consider a failing fan as "degradation".
- renonce 4y ago
  You should compare EC2 with Hetzner Cloud. Hetzner Cloud do send such alerts occasionally.

adql 4y ago

Maybe he detected 0 RPM without looking at temperature.... some controllers will just stop fans when there is enough cooling. We had a bunch of racks and fan failures gotta be one of rarest one. Even on my personal junk I had zero actual failures, just one getting noisy > Was still worth it because it's cheap, but they're cheap because it's not new and reliable hardware. Not really different than cloud, they aren't buying top of the line servers, they are making their own just like hetzner, for cheapest per performance unit.

lloydatkinson 4y ago
Are controllers that throttle down to zero common in the data centre space though? I don't think I've ever heard a quiet or silent server, except for the ones tech people make themselves at home.
- usrusr 4y ago
  Hetzner is known to do their own thing instead of blindly following datacenter conventions. E.g. having the physical architecture (how their buildings are designed) so optimized for convective flow that they get sufficient air flow without extra fans most hours of the year and only need active "heat pump" style cooling in rare exceptions. They do have board-level fans, but it it would be almost out-of-character if those did not throttle to zero whenever possible. Big german language report: https://www.golem.de/news/besuch-im-rechenzentrum-so-betreibt-hetzner-tausende-server-effizient-2211-169943.html https://www.golem.de/news/besuch-im-rechenzentrum-so-betreib...