3 ms·

Overheating machines is very common. Even if you only own ten thousand machines it’s pretty much assured you’ve got one machine out there with a cracked heat si

• mrep 7y ago
  But at google scale, you'd think they would have something for that. At a previous company I worked at, we had a guy whose entire job was to monitor temperatures throughout all of our data centers. He built a service that collected minute by minute temperature metrics on every server in every datacenter. Once collected it would create automated tickets for data center technicians to investigate when servers and racks got too hot. He even built this cool website where you could view real time maps of each rack and their temperature in every datacenter so that operations teams could visualize the high temperature areas in the data centers.
  • thedance 7y ago
    It's not that they don't know, it's that they don't care. Hot machines go into an attention queue and some tech eventually fixes it. Meanwhile, they serve. As I commented elsewhere, I think it's an effective incentive for software to grow more sophisticated. In fleets that are promptly fixed, there may be a tendency to assume that, for example, all backends are equally suited to receive an RPC. Having a lot of ambient brokenness can be helpful, as Netflix advocates with "chaos monkey".