4 ms·

Get a thermometer and check the ambient temp. It could be a bug on the mobo/BIOS. As an AWS SRE, we once had an issue where we had waves of racks being unavaila

by M_bara 8y ago

Get a thermometer and check the ambient temp. It could be a bug on the mobo/BIOS. As an AWS SRE, we once had an issue where we had waves of racks being unavailable. They'd go offline, were booted by automation, whereupon they'd come up only to go offline again. Our usual approach to solving such issues didn't work (we had a concall spanning 3 days to triage this issue). We typically relied on IPMI sensors for thermal events in the fleet. The typical script fetched the IPMI temp and if it was too high raised a ticket. What we never factored was a temp going too low. So, it turns out, that the datacenter in question would allow air from the outside in winter to help with cooling during winter. One of the air vents got stuck open and servers went cold... (IN deg C): 5, 4, 3, 2, 1, 0, 255 << oops bios initiates a server shutdown due to a very high chassis temp. Server vendor issue a patch a few weeks later. We had to put in an alarm for temp going too low in our monitoring!

Scriptor 8y ago
He does measure the ambient temperature in the article. Happens around 60 degrees Fahrenheit, so well over freezing.
westmeal 8y ago
Wow I can't believe they used an unsigned 8 bit integer for temps
- pgt 8y ago
  I recently had to parse string-based temperature readings coming in over UART (e.g. 25.4C), and I forgot to include the leading minus character in my regular expression. I think generative testing is a great option here if you run the regex "in reverse".
healsjnr1 8y ago
Interesting. A decade ago I observed something similar to the article while traveling in Nepal / Tibet. Me and friend both had iPods (of different generations) that got 'altitude sick'. At two points in the trip we got about 5,000m. Each time both iPods went totally unresponsive. Both times we'd only be there for a few hours and once we descended they'd stay working again. Obvious thing was something to do with the cold and battery, but both times the battery came at the same charged once they started working. Can't remember exactly what the ambient temperature was (we were in cars both times) but it's puzzled me whether it was cold or altitude.
- goodcanadian 8y ago
  A quick trip to Wikipedia suggests that iPod Classics had a spinning disk for storage. The disk in hard drives "floats" on a cushion of air. When the outside air pressure gets lower, the internal pressure is relatively higher and the gap increases. A big enough pressure difference and the disc can jam and stop spinning altogether. I do not know if that is what affected your iPods, but I do know that hard drive failures at the summit of Mauna Kea (~4000m) are common and it seems to mostly be down to luck which ones work and which don't (once you have a working one, it tends to stay working).
  - healsjnr1 8y ago
    Wow, that's awesome. No idea if that's actually what was happening, but it fits the picture of why these were getting 'altitude sickness'

by M_bara 8y ago

Scriptor 8y ago

He does measure the ambient temperature in the article. Happens around 60 degrees Fahrenheit, so well over freezing.

westmeal 8y ago

Wow I can't believe they used an unsigned 8 bit integer for temps

pgt 8y ago
I recently had to parse string-based temperature readings coming in over UART (e.g. 25.4C), and I forgot to include the leading minus character in my regular expression. I think generative testing is a great option here if you run the regex "in reverse".

healsjnr1 8y ago

Interesting. A decade ago I observed something similar to the article while traveling in Nepal / Tibet. Me and friend both had iPods (of different generations) that got 'altitude sick'. At two points in the trip we got about 5,000m. Each time both iPods went totally unresponsive. Both times we'd only be there for a few hours and once we descended they'd stay working again. Obvious thing was something to do with the cold and battery, but both times the battery came at the same charged once they started working. Can't remember exactly what the ambient temperature was (we were in cars both times) but it's puzzled me whether it was cold or altitude.

goodcanadian 8y ago
A quick trip to Wikipedia suggests that iPod Classics had a spinning disk for storage. The disk in hard drives "floats" on a cushion of air. When the outside air pressure gets lower, the internal pressure is relatively higher and the gap increases. A big enough pressure difference and the disc can jam and stop spinning altogether. I do not know if that is what affected your iPods, but I do know that hard drive failures at the summit of Mauna Kea (~4000m) are common and it seems to mostly be down to luck which ones work and which don't (once you have a working one, it tends to stay working).
- healsjnr1 8y ago
  Wow, that's awesome. No idea if that's actually what was happening, but it fits the picture of why these were getting 'altitude sickness'