I came across this experiment in the book "Inventing Temperature" by Hasok Chang. Chang seems to present it as if it is a phenomenon that modern science can't

I came across this experiment in the book "Inventing Temperature" by Hasok Chang. Chang seems to present it as if it is a phenomenon that modern science can't explain (possibly I misunderstood him). But I think the explanation is basically that the cold object is blocking "ambient heat" (ambient infra-red light), the same way that a dark object blocks ambient visible light. So if you look at a dark object you're not seeing "rays of dark" hitting your eyes, you're just noticing the absence of the ambient "rays of light" that you're accustomed to seeing all the time. Similarly for heat: the thermometer gets cold because it is in the heat-shadow of the cold object. The heat that the thermometer is radiating is not being topped up by the ambient heat, because it is in the shadow.