5 ms·
There seems to be little attempt to ensure the ambient environment of the processors is isothermal. That is, there's likely significant chassis to chassis ther
by mlyle 7y ago
There seems to be little attempt to ensure the ambient environment of the processors is isothermal. That is, there's likely significant chassis to chassis thermal variation that is as big or larger than any difference in thermal budget between processors.
Even things like the characteristics of the application of the thermal paste and the roughness of the individual fan ducts can matter, beyond the obvious bottom of rack vs. top of rack, position of processor in individual chassis, etc effects.
tl;dr-- probably most of what is measured is not silicon to silicon variation.
- kev009 7y agoWell thought out response, that is my gut feeling as well. I think packaging could have a larger than expected effect too, especially CPUs with a thermal cap. This kind of in situ measurement might be useful to feed into a job scheduler's weight or something but at the end of the day "environment is different" and maybe you have to derate your cluster's expected performance to account?
- dchichkov 7y agoThe cited paper doesn't attribute performance variation to silicon to silicon variation. Only the blog post does.
- mlyle 7y agoFrom the abstract: > We show that this variation is further magnified under a hardware-enforced power constraint, potentially due to the increase in number of cores, inconsistencies in the chip manufacturing process and their combined impact on processor’s energy management functionality and recommendations: > • Characterizing node performance based on averaged performance distorts the true impact of manufacturing variation on processors on the node and therefore should be avoided
- rwem 7y agoI wonder how much of this is due to measurement error of temperature. Core frequency and voltage control are governed by some suspiciously round numbers like Tj(max) == 90C. But when the controller thinks Tj == 90C, what's the measurement error?
- mlyle 7y agoA moderate amount. Some of the frequency curve involves the thermal diode, but much less so when TDP capping is used as in the paper. But the temperature of the silicon itself has a lot to do with the performance you get at a given power level.
- dchichkov 7y agoWell, yes, they do speculate in the abstract, note potentially there. It looks like authors do believe the variation is caused by manufacturing variation. But nothing in the paper actually shows it. There's no attempt to determine the cause of observed performance variation in the paper. An empirical survey. Interestingly, that recommendation could be interpreted "empirical studies based on averaged node performance" give a very distorted view on "the true impact of manufacturing variation on processors".
- mlyle 7y agoThe entire study seems to presuppose that most of what is measured is processor manufacturing variation. There's further recommendations about removing the variation with processor binning, etc. It's an interesting set of measurements, but the assumed source of the variation is dubious, and it's not clearly what, if any, actions it really supports.
- dchichkov 7y agoYeah. Maybe authors know something we don't. Or maybe were simply trying to get the paper accepted into a silicon-related conference.
- Zenst 7y agoNot just that, but also varitions in motherboard (chipset and anxilary chips), memory, timing crystals - many factors that are equally suspectable to micro-variations. WHich cumulativly, stack up. Then there is that lovely gotcha that unless all your components are from the same batch, you may well see small changes in things like ancillary chips upon motherboards due to chip supplies being from multiple vendors or even vendor variation. Many variables that widen the margin of error and would of been nice if they at least had one system in which they tested a few CPU's, to at least eliminate those extra variables. Ideally, look at the results they got, get the 5 slowest and 5 fastest CPU's based upon the results and retest those upon one system and compare the results. That would of been great and made the difference.
- fyp 7y agoI have a pair of "identical" GPUs which have dramatically different performance. Swapping the order didn't help. It turns out it was because one was blowing hot air into the other. Sigh.
- ddingus 7y agoDamn. When I trip up on that kind of thing, I always just wonder at how absurd it is. Two modern miracles. Insane complex and capable tech. Simple. Hot. Air. Basics. Oh well, maybe laugh. I would.
- pacaro 7y agoSeymour Cray referred to himself as an "overpaid plumber" for good reason. Heat management has been a critical element of high performance computing for a long time
- tonyarkles 7y agoIf I recall, one iteration of their products sprayed a shower of cooled liquid Fluorinert[1] directly into the processor cards, with no heatsinks at all. [2] [1] https://en.m.wikipedia.org/wiki/Fluorinert https://en.m.wikipedia.org/wiki/Fluorinert [2] https://en.m.wikipedia.org/wiki/Cray-2 https://en.m.wikipedia.org/wiki/Cray-2
- Baeocystin 7y agoIt was fully immersed! The Fluorinert was circulated through the stacked cards, then re-chilled with a cold water heat exchanger.
- o-__-o 7y agoI laughed so hard at this and then realized this was probably the source of my SLI woes way back in the day. But for real, I recently replaced the thermal paste on four 10 year old servers and performance improved by 20%!! A full core of performance by ensuring proper heat transfer
- different_sort 7y agoThere's a few SaaS/normal software out there that capitalize on the per chassis/compute unit performance difference amongst identical SKUs. My employer runs a few HPC grid workloads on AWS and Azure and these softwares would benchmark the instances on boot and drop the ones that were below a certain threshold, and we'd keep spawning instances until the fleet was up past our threshold. (Ultimately, we didn't buy the software). I wondered (not too hard) why there could be such a difference in performance, but your comment has given some good insight into that.
- judge2020 7y agodon't think this would work on GCP since they regularly preform 'live migrations' https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati...
- eru 7y agoI think they don't live-migrate pre-emptible instances: https://cloud.google.com/compute/docs/instances/live-migration#preemptiblemaintenance https://cloud.google.com/compute/docs/instances/live-migrati...
- praptak 7y agoThis particular solution doesn't sound feasible in the long term. If a significant portion of customers start using this, the benefit is nullified. And providers will probably react. It's very similar to the "download boosters" of old which worked until they got popular which is when download sites started throttling.
- naikrovek 7y agoWhy is it that people on this site seem to always believe that they know better than the person who took the time to do the work that is cited? The amount of "armchair expertise" here is sickening. Do we really think that someone who is looking at this type of thing in enough detail to produce the graphs in the article wouldn't think of ambient temperature differential? REALLY?! Something not being documented well enough for us does not mean that it wasn't considered... Some thing that we thought of going unmentioned does not indicate that said thing was not considered or tried. This article doesn't mention power supply equivalency or anything about how many people were present in the datacenter during a given test. Here comes the patented HN 'hot take': "uh well actually they don't mention power supplies - are we sure these devices were even powered on?"
- mlyle 7y agoLOL. Maybe have a coffee or find another way to chill out. The paper itself is pretty clear with the methodology: the throughput of each processor was measured in an installed supercomputing cluster. There are obviously sources of variation in processor performance in a supercomputing cluster beyond the actual silicon plugged into the socket. The experiment has no ability to control for this variation, and no attempt was made. The reason why we have research and research papers is so we can read about the methodology and think about the limitations of what is measured. It's an interesting measurement that the researchers made in situ; but it also has obvious limitations.
- mlyle 7y agoJust further lols for you to consider-- the data shows a large difference with the processors in slot 0 being significantly slower than the processors in slot 1. Clearly silicon variation /s. > Third, the median processor performance (shown by the green, blue and red lines) between processor 0 and processor 1 on Sandy Bridge show up to 1% difference. However, that difference in median processor performance increases to up to 5% on Broadwell This socket 1 vs socket 2 variability alone explains 1/3rd of the magnitude of the difference measured in the study-- which makes it quite clear we're not just measuring silicon variability.