12 ms·
AMD 3rd Gen EPYC Milan Review
- modzu 6y agohow is a 300W cpu cooled in a server environment? just high rpms and good environmentals? ive stayed on intel with my workstation so i can keep a virtually passive and quiet heatsink without having to go water
- Latty 6y agoYeah, noise is relatively unimportant so they tend to use a ton of extremely high RPM fans and big hunks of copper, from what I've seen.
- wffurr 6y agoAnd the DC staff wears hearing protection when they're working among the racks.
- myself248 6y agoThat still blows my mind. Coming from telecom where everything prior to the #5 ESS was convection cooled, a happy office is a quiet office. Data got weird.
- rbanffy 6y agoI'm assuming telecom had very different volume-power requirements. Where I grew up there were many mid-city phone switches that were large, concrete exterior, almost windowless, buildings.
- singlow 6y agoI wish that hearing protection had been required or at least offered when I used to visit data centers frequently. They made a big deal of the fire suppression training, but never even suggested ear plugs. 20 year-old me had no idea how bad that noise was for my ears. I hope the staff there were wearing plugs, but it was never apparent.
- mhh__ 6y agoYou only get one set of ears. I suspect I still have the record for most expletives used in front of the headmaster at my old school because someone turned a few kW speaker on while I was wiring something under the stage - i.e. I wasn't pleased.
- willis936 6y agoServers are in 4U rack mounted enclosures with (relatively) low height heatsinks and huge amounts of airflow. Intake in front, exhaust in rear. Most clients will have beefy air conditioning to keep the ambient intake temp and humidity low.
- wtallis 6y agoI think 4U servers are quite rare these days, except for models designed to accommodate large numbers of either GPUs or 3.5" hard drives. Most 2-socket servers with up to a few dozen SSDs are 2U designs.
- myrandomcomment 6y agoAgreed. My last company we used a 1RU with 2 blades (servers) in that space. Front to back cooling, quite loud. No need to worry about the noise in the DC. Power density in a rack vs computer density is all that matters.
- touisteur 6y agoY'all can try to pry my quad-cpu 3-upi 4U HPE DL.580 from my cold dead hands.
- deleted 6y ago[deleted]
- epmaybe 6y agoPerformance per watt is better on amd right now, at least last I heard.
- unicornfinder 6y agoPerformance per watt is better but unfortunately the idle power consumption is fairly high.
- epmaybe 6y agoOh I see that in the article now. It also seems like the performance gains you saw on other Zen3 chips doesn't correlate with EPYC due to the sheer number of cores and other components on chip.
- kllrnohj 6y agoEpyc's idle power consumption is fairly high but Ryzen's isn't. The more workstation-focused Threadripper & Threadripper Pro is also still significantly better than Epyc here.
- bsoft16385 6y agoZen2/Zen3-based, non-APU Ryzen still has pretty awful idle power consumption compared to Intel desktop CPUs. My 5900X at idle draws around 22W. Modern Intel desktop CPUs (which are basically all Skylake refreshes right now) are 8-10W at idle.
- kllrnohj 6y agoSure, but 22W is still a trivial amount of heat & power in the form factor of a desktop system. Fans don't need to spin for that (or if they do it's at the lowest RPM - still silent). Fans do need to spin, and with some moderate oomph, to handle 80W, though.
- fomine3 6y agoIs TR Pro really better? I suspect that it's just difference between motherboard because it's basically same as EPYC.
- formerly_proven 6y agoTypical servers are 1U or 2U high boxes. 1U=44.5 mm and each box is "full depth", so around 700 mm long (sometimes more), and ~450 mm wide. The 1U boxes typically just have a block of copper with a dense bunch of fins on them as a CPU cooler, while 2U designs usually incorporate heatpipes. Lots of airflow. High-density systems are usually 2U, but with four dual-socket systems per chassis.
- ben-schaaf 6y agoTo put it in perspective, the Intel Xeon 6258R is rated at 205W. Servers have oodles of (likely airconditioned) airflow so this isn't a problem.
- folago 6y agoWhen I worked in HPC in Tromsoe, arctic Norway, one of local supercomputers was liquid cooled and the heat used warmed up some buildings on the university campus.
- deleted 6y ago[deleted]
- Out_of_Characte 6y agoa cooling unit provides a delta between surface area temparature and the ambient temparature. Epyc chips are significantly larger and more spread out with the chiplets so the heat density of these chips are relatively similar. So the cooler doesn't have to provide a larger temparature delta and ony has to be slightly larger if at all. Overall heat density on chips has increased due to litography changes so the chiplet architecture in a way is only a stopgap.
- numpad0 6y agoBy an array of fans specced like 8cm 10W 10krpm, like six fans wide two fans deep, blowing into a passive heatsink, with help of air ducts inside the chassis Intel or AMD or SPARC or ARM the setup is all the same for rackmount hardwares, pure copper passive heatsink and high power axial fans
- rodgerd 6y agoServers are horrifyingly loud, and datacentres will destroy your hearing in no time flat. Airflow management in datacentres is quite the art form, as well: proper packing of the racks for airflow, height of raised floors to accommodate blown air, and so on; some people are starting to go back to the future with things like liquid-cooled racks as well.
- greggyb 6y agoDon't forget that you could feasibly use a 280W 64-core processor to replace a dual-socket 2x205W 2x28-core Intel box with 8 cores to spare. Obviously, there will be performance deltas based on specific use cases, but this should help frame the comparison.
- rbanffy 6y agoWhy so little L3 cache on the competition?
- ChuckNorris89 6y agoLimitations due to die size and power consumption since Intel Xeon is still on the ye olde 14nm++ process. Also since Xeon dies are monolithic, unlike AMDs chiplet design, means that increasing the size of certain components on the die, like cache for example, increases the risk of defects wich reduces the yields, making them unprofitable.
- rbanffy 6y agoTrue, but the ARM ones have just 32MB for up to 80 threads. I wonder if we could get numbers for L3 misses and cycles spent waiting for main memory under realistic workloads.
- dragontamer 6y agoThat information changes with every application. Literally every single program in the world has its own cache characteristics. I suggest learning to read performance counters, so that you can get information like this yourself! L3 cache is a bit difficult for AMD processors (because many cores share the L3 cache), but L2 cache is pretty easy to work with and profile. General memory-reads / memory latency is pretty easy to read with various performance counters. Given the amount of latency, you can sorta guess if its in L3 or in DDR4.
- greggyb 6y agoFYI, 'ye olde' is actually using the letter 'y' to represent an Old English 'thorn' character. The thorn character represents the 'th' sound. Printing presses gained prominence in non-English speaking regions earlier than in English speaking regions. The 'y' was substituted for the 'thorn' due to some amount of similarity in shape. Thus, 'ye olde' would be pronounced 'the old'. 'Ye' is just a different form of 'the' from a time when 'y' represented 'th'. So, 'the ye olde' would be read as 'the the olde'.
- rbanffy 6y agoThere must be a typo on the 74F3 price. US$2900 for it is a steal.
- tecleandor 6y agoI found it in the original press release (price for !K units, of course) https://ir.amd.com/news-events/press-releases/detail/993/amd-epyc-7003-series-cpus-set-new-standard-as-highest https://ir.amd.com/news-events/press-releases/detail/993/amd...
- masklinn 6y agoStill seems like a typo, it doesn't make any sense that the 24 / 48 would be priced between the 8 / 16 and the 16 / 32. Either the prices of the 73 and 74 were swapped or the tag is just plain wrong. "2900" is also very suspiciously round compared to every other price on the press release.
- fvv 6y agoright, I think 3900 may be the correct price
- dragontamer 6y agoHow is it suspicious? 256MB L3 (or really, 8 x 32MBs L3) and 24-cores suggests the bottom-of-the-barrel 3 cores active per 8-core CCX. 8x CCX with 3-cores. The yields on those chips must be outstanding: its like 62.5% of the cores could have critical errors and they can still sell it at that price. EDIT: My numbers were wrong at first. Fixed. Zen3 is double-sized CCX (32MBs / CCX instead of 16MBs/CCX) --------- In contrast, the 28-core 7453 is $1,570. I personally would probably go with the 28-core (with only 2x32MB L3 cache, or 64MBs) rather than the 24-core with 256MBs L3 cache. For my applications, I bet that having 7-cores share an L3 cache (and therefore able to communicate quickly) is better than having 1 or 2 cores having 32MBs of L3 to themselves. There are also significant price savings, as well as significant power / wattage savings with the 28-core / 64MBs model.
- stillbourne 6y agoI'm waiting for news on genesis peak, I'd love to get 4th gen threadripper on my next box.
- MrFoof 6y agoComputex 2021, June 1st. * Computex 2017 is when Zen Threadripper was announced. * Computex 2019 is when Zen 2 Threadripper was announced. * Computex 2020 is when Zen 2 Threadripper PRO was going to be announced. After the show was delayed, and then officially canceled, AMD did their own announcement shortly after. So odds are it'll be Computex 2021 on June 1st. This fits in with the usual release timing, as Threadripper tends to follow EPYC by a few months. I'd expect retail channel availability for Threadripper as early as late July, as there's usually a few months between announce and channel availability. Trust me, I've been waiting to build a 5970X as well. :)
- stillbourne 6y agoDude, thanks, I've been looking for any scrap of rumor or release date.
- whatshisface 6y agoThat logarithmic curve fit to the samples from a step function...
- bryanlarsen 6y agoHopefully they can fix their idle power consumption with a firmware tweak or a new stepping, that's a massive regression. It looks like that causes a significant performance degradation too -- more of the power budget going to IO means less to the compute cores.
- ece 6y ago> ...The core-to-core latency within a quadrant this generation has improved from a worst-case 112ns to 99ns, about a 10ns improvement. Access to remote quadrants has been reduced from up to 142ns to up to 114ns, which is actually a 24% improvement, which is considerable. > What’s really interesting is that inter-socket latencies have also seen very notable reductions. Whereas the Rome part these went up to 272ns, the new Milan part reduces this down to 202ns, again a large 25% improvement between generations. Good I/O is going to have a cost in terms of die area and power. Granted what Anandtech observed is interesting for 2P systems. Maybe there's a NPS4 vs NPS1 difference at work here, or AMD just isn't doing as much power gating on the IOD as it could in one or either case.
- temptemptemp111 6y agoBut why won't AMD let us do any secure booting?
- zhdc1 6y agoIt looks like Zen 2 processors about about to become even more of a bargain than they already are. I'll take a 7702P at $2-3K over a 7713P at $5K ten times out of ten.
- dragontamer 6y agoConsidering both are made from the 7nm TSMC process, AMD probably aren't going to make any more Zen2 processors at this point. I think you're right: that buying a generation old or so can offer gross cost savings. But that's only true for the time-period where those chips are available.
- IanCutress 6y agoAMD is going to keep Zen 2 EPYC sales going for a good while yet. Both families will co-exist in the market.
- blagie 6y agoI suspect so. A lot of the commercial market wants stability. Once I've validated a server config for a particular use, I want to be able to continue building those servers for a long time (often long past obsolescence). That may seem odd, but a lot of safety-critical applications (e.g. medical, military, aerospace, etc.) require spending tens of thousands, hundreds of thousands of dollars, or even millions of dollars (not to mention months of time) re-validating a system with any substantive change. Even for less critical applications, spending $2000 extra on each CPU is a bargain compared to re-validating a system. If AMD wants to be a credible presence in those markets, and I'm pretty sure it does, it needs to chips with many year lifespans before EOL. Some companies manage this by having a subset of devices or of software which is LTS.
- derefr 6y agoRather than buying new-old-stock CPUs, why not just buy all the CPUs the long-term program will ever need when they're still cheap, and stockpile them? It's not like they go bad.
- blinkingled 6y agoINVLPGB New instruction to use instead of inter-core unterrupts to broadcast page invalidates, requires OS/hypervisor support VAES / VPCLMULQDQ AVX2 Instructions for encryption/decryption acceleration SEV-ES Limits the interruptions a malicious hypervisor may inject into a VM/instance Memory Protection Keys Application control for access- disable and write-disable settings without TLB management Process Context ID (PCID) Process tags in TLB to reduce flush requirements Interruptions (Instructions) and Unterrupts (Interrupts) aside (the article obviously was pushed out as fast as AT could lol) - these additions seem like they would help with performance when it comes to mitigations of all the speculation vulnerabilities in an hypervisor env?
- fvv 6y agoBenchmarks https://www.phoronix.com/scan.php?page=article&item=epyc-7003-linux-perf&num=1 https://www.phoronix.com/scan.php?page=article&item=epyc-700...
- IanCutress 6y agoThe article linked at the top has pages of benchmarks. Did.... you miss them?
- mattashii 6y agoThere are benchmarks, yes, but Anandtechs' bechmarks are mostly synthetic benches from systems almost no-one uses (I don't run SPEC benchmarks daily, and ok, I compile stuff, but not clang, nor using clang) But Phoronix has synthetic benchmarks from systems quite a few people use (e.g. databases, rendering, simulation). SPEC is nice, but I wouldn't know where to look if I would want to apply that to e.g. postgres. With Phoronix' results, at least there is an idea of how I could apply the results of the benchmark to my usage. So I find this link quite useful. That's not to say that I think Anandtechs benchmarks are not valuable (I love reading the nitty gritty details like cache timing specifics), but I can't apply them to anything other than a generic CPU workload, not even slightly in the direction of my specific workload.
- dragontamer 6y agoSPEC benchmarks are hardly synthetic. They're Perl programs like SpamAssassin, or GCC-compile times, and other "server-like" tasks. So one can maybe argue that those workloads don't match your use case. But... SPECINT is the "standard server benchmark" for a reason. I think there's been some suggestions that SPEC benchmarks are a bit small these days and maybe bigger code would be more realistic. But the actual programs they use are decent.
- mattashii 6y ago> SPEC benchmarks are hardly synthetic. They're Perl programs like SpamAssassin, or GCC-compile times, and other "server-like" tasks. That I didn't know. SPEC doesn't really do much in regards to giving descriptive names to benchmarked items, so its hard to determine what the numbers indicate (other than "this benchmark, which emulates some workload, now has X higher performance"). Some names are descriptive-ish (I think I recognise 7 out of 22 names) but it's all gibberish without a link to why those benchmarks are run and what they are based on (at least not in a way similar to how SPECjbb is explained). It might just be me failing to find it, but searching SPEC in the page (or their website) doesn't seem to give me relevant links. Does Anandtech have a page detailing the rationale behind their current (CPU) benchmark suite?
- ChuckMcM 6y agoNice bump in specs. Perhaps now they will announce the Zen3 Threadripper :-). As others have mentioned the TR can starve itself on memory accesses when doing a lot of cache invalidation (think pointer chasing through large datasets). If the EPYC improvement of having the chiplet CPUs all share L3 cache moved into the TR space (which one might assume it would[1]) then this could be a reason to upgrade. [1] I may be wrong here but the TR looks to me like an EPYC chip with the mult-CPU stuff all pulled off. It would be interesting to have a decap with the chiplets identified.
- mcdevilkiller 6y agoIn fact, all Zen chips are equal in the cores part (except for clockspeed and other tuning) thanks to the reusability of chiplets. The only thing that differs between Ryzen, TR amd EPYC is the IO die. And the pins (TR and EPYC use the same package, yep).
- kllrnohj 6y agoWith the notably exception of ryzen mobile, which doesn't use chiplets.
- gameswithgo 6y agoyes TR will have the new cache configuration, just like regular ryzen and epyc do.
- eqvinox 6y agoKinda wish there was a 3rd gen 7282 equivalent, for those of us just building their home servers. 2 sockets with <$750/CPU was kinda the sweet spot there. (... really need to replace my 2×E5-2630v2, it's getting quite old... but I guess a 2×7282 will do.) [Ed.: nevermind, hadn't read to the "idle consumption" graphs yet... nope, I'll stick with Rome ...]
- wmf 6y agoWhy would two weak sockets be better than a single-socket system?
- jedbrown 6y ago16 channels of DDR4-3200, versus 8 in single socket EPYC or 4 for Threadripper.
- coder543 6y agoI think someone concerned with paying "<$750/CPU" would be unlikely to want to fill 16 channels of RAM... that's a lot of money, relatively speaking. But, maybe their home server's singular purpose is to have a lot of RAM?
- eqvinox 6y agoThe memory (64GB ea) for each socket in my old server cost about as much as the CPU for the respective socket, about 550€ each. (i.e. 1100€ per socket.) For a 7282 box I'd spend about as much on the RAM as on the CPU again; 16GB DDR4-3200R sticks are about 80€ each, so filling one socket is 640€. The 7282 currently sells for ~700€ currently so that lines up all right. (This would give me 2×128GB total; 32GB RAM sticks are unfortunately indeed still a bit too expensive, especially at 3200MHz rating. But maybe I should go with 32GB and half fill each socket & fill it out later when it's cheaper...)
- eqvinox 6y agoHave you even looked at the offerings? The P-series/single socket CPUs are, at the same specs, only slightly cheaper than the dual socket ones. 2×7282 gives me 32 cores at $1300 list price. 7443P is $1337 for 24 cores. 7502P is even more expensive. Two 7282s is pretty much the best perf/power/money option as far as I can see.
- fomine3 6y agoThis is most interesting part of interview posted at the same time. https://www.anandtech.com/show/16548/interview-with-amd-forrest-norrod-milan https://www.anandtech.com/show/16548/interview-with-amd-forr... > IC: While AMD increases the performance on its processor product line, the bandwidth out to DRAM remains constant. Is there an ideal intercept point where higher bandwidth memory makes sense for a customer? > FN: I think you’re absolutely right, and really at the top of the stack, depending on the workload, that can be the performance limiter. If you’re comparing top of the stack parts in certain workloads, you’re not going to see as much of a performance gain from generation to generation, just because you are memory bandwidth limited at the end of the day. > That’s going to continue as we keep increasing the performance of cores, and keep increasing the number of cores. But you should expect us to continue to increase the amount of bandwidth and memory support. DDR5 is coming, which has quite a bit of headroom to DDR4. We see more and more interest in using high bandwidth memory, for an on-package solution. I think you will see SKU’s in the future from a variety of companies incorporating HBM, especially for AI. That will initially be fairly specialized to be to be candid, because HBM is extremely expensive. So for most the standard DDR memory, even DDR5 memory, means that HBM is going to be confined initially to applications that are incredibly memory latency sensitive, and then you know, it’ll be interesting to how it plays out over time.
- ece 6y agoI would love to see desktop systems move to 4 channels. 2 DIMMs/channel can die as far as I'm concerned, lets have 2 channels/DIMM for laptops. Memory channel density needs to increase along with core density (wider and/or more cores).
- wmf 6y agoThey won't. DDR5-6400 doubles memory bandwidth and that's what we'll get for the next decade or so.
- ksec 6y agoThey are still using GF 14nm on their IOD due to Wafer Supply Agreement. We will have to wait and see how they fit their iOD with PCI-Express 5 and DDR5 with Zen 4. And I cant wait to see Netflix plays around with these in their FreeBSD box. It is interesting the biggest upgrade from Zen 4 won't actually the Core part but the iOD. As I am expecting Zen 4 to be some tweaks and a Die Shrink to 5nm. TSMC also somehow unexpectedly announce doubling their 5nm capacity with new Fab capacity being built. My guess would be an aggregate demand from Apple and AMD exceed certain threshold to be worth doing.
- freedom42 6y agoA bit offtopic. All AMD, Intel and ARM processors come with binary blobs that can't be disabled or removed which could potentially control your whole computer (Intel Management Engine, AMD Platform Security Processor). Really powerful desktop and server-grade processors are available with fully open source firmware and hardware (even fully open BMC). It's not that expensive, comes with PCIe 4.0 and other modern features [1]. They are POWER9 processors, not x86_64. But many software work on it, checkout 'talos-workstation' on IRC to chat with the users [can access IRC through Element Matrix app] - someone is running Android Studio on it (yeah so much for privacy but still). [1] https://www.raptorcs.com/ https://www.raptorcs.com/ These systems are currently being used by businesses and developers alike, including Google.
- wmf 6y agoHonestly, Power9 is not really powerful compared to Milan; it's 1/3 to 1/2 the performance at best. https://openbenchmarking.org/vs/Processor/AMD%20EPYC%207763%2064-Core,POWER9%2044-Core https://openbenchmarking.org/vs/Processor/AMD%20EPYC%207763%... The desktop-grade Power9 gets rekt by a Core i3: https://openbenchmarking.org/vs/Processor/POWER9%204-Core,Intel%20Core%20i3-10100,AMD%20Ryzen%205%205600X%206-Core https://openbenchmarking.org/vs/Processor/POWER9%204-Core,In...
- freedom42 6y agoEdit: Commented something without reading, deleted. The advantage of POWER9 is fully open firmware and hardware while being really powerful. They won't beat the latest AMD in performance. But AMD can never be open. The RaptorCS products are FSF RYF (respects your freedom) certified. https://ryf.fsf.org/vendors/raptor https://ryf.fsf.org/vendors/raptor Some benchmarks from 2019: https://www.phoronix.com/scan.php?page=article&item=rome-power9-arm&num=1 https://www.phoronix.com/scan.php?page=article&item=rome-pow...