9 ms·
Cray, AMD to Extend DOE’s Exascale Frontier
- arcanus 7y ago"greater than 1.5 exaflops" of performance will likely correspond to greater than 1 Exaflop of sustained performance on HPL (used for the top-500 ranking), making this a likely candidate for the first 'true' exascale computer.
- shifto 7y agoBut will it run Crysis?
- azhenley 7y agoThis is just a few miles down the road from me. I can try to see if they will let me run it...
- kincl 7y agoThe call for proposals for INCITE (one of the programs we provide cycles for) is open for 2020 but this would be for Summit not Frontier this time around :) http://www.doeleadershipcomputing.org/proposal/call-for-proposals/ http://www.doeleadershipcomputing.org/proposal/call-for-prop...
- gnode 7y agoTaking this idea more seriously than intended: is this practical? I've heard of CPU clusters being used for network distributed realtime raytracing. Is there a DirectX / OpenGL API implementation for this?
- opportune 7y agoOut of curiosity what makes this an exascale computer and not, say, an AWS or Azure datacenter? Just the fact that they are open about benchmarking #pflops?
- arcanus 7y agoGood question. Technically speaking just running HPL on a cluster is sufficient to get it ranked on top-500. In reality, most HPC sites use high performance interconnects, such as infiniband or a proprietary high performance ethernet. They are also less likely to use virtualization, and every node looks 'bare metal'. The software stacks are very different, from everything between the distributed memory model, compilers, to the system schedulers and diagnostic software. Nevertheless, there is a great deal of convergence happening between HPC and hyperscale data centers, particularly as hyperscale uses more machine learning, which has a similar flavor to HPC. Many believe that the FANG companies have exaflop capabilities already, but they just aren't well optimized for scientific workloads.
- berbec 7y agoWhat's the advantages of infinity fabric over pcie 4 for cpu/gpu? What interconnects do these sorts of machines use? I assume even 100GbE isn't enough? Just curious. It's interesting what exists in the "so far beyond my price range as to be ludicrous" category.
- Symmetry 7y agoPCIe provides communication but isn't intended to provide memory coherency. There's a lot of work that goes on in figuring out which cache(s) have a copy of which cache line and figuring out how to resolve conflicting access needs.
- gnode 7y agoInfinity Fabric / HyperTransport is generally lower level and lower latency than PCIe. It's aimed more for use as a front-side bus than a peripheral interconnect. A better analogue would be Intel's QuickPath Interconnect.
- Grazester 7y agoAs to the interconnect question. They usually use InfiniBand but Cray uses their proprietary interconnect tech.
- tntn 7y agoIMO the coolest thing about Summit/Sierra is that the GPUs and CPUs have a fully coherent single address space with all memory available to the GPUs by default, meaning that your stack- and malloc-allocated variables can be used directly from the GPUs. I wonder if that will be the case on Frontier.
- jacobush 7y agoAmiga 500 all over again. :) I feel these things always go back and forth in cycles in the industry.
- tntn 7y agoI think you'd have a hard time making a case that Summit is a cyclical return to any part of the Amiga 500 :) "fully coherent single address space with all memory available to [all processors]" is a lot easier if there is no virtual memory, no caching, and all processors are on a single shared bus to the same memory chips.
- wahern 7y agoYou left out the qualifier "Single-socket nodes". It's much easier when all the components are on the same die.
- jacobush 7y agoWhich system are you referring to here?
- wahern 7y agoFrom the article, > Single-socket nodes will consist of one CPU and four GPUs, connected by AMD’s custom high bandwidth, low latency coherent Infinity fabric. The article, at least, only describes coherency in the context of single-socket nodes. I suppose it's ambiguous whether Infinity fabric only connects components on the node or connects nodes to each other, but in fact Infinity fabric is the interconnect used with Zen CPUs, so I think it's pretty clear that in this case it only connects and provides memory coherency for 1xCPU and 4xGPUs. Performant memory coherency across nodes would be quite an achievement, but not one they seem to have achieved. Rather, it seems they've simply revived the PS4 model.
- foobard 7y ago> In a media briefing ahead of today’s announcement at Oak Ridge, the partners revealed that Frontier will span more than 100 Shasta supercomputer cabinets, each supporting 300 kilowatts of computing. So 30 megawatts of computing, plus cooling and other supporting services. How do you power something like this? Does ORNL have their own power station (given they have reactor(s) on site)? If power comes from an external station do they coordinate with the station operator when bringing a system like this online?
- noahl 7y agoOak ridge national laboratory was built where it is partly because they could get lots of cheap power from the TVA, so probably from that. (TVA is a regional electricity provider that operates a lot of hydro plants.)
- nthompson 7y agoThey do not have their own power station. They have the Bull Run coal plant and hydro plants in the area. They do coordinate with TVA before a run.
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- petschge 7y agoThey might coordinate with TVA for transients (i.e. going from an idle machine to a full-machine run), but in normal operations these systems are at least 50% full. In my experience as a user these machines are more than 80% most of the time. I don't have hard numbers on the average utilization over the last week/month/year (these might even be classified), but you don't by and build such a machine to be idle. I could find the numbers for two German super computers I have used in the past. SuperMUC (Phase 1 and 2 combined) had "above the desired 85%" utilization in 2017 and Hazel Hen at HLRS in Stuttgart reported a utilization "between 92% and 98%" in 2017.
- BooneJS 7y agoLooks like Frontier will use Cray’s Slingshot network. https://www.cray.com/products/computing/slingshot https://www.cray.com/products/computing/slingshot https://www.anandtech.com/show/14302/us-dept-of-energy-announces-frontier-supercomputer-cray-and-amd-1-5-exaflops https://www.anandtech.com/show/14302/us-dept-of-energy-annou...
- gok 7y ago1.5 exaflops, 30 megawatts, around 50 GFLOPs per watt? Impressive if true; that's 3x more efficient than the current top of the Green500.
- Symmetry 7y agoMoore's law might have stopped with the clock gains when Denard scaling gave out but we've still got energy efficiency gains. Koomey's law[1] is holding strong. I don't know if it'll get us all the way to Landauer's Limit[2] but I hope so. [1]https://en.wikipedia.org/wiki/Koomey%27s_law https://en.wikipedia.org/wiki/Koomey%27s_law [2]https://en.wikipedia.org/wiki/Landauer%27s_principle https://en.wikipedia.org/wiki/Landauer%27s_principle
- tntn 7y agoIt's probably not a great comparison to compare the theoretical numbers of Frontier to the achieved numbers of the green 500. The achieved flops is pretty much always considerably lower than the theoretical flops. Titan is a 27 PFLOPS machine that achieves 17.6 PFLOPS, sequoia is a 20 PFLOPS machine that achieves 17, summit is a 200 PFLOPS machine that achieves 143, ... ~ 37 GFLOPS/W is probably a better projection if we assume (out of nowhere) that the theoretical/achieved ratio of Frontier is comparable to Summit (75%). Still very impressive.
- gok 7y agoWell they don't usually hit their peak power usage either; Summit is rated at 13 MW but used less than 10 during its LINPACK run. But fair point.
- kincl 7y agoGPUs help considerably here, taking a look at previous ORNL machines: Jaguar | 2.3 PF XT5 (CPU-only) | 7 MW for HPL Titan | 27 PF XK7 (1:1 CPU to GPU) | 8.2 MW Summit | 186 PF (2:6 CPU to GPU | 8.8 MW Overall, substantial changes in computing power and 10%-20% increase in power.
- cr0sh 7y agoSo - on a more "applies to ordinary mortals" level - the fact that they are going to use all AMD components is intriguing. In reference to AI, NVidia has things "locked up" with CUDA, versus 2nd cousin AMD's OpenCL. From what I understand, it is possible to recompile TensorFlow (for instance - not that ORNL will be using TF) for OpenCL - but I don't know how well it works. Personally, I've only used TF with CUDA. Does this mean we might see greater/better support for OpenCL in the AI realm? Might we seem it become on-par with CUDA because of this collaboration for this HPC? Or will things stay as-is, at least "down here" in the consumer/business realm of AI hardware and applications? Do things like this trickle down, or are things so customized and/or proprietary for the needs of HPC at ORNL (or elsewhere) that anything to do with AI on this machine will have little to no bearing outside of the lab? Ultimately, I'd just like to see another choice (a lower cost choice!) for GPU in the world of consumer/enthusiast/hobbyist AI/DL/ML - while today's higher-end GPUs, no matter the manufacturer, tend to be fairly expensive, AMD still has an edge here that make them attractive to users (not to mention the fact that their Linux drivers are open-source, which is also a plus).
- petschge 7y agoI doubt they are going to run much AI on that machine. The national labs mostly run "traditional HPC" workloads such as fluid codes that simulate (magneto)hydrodynamics in one way or another.
- tntn 7y ago... except for on Summit. "Exascale Deep Learning for Climate Analytics" won the 2018 Gordon bell using summit.
- petschge 7y agoThey have done that and it got an award because it was new and cool, but that is not where the majority of cycles goes. That is spent on fluid codes such as Raptor, XGC or Flash and to some extend kinetic simulations with codes such as VPIC or quantum codes such as QFCPACK. There should be more information at https://www.olcf.ornl.gov/leadership-science/ https://www.olcf.ornl.gov/leadership-science/
- ksec 7y agoI wonder if this will help the adoption of ROCm. It seems everything I read settled on CUDA.