6 ms·
OpenCAPI Unveiled: AMD, IBM, Google, Xilinx, Micron and Mellanox Join Forces
- zmanian 10y agoAnand Tech seems quite excited about OpenPower and it does tend make me more interested in the Talos platform. https://www.raptorengineering.com/TALOS/prerelease.php https://www.raptorengineering.com/TALOS/prerelease.php
- eeks 10y agoHopefully this effort will get us rid of PCIe for good, unlike the version of CAPI available on POWER8.
- algorithmsRcool 10y agoWhat is wrong with PCIe? Hasn't it been extremely successful?
- sargun 10y agoWhat's wrong with PCIe? The devices are ubiquitous. It's point-to-point, allowing for device to device connectivity. We have external PCIe enclosures and cables. We have a healthy set of PCIe switches. This is similar. It's also point-to-point, and it has an external story. Skimming the spec, they've even thought about higher latency (200ns) links, and optical. Even with these advantages, I'm unsure how it'll work due to the lack of IP available as compared to PCIe. Interesting, though. Will be fun to see it play out. It bums me out a little bit though that POWER isn't easily available yet.
- eeks 10y agoPCIe is still rather slow. A single-packet transaction on PCIe costs 120ns. The lack of IP is the reason why P8/CAPI stuck to PCIe. In the original design, CAPI was to simply reuse the PCI link layer, not the transaction layer. With a cache-coherent system based on PCIe, especially when the coherency layer is set at L3 like on POWER8, you are looking at ~500ns latency for a single cache line. This kind of latency is just too much for many applications.
- theandrewbailey 10y agoIt depends on how low latency they mean by 'low latency'. If it can't drive fat gaming GPUs to full utilization, PCIe will be around for a while still. Also, Intel isn't joining up, so PCIe is absolutely sticking around.
- sargun 10y agoIn the spec, they say the maximum acceptable network delay is 200ns. The smallest network delay is 5ns.
- codebook 10y agoIt is really hard to believe that OpenCAPI can achieve this short latency with 25Gbps per lane interface and off-chip connectivity.
- iheartmemcache 10y agoXilinx offers 25gbps single-lanes that can bond up to 4x to get IEEE 802.3-2012 spec compliance for free* with their suite. Sure, you're going to need to control those trace impedences and your board won't be something coming out of OSH Park, but those are definitely attainable speeds for the consumer (e.g., in the single-thousands of dollars; not 800k Cisco VXR tier-1 infrastructure). You can configure it in CAUI-10 (10 lanes x 10.3125G) or CAUI-4 (4 lanes x 25.78125G), either way, it's been production-ready for quite some time now. (The docs have numbers, but trust me, you can get full throughput within that 200 ns). There's even production Agilent off-the-shelf test equipment out there that can fully sample at those speeds (none of that over-sampling tomfoolery, we're talking live, Bill O'Reilly style). In 1989, UltraSPARC had similar facilities (SBus) to push 100MBit between other Sun machines, so I mean, not too insane comparatively. * Free with purchase of Virtex® UltraScale™ and Kintex® UltraScale FPGA required haha.
- codebook 10y agoThanks for the info. I would like to search the term and understand how they can achieve 200ns latency. Maybe am I the only one who thinks 200ns latency as completion latency?
- codebook 10y agoStill, many devices weren't able to fully utilize PCIe bandwidth and also latency. Turn around time is 1us to 1.5us for typical PCIe IP. IMO, I'm a little bit pessimistic for OpenCAPI to reduce this latency much, unless it directly connects inside the chip.
- KSS42 10y agoSee also : http://www.ccixconsortium.com http://www.ccixconsortium.com
- KSS42 10y agoSee also : http://www.ccixconsortium.com http://www.ccixconsortium.com
- KSS42 10y agoSee also : http://www.ccixconsortium.com http://www.ccixconsortium.com
- samfisher83 10y agoNo Intel ? Without them joining on board this might not be as useful.
- nickpsecurity 10y agoWe have every big name in this, including an x86 vendor. We don't need Intel. That's not a statement I can make often. :)
- samfisher83 10y agoIntel has 99% of the server market.
- DannyBee 10y agoThe people involved or wanting to use this buy so many chips from intel that if intel doesn't get on board, it's going to likely turn out badly for intel. I note that Facebook and Intel are missing, which makes me wonder if they are off in a corner somewhere doing their own thing.
- manawy 10y agoor it could turn badly for the others, it will not gain attraction if intel products are not supported Inertia is a very strong decisive factor, especially when you need to make sure that 30+ year-old code still work like it's the case in HPC
- nickpsecurity 10y ago"especially when you need to make sure that 30+ year-old code still work like it's the case in HPC" Most code that works on Intel works on AMD. It's rare that it doesn't. The HPC vendor will have low risk on migration + get a bunch of competing accelerators at various price points for their problem. This is quite an incentive to move even if there would be stragglers.
- madenine 10y ago> "NVIDIA is a member of the OpenCAPI consortium, at the "contributor level", which is the same level Xilinx has. The same is true for HPE (HP Enterprise)" Awesome.
- KevinEldon 10y agoHewlett-Packard Enterprise is also a member of this consortium (the source article was updated to reflect this)
- PhantomGremlin 10y agoWas "anyone" as "annoyed" as I was about the "scare quotes" littered "throughout" the article? It was "hard" to read the "story". I've been guilty of using too many quotes in the recent past. Someone on HN called me on it, and I've since toned it down. Now it's something that sticks out at me.
- joblessjunkie 10y agoIt makes it seem like the "journalist" doesn't believe his own "reporting".
- bch 10y agoIt felt so buried in the article, it's worth pointing out: OpenCAPI == "Open Coherent Accelerator Processor Interface"
- X86BSD 10y agoThank you. I wasn't about to dig just to find that out. Thanks :) +1
- mablap 10y agoThis is what I was looking for before deciding to read the article or not. Didn't find it, came back here, and now I just won't bother. Unfortunate.
- deleted 10y ago[deleted]
- appleflaxen 10y agoThanks for posting this. What does "coherent" mean in this context, and why is it special?
- cmrx64 10y agoBasically, https://en.wikipedia.org/wiki/Cache_coherence https://en.wikipedia.org/wiki/Cache_coherence. You want to ensure all the devices on the bus have the same view of memory contents.
- crudbug 10y agoExciting news on the hardware bus side. Now waiting for AMD Power9 processors.
- visionscaper 10y agoHmmm, interesting. I wonder what this means for the new Intel Xeon Phi Knights Landing? I liked the approach of many cores on one bootable chip, all having a reasonable amount of local memory, and high bandwidth interconnects: no need to offload data to a peripheral (GPU) device. However with this standard the currently limited bandwidth between peripherals and the main cpu will improve a lot. To me it is obvious why Intel is not joining this party.
- convolvatron 10y agowell, i think its a bit more about market segments and interoperability than anything else. currently overall systems from ibm, and, and intel are fundamentally incompatible. the PCI-E bus that knights landing hangs off of is a qualitatively different thing than the kinds of memory-coherent inter-cpu busses that are being addressed with the CAPI proposal. Intel has their own proprietary QPI. AMD has a quasi-open hyper transport (still?). If this is done properly it means you could make generic motherboards, and generic memory controllers, and all sorts of different accelerators and mix and match them from various vendors. So its no surprise that the smaller players in the market and trying to gang together and the larger player is trying to keep lock-in. Knights Landing as it stands would already integrate in systems better if it were using an inter-cpu bus than a peripheral bus as well as GPUs, FPGAs, and certainly RDMA/memory window systems like Mellanox. Inherent distrust of standards aside, this could be a great win for people putting together bespoke systems in interesting configurations (i.e. Google), I don't think there is any downside in theory for Intel except more competition.
- gpderetta 10y agoThere are KNL variants that sit on a cpu socket and talk to the rest of the system via QPI.
- convolvatron 10y agoI haven't caught up with the latest phi releases, but I'm really interested to do so. I haven't been able to find any discussion about a coherent QPI on KNL, but have found reference to OmniPath, which looks like a non-coherent large scale memory network. Is that what you were thinking of, or maybe could you post a reference?
- honkhonkpants 10y agoGiven AMD's involvement, what is the difference between this and coherent hypertransport?
- wmf 10y agoThe difference is that NVIDIA, Mellanox, and Xilinx never adopted coherent Hypertransport.
- honkhonkpants 10y agoThere was at one point cHT IP available for Xilinx. But that is what I am kinda getting at. Did cHT fail for reasons of pure timing? Too far ahead of the market?
- praseodym 10y agoRelated post on the Google Cloud Platform Blog: https://cloudplatform.googleblog.com/2016/10/introducing-Zaius-Google-and-Rackspaces-open-server-running-IBM-POWER9.html https://cloudplatform.googleblog.com/2016/10/introducing-Zai...
- valarauca1 10y agoI really enjoy the way AMD and Nvidia fight. Nvidia makes GSync. So the GPU can control and adjust display refresh rate on the fly. Proprietary, closed source, requires private Nvidia License. AMD makes FreeSync. FreeSync open sourced, added to the HDMI, and Display Port standards. Now we see Nvidia makes NvLink. Proprietary, very fast, requires private Nvidia license. AMD partners with IBM, Google, etc., etc. to make OpenCAPI an open standard that can revise/replace PCIe3.0 Why does this feel like Microsoft vs Linux but with Hardware Standards.
- asimuvPR 10y agoLinus gives the best answer: https://youtu.be/IVpOyKCNZYw https://youtu.be/IVpOyKCNZYw tl;dr: Nvidia, fuck you.
- arcanus 10y ago> I really enjoy the way AMD and Nvidia fight That's why I was so sad AMD was having trouble later year and hope they pull through. Competition is valuable and benefits the consumer!