13 ms·
PCI-Sig Releases 256GBps PCIe 6.0 X16 Spec
- synergy20 5y agowhat a coincidence, my boss just asked me to buy the memerbership. keywords: PAM4 and 128GBps/per x16 lane(full width, per my understanding, not 256Gbps as title says), which mean it can be used to make a 1Tbps NIC, or any traffic at the scale. from my reading, we will see pcie6 products mid-2023.
- magicalhippo 5y ago> my boss just asked me to buy the memerbership I just spoke to a friend who, as a consultant, was tasked with verifying that a "motherboard" for a projector conformed to the HDMI specs. He'd gotten a bunch of measurements and numbers in a document, and asked if he couldn't get the full spec as it would be easier to understand what to do. He was told they had the specification but it was at headquarters in Belgium, on a computer that was not connected to the internet. He'd have to fly there, view it on the computer and write off what was needed... which was exactly what someone had done to make the document he'd gotten.
- formerly_proven 5y agoHDMI essentially only exists for DRM reasons (i.e. controlling who can make compatible devices); open specs are antithetical to that purpose.
- vel0city 5y agoThe DRM often conflated with HDMI (HDCP) existed years before the first HDMI cable was sold. Or were you saying DRM when it comes to a physical port specification? In which case, that wouldn't really be digital rights management.
- xvilka 5y agoYes, DisplayPort is the only good standard. Hopefully, HDMI will disappear one day.
- ShroudedNight 5y agoWhat does this accomplish? Is there some existentially significant licensing concern that justifies requiring such an onerous process to vet access?
- magicalhippo 5y agoFrom what I gather the HDMI group wants to ensure something marked HDMI just works when plugged into another HDMI device. I guess this gate-keeping is part of it.
- userbinator 5y agoIronically enough, a quick Google found the HDMI spec --- even in this day of horrible search results.
- magicalhippo 5y agoRight, but for a device that's to be sold on the market, would you trust some random PDF from Google? Low volume, expensive products like the one my friend got the job for or consumer products, having to do a recall and board replacement will not be a fun experience for those involved.
- Dylan16807 5y ago5.0 transmits 1 bit at a time per lane, 32 billion times a second. 6.0 transmits 2 bits at a time per lane, 32 billion times a second. So why are they claiming it's 64 gigatransfers per second? Do I misunderstand the term, or are they trying to pull something cheeky?
- Klinky 5y agoMaybe they are considering both directions, i.e. 128GB/sec full duplex = 256GB/sec total bandwidth. Kinda cheeky. https://pcisig.com/blog/pcie%C2%AE-60-specification-released-members-double-bandwidth-next-generation-applications https://pcisig.com/blog/pcie%C2%AE-60-specification-released... > a maximum bidirectional bandwidth of up to 256 GB/s for x16 lanes
- Dylan16807 5y agoThey're definitely considering both directions for the total bandwidth number. But I don't think that's what's happening with the gigatransfers. When they launched 5.0, they were clearly counting one differential pair of pins: "Delivers 32 GT/s raw bit rate and up to 128 GB/s via x16 configuration"
- Klinky 5y agoI am pretty sure that 5.0 128GB/sec figure is also total bidirectional bandwidth with each direction being approximately 64GB/sec
- Dylan16807 5y agoYes, it is. But my point is, even though they were already doing that, they were claiming 32GT/s last generation. But it's 64GT/s now. I don't think that's counting more pins, I think they're saying that each transfer is two transfers.
- shaklee3 5y ago
- vmception 5y agowhat use case does this level of bandwidth open up, and I am not able to understand why the article thinks SSDs are one of them. PCIe already provides more than enough bandwidth for the fastest SSDs, am I missing some forward-looking advancement?
- smiley1437 5y agoFrom what I understand, internally an SSD's bandwidth can be easily scaled by spreading reads and writes across arbitrarily large numbers of NVRAM chips within the SSD. So, you can just create SSDs that saturate whatever bus you connect them to. In a sense then, it is the bus specification itself that limits SSD throughput.
- zionic 5y ago120hz star citizen
- dragontamer 5y agoPCIe 4.0 x4 lane provides enough bandwidth. Ish... we've already capped out with 8GBps SSD actually. PCIe 4.0 x4 lane is now the limiting factor. PCIe 6.0 1x lane would provide the same bandwidth, meaning you run 1/4th as many wires and still get the same speed. Alternatively, PCIe 6.0 4x lane will be 4x faster than 4.0, meaning our SSDs can speed up once more.
- AnotherGoodName 5y agoInternally passing around multiple 4k video outputs would use this. Maybe you don't want the output via the port on the back of the card but want to pass it through internally to some other peripheral? I think this is how thunderbolt ports work right (happy to be corrected)?
- adgjlsfhk1 5y agocurrent top of the line SSDs are close to maxing out 4 lanes of PCIE gen 4. Gen 6 will make it a lot easier to find room for a few 100gb/s ethernet connections which are always nice for faster server to server networking, as well as making it easier to use pcie only storage.
- alberth 5y agoNetflix, I imagine, would love to have this kind of I/O bandwidth.
- SahAssar 5y agoIsn't netflix pretty much capped at network I/O, not disk I/O? All the posts I've read about them have been focused on network.
- willcipriano 5y agoIsn't this just pure I/O? You could have a PCIe 6.0 raid controller or network card.
- toast0 5y agoThe headline number is network I/O, but IIRC, the bottleneck is really in/around system RAM. Since most of the content isn't in cache and neither the disks nor the network cards have sufficient buffering, you need to have the disk write to system RAM and indicate readiness, then the network card reads from system RAM. NUMA bandwidth and latency can be an issue there too. But, I don't think there was room on the pci-e lanes to do something like put a gpu in to use as a DMA buffer to get more ram bandwidth. I think the next Epyc generation should have PCIe 5 and DDR5, both of which should help.
- martinald 5y agoSurely the disk would be more of a problem than RAM?
- toast0 5y agoThere are a lot of disks in a Netflix content appliance. The latest slide says 18 drives, each with PCIe 3.0 x4. There are 4 PCIe 4.0 x 16 nics, and there's all your PCIe lanes. You could get PCIe 4 drives, but it's not the bottleneck at the moment. But, RAM needs at least twice the bandwidth as your network, because you can't have the NIC read from the disk directly, you need to have the disk DMA to ram, and the NIC DMA from ram, and (normal system) ram isn't dual ported, so reads and write contend. If you need to do TLS in software, you touch ram 4 times (disk read, cpu read, cpu write, nic read), so ram bandwidth is an even bigger bottleneck.
- AdrianB1 5y agoThe problem with PCIe is not bandwidth, it is the limit in lanes on consumer PCs: 20 lanes from CPU and a few more from the SouthBridge is not enough when the GPU is usually linked on a 16 lane connection. The easy way out is to reduce GPU lanes to 8, that leaves plenty of bandwidth for nVME SSDs and maybe for 10 or 25 Gbps NICs (it's about time). For servers it is a different story, but the recent fast move from PCIe ver 3 to ver 5 improved the situation 4x, doubling again is nice, but it does not seem that much of a deal. Maybe moving NICs from the usual 8 lane to a lot less (8 lanes of ver 3 means 2 lanes of ver 5 or a single lane of ver 6) will also make some difference.
- alberth 5y agoIsn't this only an artificial problem Intel created to segment the market, a problem that AMD doesn't have.
- csdvrx 5y agoYes they do have that problem, even the upcoming AM5 will only have 28 lanes given the annoucements: https://www.hwcooling.net/en/more-on-amd-am5-tdp-to-reach-120-w-more-pcie-lanes-cpu-image/ https://www.hwcooling.net/en/more-on-amd-am5-tdp-to-reach-12...
- wmf 5y agoAMD has the same or slightly more lanes than Intel.
- the8472 5y agoEPYCs have 128 PCIe Gen4, recent Xeons have 64 PCIe Gen4. And Intel introduced Gen4 later than AMD.
- zamadatix 5y ago> The problem with PCIe is not bandwidth, it is the limit in lanes on consumer PCs... For servers it is a different story EPYC/Threadripper is awesome but also explicitly not the segment being discussed. But since it was mentioned it's 64 PCIe lanes per socket on Intel (up to 8 sockets/512 lanes) instead of 128 lanes for single socket/up to 160 lanes for dual socket on Epyc. Gonna cost you a damn arm and leg and you better know your NUMA though. In the consumer segment AMD has 16 PCIe lanes for the graphics and up to 4 PCIe lanes for NVMe drives, the chipset link is and additional x4 PCIe instead of a different interconnect. This comes to 20 PCIe lanes for user devices and 4 lanes worth of PCIe bandwidth off the chipset. https://i.imgur.com/8Aug02l.png https://i.imgur.com/8Aug02l.png Intel offers something similar, 16 PCIe lanes for graphics and 4 PCIe lanes for NVMe. They opt for a proprietary DMI connection to the chipset which is equivalent to 8 lanes of PCIe 3.0 bandwidth or 4 lanes of PCIe 3.0 bandwith. https://i.pcmag.com/imagery/reviews/070BdprI2Ik2Ecd2wzo0Asi-6.fit_lim.size_768x.png https://i.pcmag.com/imagery/reviews/070BdprI2Ik2Ecd2wzo0Asi-... Each offers splitting the x16 for the GPU into 2 x8 as well as oversubscribing the downstream PCIe bandwidth from the chipset, devices through the chipset on each obviously hit different latency penalties than CPU direct lanes as well. In the end the offerings are both pretty much identical in the consumer space.
- rjzzleep 5y agoHow long does it usually take to get consumer products of new PCIe specs? Fast PCIe Gen 4 is only just getting affordable. Like $350 for 2 TB NVMe ssds. Also, I remember playing around with PCI implementation on FPGAs over a decade ago and timing was already not easy. What goes into creating a PCIe Gen4/5 device these days? How can you actually achieve that when you're designing it? Are people just buying the chipsets from a handful of producers because it's unachievable for normal humans? EDIT: What's inside the spec differences between say gen 3 and 6 that allows for so many more lanes to be available?
- dragontamer 5y ago> How long does it usually take to get consumer products of new PCIe specs? Like 2 years. When PCIe 3.0 was getting popular, 4.0 was finalized. When 4.0 was getting popular, 5.0 was finalized. Now that PCIe 5.0 is coming out (2022, this year), PCIe 6.0 is finalized.
- jeffbee 5y agoPCIe 5 was standardized May 2019 and you could buy it at retail in late 2021. 2 years good rule of thumb.
- formerly_proven 5y agoThere was a much bigger gap between 3.0 and 4.0. PCIe 3.0 was available with Sandy or Ivy Bridge, so 2011/2012. PCIe 4.0 was introduced with Zen 2 in 2019. We seem to be back to a faster cadence now however.
- zamadatix 5y agoThe large delay between 3.0 to 4.0 was a gap between specifications (2010 to 2017) not a gap between specification to implementations (2017 to 2019).
- dragontamer 5y agoWith the rise of GPU-compute, a lot of the supercomputers are playing around with faster I/O systems. IBM pushed OpenCAPI / NVLink with Nvidia, and I think that inspired the PCIe ecosystem to innovate. PCIe standards are including more and more coherent-memory options. It seems like PCIe is trying to become more like Infinity Fabric (AMD) / UltraPath Interconnect (Intel).
- NKosmatos 5y agoGood news for everyone and hopefully this will not interfere with the PCIe 5.0 rollout, even though the 4.0 one is not fully adopted by the market yet. Not that I’d be able to use it, but it’s a pity they make the spec available to members only (4000$/year membership) or they sell it at ridiculous prices https://pcisig.com/specifications/order-form https://pcisig.com/specifications/order-form I know that other specifications and even ISO standards are provided for a fee https://www.iso.org/store.html https://www.iso.org/store.html and perhaps something similar should be applied for open source software to avoid similar issues like faker.js and colors.js
- wmf 5y agoYou might be better off with a book than the spec anyway. Unfortunately I only see books covering PCIe 3.0.
- nwmcsween 5y agoSo my understanding of DDR5 has on chip ECC is needed due to the ever increasing need for -Ofast, will/does PCIe have the same requirements?
- ksec 5y agoYes. Forward Error Correction (FEC) [1] . The Anandtech article wasn't in my feed when I submitted this. As it offers much more technical details. [1] https://www.anandtech.com/show/17203/pcie-60-specification-finalized-x16-slots-to-reach-128gbps https://www.anandtech.com/show/17203/pcie-60-specification-f...
- Dylan16807 5y agoYes it has that, but also no it's not the same. DDR5 has no link-level error correction. The faster and faster DDR5 bus is entirely unprotected on consumer devices.
- monocasa 5y agoThe on chip ECC for DDR5 isn't because of faster memory; it's because of denser memory. It can rowhammer itself, and fox it up silently. And the cheaper brands can start shipping chips with defects like they do with flash, relying on the ECC to paper over it.
- Taniwha 5y agoreading the articles I think that FEC is being used to protect link integrity - it's different from ECC on DRAM which is also protecting the contents (against things like row-hammer and cosmic rays)
- loeg 5y agoPCIe packets have always had error detection at the DLLP layer.
- ChuckMcM 5y agoOkay, that is just stupidly wide bandwidth :-). As a systems guy I really don't see the terabytes of bandwidth in main memory coming in the next 5 years. GDDR6, the new "leading edge" is nominally 768GBs, which sounds great but how many transactions per second can you push through a CPU/GPU memory controller? Are we going to see 1152 bit wide memory busses? (1024 bits w/ECC) That is over 1/3 of all the pins on a high end Xeon and you're going to need probably another 512 pins worth of ground return. HBM soldered to the top of the die? Perhaps, that is where GPUs seem to be headed, but my oh my. I'm sure on the plus side there will be a huge resale market for "combination computer and room heater" :-)
- Terry_Roll 5y agoDatabase running in ram will benefit enormously, just running an ISAM db on a ram drive massively increases performance. RDBMS run alot of stuff in ram as well so all in all I think we will see some decent DB performance gains.
- jlgaddis 5y agoMaybe I'm missing something but how does a newer version of the PCI-e spec directly lead to enormous benefits for a database that's running entirely in RAM? In fact, on the surface, I would think that "traditional" databases (i.e., those residing on actual storage devices) would be more likely to benefit enormously as a result of this "upgrade" -- because of the massive increase in available bandwidth for storage and networking, of course. -- (I'll certainly admit thatI don't follow or keep up with developments in new generations / iterations of Intel processors / hardware architectures, but I'm assuming that Intel's Xeon CPUs still access RAM via their own memory controller. Intel has definitely done (more than) their fair share of stupid shit over the years but I'm guessing they haven't recently decided that, now, the proper place for system memory is on the PCI-e bus!)
- timpattinson 5y agoYeah nothing has changed after the late 2000s when the memory controller moved to the CPU die rather than the northbridge
- gjsman-1000 5y agoWow, it's like the PCIe Forum woke up and decided to invent in the last two years the same amount of standards they made over the last fifteen.
- tinco 5y agoI think PCIe bandwidth has suddenly become super relevant. It used to be that videogames were not bottlenecked by PCIe bandwidth, and GPUs were the most data intensive thing on the bus. Nowadays there's all sorts of business cases that require high bandwidth, and the devices to fill them also exist. From NVMe arrays to crazy interconnects to hyper connected machine learning accelerators, PCIe is enabling it all.
- gjsman-1000 5y agoNow if only PCIe had woken up two years earlier and spent a bit more time on development… then PCIe 4 would be the equivalent to this new PCIe 6 standard.
- ksec 5y agoNot strictly true. The only thing that got huge delay was 4.0, which was due to technical difficulties. 5.0 is by and large an extension of 4.0, and arrived roughly on schedule. PCI-E 6.0 was in research for long because they knew 5.0 would be the end of the way things were working previously.
- azalemeth 5y agoStupid question from me: how the hell do they prototype and debug this? It's very far from my EE knowledge. Communication analysers will show the eye diagram and tell you a lot about stray capacitance and inductance, but I know that most 10GSa/s+ devices cost more than a house and have a tendency to live in a lab for a long time. It seems to my mind like there are so many difficulties in driving pci-e fast, from timing to voltage drop and current limitations. All of this must be implemented, somehow, if the spec is finalised. Right? They wouldn't write it up otherwise (right?!). So -- how do you prove that your infrastructure can blow the socks off everything without building two of them and testing?
- wmf 5y agoThey do build prototypes and test them.
- userbinator 5y ago...using very, very expensive equipment.
- StillBored 5y agoModern high end serdes have very complex signal mgmt and monitoring tools built in. You can detect the link quality, pull eye diagrams/etc from them. So a lot of the reason for using some of these high end scopes/analyzers is bypassed by the testing/diagnostics functions on the serdes itself. And then of course there are a ton of signal tuning parameters which get programmed during link training. Even modern analyzers have problems with this, and I've seen vendors that instead of even trying to reconstruct the signal at the endpoint use virtual transmission models and require one to probe at the source. This is one of those areas that I bump into every couple years, and its frequently quite educational.
- RF_Savage 5y agoI wonder how flexible the serdes built in test equipment stuff is. A BNC-to-PCIe-lane board with some signal conditioning would make for a hilarious oscilloscope.
- gautamcgoel 5y agoOne question I always have about new PCIe standards is how much cruft is caused by the insistence on backwards compatibility with previous PCIe versions? If you were designing a brand-new protocol to do the same things that PCIe does without having to maintain backwards compatibility, what would that new, optimized protocol look like? How much more performant could it be relative to PCIe? EDIT: Changed PCI to PCIe.
- vel0city 5y agoFWIW, PCI-e has never had backwards compatibility with PCI. They're two vastly different protocols that were somewhat standardized by the same standards body. So, PCI-e has always been vastly more performant than PCI since 2004.
- mikepurvis 5y agoThey're not very relevant nowadays, but PLX and others make bidirectional PCI-PCIe bridges that I think were mostly used so that companies could ship transitional products— eg, do a new board layout that accommodates the PCIe slot/power/whatever, but add the bridge IC to it so that you don't have respin your super fancy custom ASIC.
- simcop2387 5y agoCheck out CXL, it uses the same electrical standards but new protocols on top. There's a good servethehome video on it too. The end goal is to enable non-local memory pooling and other flexibilites to devices. It's a bit of the next evolution of things like DPUs and other datacenter mish-mashing of hardware. https://youtu.be/Mp9L7OClb2U https://youtu.be/Mp9L7OClb2U
- Tuna-Fish 5y ago> what would that new, optimized protocol look like? Exactly like PCIe 6.0, but with a slightly simpler negotiation process and none of the fallback modes. > How much more performant could it be relative to PCIe? It would have the same performance, but be slightly cheaper to implement. When two PCIe devices are connected to each other, they don't start in any data transfer mode, they start in a complex autonegotiation mode [0], where they find out the width of the link, the maximum protocol version that each side support, and the maximum speed that the connection between them can support. They were forward-thinking enought to make sure that this process can support pretty much any kind of protocol that does packet switching over LVDS signaling. Since packet switching over LVDS is still the best game in town, they are not overly burdened by backwards compatibility, other than having to also carry all the previous versions with them in the interface controllers. 6.0 changed to a completely different kind of packet system to support FEC, and no-one other than the implementors of the interface controllers have to care. Because PCIe is so flexible, it won't be unseated until there is some radically new kind of signaling that is better than LVDS on copper, say some kind of optical link. And even then, the successor is almost certain to be a variant of PCIe, just with a new physical layer. [0]: PCIe LTSSM. I sadly cannot link the spec here because it requires registration but have the diagram someone kindly ripped for SO: https://i.stack.imgur.com/QYyCM.png https://i.stack.imgur.com/QYyCM.png
- aloer 5y agoAre there similar physical limitations in the near future to technologies like this just as there are expected limits for silicon CPUs?
- ancharm 5y agoProbably worthwhile to link this as well -- Intel just demoed 224G SerDes on one of their FPGA platforms https://www.intel.com/content/www/us/en/products/docs/programmable/next-gen-fpga-224g-pam4-lr-transceiver-video.html https://www.intel.com/content/www/us/en/products/docs/progra...