10 ms·
How PCI-Express works (2020)
- jokowueu 5y agoDidn't realize PCIe gen 6 was this close . Seems like they should have skipped a few generations like CAT8 instead of making new versions so close to each other .
- terafo 5y agoThere was 7 year gap between 3 and 4 and they're catching up.
- jilijeanlouis 5y agoalso consider that you need the CPU to be compatible and this is where the innovation struggles to be implemented in real life.
- arghwhat 5y agoEach upgrade makes it harder to make a chip IP/motherboard that is in spec due to the increasing frequency and signal integrity requirements. There's a reason gen4 took so long to roll out - it has been finalized for 4 years (with he manufacturers working on support for much longer), and it's still not ubiquitous. I'm sure you'd rather have a gen4 or gen5 board when gen6 is rather than be stuck on gen3. (Plus, for gaming loads there are diminishing returns - outside reducing texture upload times, GPUs in gaming scenarios do not use a lot of PCIe bandwidth.)
- deleted 5y ago[deleted]
- Dylan16807 5y agoI wouldn't really hold ethernet cables up as a model of skipping unnecessary generations. There's 6, then there's 7, then they came out with 6A and 7A, and 7 is unpopular and 7A unheard of, and then there are apparently two kinds of 8 with different plug and shielding requirements?
- sebazzz 5y agoI've never given much thought to how much PCIe lanes are available to my hardware. However, with the advent of NVME drives I realise that there are only so many drives one can install.
- arghwhat 5y agoPCIe switches are a thing, and devices don't need all their lanes if you're okay with less max bandwidth. It ends up being the same as any other interface: there's a limit to how many devices can be connected before the limit for non-blocking simultaneous bandwidth is reached, but that merely means that bandwidth can be a bottleneck if you go above that.
- dogma1138 5y agoBelow switches which on the high end can have their own root complex, the more common thing is lane bifurcation and multiplexing. This holds especially true for consumer motherboards and PCIe breakout boards.
- arghwhat 5y agoSpeaking from experience: bifurcation is very uncommon and support is absolutely terrible, with most motherboards lacking the feature or having it be broken. Even when it is supported it's usually a pain... Multiplexers just lets you switch between connected devices - it won't let you use both simultaneously. It's hot-swapping without having to physically change devices.
- dogma1138 5y agoBifurcation is much better supported these days due to the prevalence of mining. And yes multiplexing is basically “time sharing” and the PLX chips operate on the physical layer only. PCIe switches have an internal bus, buffers and actual decode the packets to know where to send them too, they can also mediate between different versions of PCIe connected to the same switch so connecting a PCIe 2.0 device to a switch would not impact other 3.0/4.0 devices whilst a multiplexer would always operate at the lowest “speed”. But PLX chips were pretty much the only thing you can get on consumer motherboards at least back when multi GPU setups were common and SLI/XFire motherboards we’re a thing (that said PLX did help quite a bit more with XFire setups than SLI due to the lack of an external cross bridge between GPUs in later XFire revisions). The chipset on your motherboard can also support its own PCIe lanes however at least on Intel chipsets it’s not a classical PCIe switch its closer to the silicon in the CPU that runs the root complex.
- sharmin123 5y agoWhy Is It Necessary To Hire A Professional Hacker?: https://www.hackerslist.co/why-is-it-necessary-to-hire-a-professional-hacker/ https://www.hackerslist.co/why-is-it-necessary-to-hire-a-pro...
- superjan 5y agoSo one thing I still wonder about is why PCIe peripherals often present themselves as a region of memory to the main processor. If the chipset/CPU is buffering writes for fast transmission, does it guess when you are done writing? Is any regular of the normal RAM caches used? If so what happens when write only a partial cache line? It seems so backward to me.
- avianlyric 5y agoThis enables DMA (direct memory access) which is a much faster way of moving data around. It allows the CPU (or any other device in the PCI bus) to write/read data to/from the device when it’s convenient for the sender, without having to interrupt whatever the receiving device is currently doing. You still need to coordinate with the device to tell it that you’ve written to its memory, or read from it. But that’s a pretty cheap operation. The alternative is the both CPU and GPU would have to stop what their doing and manage the data copy while doing nothing else. So it’s basically the difference between sending someone an email vs giving them a call. With DMA your emailing a big document, the later calling to make sure they received it. Without DMA it would be like calling them and reading the document out over the phone. One is clearly better for everyone’s productivity.
- superjan 5y agoThanks! I could still read this two ways however: one where the memory is on the peripheral, and one where the memory is main memory, where the peripheral is copying to/from using DMA. Which one is it?
- __padding 5y agoTypically with devices like network cards (that also operate over PCI-E) You send the device a circular list of descriptors (pointers) to a region of main memory. In order to send data to the device, you write your network packet to the memory region associated with the pointer of the current ‘head’ of the descriptor list. So far, you have a ring of pointers, one of those pointers points to a location you just wrote to in ram. You then tell the device that the head of the list has changed (as you just wrote some data to the region that the head of the list is pointing to - so it can consume that pointer), the device then goes ahead and copies the data from ram into an internal buffer on the card. Once the data is consumed, the tail pointer of the ring buffer is updated to indicate that the card is finished with that memory region.
- _V_ 5y ago> A lane is composed of 2 wires: one for inbound communications and one, which has double the traffic bandwidth, for outbound. I might be wrong, but I believe PCIe lanes are differential. Meaning you have to have 2 pairs of wires (1 for each direction), not 1 totaling 4 physical wires per lane. Also both pairs in the lane should have the same bandwidth but since they are "full duplex" system, you might say it doubles the overall bandwidth of the lane but it does not mean that one of them is twice as fast.
- dogma1138 5y agoYou are not wrong I can’t tell if this article is some machine learning written garbage or a Dunning–Kruger effect thesis. It literally starts with a discussion about add-in cards form factors like it even matters. Then has tons of blunders, like the full/half duplex one, calling transfers / second frequency which it isn’t how many transfer a second has little to do with the clock frequency because it’s dependent on many other factors such as the width of the channel and how many times per clock cycle you can transfer data. It doesn’t even begin to go over the basic concepts of the PCIe root complex and the structure of a switched fabric. Then I has some gems like “having a nice GPU with 16 PCIe Lanes and having a CPU with 8 PCIe Bus lanes will be as efficient as throwing away half your money because it doesn’t fit in your wallet.” which really isn’t true…
- lizknope 5y agoYes. Every lane has 4 data pins. TX+, TX-, RX+, RX- The data is transmitted and received as differential pairs because this helps in signal integrity issues. You are also correct that you can transmit and receive at the same time but whether this helps our now depends on what you are running. The Serdes used for PCI-E are also used for other protocols like SATA and USB3. They are extremely similar at the physical layer but the higher layer protocols are very different. USB 1/2 has a single differential pair so you can only transmit or receive but not at the same time. USB 3 has differential pairs for RX and TX so you can do both at the same time.
- jlokier 5y ago> USB 1/2 has a single differential pair so you can only transmit or receive but not at the same time > > USB 3 has differential pairs for RX and TX so you can do both at the same time. Just in case this leaves some readers thinking it's only possible to transmit or receive on a differential pair, not so: Copper gigabit ethernet (1000Base-T) transmits and receives at the same time over all four differential pairs, using adaptive equalisation and echo cancellation to separate the outgoing and incoming signals on each pair. The signal processing is quite power hungry, and doesn't work so well at higher speeds. So it makes sense that PCIe wouldn't do this.
- avianlyric 5y agoMinor correction to the article. It mentions that each PCIe lane has two wires. One for inbound and one for outbound. I think this should really read that each lane has two differential pairs (interestingly the diagram shows twisted pairs). One for inbound, one for outbound. Which sums to four wires (or traces) in total for each lane. I’m also a little confused about the whole > A lane is composed of 2 wires: one for inbound communications and one, which has double the traffic bandwidth, for outbound. Not really sure what the author is trying to say. But the ideas that outbound has double the bandwidth doesn’t really make much sense, as outbound is different depending on perspective as a device. So I read this statement as saying that each device transmits at twice the speed the receiving device can receive data. Which is clearly non-sensical.
- phendrenad2 5y agoMaybe by outbound they mean root-to-peripheral.
- prabhu-yu 5y agoYes, I too confused on the same points you mentioned. Thanks for clarification.
- cwalv 5y agoI would guess they mean it's full duplex
- jilijeanlouis 5y agoI meant full duplex ;-) sorry guys for the confusion, I'll correct !
- dogma1138 5y agoThis is a very low quality article with a lot of errors and completely missing on key concepts of PCIe like the root complex.
- kabdib 5y agoIt's a fine article if it gets you excited to learn more about PCIe (which is pretty dang cool). The MindShare book on it is quite good, very detailed and comprehensive, and doubles as a doorstop when not in use. But yeah, I spotted the "two wires" stuff and a bunch of other problems. Credit for enthusiasm, perhaps, but it's not terribly accurate.
- dogma1138 5y agoIt reads like marketing branding fluff… Based on other articles by the author https://www.ovh.com/blog/understanding-the-anatomy-of-gpus-using-pokemon/ https://www.ovh.com/blog/understanding-the-anatomy-of-gpus-u... it seems all of them are quite terrible word salads with no substance.
- magicalhippo 5y agoOne neat thing about PCIe is that every PCIe compliant device must support using less lanes than it can. So your GPU which would like to work with x16 must support x1 etc. So in theory one could install a GPU or HBA in one of those x1 slots on the motherboard if one isn't terribly concerned about bandwidth. Except most of the motherboards use x1 connectors which doesn't have an "open back" for larger cards... so much for that[1]. Another thing I didn't see it mentioning was this. Say you have a x16 slot on your motherboard, but you really want to install 4 NVME m.2 drives. Well each of those need a x4, but you have an x16 so per above you should be fine right, just plop in a "dumb" four socket m.2 PCIe card? Well, no, not unless the motherboard supports bifurcation[2]. If not, you'll need a card with a PCIe switch which can turn those four x4's into one x16. [1]: yes I know about risers [2]: https://www.10gtek.com/new-1414 https://www.10gtek.com/new-1414
- FractalParadigm 5y agoFor point [1] it's also trivial to file or Dremel the slot to be open-backed, if you're careful. I've done this successfully a handful of times, unsuccessfully a few more than that...
- magicalhippo 5y agoTrue, I'm just really annoyed an open-back socket isn't the default choice for the manufacturers. Not like I enjoy using a Dremel on my motherboard...
- dogma1138 5y agoYou’ll find many cases where lack of open back slots usually also means there will be components behind the slot that will interfere with a full/half length card. Also don’t use a Dremel or even a file a hobby knife or flush cutters work just fine. If you have a steady hand then a hot knife attachment for a soldering iron is fine too. The cutting wheel on a dremel tool can run off and the dust it produces can be conductive and short something.
- krisoft 5y ago
- gameshot911 5y agoI've always been interested in understanding the hardware side of PCI-E better. By that I mean, how is it just so much faster than every other type of mobo/CPU interface (e.g. SATA, DVI, USB). Why not just use PCI-E _everywhere_ if it's so much faster?
- Taniwha 5y agoMostly because it's more channels in parallel. These days all of these interconnects (and high speed ether) are in the process of a bit of a convergence into what essentially just versions of raw serdes interfaces on chips - the same wires coming off the die can be 10Gb ether, PCIE, etc etc - they all just self clocked differential pairs
- dsr_ 5y agoBecause PCIe allows direct access to everything else. A USB3 port is already pretty dangerous: you can plug in something that will generate keystrokes or mouse movements and also present storage, so a malicious device can mount itself, copy over a payload, run it, and then pretend to be a cup-warmer again. Plug in a PCIe device and it gets to control your system.
- fomine3 5y agoThe DMA problem was solved. https://docs.microsoft.com/en-us/windows/security/information-protection/kernel-dma-protection-for-thunderbolt https://docs.microsoft.com/en-us/windows/security/informatio...
- dsr_ 5y agoFrom your link: System compatibility Kernel DMA Protection requires new UEFI firmware support. This support is anticipated only on newly-introduced, Intel-based systems shipping with Windows 10 version 1803 (not all systems). So, it's CPU specific, motherboard specific, firmware specific, and OS version specific. That's not really solved, is it?
- 5y ago
- jleahy 5y agoIf you really want to know how PCI Express works I would instead recommend these: http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1 http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tu... http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-2 http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tu...
- gautamcgoel 5y agoIt seems to me there's a lot of overhead in this protocol. How much more performance could a new-from-scratch protocol achieve?
- Lramseyer 5y agoIndeed there is, and you would be correct in that assumption! Take a look at CXL. It's currently optimized for NVMe, Ethernet, or co-processors (GPUs.) It uses the PCIe Gen 5.0 physical layer (but then again most SerDes IPs are general purpose and go into all different types of controllers.) It's essentially an alternate protocol that is negotiated in PCIe link up. What's nice is that it still keeps everything good about PCIe - like the electrical and mechanical spec, but does away with the old legacy things that nobody likes about PCIe. However, I'm not super familiar with CXL, so maybe it just looks nice from far away.
- caserocK 5y agoWHATS THIS ALL ABOUT?