26 ms·

I've always been interested in understanding the hardware side of PCI-E better. By that I mean, how is it just so much faster than every other type of mobo/CPU

I've always been interested in understanding the hardware side of PCI-E better. By that I mean, how is it just so much faster than every other type of mobo/CPU interface (e.g. SATA, DVI, USB). Why not just use PCI-E _everywhere_ if it's so much faster?

Taniwha 5y ago
Mostly because it's more channels in parallel. These days all of these interconnects (and high speed ether) are in the process of a bit of a convergence into what essentially just versions of raw serdes interfaces on chips - the same wires coming off the die can be 10Gb ether, PCIE, etc etc - they all just self clocked differential pairs
dsr_ 5y ago
Because PCIe allows direct access to everything else. A USB3 port is already pretty dangerous: you can plug in something that will generate keystrokes or mouse movements and also present storage, so a malicious device can mount itself, copy over a payload, run it, and then pretend to be a cup-warmer again. Plug in a PCIe device and it gets to control your system.
- fomine3 5y ago
  The DMA problem was solved. https://docs.microsoft.com/en-us/windows/security/information-protection/kernel-dma-protection-for-thunderbolt https://docs.microsoft.com/en-us/windows/security/informatio...
  - dsr_ 5y ago
    From your link: System compatibility Kernel DMA Protection requires new UEFI firmware support. This support is anticipated only on newly-introduced, Intel-based systems shipping with Windows 10 version 1803 (not all systems). So, it's CPU specific, motherboard specific, firmware specific, and OS version specific. That's not really solved, is it?
    - fomine3 5y ago
      Should I say "mitigated"? But DMA itself is a core part to improve transfer performance, and itself is a security hole. Anyway mitigation is considered to enough TB/USB4 to be adopted for new devices.
    - opencl 5y ago
      It's CPU specific in the sense that the CPU needs IOMMU instructions (almost everything made in the past 10 years) and OS version specific in that Windows and Linux have both supported it for about 3 years. The problematic part is that the UEFI needs to support it, it seems most systems with Thunderbolt have enabled it since 2018 and systems without Thunderbolt still don't bother.
    - Dylan16807 5y ago
      Most hardware features need that kind of adoption to work. It's mandatory for thunderbolt 4.
Nextgrid 5y ago
It's susceptible to interference so there's probably no need to use something like this where something simpler (and more robust) will fit the bill. But PCI-E is used externally - that's what Thunderbolt is.
fomine3 5y ago
Now it's going to everywhere, isn't it? * NVMe 2.0 going to support HDD * Thunderbolt, USB 4.0 supports PCIe based connection * CFexpress, SD express is based on PCIe
loeg 5y ago
Price and range, in addition to the security issues related to DMA others have brought up. Max cable length for the very high signaling rates is very short: https://superuser.com/questions/885232/what-is-the-maximum-length-of-a-pci-express-flexible-extension-cable https://superuser.com/questions/885232/what-is-the-maximum-l...
- wtallis 5y ago
  Max cable length is very short, if you insist on using cheap ribbon cables with the relatively ancient PCIe CEM slot as a connector. Or you can do PCIe 4.0 x8 over a 2M cable, if you use quality twinaxial cabling with more modern connectors, eg. https://www.serialcables.com/product-category/pcie4-oculink-cables/ https://www.serialcables.com/product-category/pcie4-oculink-...
  - loeg 5y ago
    I mean, there's the length covered by the standard, which is short, and then these longer cables. If they work, great, but I don't believe it's required by PCIe. 2m is also a fairly short cable, if not as short as a few inches.
    - wtallis 5y ago
      I don't believe I've ever run across a section of the PCIe standard dictating cable or PCB trace length requirements. All I've ever seen are signal integrity requirements (ie. dB loss) and timing requirements. The timing requirements are not tight enough to preclude long cables, and while the signal loss requirements are chosen based on what's practical and affordable to achieve with a PCB, they don't directly concern themselves with setting minimum or maximum reach lengths.
Dylan16807 5y ago
In 2003 SATA did 150MB/s and PCIe x1 did 250MB/s. In 2004 SATA did 300MB/s. In 2007 PCIe x1 did 500MB/s. In 2008 SATA did 600MB/s. In 2010 PCIe did 1GB/s. The only reason SATA fell behind on a per-lane basis is because they stopped updating it. PCIe has no special sauce for running faster, just more lanes. It's taking over drives, but mostly for other reasons. DVI was replaced by HDMI. HDMI 2.1, which came out in 2017, is about 2/3 as fast per lane as PCIe gen4, which also came out in 2017. For an external passive cable that's really good. USB 3.1 jumped to 1.2GB/s in 2013, which means it was faster than PCIe for four years. And USB4 is twice that speed per lane, making it faster than PCIe gen4. So the constraining factor is not the type of signalling, it's the number of wires and transceivers. You can make anything go faster with more wires, but that's expensive.
avianlyric 5y ago
Thunderbolt is basically PCIe in a cable. But the big issue you face is that PCIe is fast because the physical layer is so tightly controlled. Lane lengths to the same port are all approximately the same length. All the lanes are carefully shielded with a complex mix of grounded traces and large ground planes above and below. The length of the traces is limited to make sure the speed of light doesn’t introduce too much delay. So PCIe speed doesn’t come from fancy algorithms, but from fancy electronics, and a very tightly controlled physical layer. Extending that layer outside of a computer is tricky. Just like trains are much faster than cars, PCIe is much faster the DisplayPort. But trains require tracks, that are long and straight, and car don’t. Equally PCIe requires electrical connections that are short and straight, and DisplayPort doesn’t. Thunderbolt kinda splits the difference. It’s basically PCIe, but heavily downrated so it can handle being outside of a motherboard.
- wtallis 5y ago
  Recent versions of PCIe have short reach within a system precisely because the physical layer is not all that tightly controlled. Cheap PCB materials are a much worse physical medium for high speed signals than twinaxial cables, which are used in a lot of rackmount servers to bring PCIe from the back of the system where the CPUs and expansion slots are up to a SSD backplane in the front of the system. Thunderbolt goes for even longer cable runs by using active cables, while long PCIe runs use redrivers or retimers that aren't integrated into the cables. Thunderbolt is also not really a downrated PCIe in any way; Thunderbolt transmits data faster than PCIe 4.0 lanes, but doesn't scale to as many lanes as internal PCIe links.

Taniwha 5y ago

Mostly because it's more channels in parallel. These days all of these interconnects (and high speed ether) are in the process of a bit of a convergence into what essentially just versions of raw serdes interfaces on chips - the same wires coming off the die can be 10Gb ether, PCIE, etc etc - they all just self clocked differential pairs

dsr_ 5y ago

Because PCIe allows direct access to everything else. A USB3 port is already pretty dangerous: you can plug in something that will generate keystrokes or mouse movements and also present storage, so a malicious device can mount itself, copy over a payload, run it, and then pretend to be a cup-warmer again. Plug in a PCIe device and it gets to control your system.

fomine3 5y ago
The DMA problem was solved. https://docs.microsoft.com/en-us/windows/security/information-protection/kernel-dma-protection-for-thunderbolt https://docs.microsoft.com/en-us/windows/security/informatio...
- dsr_ 5y ago
  From your link: System compatibility Kernel DMA Protection requires new UEFI firmware support. This support is anticipated only on newly-introduced, Intel-based systems shipping with Windows 10 version 1803 (not all systems). So, it's CPU specific, motherboard specific, firmware specific, and OS version specific. That's not really solved, is it?
  - fomine3 5y ago
    Should I say "mitigated"? But DMA itself is a core part to improve transfer performance, and itself is a security hole. Anyway mitigation is considered to enough TB/USB4 to be adopted for new devices.
  - opencl 5y ago
    It's CPU specific in the sense that the CPU needs IOMMU instructions (almost everything made in the past 10 years) and OS version specific in that Windows and Linux have both supported it for about 3 years. The problematic part is that the UEFI needs to support it, it seems most systems with Thunderbolt have enabled it since 2018 and systems without Thunderbolt still don't bother.
  - Dylan16807 5y ago
    Most hardware features need that kind of adoption to work. It's mandatory for thunderbolt 4.

Nextgrid 5y ago

It's susceptible to interference so there's probably no need to use something like this where something simpler (and more robust) will fit the bill. But PCI-E is used externally - that's what Thunderbolt is.

fomine3 5y ago

Now it's going to everywhere, isn't it? * NVMe 2.0 going to support HDD * Thunderbolt, USB 4.0 supports PCIe based connection * CFexpress, SD express is based on PCIe

loeg 5y ago

Price and range, in addition to the security issues related to DMA others have brought up. Max cable length for the very high signaling rates is very short: https://superuser.com/questions/885232/what-is-the-maximum-length-of-a-pci-express-flexible-extension-cable https://superuser.com/questions/885232/what-is-the-maximum-l...

wtallis 5y ago
Max cable length is very short, if you insist on using cheap ribbon cables with the relatively ancient PCIe CEM slot as a connector. Or you can do PCIe 4.0 x8 over a 2M cable, if you use quality twinaxial cabling with more modern connectors, eg. https://www.serialcables.com/product-category/pcie4-oculink-cables/ https://www.serialcables.com/product-category/pcie4-oculink-...
- loeg 5y ago
  I mean, there's the length covered by the standard, which is short, and then these longer cables. If they work, great, but I don't believe it's required by PCIe. 2m is also a fairly short cable, if not as short as a few inches.
  - wtallis 5y ago
    I don't believe I've ever run across a section of the PCIe standard dictating cable or PCB trace length requirements. All I've ever seen are signal integrity requirements (ie. dB loss) and timing requirements. The timing requirements are not tight enough to preclude long cables, and while the signal loss requirements are chosen based on what's practical and affordable to achieve with a PCB, they don't directly concern themselves with setting minimum or maximum reach lengths.

Dylan16807 5y ago

In 2003 SATA did 150MB/s and PCIe x1 did 250MB/s. In 2004 SATA did 300MB/s. In 2007 PCIe x1 did 500MB/s. In 2008 SATA did 600MB/s. In 2010 PCIe did 1GB/s. The only reason SATA fell behind on a per-lane basis is because they stopped updating it. PCIe has no special sauce for running faster, just more lanes. It's taking over drives, but mostly for other reasons. DVI was replaced by HDMI. HDMI 2.1, which came out in 2017, is about 2/3 as fast per lane as PCIe gen4, which also came out in 2017. For an external passive cable that's really good. USB 3.1 jumped to 1.2GB/s in 2013, which means it was faster than PCIe for four years. And USB4 is twice that speed per lane, making it faster than PCIe gen4. So the constraining factor is not the type of signalling, it's the number of wires and transceivers. You can make anything go faster with more wires, but that's expensive.

avianlyric 5y ago

Thunderbolt is basically PCIe in a cable. But the big issue you face is that PCIe is fast because the physical layer is so tightly controlled. Lane lengths to the same port are all approximately the same length. All the lanes are carefully shielded with a complex mix of grounded traces and large ground planes above and below. The length of the traces is limited to make sure the speed of light doesn’t introduce too much delay. So PCIe speed doesn’t come from fancy algorithms, but from fancy electronics, and a very tightly controlled physical layer. Extending that layer outside of a computer is tricky. Just like trains are much faster than cars, PCIe is much faster the DisplayPort. But trains require tracks, that are long and straight, and car don’t. Equally PCIe requires electrical connections that are short and straight, and DisplayPort doesn’t. Thunderbolt kinda splits the difference. It’s basically PCIe, but heavily downrated so it can handle being outside of a motherboard.

wtallis 5y ago
Recent versions of PCIe have short reach within a system precisely because the physical layer is not all that tightly controlled. Cheap PCB materials are a much worse physical medium for high speed signals than twinaxial cables, which are used in a lot of rackmount servers to bring PCIe from the back of the system where the CPUs and expansion slots are up to a SSD backplane in the front of the system. Thunderbolt goes for even longer cable runs by using active cables, while long PCIe runs use redrivers or retimers that aren't integrated into the cables. Thunderbolt is also not really a downrated PCIe in any way; Thunderbolt transmits data faster than PCIe 4.0 lanes, but doesn't scale to as many lanes as internal PCIe links.