6 ms·

The Sneaky Standard – Intel PCI Standardization History

61 points by ecliptik 3y ago

wmf 3y ago
PCI was so far ahead of its time it was crazy. When it came out, PCI was 10x faster than SCSI and 100x faster than Ethernet.
- RiverCrochet 3y ago
  Well, you couldn't plug a cable from a peripheral directly into a PCI port in 1993. It wasn't an external interface at all. PCI was an internal "close to the CPU" communications method mostly routed through a motherboard, superseding ISA which was the CPU bus physically extended into slots. ISA ran at 8Mhz (maybe 10Mhz or 16Mhz, memory is funny) which was also pretty fast in 1993. It wasn't until PCIe introduced switching and Thunderbolt that PCI-anything became more like switched Ethernet and something that could be funneled through cables.
  - NikkiA 3y ago
    There were actually PCI and PCI-X external chassis boxes (in all 4 forms), they were expensive as hell, but they were a critical tool for developing PCI/PCI-X devices. They usually used 1 or 2 scsi-3 (80-pin) connector to connect from the host to the box (I think you could get away with 1 connector on either 32bit standard if you dropped some of the non-vital signals).
  - mjg59 3y ago
    > ISA ran at 8Mhz (maybe 10Mhz or 16Mhz, memory is funny) The answer is "It depends", but it also depends on what you mean by ISA? The slots in the PC, XT, and AT (and clones of the appropriate vintage) tended to run synchronously with the CPU, which meant the performance of a card would depend on how fast your CPU was - well, up until the point where it was excessively overclocked and stopped working, of course. This meant that old cards wouldn't necessarily work in newer machines, given they may have been designed to work with a 4.77MHz PC. Later systems supported dividing the CPU clock to get back down to around the 8MHz range, and eventually we get to the point where the ISA slots aren't connected to the CPU at all but are bridged to PCI. But the term ISA wasn't really a thing until vendors introduced EISA as a royalty-free competitor to IBM's MCA slot architecture, and retconned the AT slot into ISA, and at that point things had pretty much settled on 8MHz. So, in that sense, ISA has always been 8MHz, even if the thing that was identical to ISA wasn't.
  - aidenn0 3y ago
    16 bit ISA@8.33MHz was about 15x slower than PCI. A 10BASE2 card used a significant fraction of the bus.
    - rasz 3y ago
      To expand on the above, absolute theoretical 16bit ISA limit is 2 cycles per transaction (0WS), (2 Bytes x clock MHz)/2 cycles = ~8MB/s at 8MHz. In practice best period correct VGA cards reach ~4-6MB/s write speed. Slowest VLB cards do ~10MB/s fastest ~30MB/s, slowest PCI VGA ~15MB/s fastest again ~30MB/s.
- rasz 3y ago
  It was 1x speed of 2 years younger VLB and did Autoconfig like 1986 Amiga :)
userbinator 3y ago
Ironically, VLB was based on the 486's local bus. Many people may know Intel also contributed to the initial USB standard, but they also created AC97, HD Audio, and AMR/CNR.
- jauntywundrkind 3y ago
  The article goes into that some. > To put that all another way, VESA came up with a slightly faster bus standard for the next generation of graphics cards, one just fast enough to meet the needs of 486 users. Intel came up with an interface designed to reshape the next decade of computing, one that it would even let its competitors use. It's crazy how useful PCI was. Intel kept being the company that made standards, that created places for cooperation. In an industry where historically it felt like every company wanted to take on the world solo. Its hard to think of a company that has done more to grow the pot in an industry than Intel. (Intel also created nvme!)
  - crotchfire 3y ago
    > (Intel also created nvme!) To be fair they did that because there were already several vendor-specific proprietary PCIe-attached-SSD products (notably in Apple's macbooks) and the lack of a standard way of doing this was a major fragmentation risk. PCI and PCIe were generous gifts from Intel to the rest of the industry and involved giving away a fair amount of hard-won development results (like the pcie serdes electrical parameters). NVMe was more a case of Intel being the referee because we needed somebody to make an arbitrary decision.
    - wtallis 3y ago
      > there were already several vendor-specific proprietary PCIe-attached-SSD products (notably in Apple's macbooks) and the lack of a standard way of doing this was a major fragmentation risk. Are you confusing the protocol for the mechanical connector and form factor? Prior to using NVMe, Apple was using PCIe SSDs that spoke AHCI, the Intel-supported industry standard for storage controllers used by almost all SATA controllers. The M.2 card connector and form factor were standardized by PCI-SIG and are unrelated to NVMe.
chem83 3y ago
I'd have included a 6th entry into the list of offshoots: OCulink.
sponaugle 3y ago
I was an engineer at Intel in the 90s and worked on a PCI card for audio codec support. I remember going into one of the PCI test labs the first time and seeing this fantastic hardware setup where all of the PCI signals were mapped into a logic analyzer that could do full rate recording. The processor on the board was also mapped into a LA so you can see CPU side transactions. No doubt in the 90s this equipment was well in the hundreds of thousands of dollars. Like any good setup of the time the development machine had a VGA monitor on a Mach 64 as well as a monochrome monitor for SoftICE. It was running in Windows NT 3.51. Ah such good times for a young engineer! Perhaps best of all, if I had a question the people who designed PCI were on the floor above us.
- rasz 3y ago
  You might enjoy hour long red team R&D lab tour, includes similar PCIE snooping setups :) "Secrets of a $182 Billion Chip Maker: AMD's Labs | Full Documentary" - Gamers Nexus https://www.youtube.com/watch?v=7H4eg2jOvVw https://www.youtube.com/watch?v=7H4eg2jOvVw
rasz 3y ago
>(VL-Bus) ... It wasn’t a massive leap, more like a stopgap improvement on the way to better graphics. >VESA came up with a slightly faster bus standard for the next generation of graphics cards, one just fast enough to meet the needs of 486 users. from 5MB/s to 133MB/s not a massive leap, just slightly faster, ok :) >Intel didn’t exactly make that point clear to the companies supporting the VESA standards body until it was too late for them to react. Those companies didnt care. Adding VLB to a 486 motherboard cost below 1$ by reusing excess stock of failed standard MCA connectors. Adding PCI meant big expensive PCI bridge chip or developing next generation chipset and impedance matched PCB to satisfy PCI reflected wave switching requirements. VLB was a quick hack to give users what was needed at the time at the price they were willing to pay. Market wasnt ready to pay for PCI in 1992, two years later cost came down to earth.
- yuhong 3y ago
  [dead]
- hyperman1 3y ago
  5mb/s is on the low side. Isa ran on the same clock as the cpu. We started there, but things got better quickly. I think the hard limit was a bit below 40MHz, as people still used 74HC chips for glue logic. Combined with EISA 16 bits = 2 bytes per cycle this might give you a 80MB/s theoretical EISA peak. I had an EISA based 386 running 40MHz at the time. Interestingly, VESA's 133MHz is double this, so maybe the majority of the speed gain was doubling bus width to 32bit?
  - rasz 3y ago
    We started at 8 bit 4.77MHz with 4 clocks minimum so 1MB/s. We ended at around 12MHz in some crazy Turbo XT/AT clones, anything faster was back to ~8MHz with fancier BIOSes giving you option to overclock. You are thinking cpu fsb, ISA never ran at 40MHz. Even EISA ran at a fixed ~8MHz. Btw EISA is not those longer black ISA slots on normal boards, EISA is brown slots with too many pins between normal pins because its secretly a 32bit bus, found pretty much only on workstation/server boards from 1989-1994.
peter_d_sherman 3y ago
>"Key to PCIe’s approach is the use of “lanes” of data transfer speed, allowing high-speed cards like graphics adapters more bandwidth (up to 16 lanes) and slower technologies like network adapters or audio adapters less. This has given PCIe unparalleled backwards compatibility — it’s technically possible to run a modern card on a first-gen PCIe port in exchange for lower speed — while allowing the standard to continue improving." Backwards compatibility -- seems to have been the key to Microsoft Windows OS's popularity as well... Anyway, an excellent computer history article!
vrinsd 3y ago
Intel also had quite a few issues (as usual) getting their own PCI standard working. Almost all of the early Intel PCI chipsets (i.e. Mercury, Neptune ) had serious bugs. Intel actually used DEC PCI chipsets or licensed their designs to actually get a more robust PCI implementation (IIRC Triton). Until then VLB was still a better choice. PCIe retains a lot of "weirdness" that PCI introduced which includes a lack of a unified or "properly architected" device-to-host and host-to-device centralized DMA agent or broker. Even today it's still a challenge to extract more than 50-60% of the theoretical bandwidth PCIe offers without a lot of tuning and mental churn.
- wtallis 3y ago
  > it's still a challenge to extract more than 50-60% of the theoretical bandwidth PCIe offers without a lot of tuning and mental churn. Are you referring to using all the bandwidth across all of a CPU's PCIe lanes? Because it's clearly not a challenge to do way better than 50-60% utilization for transfers in one direction at a time to one device. All kinds of cheap crappy products have passed that threshold on their first try.
  - vrinsd 3y ago
    The issue is not lack of raw bandwidth, it's getting the hardware, software, drivers and OS "to do the right thing". PCIe gives you the building blocks of posted transactions and non-posted transactions but doesn't help you use them effectively. There is no coordinated or designated DMA subsystem to help move data between the root-complex("host") and end-point("device)". So, if you have to design a new PCIe end-point (target in original PCI terms) using an FPGA or ASIC then trying to actually sustain PCIe throughput in either "direction" isn't trivial. Posted transactions ("writes") are 'fire and forget' and non-posted transactions ("reads") have a request/acknowledgement system, flow-control, etc. If you can get your "system" to use ONLY posted writes (fire and forget) with a large enough MPS (payload size), usually >128 Bytes, then you can get to 80%-95% of theoretical throughput (1). The real difficulty is if you need to do a PCIe 'read' this breaks down into a read-request (MRd) and a Completion with Data (CplD). The 'read' results in a lot of back and forth traffic and tracking the MRds/CplDs becomes a challenge (2). Often an end-point can use 'posted writes' to blast data to the PCIe root-complex (usually the CPU/host) maximizing throughput since a host usually has hundreds of MegaBytes of RAM to make use of for buffers. Unfortunately to transfer data from the root-complex(host) to the end-point(device), the host usually will have the device's DMA controller initiate a 'read' from the host's memory which results in these split transactions since end-points don't often carry hundreds of MB of RAM. This also means bespoke drivers, tying into the OS PCIe subsystems and hopefully not loosing any MSI-X interrupts. To re-iterate in the modern "Intel way" the CPU houses the PCIe root complex but does not house ANY DMA controller. So to get "DMA" working means each PCIe end-point's implementation has some kind of DMA "controller" which is different than the DMA controller of all other end-points, rather than Intel having spec'd out an "optional" centralized a DMA controller in the root complex. 1: https://cdrdv2-public.intel.com/666650/an456-683541-666650.pdf https://cdrdv2-public.intel.com/666650/an456-683541-666650.p... 2: https://www.intel.com/content/www/us/en/docs/programmable/683541/current/throughput-for-reads.html https://www.intel.com/content/www/us/en/docs/programmable/68...
chem83 3y ago
VLB felt like a stopgap much in the same way that AGP lingered just enough for the PCI-X v PCIe mess to be sorted out.

61 points by ecliptik 3y ago

wmf 3y ago

PCI was so far ahead of its time it was crazy. When it came out, PCI was 10x faster than SCSI and 100x faster than Ethernet.

RiverCrochet 3y ago
Well, you couldn't plug a cable from a peripheral directly into a PCI port in 1993. It wasn't an external interface at all. PCI was an internal "close to the CPU" communications method mostly routed through a motherboard, superseding ISA which was the CPU bus physically extended into slots. ISA ran at 8Mhz (maybe 10Mhz or 16Mhz, memory is funny) which was also pretty fast in 1993. It wasn't until PCIe introduced switching and Thunderbolt that PCI-anything became more like switched Ethernet and something that could be funneled through cables.
- NikkiA 3y ago
  There were actually PCI and PCI-X external chassis boxes (in all 4 forms), they were expensive as hell, but they were a critical tool for developing PCI/PCI-X devices. They usually used 1 or 2 scsi-3 (80-pin) connector to connect from the host to the box (I think you could get away with 1 connector on either 32bit standard if you dropped some of the non-vital signals).
- mjg59 3y ago
  > ISA ran at 8Mhz (maybe 10Mhz or 16Mhz, memory is funny) The answer is "It depends", but it also depends on what you mean by ISA? The slots in the PC, XT, and AT (and clones of the appropriate vintage) tended to run synchronously with the CPU, which meant the performance of a card would depend on how fast your CPU was - well, up until the point where it was excessively overclocked and stopped working, of course. This meant that old cards wouldn't necessarily work in newer machines, given they may have been designed to work with a 4.77MHz PC. Later systems supported dividing the CPU clock to get back down to around the 8MHz range, and eventually we get to the point where the ISA slots aren't connected to the CPU at all but are bridged to PCI. But the term ISA wasn't really a thing until vendors introduced EISA as a royalty-free competitor to IBM's MCA slot architecture, and retconned the AT slot into ISA, and at that point things had pretty much settled on 8MHz. So, in that sense, ISA has always been 8MHz, even if the thing that was identical to ISA wasn't.
- aidenn0 3y ago
  16 bit ISA@8.33MHz was about 15x slower than PCI. A 10BASE2 card used a significant fraction of the bus.
  - rasz 3y ago
    To expand on the above, absolute theoretical 16bit ISA limit is 2 cycles per transaction (0WS), (2 Bytes x clock MHz)/2 cycles = ~8MB/s at 8MHz. In practice best period correct VGA cards reach ~4-6MB/s write speed. Slowest VLB cards do ~10MB/s fastest ~30MB/s, slowest PCI VGA ~15MB/s fastest again ~30MB/s.
rasz 3y ago
It was 1x speed of 2 years younger VLB and did Autoconfig like 1986 Amiga :)

userbinator 3y ago

Ironically, VLB was based on the 486's local bus. Many people may know Intel also contributed to the initial USB standard, but they also created AC97, HD Audio, and AMR/CNR.

jauntywundrkind 3y ago
The article goes into that some. > To put that all another way, VESA came up with a slightly faster bus standard for the next generation of graphics cards, one just fast enough to meet the needs of 486 users. Intel came up with an interface designed to reshape the next decade of computing, one that it would even let its competitors use. It's crazy how useful PCI was. Intel kept being the company that made standards, that created places for cooperation. In an industry where historically it felt like every company wanted to take on the world solo. Its hard to think of a company that has done more to grow the pot in an industry than Intel. (Intel also created nvme!)
- crotchfire 3y ago
  > (Intel also created nvme!) To be fair they did that because there were already several vendor-specific proprietary PCIe-attached-SSD products (notably in Apple's macbooks) and the lack of a standard way of doing this was a major fragmentation risk. PCI and PCIe were generous gifts from Intel to the rest of the industry and involved giving away a fair amount of hard-won development results (like the pcie serdes electrical parameters). NVMe was more a case of Intel being the referee because we needed somebody to make an arbitrary decision.
  - wtallis 3y ago
    > there were already several vendor-specific proprietary PCIe-attached-SSD products (notably in Apple's macbooks) and the lack of a standard way of doing this was a major fragmentation risk. Are you confusing the protocol for the mechanical connector and form factor? Prior to using NVMe, Apple was using PCIe SSDs that spoke AHCI, the Intel-supported industry standard for storage controllers used by almost all SATA controllers. The M.2 card connector and form factor were standardized by PCI-SIG and are unrelated to NVMe.

chem83 3y ago

I'd have included a 6th entry into the list of offshoots: OCulink.

sponaugle 3y ago

I was an engineer at Intel in the 90s and worked on a PCI card for audio codec support. I remember going into one of the PCI test labs the first time and seeing this fantastic hardware setup where all of the PCI signals were mapped into a logic analyzer that could do full rate recording. The processor on the board was also mapped into a LA so you can see CPU side transactions. No doubt in the 90s this equipment was well in the hundreds of thousands of dollars. Like any good setup of the time the development machine had a VGA monitor on a Mach 64 as well as a monochrome monitor for SoftICE. It was running in Windows NT 3.51. Ah such good times for a young engineer! Perhaps best of all, if I had a question the people who designed PCI were on the floor above us.

rasz 3y ago
You might enjoy hour long red team R&D lab tour, includes similar PCIE snooping setups :) "Secrets of a $182 Billion Chip Maker: AMD's Labs | Full Documentary" - Gamers Nexus https://www.youtube.com/watch?v=7H4eg2jOvVw https://www.youtube.com/watch?v=7H4eg2jOvVw

rasz 3y ago

>(VL-Bus) ... It wasn’t a massive leap, more like a stopgap improvement on the way to better graphics. >VESA came up with a slightly faster bus standard for the next generation of graphics cards, one just fast enough to meet the needs of 486 users. from 5MB/s to 133MB/s not a massive leap, just slightly faster, ok :) >Intel didn’t exactly make that point clear to the companies supporting the VESA standards body until it was too late for them to react. Those companies didnt care. Adding VLB to a 486 motherboard cost below 1$ by reusing excess stock of failed standard MCA connectors. Adding PCI meant big expensive PCI bridge chip or developing next generation chipset and impedance matched PCB to satisfy PCI reflected wave switching requirements. VLB was a quick hack to give users what was needed at the time at the price they were willing to pay. Market wasnt ready to pay for PCI in 1992, two years later cost came down to earth.

yuhong 3y ago
[dead]
hyperman1 3y ago
5mb/s is on the low side. Isa ran on the same clock as the cpu. We started there, but things got better quickly. I think the hard limit was a bit below 40MHz, as people still used 74HC chips for glue logic. Combined with EISA 16 bits = 2 bytes per cycle this might give you a 80MB/s theoretical EISA peak. I had an EISA based 386 running 40MHz at the time. Interestingly, VESA's 133MHz is double this, so maybe the majority of the speed gain was doubling bus width to 32bit?
- rasz 3y ago
  We started at 8 bit 4.77MHz with 4 clocks minimum so 1MB/s. We ended at around 12MHz in some crazy Turbo XT/AT clones, anything faster was back to ~8MHz with fancier BIOSes giving you option to overclock. You are thinking cpu fsb, ISA never ran at 40MHz. Even EISA ran at a fixed ~8MHz. Btw EISA is not those longer black ISA slots on normal boards, EISA is brown slots with too many pins between normal pins because its secretly a 32bit bus, found pretty much only on workstation/server boards from 1989-1994.

peter_d_sherman 3y ago

>"Key to PCIe’s approach is the use of “lanes” of data transfer speed, allowing high-speed cards like graphics adapters more bandwidth (up to 16 lanes) and slower technologies like network adapters or audio adapters less. This has given PCIe unparalleled backwards compatibility — it’s technically possible to run a modern card on a first-gen PCIe port in exchange for lower speed — while allowing the standard to continue improving." Backwards compatibility -- seems to have been the key to Microsoft Windows OS's popularity as well... Anyway, an excellent computer history article!

vrinsd 3y ago

Intel also had quite a few issues (as usual) getting their own PCI standard working. Almost all of the early Intel PCI chipsets (i.e. Mercury, Neptune ) had serious bugs. Intel actually used DEC PCI chipsets or licensed their designs to actually get a more robust PCI implementation (IIRC Triton). Until then VLB was still a better choice. PCIe retains a lot of "weirdness" that PCI introduced which includes a lack of a unified or "properly architected" device-to-host and host-to-device centralized DMA agent or broker. Even today it's still a challenge to extract more than 50-60% of the theoretical bandwidth PCIe offers without a lot of tuning and mental churn.

wtallis 3y ago
> it's still a challenge to extract more than 50-60% of the theoretical bandwidth PCIe offers without a lot of tuning and mental churn. Are you referring to using all the bandwidth across all of a CPU's PCIe lanes? Because it's clearly not a challenge to do way better than 50-60% utilization for transfers in one direction at a time to one device. All kinds of cheap crappy products have passed that threshold on their first try.
- vrinsd 3y ago
  The issue is not lack of raw bandwidth, it's getting the hardware, software, drivers and OS "to do the right thing". PCIe gives you the building blocks of posted transactions and non-posted transactions but doesn't help you use them effectively. There is no coordinated or designated DMA subsystem to help move data between the root-complex("host") and end-point("device)". So, if you have to design a new PCIe end-point (target in original PCI terms) using an FPGA or ASIC then trying to actually sustain PCIe throughput in either "direction" isn't trivial. Posted transactions ("writes") are 'fire and forget' and non-posted transactions ("reads") have a request/acknowledgement system, flow-control, etc. If you can get your "system" to use ONLY posted writes (fire and forget) with a large enough MPS (payload size), usually >128 Bytes, then you can get to 80%-95% of theoretical throughput (1). The real difficulty is if you need to do a PCIe 'read' this breaks down into a read-request (MRd) and a Completion with Data (CplD). The 'read' results in a lot of back and forth traffic and tracking the MRds/CplDs becomes a challenge (2). Often an end-point can use 'posted writes' to blast data to the PCIe root-complex (usually the CPU/host) maximizing throughput since a host usually has hundreds of MegaBytes of RAM to make use of for buffers. Unfortunately to transfer data from the root-complex(host) to the end-point(device), the host usually will have the device's DMA controller initiate a 'read' from the host's memory which results in these split transactions since end-points don't often carry hundreds of MB of RAM. This also means bespoke drivers, tying into the OS PCIe subsystems and hopefully not loosing any MSI-X interrupts. To re-iterate in the modern "Intel way" the CPU houses the PCIe root complex but does not house ANY DMA controller. So to get "DMA" working means each PCIe end-point's implementation has some kind of DMA "controller" which is different than the DMA controller of all other end-points, rather than Intel having spec'd out an "optional" centralized a DMA controller in the root complex. 1: https://cdrdv2-public.intel.com/666650/an456-683541-666650.pdf https://cdrdv2-public.intel.com/666650/an456-683541-666650.p... 2: https://www.intel.com/content/www/us/en/docs/programmable/683541/current/throughput-for-reads.html https://www.intel.com/content/www/us/en/docs/programmable/68...

chem83 3y ago

VLB felt like a stopgap much in the same way that AGP lingered just enough for the PCI-X v PCIe mess to be sorted out.