7 ms·

How long does it usually take to get consumer products of new PCIe specs? Fast PCIe Gen 4 is only just getting affordable. Like $350 for 2 TB NVMe ssds. Also,

How long does it usually take to get consumer products of new PCIe specs? Fast PCIe Gen 4 is only just getting affordable. Like $350 for 2 TB NVMe ssds. Also, I remember playing around with PCI implementation on FPGAs over a decade ago and timing was already not easy. What goes into creating a PCIe Gen4/5 device these days? How can you actually achieve that when you're designing it? Are people just buying the chipsets from a handful of producers because it's unachievable for normal humans? EDIT: What's inside the spec differences between say gen 3 and 6 that allows for so many more lanes to be available?

dragontamer 5y ago
> How long does it usually take to get consumer products of new PCIe specs? Like 2 years. When PCIe 3.0 was getting popular, 4.0 was finalized. When 4.0 was getting popular, 5.0 was finalized. Now that PCIe 5.0 is coming out (2022, this year), PCIe 6.0 is finalized.
- jeffbee 5y ago
  PCIe 5 was standardized May 2019 and you could buy it at retail in late 2021. 2 years good rule of thumb.
- formerly_proven 5y ago
  There was a much bigger gap between 3.0 and 4.0. PCIe 3.0 was available with Sandy or Ivy Bridge, so 2011/2012. PCIe 4.0 was introduced with Zen 2 in 2019. We seem to be back to a faster cadence now however.
  - zamadatix 5y ago
    The large delay between 3.0 to 4.0 was a gap between specifications (2010 to 2017) not a gap between specification to implementations (2017 to 2019).
  - dragontamer 5y ago
    With the rise of GPU-compute, a lot of the supercomputers are playing around with faster I/O systems. IBM pushed OpenCAPI / NVLink with Nvidia, and I think that inspired the PCIe ecosystem to innovate. PCIe standards are including more and more coherent-memory options. It seems like PCIe is trying to become more like Infinity Fabric (AMD) / UltraPath Interconnect (Intel).
    - Zenst 5y ago
      One bus to rule them all! Would make some sense and if it's fast enough, not saying we will see memory in PCIe form factor, though, size of memory progression, maybe down the line it may go that way.
    - mikepurvis 5y ago
      With sufficiently fast memory bandwidth, does dedicated GPU memory become irrelevant? AFAIK discrete graphics on laptops still share the main memory, but at what point does that become a standard setup on desktops too?
      - dragontamer 5y ago
        DDR5 is far slower than GPU VRAM. If anything, the higher-bandwidths of future PCIe-specs would allow the CPU to access the dedicated GPU VRAM at its full speed, rather than PCIe-limited speeds.
    - jabl 5y ago
      > It seems like PCIe is trying to become more like Infinity Fabric (AMD) / UltraPath Interconnect (Intel). So far CXL is pretty strictly a leader/follower type approach, not really suitable for a more symmetric relationship like you have in a multi-socket system. But maybe one day..
jiggawatts 5y ago
I love how exponential growth can be utterly terrifying or unfathomably amazing. Just a few years ago I was trying to explain to an IT manager that 200 IOPS just doesn’t cut it for their biggest, most important OLAP database. He asked me what would be a more realistic number. “20,000 IOPS is a good start” “You can’t be serious!” “My laptop can do 200,000.”
- KennyBlanken 5y ago
  > Just a few years ago > “My laptop can do 200,000.” Only now (PCIe 4 and very recent controllers etc) are the very latest top-end NVME drives hitting around 150k IOPS (which isn't stopping manufacturers from claiming ten times that; WD's NVME drive tests at around 150-200k IOPS and yet they claim 1M) and only in ideal circumstances...reads and writes coming out of the SLC cache, which typically under 30GB, often a lot smaller except in the highest-end drives. Many drives that claim to reach that sort of performance are actually using Host Backed Cache, ie stealing RAM. IOPS on SSDs drops precipitously once you exhaust any HBC, controller ram, SLC cache, mid-level MLC cache...and start having to hit the actual QLC/TLC. In the case of a very large database, a lot of IO would be outside cache (though certainly any index, transaction, logging, etc IO would likely be in cache.)
  - Cullinet 5y ago
    I would love to pick up from the 200k IOPS laptop quote and demo a RAM drive and then saturate the RAM drive into swapping - I don't know how you could do this on stock distros or Windows but it would make a great executive suite demo of the issues.
  - wtallis 5y ago
    More than 500k IOPS was downright common for PCIe gen3 SSDs. 1M IOPS is totally achievable with one high-end gen4 SSD. If you meant IOPS with a queue depth of one, or sustained write IOPS, then you need to specify those extra conditions before calling vendors liars. No high-end drives implement the NVMe Host Memory Buffer feature, let alone rely on it for maximum performance. You can buy an Enterprise QLC drive that is fully capable of 800k IOPS for reads. TLC drives that can do twice that are available. Those drives don't have SLC caching.
cjensen 5y ago
PCIe Gen5 is now available in the latest Intel desktop processors. There are very few lanes, so it can really only run a single GPU, but that's covers a lot of the potential market.
- eigen 5y ago
  Look like Desktop Chipset 600 series just supports gen 3 & 4[1] and PCIe gen5 ports are only available on AlderLake Desktop[2] not Mobile[3] processors. [1] https://ark.intel.com/content/www/us/en/ark/products/series/218828/intel-600-series-desktop-chipsets.html https://ark.intel.com/content/www/us/en/ark/products/series/... [2] https://ark.intel.com/content/www/us/en/ark/products/134598/intel-core-i912900f-processor-30m-cache-up-to-5-10-ghz.html https://ark.intel.com/content/www/us/en/ark/products/134598/... [3] https://ark.intel.com/content/www/us/en/ark/products/132214/intel-core-i912900h-processor-24m-cache-up-to-5-00-ghz.html https://ark.intel.com/content/www/us/en/ark/products/132214/...
willis936 5y ago
I've not done PHY development personally, but these interfaces are called SerDes. SerDes is short for Serial-Deserializer. Outside of dedicated EQ hardware, everything on the chips are done in parallel so nothing needs to run at a multi-GHz clock.
- deleted 5y ago
  [deleted]
- Taniwha 5y ago
  I think that these days there's a lot of convergence going on - everything is essentially serdes in some form - some chips just have N serdes lanes and let you config them for PCIe/ether/data/USB/etc as you need them, much as more traditional SoCs config GPIOs between a bunch of other functions like uarts/spi/i2c/i2s/pcm/...
ksec 5y ago
>How long does it ....... It is not just about getting a product out ( i.e PCI-E 6.0 SSD ), but also the platform support. ( i.e Intel / AMD Motherboard support for PCI-e 6.0 ) Product Launch are highly dependent on Platform support. So far Intel and AMD dont have any concrete plan on PCI-E 6.0, but I believe Amazon could be ahead of the pack with their Graviton platform. Although I am eager to see Netflix's Edge Appliance serving up to 800Gbps if not 1.6Tbps per box.
- iancarroll 5y ago
  I recently bought a few Zen 2 and Zen 3 HPE servers, and found out only via trial and error that HPE sells Zen 2 servers without Gen4 motherboard support! It seems they took the original Zen motherboards with Gen3 and just swapped out the CPU. Only the Zen 3 has a refreshed motherboard. Makes me now check things more carefully to be sure.
  - tinco 5y ago
    Same, worse my vendor sent me an outdated motherboard that officially didn't support the Zen2 CPU and it randomly wouldn't boot, and then they sent me a patched BIOS so it would boot consistently. We needed that machine yesterday, and it's seen maybe 2 minutes of downtime in the past couple years, so I couldn't send it back. My fault for not specifying I wanted the .v2 of the barebones when ordering it over the phone I guess..
jeffbee 5y ago
There's not more lanes available. The generations are getting faster just by increasing the transfer clock rate, up to PCIe 5, and in PCIe 6 by increasing the number of bits per transfer. The way they doubled the speed every generation was pretty basic: the timing tolerances were chopped in half every time. The allowable clock phase noise in PCIe 4 is 200x less than in PCIe 1. The miracle of progress, etc. That miracle is somewhat over. They're not going to be able to drive phase noise down below 1 femtosecond, so 6.0 changes tactics. They are now using a fancy encoding on the wire to double the number of bits per symbol. Eventually, it will look more like wifi-over-copper than like PCI. Ethernet faster than 1gbps has the same trend, for whatever it's worth.
- bserge 5y ago
  Speaking of which, when is 10Gbit Ethernet coming to laptops? Most have even lost the port ffs.
  - jeffbee 5y ago
    Many laptops have a thunderbolt port which serves a similar purpose. On TB4 I get 15gbps in practice, and I can bridge it to ethernet using either a dock or a PC (I use a mac mini with a 10g port to bridge TB to 10ge).
  - wmf 5y ago
    I think the real question is when will 10G Ethernet USB4 dongles drop to a reasonable price like <$150.
    - jeffbee 5y ago
      A USB4 peripheral interface is $15 and a 10g ethernet IC is like $80 at least, plus a little box, SFP+ thing, other physical items. Seems hard to get the price down that low and still make a profit. Plus the buyer still needs an expensive cable on both sides.
      - wmf 5y ago
rektide 5y ago
> How long does it usually take to get consumer products of new PCIe specs? Personally I'm expecting this spec to drive pcie 5.0 adoption into consumer space. Tbh consumers dont need this througjput. But given that consumer space has remained stuck around 20x lanes off the cpu (plus some for the chipset), the 5.0 and 6.0 specs will be great for those wanting to build systems with more peripherals. A 1x 16GBps link is useful for a lot.
- robbedpeter 5y ago
  I'd be leery of dismissing the potential consumer demand. That much throughput could be put to good use for a myriad of personal and business functions, and software trends to filing whatever hardware can provide. It's like every prediction about users not needing X amount of ram or cpu or dpi or network or storage space. Having that much throughput suggests paging and caching across multiple disks, or using giant models (ml or others) with precomputed lookups in lieu of real-time generation. At any rate, all it takes is a minor inconvenience to overcome and the niche will be exploited to capacity.
  - Dylan16807 5y ago
    > Having that much throughput suggests paging and caching across multiple disks, or using giant models (ml or others) with precomputed lookups in lieu of real-time generation. Those sound to me like situations where latency will be the bottleneck, even on something like PCIe 3.0 x4. I can currently only name one consumer use for these super high speed data transfers, and that's loading 3d assets and textures from SSD to GPU in real time based on where you're looking.
- jabl 5y ago
  Signal integrity is becoming a serious issue for these higher speed links. So it's a compromise between how many traces you need vs. the signal speed. For the server market, GPU's and other accelerators, high speed NIC's and SSD's have an unending appetite for bandwidth, so there I expect a lot of interest. But I think we'll see an increasing gap between this and the consumer market, for better or worse.

dragontamer 5y ago

> How long does it usually take to get consumer products of new PCIe specs? Like 2 years. When PCIe 3.0 was getting popular, 4.0 was finalized. When 4.0 was getting popular, 5.0 was finalized. Now that PCIe 5.0 is coming out (2022, this year), PCIe 6.0 is finalized.

jeffbee 5y ago
PCIe 5 was standardized May 2019 and you could buy it at retail in late 2021. 2 years good rule of thumb.
formerly_proven 5y ago
There was a much bigger gap between 3.0 and 4.0. PCIe 3.0 was available with Sandy or Ivy Bridge, so 2011/2012. PCIe 4.0 was introduced with Zen 2 in 2019. We seem to be back to a faster cadence now however.
- zamadatix 5y ago
  The large delay between 3.0 to 4.0 was a gap between specifications (2010 to 2017) not a gap between specification to implementations (2017 to 2019).
- dragontamer 5y ago
  With the rise of GPU-compute, a lot of the supercomputers are playing around with faster I/O systems. IBM pushed OpenCAPI / NVLink with Nvidia, and I think that inspired the PCIe ecosystem to innovate. PCIe standards are including more and more coherent-memory options. It seems like PCIe is trying to become more like Infinity Fabric (AMD) / UltraPath Interconnect (Intel).
  - Zenst 5y ago
    One bus to rule them all! Would make some sense and if it's fast enough, not saying we will see memory in PCIe form factor, though, size of memory progression, maybe down the line it may go that way.
  - mikepurvis 5y ago
    With sufficiently fast memory bandwidth, does dedicated GPU memory become irrelevant? AFAIK discrete graphics on laptops still share the main memory, but at what point does that become a standard setup on desktops too?
    - dragontamer 5y ago
      DDR5 is far slower than GPU VRAM. If anything, the higher-bandwidths of future PCIe-specs would allow the CPU to access the dedicated GPU VRAM at its full speed, rather than PCIe-limited speeds.
  - jabl 5y ago
    > It seems like PCIe is trying to become more like Infinity Fabric (AMD) / UltraPath Interconnect (Intel). So far CXL is pretty strictly a leader/follower type approach, not really suitable for a more symmetric relationship like you have in a multi-socket system. But maybe one day..

jiggawatts 5y ago

I love how exponential growth can be utterly terrifying or unfathomably amazing. Just a few years ago I was trying to explain to an IT manager that 200 IOPS just doesn’t cut it for their biggest, most important OLAP database. He asked me what would be a more realistic number. “20,000 IOPS is a good start” “You can’t be serious!” “My laptop can do 200,000.”

KennyBlanken 5y ago
> Just a few years ago > “My laptop can do 200,000.” Only now (PCIe 4 and very recent controllers etc) are the very latest top-end NVME drives hitting around 150k IOPS (which isn't stopping manufacturers from claiming ten times that; WD's NVME drive tests at around 150-200k IOPS and yet they claim 1M) and only in ideal circumstances...reads and writes coming out of the SLC cache, which typically under 30GB, often a lot smaller except in the highest-end drives. Many drives that claim to reach that sort of performance are actually using Host Backed Cache, ie stealing RAM. IOPS on SSDs drops precipitously once you exhaust any HBC, controller ram, SLC cache, mid-level MLC cache...and start having to hit the actual QLC/TLC. In the case of a very large database, a lot of IO would be outside cache (though certainly any index, transaction, logging, etc IO would likely be in cache.)
- Cullinet 5y ago
  I would love to pick up from the 200k IOPS laptop quote and demo a RAM drive and then saturate the RAM drive into swapping - I don't know how you could do this on stock distros or Windows but it would make a great executive suite demo of the issues.
- wtallis 5y ago
  More than 500k IOPS was downright common for PCIe gen3 SSDs. 1M IOPS is totally achievable with one high-end gen4 SSD. If you meant IOPS with a queue depth of one, or sustained write IOPS, then you need to specify those extra conditions before calling vendors liars. No high-end drives implement the NVMe Host Memory Buffer feature, let alone rely on it for maximum performance. You can buy an Enterprise QLC drive that is fully capable of 800k IOPS for reads. TLC drives that can do twice that are available. Those drives don't have SLC caching.

cjensen 5y ago

PCIe Gen5 is now available in the latest Intel desktop processors. There are very few lanes, so it can really only run a single GPU, but that's covers a lot of the potential market.

eigen 5y ago
Look like Desktop Chipset 600 series just supports gen 3 & 4[1] and PCIe gen5 ports are only available on AlderLake Desktop[2] not Mobile[3] processors. [1] https://ark.intel.com/content/www/us/en/ark/products/series/218828/intel-600-series-desktop-chipsets.html https://ark.intel.com/content/www/us/en/ark/products/series/... [2] https://ark.intel.com/content/www/us/en/ark/products/134598/intel-core-i912900f-processor-30m-cache-up-to-5-10-ghz.html https://ark.intel.com/content/www/us/en/ark/products/134598/... [3] https://ark.intel.com/content/www/us/en/ark/products/132214/intel-core-i912900h-processor-24m-cache-up-to-5-00-ghz.html https://ark.intel.com/content/www/us/en/ark/products/132214/...

willis936 5y ago

I've not done PHY development personally, but these interfaces are called SerDes. SerDes is short for Serial-Deserializer. Outside of dedicated EQ hardware, everything on the chips are done in parallel so nothing needs to run at a multi-GHz clock.

deleted 5y ago
[deleted]
Taniwha 5y ago
I think that these days there's a lot of convergence going on - everything is essentially serdes in some form - some chips just have N serdes lanes and let you config them for PCIe/ether/data/USB/etc as you need them, much as more traditional SoCs config GPIOs between a bunch of other functions like uarts/spi/i2c/i2s/pcm/...

ksec 5y ago

>How long does it ....... It is not just about getting a product out ( i.e PCI-E 6.0 SSD ), but also the platform support. ( i.e Intel / AMD Motherboard support for PCI-e 6.0 ) Product Launch are highly dependent on Platform support. So far Intel and AMD dont have any concrete plan on PCI-E 6.0, but I believe Amazon could be ahead of the pack with their Graviton platform. Although I am eager to see Netflix's Edge Appliance serving up to 800Gbps if not 1.6Tbps per box.

iancarroll 5y ago
I recently bought a few Zen 2 and Zen 3 HPE servers, and found out only via trial and error that HPE sells Zen 2 servers without Gen4 motherboard support! It seems they took the original Zen motherboards with Gen3 and just swapped out the CPU. Only the Zen 3 has a refreshed motherboard. Makes me now check things more carefully to be sure.
- tinco 5y ago
  Same, worse my vendor sent me an outdated motherboard that officially didn't support the Zen2 CPU and it randomly wouldn't boot, and then they sent me a patched BIOS so it would boot consistently. We needed that machine yesterday, and it's seen maybe 2 minutes of downtime in the past couple years, so I couldn't send it back. My fault for not specifying I wanted the .v2 of the barebones when ordering it over the phone I guess..

jeffbee 5y ago

There's not more lanes available. The generations are getting faster just by increasing the transfer clock rate, up to PCIe 5, and in PCIe 6 by increasing the number of bits per transfer. The way they doubled the speed every generation was pretty basic: the timing tolerances were chopped in half every time. The allowable clock phase noise in PCIe 4 is 200x less than in PCIe 1. The miracle of progress, etc. That miracle is somewhat over. They're not going to be able to drive phase noise down below 1 femtosecond, so 6.0 changes tactics. They are now using a fancy encoding on the wire to double the number of bits per symbol. Eventually, it will look more like wifi-over-copper than like PCI. Ethernet faster than 1gbps has the same trend, for whatever it's worth.

bserge 5y ago
Speaking of which, when is 10Gbit Ethernet coming to laptops? Most have even lost the port ffs.
- jeffbee 5y ago
  Many laptops have a thunderbolt port which serves a similar purpose. On TB4 I get 15gbps in practice, and I can bridge it to ethernet using either a dock or a PC (I use a mac mini with a 10g port to bridge TB to 10ge).
- wmf 5y ago
  I think the real question is when will 10G Ethernet USB4 dongles drop to a reasonable price like <$150.
  - jeffbee 5y ago
    A USB4 peripheral interface is $15 and a 10g ethernet IC is like $80 at least, plus a little box, SFP+ thing, other physical items. Seems hard to get the price down that low and still make a profit. Plus the buyer still needs an expensive cable on both sides.
    - wmf 5y ago

rektide 5y ago

> How long does it usually take to get consumer products of new PCIe specs? Personally I'm expecting this spec to drive pcie 5.0 adoption into consumer space. Tbh consumers dont need this througjput. But given that consumer space has remained stuck around 20x lanes off the cpu (plus some for the chipset), the 5.0 and 6.0 specs will be great for those wanting to build systems with more peripherals. A 1x 16GBps link is useful for a lot.

robbedpeter 5y ago
I'd be leery of dismissing the potential consumer demand. That much throughput could be put to good use for a myriad of personal and business functions, and software trends to filing whatever hardware can provide. It's like every prediction about users not needing X amount of ram or cpu or dpi or network or storage space. Having that much throughput suggests paging and caching across multiple disks, or using giant models (ml or others) with precomputed lookups in lieu of real-time generation. At any rate, all it takes is a minor inconvenience to overcome and the niche will be exploited to capacity.
- Dylan16807 5y ago
  > Having that much throughput suggests paging and caching across multiple disks, or using giant models (ml or others) with precomputed lookups in lieu of real-time generation. Those sound to me like situations where latency will be the bottleneck, even on something like PCIe 3.0 x4. I can currently only name one consumer use for these super high speed data transfers, and that's loading 3d assets and textures from SSD to GPU in real time based on where you're looking.
jabl 5y ago
Signal integrity is becoming a serious issue for these higher speed links. So it's a compromise between how many traces you need vs. the signal speed. For the server market, GPU's and other accelerators, high speed NIC's and SSD's have an unending appetite for bandwidth, so there I expect a lot of interest. But I think we'll see an increasing gap between this and the consumer market, for better or worse.