8 ms·

Stupid question from me: how the hell do they prototype and debug this? It's very far from my EE knowledge. Communication analysers will show the eye diagram an

Stupid question from me: how the hell do they prototype and debug this? It's very far from my EE knowledge. Communication analysers will show the eye diagram and tell you a lot about stray capacitance and inductance, but I know that most 10GSa/s+ devices cost more than a house and have a tendency to live in a lab for a long time. It seems to my mind like there are so many difficulties in driving pci-e fast, from timing to voltage drop and current limitations. All of this must be implemented, somehow, if the spec is finalised. Right? They wouldn't write it up otherwise (right?!). So -- how do you prove that your infrastructure can blow the socks off everything without building two of them and testing?

wmf 5y ago
They do build prototypes and test them.
- userbinator 5y ago
  ...using very, very expensive equipment.
  - StillBored 5y ago
    Modern high end serdes have very complex signal mgmt and monitoring tools built in. You can detect the link quality, pull eye diagrams/etc from them. So a lot of the reason for using some of these high end scopes/analyzers is bypassed by the testing/diagnostics functions on the serdes itself. And then of course there are a ton of signal tuning parameters which get programmed during link training. Even modern analyzers have problems with this, and I've seen vendors that instead of even trying to reconstruct the signal at the endpoint use virtual transmission models and require one to probe at the source. This is one of those areas that I bump into every couple years, and its frequently quite educational.
    - RF_Savage 5y ago
      I wonder how flexible the serdes built in test equipment stuff is. A BNC-to-PCIe-lane board with some signal conditioning would make for a hilarious oscilloscope.
      - jacquesm 5y ago
        You better make isolation a subpart of 'some signal conditioning' or you'll be looking at some expensive repairs. That's the hard part of making a scope: isolating the source and the scope guts while maintaining signal integrity.
      - willis936 5y ago
        >BNC This makes me smile. When I left PHY testing a few years ago SMA was no longer keeping up with the > 25 GBaud PAM4 standards coming out. BNC was useful for PHY development in the 90s.
      - jjoonathan 5y ago
        I suspect it's heavily sampled, and maybe not only on the time axis. They would already have a reconstructed clock available, so it would be straightforward to sync up a time aperture, and the voltage aperture could be swept by adding an offset into the output comparator(s). They could then scan the ranges of both parameters to build up the eye diagram with minimal additional hardware. This would make it a brilliant oscilloscope if you want an eye diagram of a high speed serial bus, but for just about any other task it would be a bit rubbish. There isn't a UXR hiding in every PCIe 5+ lane.
        RF_Savage 5y ago
        That's likely how it is. It would still be interesting if there was something like a glscopeclient driver for them. If nothing else it would make hardcore overclockers step up their game and start talking more about signal integrity. :)
        jjoonathan 5y ago
        Haha, yes, I'd love to see what slathering vaseline on a high speed diff pair does to the eye. Link equalization is the real MVP! It'll be fun to see how this evolves with 32Gbps and PAM4.
- ancharm 5y ago
  Yep, and the highest data rates are only really possible on the best processes (12nm, 7nm, 5nm) , which is a fun chunk of change to spend for a tape out.
dragontamer 5y ago
I'm not in EE, but I'd imagine that you start with x1 lane and work your way up. Each PCIe lane is an independent SERDES. Its not like a parallel memory controller where they share a clock, when you have x16 lanes, you have 16x independent streams you're shoving to the device downstream.
lizknope 5y ago
We build a test chip with the serdes and some control logic. One wafer might give us 50 chips. We package them and build a board with two of the chips and have them talk to each other.
Lramseyer 5y ago
Not a stupid question at all! Any communication protocol has functional layers to them, and at the very bottom is what's called the Phy layer (or physical layer.) While pretty much every layer can be implemented in digital logic, the physical layer has a lot of analog circuitry. (It's programmable analog circuitry, so it's technically mixed signal.) Most wired communication protocols that you are familiar with (including SAS/SATA, USB, HDMI/DP, PCIe/NVMe, and Ethernet) all use what's known as a "SerDes Phy" which is short for Serializer Deserializer. It takes parallel data in, and serializes it to transmit data, and does the reverse when receiving data. In the example of PCIe Gen 4, it takes 32 bits of data in at 500 Mb/s (per bit) and serializes it to 16 Gb/s. Because the SerDes Phy is such a common part of all communication protocols, it's often reused in other designs. For example - a GPU may use the same SerDes Phy design for its PCIe lanes as it does for its HDMI ports (at least on the transmit side.) They are however programmed differently. Because these designs see so much reuse in so many different chips, there are companies that specialize in designing SerDes Phys, and their development is usually ahead of protocol development. Though in today's day and age, it's super complicated, and consists of way more than shift registers and a PLL. But there are SerDes designs that exist in products that are capable of 112Gb/s (not sure if 224Gb/s exists outside of test chips yet.) You can buy modern FPGAs that have PCIe Gen 5, but with a SerDes Phy capable of going way faster. And if you want, you could bypass that hard block and talk straight to the SerDes, and program your own PCIe 6.0 controller. But to answer your question on how they prototype it and test it, they run computer simulations that are exceedingly complicated and slow, and once a test chip comes back from the fab, eye diagrams are usually collected using what's known as a repeating signal scope. Since it's insanely difficult to sample a 112Gb/s signal at any reasonable resolution, it's sampled statistically (in time) over billions of samples, and plotted by knowing the frequency of the target signal. An eye diagram is what comes out of a traditional NRZ (non return to zero) where each cycle represents either a 1 or a 0. In an eye diagram, you see all permutations of bit transitions overlayed on each other, since you're sampling billions of cycles statistically. This is important because you can see other anomalies of your signal like jitter and rise/fall times. There's a lot more to it, and this comment is already getting to be a wall of text but definitely feel free to ask more questions!
- baruch 5y ago
willis936 5y ago
In addition to the other comments: testing! PCI SIG has regular validation events and beyond that the lab where I cut my teeth is well renowned for hosting plugfests for cutting edge technologies.

wmf 5y ago

They do build prototypes and test them.

userbinator 5y ago
...using very, very expensive equipment.
- StillBored 5y ago
  Modern high end serdes have very complex signal mgmt and monitoring tools built in. You can detect the link quality, pull eye diagrams/etc from them. So a lot of the reason for using some of these high end scopes/analyzers is bypassed by the testing/diagnostics functions on the serdes itself. And then of course there are a ton of signal tuning parameters which get programmed during link training. Even modern analyzers have problems with this, and I've seen vendors that instead of even trying to reconstruct the signal at the endpoint use virtual transmission models and require one to probe at the source. This is one of those areas that I bump into every couple years, and its frequently quite educational.
  - RF_Savage 5y ago
    I wonder how flexible the serdes built in test equipment stuff is. A BNC-to-PCIe-lane board with some signal conditioning would make for a hilarious oscilloscope.
    - jacquesm 5y ago
      You better make isolation a subpart of 'some signal conditioning' or you'll be looking at some expensive repairs. That's the hard part of making a scope: isolating the source and the scope guts while maintaining signal integrity.
    - willis936 5y ago
      >BNC This makes me smile. When I left PHY testing a few years ago SMA was no longer keeping up with the > 25 GBaud PAM4 standards coming out. BNC was useful for PHY development in the 90s.
    - jjoonathan 5y ago
      I suspect it's heavily sampled, and maybe not only on the time axis. They would already have a reconstructed clock available, so it would be straightforward to sync up a time aperture, and the voltage aperture could be swept by adding an offset into the output comparator(s). They could then scan the ranges of both parameters to build up the eye diagram with minimal additional hardware. This would make it a brilliant oscilloscope if you want an eye diagram of a high speed serial bus, but for just about any other task it would be a bit rubbish. There isn't a UXR hiding in every PCIe 5+ lane.
      - RF_Savage 5y ago
        That's likely how it is. It would still be interesting if there was something like a glscopeclient driver for them. If nothing else it would make hardcore overclockers step up their game and start talking more about signal integrity. :)
        jjoonathan 5y ago
        Haha, yes, I'd love to see what slathering vaseline on a high speed diff pair does to the eye. Link equalization is the real MVP! It'll be fun to see how this evolves with 32Gbps and PAM4.
ancharm 5y ago
Yep, and the highest data rates are only really possible on the best processes (12nm, 7nm, 5nm) , which is a fun chunk of change to spend for a tape out.

dragontamer 5y ago

I'm not in EE, but I'd imagine that you start with x1 lane and work your way up. Each PCIe lane is an independent SERDES. Its not like a parallel memory controller where they share a clock, when you have x16 lanes, you have 16x independent streams you're shoving to the device downstream.

lizknope 5y ago

We build a test chip with the serdes and some control logic. One wafer might give us 50 chips. We package them and build a board with two of the chips and have them talk to each other.

Lramseyer 5y ago

Not a stupid question at all! Any communication protocol has functional layers to them, and at the very bottom is what's called the Phy layer (or physical layer.) While pretty much every layer can be implemented in digital logic, the physical layer has a lot of analog circuitry. (It's programmable analog circuitry, so it's technically mixed signal.) Most wired communication protocols that you are familiar with (including SAS/SATA, USB, HDMI/DP, PCIe/NVMe, and Ethernet) all use what's known as a "SerDes Phy" which is short for Serializer Deserializer. It takes parallel data in, and serializes it to transmit data, and does the reverse when receiving data. In the example of PCIe Gen 4, it takes 32 bits of data in at 500 Mb/s (per bit) and serializes it to 16 Gb/s. Because the SerDes Phy is such a common part of all communication protocols, it's often reused in other designs. For example - a GPU may use the same SerDes Phy design for its PCIe lanes as it does for its HDMI ports (at least on the transmit side.) They are however programmed differently. Because these designs see so much reuse in so many different chips, there are companies that specialize in designing SerDes Phys, and their development is usually ahead of protocol development. Though in today's day and age, it's super complicated, and consists of way more than shift registers and a PLL. But there are SerDes designs that exist in products that are capable of 112Gb/s (not sure if 224Gb/s exists outside of test chips yet.) You can buy modern FPGAs that have PCIe Gen 5, but with a SerDes Phy capable of going way faster. And if you want, you could bypass that hard block and talk straight to the SerDes, and program your own PCIe 6.0 controller. But to answer your question on how they prototype it and test it, they run computer simulations that are exceedingly complicated and slow, and once a test chip comes back from the fab, eye diagrams are usually collected using what's known as a repeating signal scope. Since it's insanely difficult to sample a 112Gb/s signal at any reasonable resolution, it's sampled statistically (in time) over billions of samples, and plotted by knowing the frequency of the target signal. An eye diagram is what comes out of a traditional NRZ (non return to zero) where each cycle represents either a 1 or a 0. In an eye diagram, you see all permutations of bit transitions overlayed on each other, since you're sampling billions of cycles statistically. This is important because you can see other anomalies of your signal like jitter and rise/fall times. There's a lot more to it, and this comment is already getting to be a wall of text but definitely feel free to ask more questions!

baruch 5y ago

willis936 5y ago

In addition to the other comments: testing! PCI SIG has regular validation events and beyond that the lab where I cut my teeth is well renowned for hosting plugfests for cutting edge technologies.