6 ms·
PCIe for Hackers: The Diffpair Prelude
- gaudat 4y agoHow do I get into the layers between differential pairs and drivers running on my computer? I think we need at least a mid-end FPGA board in ordor to play with all the high speed goodness. The crypto crash gave us a bunch of those boards but now supply is drying up.
- zokier 4y agoSounds like this might be a start of a series of articles on HaD, hopefully they'll cover making a custom endpoint. Because that is also my impression that you need beefy fpga and possibly even proprietary ip cores to play with pcie, but it'd be cool to be proven wrong there
- marktangotango 4y agoThis is my understanding too. Is it possible to use an fpga without built in diff pairs and serdes, and use the io's that are built with a suitable high speed clock domain for a "diy" differential pair? Maybe not as fast, but still faster than any other io on the fpga?
- WJW 4y agoGet yourself a mid-end FPGA before it is too late then. :) Tinkering with anything physical is always going to need appropriate tools, so there is probably no better way than to find some of the tools as cheaply as possible.
- pjc50 4y agoJudging from https://www.fpgadeveloper.com/list-of-fpga-dev-boards-for-pcie/ https://www.fpgadeveloper.com/list-of-fpga-dev-boards-for-pc..., your best bet is one that plugs into an M.2 slot.
- prettyStandard 4y agoI hope to see more about this. One of the things I would like to see are adapters that lets you bond multiple thunderbolt cables to get a full 16x PCIe interface. The reverse of a bifurcator. I'd like an external GPU for a high-end laptop with no compromises.
- cushychicken 4y agoTheir point about diff pairs being more resistant to noise is correct, but it’s not the primary reason for using diff pairs. Differential signals, at a physical layer level, are faster than single ended IO. It’s because you have double the current drive/sinking capability with two drivers. You’re also getting capacitive coupling between each leg of the pair working in your favor, which keeps your edge transition nice and fast. Important stuff when your unit interval is under 500ps!
- k0stas 4y ago> Their point about diff pairs being more resistant to noise is correct, but it’s not the primary reason for using diff pairs. The article is correct. This comment is wrong. > Differential signals, at a physical layer level, are faster than single ended IO. It’s because you have double the current drive/sinking capability with two drivers. The load is also double in a diff pair compared to a single wire, so the net effect is a wash compared to a single wire. > You’re also getting capacitive coupling between each leg of the pair working in your favor, which keeps your edge transition nice and fast. The opposite is true. Differential capacitance effectively appears 2X higher than the nominal capacitance to differential signals, making it a drawback of differential signaling rather than a benefit.
- CamperBob2 4y agoWhen you see a differential pair, simply imagine two independent lines that are each referenced to ground. (That's close to what happens on the PCB anyway, if a ground plane is present. Most of the return current ends up ground-referenced.) Two lines that would have exhibited 50-ohm characteristic impedance in a single-ended circuit will form a ~100-ohm diff pair. In other words, the capacitance isn't doubled, since the capacitance is split by the imaginary ground between the two lines. It looks like two caps in series, not in parallel. Same is true for the load resistance.
- k0stas 4y agoI have designed PCIe compatible transceivers and this will be the last comment I make about it because correcting hardware nonsense on HN is only of transient interest to me. Everything I wrote is 100% correct and in fact incontrovertible. Whether there is a little or a lot of differential capacitance does not change the fact that the differential portion of the capacitance has a 2X effect on the differential signal (as opposed to the common-mode signal, which it has no effect on). This is supported by basic math. If capacitance is to ground then it is not differential capacitance so it is not relevant to this discussion. It may be true that differential capacitance is not a significant contributor to the impedance of PCB differential traces but that does not change the fundamental result (similarly, the principle of photovoltaic conversion still holds true in the dark even though there is little light to convert). And PCB traces are not the only kinds of differential pairs. Diff pairs exist inside the integrated circuits that drive the PCBs where they operate less like transmission lines and more like lumped capacitances due to the frequencies of interest compared to the dimensions of the conductors. In these circuit and conductor structures, differential capacitance can be significant and this is what OP was talking about since he was talking about the legs of the driver (transistors). OP was just wrong about the differential capacitance being good for speed. It's bad for speed.
- twawaaay 4y agoFor anyone interested in the topic, I can highly recommend "Signal and Power Integrity Simplified" by Eric Bogatin. I found this to be absolutely best resource on the topic for somebody with no formal electronics education like me.
- DHaldane 4y agoI like this book a lot. Has a heavy focus on physics and an intuitive understanding of physics, rather than just giving a list of heuristics to abide by.
- twawaaay 4y agoYes, that's exactly the point. Heuristics is good until it is not. And any heuristics will eventually fail at high enough frequency... With intuitive understanding you start seeing how various design choices affect the performance and you can focus on checking what might be the cause of the problem and fixing things that have biggest potential for improvement. It allows you to learn from your mistakes whereas with heuristics you feel like a dog that is being beaten by the owner but without any idea why.
- izacus 4y agoIsn't Thunderbolt also "just" PCIe over USB-C cabling? :) PCIe seems to be a well designed standard all things considered - very fast and very robust.
- wtallis 4y agoThunderbolt is quite different at the lower layers. It's more accurate to think of Thunderbolt as its own connection technology that can carry both PCIe and DisplayPort data packets multiplexed on one cable, with the DisplayPort packets getting priority.
- jjoonathan 4y agoI've heard that Thunderbolt and USB4 "merged" -- does this mean it's Thunderbolt at the lowest layers and USB4 is encapsulated? Or are there two different signaling standards sharing the same PHY?
- wtallis 4y agoWhen USB is involved, you can always assume they chose the more complicated option. USB4 is a competing, incompatible method of encapsulating and multiplexing PCIe, DisplayPort and USB data packets over one cable. But the USB4 spec incorporates Thunderbolt as an alternate mode. Thunderbolt support is required for USB4 hubs but optional for USB4 hosts and devices. So you will often (maybe even always, in practice) be able to use a USB4 host port as a Thunderbolt port.
- izacus 4y agoIt's worth mentioning that Microsoft seems to demand support for Thunderbolt mode for Win11 certification.
- deleted 4y ago[deleted]
- rektide 4y ago
- nanumbat 4y agoFrom the comments: "Protip: don’t try to be fancy and provide a local REFCLK from an independent oscillator. Always use the host’s REFCLK as it may be spread spectrum modulated for EMI compliance in most PC systems." Someone handed me a non-functional prototype system a few years ago (the first time I'd encountered PCIe), and it had a local REFCLK and an upstream Android-style host which, indeed, had enabled spread-spectrum clocking. Took me two miserable weeks to figure that one out.
- jhallenworld 4y agoAlso it's a good point to understand just how these SERDES blocks work. There is a CDR circuit, but it is fully dependent on the refclk, which can only be off a tiny amount from the one used on the transmit side- it's not like there's a wide-bandwidth PLL that recovers the embedded clock from the encoded data. This is unfortunate, because for transporting video you would like to also transport the pixel clock, but you don't get it for free from the SERDES CDR. DisplayPort use SERDES and have to transport the pixel clock. They do it by sending a message with fractional relationship between the pixel clock and the SERDES clock to the receive side, which derives the pixel clock from the recovered SERDES CDR clock using a fractional-N PLL. I had the idea of transporting video over PCIe at one point, so I was interested in this. The reason is that PCIe is sometimes available for free, so why not use it. I wanted to use a local reference clock, so would have been forced to use the DisplayPort scheme (but no fractional-N PLL available). SERDES with actual PLL CDRs in them for video do exist, but they are different from the generic SERDES used for PCIe and networking. [edit: removed HDMI, only DisplayPort works as I said above]
- formerly_proven 4y agoiirc that’s why only very few GPUs supported more than two HDMI/DVI outputs, you need a complicated clock generator for them as the bit clock is a multiple of the pixel clock. Meanwhile DisplayPort only has like four possible link clocks.
- dfox 4y agoHDMI/DVI has dedicated pixel clock pair as do the various LVDS LCD panel interfaces (and CameraLink, which is a related PHY). These interfaces are more or less “dumb” and simply serialize the display data onto three pairs with the same timing as VGA/DPI (discrete serdeses for these interfaces tend to have internal x10 PLL for generating the bit-clock).
- DarthNebo 4y agoSomeone figure out eGPUs for Apple Silicon, maybe a kickstarter?
- exabrial 4y agoSomeday, I wish I could understand electrical impedance. I understand DC resistance. After designing and building my own Tube based electric guitar amplifier, I head to learn the relation between DC resistance and current _intimately_. What I still didn't have a grasp on is the concept of impedance. While I was able to calculate correct values for my guitar amp, I didn't know _why_ I was doing it which is a bit frustrating. For instance, I still don't understand the output transformer that "matches the impedance of the speaker to the output tube". Le sigh.
- edrxty 4y agoI'm no expert but FWIW, I find thinking about it as a resistance is unhelpful. It's much closer to a sort of "resonance coefficient". Sure it's measured in ohms but a 50 ohm cable will not "resist" the flow of signal like a 50 ohm resistor because a resistor is a load (can simulate an antenna) while the cable is just resonant with the load. What makes a given line/cable "50 ohm" is largely capacitance driven because it needs to be resonant with a 50 ohm load at one end and a 50 ohm source at the other. I find amateur radio literature to be pretty good for this kind of stuff. It's a very central issue to them.
- meatmanek 4y agoI think the OP is talking about input/output impedances of amplifiers/guitar pickups, while you're talking about transmission line characteristic impedance. Impedance of a transmission line is weird: in some ways, it acts exactly like a resistor, and in some ways it doesn't. In contrast to a resistor, an ideal transmission line doesn't convert electrical energy to heat. Ideally, 100% of electrical energy put in one end of the line will make it to the other end of the line intact. However, just like an ideal resistor, an ideal transmission line will have a real-valued impedance, not a capacitive (negative imaginary values) or inductive (positive imaginary values) impedance; nor will an ideal transmission line have any frequency dependence in its impedance: 50 ohms is 50 ohms. One way I like to think of characteristic impedance is that it's the temporary impedance a change in signal will see until current/voltage wave reflections make it back from the other end: For a 50-ohm ideal transmission line one light-second long (pretending we have velocity factor 1.0 to make the math easy), your ohmmeter would read 50 ohms for two seconds, and after that it would read whatever resistance is connected to the other end. If the other end is an open circuit, the ohmmeter would measure 50 ohms then infinite resistance; if the other end is a short circuit, the ohmmeter would see 50 ohms then 0 ohms; if the other end is terminated with a 50-ohm resistor, the ohmmeter would measure 50 ohms indefinitely. If you have an infinitely long ideal transmission line with 50 ohm impedance, and hooked up an ohmmeter to it, it would measure 50 ohms. If you hooked up an LCR meter, it would show 0 impedance and 0 capacitance. This indistinguishability between an infinitely long transmission line and a resistor is why a matched termination resistor prevents signal reflections: If you have some finite length of transmission line, and you attach either a matched resistor or an infinitely long transmission line with the same impedance to the end, the first length of transmission line cannot tell which you have attached - its behavior will be the same in either case: all the energy is passed into the next section.
- samtho 4y agoOne of the most fascinating thing about working with hardware vs software is your designs are influenced by the laws of physics, but more importantly, how we choose to implement our understanding of these laws. I've gotten away with quick and dirty prototypes with no length matching but always keeping the two traces as close to each others' path as possible with as few vias as possible (even avoiding if able), but other times you need to get it right as I suspect that some chips are more robust than others with their signal processing. For CAN bus and PCIe, you can use garbage routing and tin foil to connect your diff pairs, but for HDMI, you better get that right or suffer random, intermittent problems forever.
- DHaldane 4y agoFrom the article: > any interference affects the [diff pair] signals equally – as signals are compared to each other to receive information, this means that the information received is not affected by noise overlaid onto both of the signals. This is not true - a common myth about differential pairs. Don't rely on it for your designs. More info - see page 15 https://www.speedingedge.com/PDF-Files/DiffSigDesign.pdf https://www.speedingedge.com/PDF-Files/DiffSigDesign.pdf
- DHaldane 4y agoTo expand - the way most relevant noise propagates on a board is often assumed to couple equally into both sides of the differential pair. It doesn't.
- jhallenworld 4y agoHere is one thing I've learned about PCIe- the .1 uF series capacitors must be .1 uF, .01 uF will not work. The reason is that the link detection depends on the RC charging rate of these capacitors. Can you guess why I know this?
- westmeal 4y agoI can and I am so sorry.
- datenwolf 4y ago> First off, you want to keep both of the pair’s signals close to each other throughout their length. The closer the two signals are, the better external interference cancellation works, and the less noise they radiate (…) Wrong! This is a myth that's stubbornly difficult to weed out. The coupling between the wires of a pair is only ~12% the amount of coupling to the reference plane. On a PCB it's best to treat each signal on its own and route it as a single ended, coplanar waveguide surrounded by reference planes. This allows to use far thicker widths, which reduces inductance, which also reduces the coupling to radiating EM fields. If you want a longer talk on the topic: https://www.youtube.com/watch?v=QG0Apol-oj0 https://www.youtube.com/watch?v=QG0Apol-oj0