29 ms·

I started probing the Ethernet board's input circuit with the oscilloscope. The board was receiving the input okay, but a few gates later the signals looked kin

I started probing the Ethernet board's input circuit with the oscilloscope. The board was receiving the input okay, but a few gates later the signals looked kind of sketchy, as you can see above. Is it even possible to do this kind of debugging on modern hardware? Or is this a lost art?

al2o3cr 9y ago
Sorta? Not all the signals are available, depending on the level of integration. OTOH, modern hardware also frequently has test support via things like JTAG, which can give you a ton of information and/or control the device directly. For instance, there's ChipScope from Xilinx (https://www.xilinx.com/products/design-tools/chipscopepro.html https://www.xilinx.com/products/design-tools/chipscopepro.ht...) which allows you to debug your FPGA hardware from outside without wasting a ton of pins. It's halfway between poking the board with an oscilloscope and running GDB, as it uses the metadata from your design to label things.
subway 9y ago
If you can find the relevant documentation, sure. Modern hardware is made from black boxes on top of black boxes.
sigstoat 9y ago
> Is it even possible to do this kind of debugging on modern hardware? Or is this a lost art? EE's still have to diagnose hardware problems. Keysight will be happy to sell you diagnostic equipment for investigating problems with your 100Gbps ethernet devices. Your bank probably won't care to fund the necessary loans, though.
SAI_Peregrinus 9y ago
It's possible, but harder. A modern Ethernet card would probably be doing this step entirely on one chip. A quick search shows the Microchip ENC28J60 as a pretty typical part. The result is that you'd be able to probe the input to the chip and the output, and see the failure at the output, and replace the chip. Effectively the same as what was done here, just with more components built into the chip. Louis Rossmann[1] does macbook component-level logic board repairs, though 99% of the time he doesn't need to break out the oscilloscope to find the problem. It's almost always a power supply or liquid damage problem, or something similar that can be found with nothing more than a multimeter. [1] https://www.youtube.com/playlist?list=PLkVbIsAWN2lsHdY7ldAAgtJug50pRNQv0 https://www.youtube.com/playlist?list=PLkVbIsAWN2lsHdY7ldAAg...
- emeraldd 9y ago
  "That's not unprofessional... This is unprofessional... " That made my day! Thanks for pointing this out.
- katastic 9y ago
  I thought you needed a many-GHZ scope/logic probe to capture modern ethernet.
  - SAI_Peregrinus 9y ago
    Not really. For example the chip I mentioned (and most others) includes JTAG test capability, and several programmable modes. That makes it possible to test the chip more easily. Also some of the most common errors (overvoltage/overcurrent applied to inputs) will likely result in a short to ground, so they'll show up when checking the chip's power input with a multimeter: it will be shorted to ground! Other errors can cause other symptoms, but needing to actually view the signal is pretty rare.
- WalterBright 9y ago
  I kept my old computers from the 80's to today. A couple years ago I tried to fire them up. None older than 10 years would come up. Some would crackle and belch smoke, others weird POST errors. Oh well.
- sillysaurus3 9y ago
  https://www.youtube.com/watch?v=dK9pfbGSE4A https://www.youtube.com/watch?v=dK9pfbGSE4A is incredibly cool. Thanks for this series! It's so neat to watch a pro in a completely different domain. I know some of the basic concepts, but it's mesmerizing to watch him reason through voltage levels and pinpoint a failed resistor. That's magical to me.
  - heywire 9y ago
    Agreed. I became aware of Louis' work after looking for information on the infamous Mid-2010 Macbook Pro 15 GPU kernel panic. After watching this video [1], I was able to replace the same capacitor for ~$3 and restore a previously unusable Macbook. [1] https://www.youtube.com/watch?v=DzcgT_fiVTA https://www.youtube.com/watch?v=DzcgT_fiVTA
dfox 9y ago
Debugging this problem in this way was not possible in the time of Alto and became possible in late 80's (with LeCroy DSOs) and really practical in early to mid 90's (HP/Agilent) and then cheap relatively recently (chinese DSOs). In the days before that you relied on CRT based scope and tried to make the fault frequent enought to be visible on it. To some extent this is easier today (but not might be so in few years) because you can buy realtime DSO that is fast enough to capture almost any computer interface used today with usable resolution (albeit you can probably find some successful small-ish hardware startup that would cost you less than such scope) The art in this kind of debugging is IMHO in using woefully inadequate test equipment to come up with meaningful idea of what might be broken. And this is still done today, with TV repair shops fixing OLED TVs with nothing but DVM and RLC bridge, me looking at 400Mbps+ serial LVDS on customer site with 50MHz portable DSO (they didn't have anything else) and such.
- labcomputer 9y ago
  I don't see anything in the write-up that requires a DSO--an storage tube (analog) oscilloscope should suffice. Most storage tube oscilloscopes supported single sweep capture in bi-stable mode. That is, you could clear the screen, reset the timebase, and capture the trace of a single trigger event. In bi-stable mode, the image should last hours. Some models (like the Tek 7313) even allowed the top and bottom of the screen to be set to storage or live independently. On top of that, most scope camera shutters could be set up to automatically open during the forward seep (called "single sweep" on Tek cameras). That allows you to set up the scope, and walk away while you wait for the rare event to occur. Tektronix even sold delay lines (big coils of hardline coax) with trigger pick-offs for the express purpose of displaying before-the-trigger waveforms. No question that DSOs make this all easier, but saying it was impossible is a bit of a stretch.
planteen 9y ago
The input even on modern hardware is received by an Ethernet PHY which then has logic-level lines going into the SoC. So this level of debugging would be possible. Inside the SoC if something has failed, well, you are kind of screwed.
- srcmap 9y ago
  Fundamental engineering debugging process is similar. I recently debugged packet issues after 100gbps port speed change to 25gbps, to 10 gbps issues. Breakdown the issue to sub-system components - check phy, mac loopback, check link status on both side of the QSFP connections before and after speed changes, check counters (A LOT of them), enable debug packet to send to CPU with special command. Check all the VLAN, port settings, configuration commands over and over again. Enable debugging on kernel driver to track down every bytes/bits of every packet. Use tcpudmp on linux socket layer when one gets to that point. Instead of oscilloscope, today's SOC does have a lot of counters inside that help one identifies issues. For complex issue, one does get tremendous high when the issue was ID and resolved.
  - nerpderp83 9y ago
    > one does get tremendous high when the issue was ID and resolved I find getting a little high really helps with long debugging sessions.
  - planteen 9y ago
    Sure, that is very true during development or debugging a complex production issue as you mentioned. I meant troubleshooting and reworking along the lines of "this board used to work, now it doesn't". Most reworks I see of boards involve replacing parts, which in a modern Ethernet design would consist of the jack, some magnetics, PHY, and SoC.

al2o3cr 9y ago

Sorta? Not all the signals are available, depending on the level of integration. OTOH, modern hardware also frequently has test support via things like JTAG, which can give you a ton of information and/or control the device directly. For instance, there's ChipScope from Xilinx (https://www.xilinx.com/products/design-tools/chipscopepro.html https://www.xilinx.com/products/design-tools/chipscopepro.ht...) which allows you to debug your FPGA hardware from outside without wasting a ton of pins. It's halfway between poking the board with an oscilloscope and running GDB, as it uses the metadata from your design to label things.

subway 9y ago

If you can find the relevant documentation, sure. Modern hardware is made from black boxes on top of black boxes.

sigstoat 9y ago

> Is it even possible to do this kind of debugging on modern hardware? Or is this a lost art? EE's still have to diagnose hardware problems. Keysight will be happy to sell you diagnostic equipment for investigating problems with your 100Gbps ethernet devices. Your bank probably won't care to fund the necessary loans, though.

SAI_Peregrinus 9y ago

It's possible, but harder. A modern Ethernet card would probably be doing this step entirely on one chip. A quick search shows the Microchip ENC28J60 as a pretty typical part. The result is that you'd be able to probe the input to the chip and the output, and see the failure at the output, and replace the chip. Effectively the same as what was done here, just with more components built into the chip. Louis Rossmann[1] does macbook component-level logic board repairs, though 99% of the time he doesn't need to break out the oscilloscope to find the problem. It's almost always a power supply or liquid damage problem, or something similar that can be found with nothing more than a multimeter. [1] https://www.youtube.com/playlist?list=PLkVbIsAWN2lsHdY7ldAAgtJug50pRNQv0 https://www.youtube.com/playlist?list=PLkVbIsAWN2lsHdY7ldAAg...

emeraldd 9y ago
"That's not unprofessional... This is unprofessional... " That made my day! Thanks for pointing this out.
katastic 9y ago
I thought you needed a many-GHZ scope/logic probe to capture modern ethernet.
- SAI_Peregrinus 9y ago
  Not really. For example the chip I mentioned (and most others) includes JTAG test capability, and several programmable modes. That makes it possible to test the chip more easily. Also some of the most common errors (overvoltage/overcurrent applied to inputs) will likely result in a short to ground, so they'll show up when checking the chip's power input with a multimeter: it will be shorted to ground! Other errors can cause other symptoms, but needing to actually view the signal is pretty rare.
WalterBright 9y ago
I kept my old computers from the 80's to today. A couple years ago I tried to fire them up. None older than 10 years would come up. Some would crackle and belch smoke, others weird POST errors. Oh well.
sillysaurus3 9y ago
https://www.youtube.com/watch?v=dK9pfbGSE4A https://www.youtube.com/watch?v=dK9pfbGSE4A is incredibly cool. Thanks for this series! It's so neat to watch a pro in a completely different domain. I know some of the basic concepts, but it's mesmerizing to watch him reason through voltage levels and pinpoint a failed resistor. That's magical to me.
- heywire 9y ago
  Agreed. I became aware of Louis' work after looking for information on the infamous Mid-2010 Macbook Pro 15 GPU kernel panic. After watching this video [1], I was able to replace the same capacitor for ~$3 and restore a previously unusable Macbook. [1] https://www.youtube.com/watch?v=DzcgT_fiVTA https://www.youtube.com/watch?v=DzcgT_fiVTA

dfox 9y ago

Debugging this problem in this way was not possible in the time of Alto and became possible in late 80's (with LeCroy DSOs) and really practical in early to mid 90's (HP/Agilent) and then cheap relatively recently (chinese DSOs). In the days before that you relied on CRT based scope and tried to make the fault frequent enought to be visible on it. To some extent this is easier today (but not might be so in few years) because you can buy realtime DSO that is fast enough to capture almost any computer interface used today with usable resolution (albeit you can probably find some successful small-ish hardware startup that would cost you less than such scope) The art in this kind of debugging is IMHO in using woefully inadequate test equipment to come up with meaningful idea of what might be broken. And this is still done today, with TV repair shops fixing OLED TVs with nothing but DVM and RLC bridge, me looking at 400Mbps+ serial LVDS on customer site with 50MHz portable DSO (they didn't have anything else) and such.

labcomputer 9y ago
I don't see anything in the write-up that requires a DSO--an storage tube (analog) oscilloscope should suffice. Most storage tube oscilloscopes supported single sweep capture in bi-stable mode. That is, you could clear the screen, reset the timebase, and capture the trace of a single trigger event. In bi-stable mode, the image should last hours. Some models (like the Tek 7313) even allowed the top and bottom of the screen to be set to storage or live independently. On top of that, most scope camera shutters could be set up to automatically open during the forward seep (called "single sweep" on Tek cameras). That allows you to set up the scope, and walk away while you wait for the rare event to occur. Tektronix even sold delay lines (big coils of hardline coax) with trigger pick-offs for the express purpose of displaying before-the-trigger waveforms. No question that DSOs make this all easier, but saying it was impossible is a bit of a stretch.

planteen 9y ago

The input even on modern hardware is received by an Ethernet PHY which then has logic-level lines going into the SoC. So this level of debugging would be possible. Inside the SoC if something has failed, well, you are kind of screwed.

srcmap 9y ago
Fundamental engineering debugging process is similar. I recently debugged packet issues after 100gbps port speed change to 25gbps, to 10 gbps issues. Breakdown the issue to sub-system components - check phy, mac loopback, check link status on both side of the QSFP connections before and after speed changes, check counters (A LOT of them), enable debug packet to send to CPU with special command. Check all the VLAN, port settings, configuration commands over and over again. Enable debugging on kernel driver to track down every bytes/bits of every packet. Use tcpudmp on linux socket layer when one gets to that point. Instead of oscilloscope, today's SOC does have a lot of counters inside that help one identifies issues. For complex issue, one does get tremendous high when the issue was ID and resolved.
- nerpderp83 9y ago
  > one does get tremendous high when the issue was ID and resolved I find getting a little high really helps with long debugging sessions.
- planteen 9y ago
  Sure, that is very true during development or debugging a complex production issue as you mentioned. I meant troubleshooting and reworking along the lines of "this board used to work, now it doesn't". Most reworks I see of boards involve replacing parts, which in a modern Ethernet design would consist of the jack, some magnetics, PHY, and SoC.