3 ms·
> the time between crashes was long I know that pain. I was working on the NDIS driver (WinNT 3.5.1, later a 4.0 beta) for our HIPPI[1] PCI cards. The hardware
by pdkl95 11y ago
> the time between crashes was long
I know that pain. I was working on the NDIS driver (WinNT 3.5.1, later a 4.0 beta) for our HIPPI[1] PCI cards. The hardware was based on our very-reliable SBus cards, so when the PCI device started crashing, we assumed it must be a software error.
I probably spent 2+ months trying to find the cause of the crash. Trying to decide if your change had any affect at all when you have to wait anywhere from 5 minutes to >10 hours for the crash to happen will drive you insane. You have to fight the urge to "read the tea leaves"; you will see what you want to see if you aren't careful.
While, I never did find the problem, I did discover that MSVC was dropping "while (1) {...}" loops when they were in a macro, but compiled correctly when they macros were changed to "for (;;) {...}".
Later, a hardware engineer took the time to try to randomly capture the entire PCI interface in the logic analyzer, hoping to randomly capture what happened before the crash. After another month+ of testing, it worked. He discovered that the PCI interface chip we were using (AMCC) had a bug. If the PCI bus master deasserted the GNT# pin in exactly the same clock cycle that the card asserted the REQ# pin, the chip wouldn't notice that it lost the bus. The card would continue to drive the bus instead of switching to tri-state, and everything crashed.
Every read or write to the card was rolling 33MHz dice. Collisions were unlikely, but with enough tries the crash was inevitable.
[1] https://en.wikipedia.org/wiki/HIPPI https://en.wikipedia.org/wiki/HIPPI
- jacquesm 11y agoOk, that one should get the prize. The chances of spotting that are insanely small, kudos on making progress at all, more kudos for eventually tracing it down to the root. I really hope I'll never have anything that nasty on my path.
- userbinator 11y agoThat reminds me of this: http://www.linusakesson.net/scene/safevsp/ http://www.linusakesson.net/scene/safevsp/
- jacquesm 11y ago"A technical explanation appears in the demo as a 10-minute scroller, but the same text is provided below, for your convenience." Hehe :) That's a really neat one: if you've fixed your machine you can read it, otherwise you can't... Awesome sleuthing, and the take home is if you get down deep enough digital is analog again.
- pdkl95 11y agoMost of the credit goes to the hardware guys that were able to finally isolate the problem. We found the bug, but the months of delay (and a few other problems like losing a big contract[1]) killed the startup a few months later. While I'm annoyed the SCSI3-over-{HIPPI,ATM,FDDI} switch I got to work on was never finished, the next job doing embedded Z80 stuff was a lot more fun... and a LOT easier to debug. Incidentally, I found a picture[2] of the NIC. Note that HIPPI is simplex - you needed two cards and two 50-pin cables. This made the drivers extra "fun". [1] "no, I'm not going to smuggle a bag of EEPROMs with the latest firmware through Malaysian customs in my carry-on" (still hadn't found the bug at the time) [2] https://hsi.web.cern.ch/HSI/hippi/procintf/pcihippi/pcihipso.htm https://hsi.web.cern.ch/HSI/hippi/procintf/pcihippi/pcihipso...
- jacquesm 11y agoThat's a beautiful board. I remember the FAMP made by Herzberger & Co at NIKHEF-H, I used to hang around their hardware labs when those were built. Similar hairy debug sessions in progress. Those worked well in the end iirc and ended up at CERN.