5 ms·

I eventually solved a bug that took about 1.5 years to figure out, since we were not able to reproduce it, and it only happened on a customer's system. Long sto

I eventually solved a bug that took about 1.5 years to figure out, since we were not able to reproduce it, and it only happened on a customer's system. Long story short, it ended up being a by-product of sending a (256*N)+1 byte packet through the system, and the fpga asserted a signal 2 clocks later that did not assert in time on those sizes. This resulted in a single buffer leaking, but eventually it built up exponentially. Many days and nights of a logic analyzer and other equipment led to a single accident in the lab where I was able to reproduce it. It's hard to describe the feeling of reproducing something like that, after haunting me for so long.

ComputerGuru 8y ago
I feel you. A bug in our recovery software would result in the live CD being unable to open the Windows kernel file after having located it on disk. No amount of logging we added to the code made sense. In the end, after more than a year, there was a customer in the UK that could reproduce the problem each time on a clean Windows 10 install (so no private info). I mailed him a new SSD to replace his old HDD and paid for him to overnight me his hard disk. Problem solved the very next day.
- shaklee3 8y ago
  That's awesome!
nathan_f77 8y ago
I think that's the craziest one I've heard so far. I can't imagine trying to reproduce a hardware bug in an FPGA with a logic analyzer. How much time did you spend on the bug? Was it something that you ignored for a long time, and then you decided to dive in and spend a couple of weeks on it?
- shaklee3 8y ago
  It was about 8 years ago, so my memory is a bit foggy. At one point we had everyone on our entire project working on it, which was about 20 people. We had daily calls with the customer since it was causing frequent outages on their network. I was full time on the bug for a while, and most of that time was spent making meticulous documentation for what happens in every clock cycle in the fpga. I visited the customer's site about 3 times during that period with a very expensive logic analyzer. By that time I'd added a ton of debug code in the fpga, and the logic analyzer was set to trigger if any of those counters hit. A big breakthrough was we were counting packets at each point in the pipeline, and that debug was showing that at a certain point, the counters were off. This was huge since it was the first clue into what was actually happening. Prior to that we just knew that there was an exponential increase in retransmitted packets until the whole thing came crashing down. Looking back on it, I'm really happy I ended up writing a large post-mortem with screen shots of the logic analyzer and everything. I can go back and see exactly what caused it, which is really interesting even today.
  - nathan_f77 8y ago
    Wow, that blows my mind. I don’t even know how to ask the right question, but is there some kind of “formal verification” or “type-checking” that could have caught this issue earlier? Or is that just too difficult, and reserved for medical equipment, nuclear reactors, etc.?
    - shaklee3 8y ago
      It was a problem with a clock domain crossing with IP from another company into ours. There were testbenches/simulations for verification, but they didn't test every packet size. In hindsight, everything can be caught, but it's a matter of how extensive the testing is. For example, passing every packet size through would not have showed the symptom. Everything looked normal, and all packets succeed. The key was that the meta data saying whether a buffer was free after transmitting was incorrect and never showing that it was free. So to really see the issue you had to send enough packets of that size to totally deplete the buffers. With a standard packet distribution, such as imix, you never hit that size.
      - nathan_f77 8y ago
        Thanks a lot for sharing this story! I started reading about clock domain crossing and metastability issues, and for the past few days I've been thinking a lot about circuit designs and FPGAs. I posted on /r/FPGA on Reddit [1], and I asked about the best development board to start with. I was wondering if you had any thoughts on that? I think I'm going to order a snickerdoodle black [2] and a breakout board, and start learning VHDL or Verilog. I think it would be a lot of fun to eventually design my own soft-core processor and compile a Rust program that runs on it. And after reading about clock domain crossing issues, I think it would also be really interesting to experiment with asynchronous circuits and clock-less CPUs. Thanks for introducing me to this! [1] https://www.reddit.com/r/FPGA/comments/9yutk8/best_100300_fpga_development_board_in_2018/ https://www.reddit.com/r/FPGA/comments/9yutk8/best_100300_fp... [2] https://www.crowdsupply.com/krtkl/snickerdoodle https://www.crowdsupply.com/krtkl/snickerdoodle
        8y ago

ComputerGuru 8y ago

I feel you. A bug in our recovery software would result in the live CD being unable to open the Windows kernel file after having located it on disk. No amount of logging we added to the code made sense. In the end, after more than a year, there was a customer in the UK that could reproduce the problem each time on a clean Windows 10 install (so no private info). I mailed him a new SSD to replace his old HDD and paid for him to overnight me his hard disk. Problem solved the very next day.

shaklee3 8y ago
That's awesome!

nathan_f77 8y ago

I think that's the craziest one I've heard so far. I can't imagine trying to reproduce a hardware bug in an FPGA with a logic analyzer. How much time did you spend on the bug? Was it something that you ignored for a long time, and then you decided to dive in and spend a couple of weeks on it?

shaklee3 8y ago
It was about 8 years ago, so my memory is a bit foggy. At one point we had everyone on our entire project working on it, which was about 20 people. We had daily calls with the customer since it was causing frequent outages on their network. I was full time on the bug for a while, and most of that time was spent making meticulous documentation for what happens in every clock cycle in the fpga. I visited the customer's site about 3 times during that period with a very expensive logic analyzer. By that time I'd added a ton of debug code in the fpga, and the logic analyzer was set to trigger if any of those counters hit. A big breakthrough was we were counting packets at each point in the pipeline, and that debug was showing that at a certain point, the counters were off. This was huge since it was the first clue into what was actually happening. Prior to that we just knew that there was an exponential increase in retransmitted packets until the whole thing came crashing down. Looking back on it, I'm really happy I ended up writing a large post-mortem with screen shots of the logic analyzer and everything. I can go back and see exactly what caused it, which is really interesting even today.
- nathan_f77 8y ago
  Wow, that blows my mind. I don’t even know how to ask the right question, but is there some kind of “formal verification” or “type-checking” that could have caught this issue earlier? Or is that just too difficult, and reserved for medical equipment, nuclear reactors, etc.?
  - shaklee3 8y ago
    It was a problem with a clock domain crossing with IP from another company into ours. There were testbenches/simulations for verification, but they didn't test every packet size. In hindsight, everything can be caught, but it's a matter of how extensive the testing is. For example, passing every packet size through would not have showed the symptom. Everything looked normal, and all packets succeed. The key was that the meta data saying whether a buffer was free after transmitting was incorrect and never showing that it was free. So to really see the issue you had to send enough packets of that size to totally deplete the buffers. With a standard packet distribution, such as imix, you never hit that size.
    - nathan_f77 8y ago
      Thanks a lot for sharing this story! I started reading about clock domain crossing and metastability issues, and for the past few days I've been thinking a lot about circuit designs and FPGAs. I posted on /r/FPGA on Reddit [1], and I asked about the best development board to start with. I was wondering if you had any thoughts on that? I think I'm going to order a snickerdoodle black [2] and a breakout board, and start learning VHDL or Verilog. I think it would be a lot of fun to eventually design my own soft-core processor and compile a Rust program that runs on it. And after reading about clock domain crossing issues, I think it would also be really interesting to experiment with asynchronous circuits and clock-less CPUs. Thanks for introducing me to this! [1] https://www.reddit.com/r/FPGA/comments/9yutk8/best_100300_fpga_development_board_in_2018/ https://www.reddit.com/r/FPGA/comments/9yutk8/best_100300_fp... [2] https://www.crowdsupply.com/krtkl/snickerdoodle https://www.crowdsupply.com/krtkl/snickerdoodle
      - 8y ago