5 ms·

A game controller I worked on was accidentally sensitive to physical strain. We discovered this after the first prototypes/consoles were sent to early reviewer

A game controller I worked on was accidentally sensitive to physical strain. We discovered this after the first prototypes/consoles were sent to early reviewers. They kept seeing weird in-game behavior no one could replicate, even when affected reviewers tried to demonstrate. Turns out they were gripping hard enough to minutely bend the PCB during intense moments in games, which created voltage spikes the software mistook for button presses. You needed to be angry to replicate.

HowardStark 3y ago
Stories like these are my catnip. The 500 mile email is another great one. How did you end up debugging/reproducing? I conjured a picture in my head of an engineer comically exasperated and squeezing the controller followed by an “Aha!”
- AlotOfReading 3y ago
  There was bare PCB prototype running on my desk that was in the way. I started cleaning my desk to work on something less frustrating and it got knocked off by accident. The logger went haywire immediately. That was the hint I needed.
  - RGamma 3y ago
    Accident-driven development.
    - davidmurdoch 3y ago
      Aka my career
ShadowBlades512 3y ago
This is definitely one of the reasons I wrote this article. To be able to see the cause/effect when a problem is very convoluted is something difficult to learn and difficult to teach. You just have to consume a lot of second hand stories and experience a few yourself. A story I have heard several times in different instances is some light sensitivity, I think this happened to the Raspberry Pi team (PMIC transient induced by camera flash).
- bayindirh 3y ago
  I think one of the Raspberry Pis were resetting on flash because of an exposed/uncapped chip. Yes. It’s the Raspberry Pi 2: https://hackaday.com/2015/02/08/photonic-reset-of-the-raspberry-pi-2/ https://hackaday.com/2015/02/08/photonic-reset-of-the-raspbe...
robomartin 3y ago
If you work on hardware long enough you run into all kinds of things that make you realize just how little you know. I had an interesting case around fifteen years ago. I had designed a real time image/video processor using a large Xilinx FPGA at the center of it. The system worked very well, except that it would have problems when flown on a helicopter. Actually, only certain kinds of helicopters, with certain types of cameras. I scratched my head for months trying to solve this riddle. Months later, while working on a completely unrelated problem, I touched the FPGA and burned my finger. It was surprisingly hot. This thing ran cool as can be, even without a heatsink. What happened? At first I thought I had a bad board, so I swapped it out. Sure enough, it got very hot, very quickly. Huh? That's when I realized I was using footage from one of the helo's. For some reason my brain made the connection. I instantly knew why we were having problems with helicopter-mounted cameras and why I burned my finger: Vibration. CMOS consumes power on state transitions, not so much once you reached the final state (low or high). More transitions per unit of time means more power. If the transitions are high speed edges (which is the case for real-time image processing) even worse. The helicopters in question did not have motion-stabilized cameras. They were hard-mounted to the verticals of the skids and that was that. Which means that every bit of vibration from the helo made it onto the camera. Add that to the helo travelling and you have a perfect storm: Every pixel was changing on every frame at 120 frames per second. The translation was a massive increase in power consumption and heat dissipation. Hence my burned index finger and the problems we had with helo-mounted systems. So, yeah, the power consumed by an imaging system is a direct function of the rate of change per unit of time of pixel data. A still frame consumes nothing when compared to a busy sequence where every pixel is changing. Logical, yet not obvious when you are not thinking in those terms. You have to burn your finger (or knock stuff off your desk) to learn this kind of thing.
- AlotOfReading 3y ago
  Funny you should use that example. I have a system where that could be an issue we hadn't anticipated. Now I have to check, thanks.
  - robomartin 3y ago

HowardStark 3y ago

Stories like these are my catnip. The 500 mile email is another great one. How did you end up debugging/reproducing? I conjured a picture in my head of an engineer comically exasperated and squeezing the controller followed by an “Aha!”

AlotOfReading 3y ago
There was bare PCB prototype running on my desk that was in the way. I started cleaning my desk to work on something less frustrating and it got knocked off by accident. The logger went haywire immediately. That was the hint I needed.
- RGamma 3y ago
  Accident-driven development.
  - davidmurdoch 3y ago
    Aka my career

ShadowBlades512 3y ago

This is definitely one of the reasons I wrote this article. To be able to see the cause/effect when a problem is very convoluted is something difficult to learn and difficult to teach. You just have to consume a lot of second hand stories and experience a few yourself. A story I have heard several times in different instances is some light sensitivity, I think this happened to the Raspberry Pi team (PMIC transient induced by camera flash).

bayindirh 3y ago
I think one of the Raspberry Pis were resetting on flash because of an exposed/uncapped chip. Yes. It’s the Raspberry Pi 2: https://hackaday.com/2015/02/08/photonic-reset-of-the-raspberry-pi-2/ https://hackaday.com/2015/02/08/photonic-reset-of-the-raspbe...

robomartin 3y ago

If you work on hardware long enough you run into all kinds of things that make you realize just how little you know. I had an interesting case around fifteen years ago. I had designed a real time image/video processor using a large Xilinx FPGA at the center of it. The system worked very well, except that it would have problems when flown on a helicopter. Actually, only certain kinds of helicopters, with certain types of cameras. I scratched my head for months trying to solve this riddle. Months later, while working on a completely unrelated problem, I touched the FPGA and burned my finger. It was surprisingly hot. This thing ran cool as can be, even without a heatsink. What happened? At first I thought I had a bad board, so I swapped it out. Sure enough, it got very hot, very quickly. Huh? That's when I realized I was using footage from one of the helo's. For some reason my brain made the connection. I instantly knew why we were having problems with helicopter-mounted cameras and why I burned my finger: Vibration. CMOS consumes power on state transitions, not so much once you reached the final state (low or high). More transitions per unit of time means more power. If the transitions are high speed edges (which is the case for real-time image processing) even worse. The helicopters in question did not have motion-stabilized cameras. They were hard-mounted to the verticals of the skids and that was that. Which means that every bit of vibration from the helo made it onto the camera. Add that to the helo travelling and you have a perfect storm: Every pixel was changing on every frame at 120 frames per second. The translation was a massive increase in power consumption and heat dissipation. Hence my burned index finger and the problems we had with helo-mounted systems. So, yeah, the power consumed by an imaging system is a direct function of the rate of change per unit of time of pixel data. A still frame consumes nothing when compared to a busy sequence where every pixel is changing. Logical, yet not obvious when you are not thinking in those terms. You have to burn your finger (or knock stuff off your desk) to learn this kind of thing.

AlotOfReading 3y ago
Funny you should use that example. I have a system where that could be an issue we hadn't anticipated. Now I have to check, thanks.
- robomartin 3y ago