The most painful bug I encountered had to do with a visitor access kiosk I had designed and written the software for at my previous company. The workaround was

The most painful bug I encountered had to do with a visitor access kiosk I had designed and written the software for at my previous company. The workaround was to block access to a set of sites for the entire company to keep the two kiosks from failing. About every few months the web cam would just ... randomly stop working. This would cause the kiosk application to crash while attempting to take a visitor badge photo of the visitor, rebooting the machine. Because of the nature of the device[0], it was very difficult to identify the root cause and the fix was to physically visit the kiosk, unplug the web cam, remove the driver, install the latest driver and plug it back in. Eventually, I took some time and set-up one in my office and watched it. Something odd about the web cam was that the driver would never work if the web cam was plugged in while the driver installation ran. The installer instructed clearly, on a separate page of the installer to unplug the camera before "proceeding" and in what I have come to believe is one of the dumbest designs for driver software, it would periodically look for updates over the internet and silently install them, yielding a completely broken web cam. I spent about a month's time diagnosing the problem, mainly because that wasn't where I expected it to be, since I had other, more likely targets[1] (and I hadn't handled the OS install/driver setup). To make matters more entertaining, the guy who maintained the hardware had added the hostnames and IPs to hosts and configured it to resolve to 127.0.0.1 but the driver service helpfully ignored that file (as far as I was told[2]) and turning on the corporate firewall (Symantec Endpoint Protection) caused blue screens. Since this driver started feeling a lot like fighting malware, we ended up attacking it as such and shut down all communication with the updates servers and IPs via the perimeter ... for the whole company[3]. [0] It was an Office Communications Server solution written in a very old API and the kiosk ran Windows XP, which we stripped of nearly everything and forced the device to use the kiosk application as its shell (which would boot itself if it encountered any problem). [1] I had to write a component for the software to work with the web cam in C++, a language I hadn't touched in years, so my gut feeling was that it was related to that component. [2] It could be that we missed some of the IP addresses it polled, or it could be that it just ignored the hosts file in windows. I didn't do this work so I'm not entirely sure. [3] For whatever reason, security wouldn't/couldn't block the IPs just for the kiosk itself (something about it being setup to not require authentication to access the internet and our perimeter proxy server -- at the time -- being unable to be configured to block specific external IPs for specific internal IPs. My bet is that it was more a "not willing to" than an "unable to", but who knows?). The practical upshot is that we had some of these devices on peoples' desks within the company and they experienced the same problem so once it was banned, we received fewer help desk calls for broken web cams.