3 ms·
Once had an issue with a network of wireless mesh devices that I had to try to debug from across the country (I was in Philadelphia, the client was having an is
by moron4hire 13y ago
Once had an issue with a network of wireless mesh devices that I had to try to debug from across the country (I was in Philadelphia, the client was having an issue with a network installed in a hotel in Las Vegas). Getting ready to leave for the a long weekend, I got a call from a client at 4:45pm (knowing this client, I suspected they did it on purpose. They had called me on my cell, so clearly they didn't expect me to be in the office). Basically, every time you'd try to reset the room for a new guest, the blinds would shake, half the lights would fail to turn on, and the door would unlock. If you kept at it, it would eventually work. But then, setting the thermostat would sometimes make half the lights go out. And the door would unlock.
They had a packet sniffer that we had built for them, so we went to trying to diagnose the issue. I'd send them new versions of the programmer tool, they'd flash the room (which required pinging each and every device in the room and tapping a physical button on it, in one of the largest single-room installations of this brand of mesh network in the world), and they'd send me the logs of the sniffed packets. We could see that the packets for any device you happened to be standing next to would be completely fine, but if you turned your back on it and walked across the room, it started acting up. But they would be fine again if you walked back to it. "It's like it knows you're watching it", said the guy on the phone.
They kept insisting that I had let a virus into their network. Never mind that it was only possible to rewrite the configuration ROM over the air, rewriting the ROM required physical access to the board in the case, clearly it was my fault for all the "unnecessary fiddling" I had been doing recently (i.e. a slew of bug fixes they had requested all involving my predecessor's lack of understanding of the pitfalls of threading and UI in .NET).
I kept telling the client that all the symptoms suggested radio interference from an outside source. They insisted they had never heard of such a thing, ever, in any context, including static on their car's radio. I being "merely" an applications developer and not an electrical engineer, they lacked faith in my explanation and insisted that I "just fix it". How I was supposed to be knowledgeable enough to fix it if I was apparently not knowledgeable to understand what was going on with it was beyond me, but whatever.
"Put it back to what it was before you started f*ing around with it." Revert the code through source control (thank God I had installed SVN when I first arrived at that company, because apparently EEs don't understand that it's not a good idea to keep dated copies of code directories around as "backups"). "This isn't working, I said give me the old version." Send them links to the installer on their own server. "You must have changed it on our server! How did you get access to our servers? This isn't working!"
I finally gave up around midnight and drove the 3 hours to my parents' house for Thanksgiving the next day. I nearly got fired for it.
We shipped them a radio spectrum analyzer and determined for sure, it was radio interference. The hotel opened the room next door and found a baby monitor, still on, fallen behind the dresser. They turned it off and the room responded flawlessly.
I should have quit then, but I needed the money and I was going through some depression issues so I really thought it was my fault. I eventually did get fired from that place, the only place I've ever gotten fired from, for "not working enough overtime" because I was only doing 50 hours a week when the intern fresh out of college needed 60 to get his much simpler tasks done (and often leaving me blocked because of it, but I wasn't allowed to help him with anything because "you're not an electrical engineer", where apparently only electrical engineers know how to code in C). I don't regret it, biggest piece of shit place I've ever been, and just the motivation I needed to get off my ass and finally change my relationship to work. I've been freelancing every since.
I guess that's not so much "my hardest bug", but I did actually fix a bunch of bugs in the process of trying to convince them it was interference and not some mythical radio virus that could corrupt packets in mid-air. And all of it based on phone calls and emails with hex dumps of sniffed radio packets, with being "nothing more" than a lowly applications programmer.