I encountered a very annoying-to-track-down "pseudo" Heisenbug in a mobile application I'm working on a few weeks ago. Once you logon to the application, it do

I encountered a very annoying-to-track-down "pseudo" Heisenbug in a mobile application I'm working on a few weeks ago. Once you logon to the application, it downloads product data (ID, description, quantity) for sales reps for that day. To cut a long story short, I'd just finished implementing a new piece of functionality and I logged on to the app to test it. The application started behaving very unexpectedly. I double-checked my code, but couldn't see anything that make it act the way it did. Then I tried logging on as a different user, et voila! It worked! Tried the old user and the problem occurred again. Tried a 3rd user and I got different behaviour again, but not the same as with the original user I was testing with. I thought it had to be a data issue, so I uninstalled the application and cleared the cache and app data, reinstalled, and the bug was gone. Whatever I tried I could not replicate the problem. I shrugged it off as local database corruption and continued working. A few days later, the same thing happened. Same symptoms, but this time the bug disappeared without me even having to reinstall the app. After a few hours of frustrating debugging and source code reviews, I remembered something. I ran into the problem at the same time of the morning as I did before - around 12am. It turns out that the process that populates the server with daily product information that usually ran at 11pm was rescheduled to run at 12am (we weren't informed about this change). It ran for 5 to 8 mins. At the time I logged on, the data for the first user was not available yet, but the data for the second user was. To complicate things, if the server can't find daily product information for that day, it sends data from the previous week (the products are essentially the same, but the quantities may be slightly different and the IDs are tied to older stock batches). But the user I was testing with didn't exist 7 days ago so he didn't have data to fall back on. The 3rd user's daily product data also wasn't available yet, but he did exist 7 days ago so he received data with product IDs that I did not expect. In the end I "fixed" the problem using a scheduled task that emailed tech support if the product data was not in our databases by 11:30pm.