A decade ago i worked with sortation devices for postorder companies, and one of our clients reported that they sometimes had issues with items that were sorted

A decade ago i worked with sortation devices for postorder companies, and one of our clients reported that they sometimes had issues with items that were sorted wrong, but were unable to reproduce it. They used trays for sorting, and each tray had a barcode with a uniqe id. I spent a LONG time looking at logs until i ended up enabling debug logging, and because the site was on a 1200 baud modem i had the client burn the logs to a DVD media and ship them to us. I ended up writing i piece of perl code to parse the logs and insert them into a MySQL database where i could then trace the individual sorter trays by id, and by some obscure miracle of sleep deprevation and too much coffee, i manage to find a correlation. Turns out the bug only showed up when a tray had been used for sorting inbound items, then reused for sorting outbound items, and when used for sorting inbound items again (not outbund, which would reset it), then the bug would happen. The fix was traced to a single line in an if/else statement. Time to fix : around 1 hour including tests. Time to find bug : around 300 hours. Of something more relevant to the article, i used to write operating systems for mobile phones, and we spent A LONG time debugging an issue where our brand new display driver was acting up. After attaching a lauterbach debugger we finally managed to track it down to the compiler. Turns out : int i = 1+2+3; would mean i=3 in the code as the compiler only considered the first two variables in the assignment list. Another fun feature of that compiler was the fact that when you incremented heap memory past the memory page, it would forget to increment the page pointer, meaning it simply just wrapped to 0 and the memory you referenced was nowhere near what you'd expect :)