I used to work at a regional telecom company, and one of the jobs I did there involved keeping part of the order processing system alive. It consisted of somet

I used to work at a regional telecom company, and one of the jobs I did there involved keeping part of the order processing system alive. It consisted of something on the order of 100 PCs running OS/2 (this was in the mid-90s), with 3270 emulators that would connect to the mainframes, scrape information on the order forms, and plug them into forms on Unix boxes for another part of the order process. And the reverse. But, this software was notoriously unreliable. There were two of us trying to keep this system moving, and we couldn't go to lunch together because we couldn't step away from the system for more than half an hour without getting paged. We had one person at the data center where these systems lived that would walk the aisles of machines, looking at the screens, and trying to decide if the PC was hung and needed rebooting. I wrote some code that would watch logfiles these machines generated and would alert if the log for a specific system hadn't updated in ~30 minutes, and we could then ask the person on the floor to check on it. But, some orders could take an hour to process, and weren't really hung. So the person would have to go eyeball the screen and see if it looked like it was working or not. This was developed by the same team that spent a year developing a tool called "the blue GUI", and I went with them out to the service center that was going to be using it so they could show it off. If memory serves me, it was a giant window with a shitton of input fields laid out in columns and rows. How this was different from a spreadsheet, I really never understood, but a team of 10 had been working on this for a year. The lead developer I went with was sour on the 4 hour drive back, because the management of the service center declared that it was totally useless.