I was once flown out to look at a significant performance problem. The customer could not do a days worth of work in a day. We’d told them to buy a fully loaded

I was once flown out to look at a significant performance problem. The customer could not do a days worth of work in a day. We’d told them to buy a fully loaded Sun E15k, and it couldn’t keep up. I instrumented everything (recompiled everything so that the binaries spat out gprof data, iostat output, vmstat output, sybase monitoring) and it quickly became obvious that the DB driven config was configured in a way we’d not anticipated. The fix was easy (just use a btree in a couple of places instead of a linear array walk), but then we had a huge problem. With the fix, the CPU bottleneck became a 15% CPU utilisation. The system had been sized based upon our advice. Whoops. What happened next was political and I won’t go into it. Deliberately vague on the industry and customer because of the 7 figure cost involved!